www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Dub for internal dependency management: Git repo as package supplier?

reply FeepingCreature <feepingcreature gmail.com> writes:
We (Funkwerk) are currently using rake as a build system. Since 
rake does not offer package management, we have to list all 
transitive dependencies of every package, which is obviously not 
satisfactory. Because of this, we're looking into switching to 
dub and dub.selection.json as a "source of truth" for the used 
set of library dependencies.

We're currently using git submodules for local dependencies. If 
they were migrated to dub, we'd run into the problem of what to 
do while working on a dependency. Often changes to a dependency 
only make sense in the context of the project that needs the 
dependency, so we have to be able to set dub to use the git 
branch that we're working on. However, while this can be made to 
work locally by using dub add-override to point dub at a folder 
that has the branch checked out, this cannot work on our Jenkins 
system because obviously that doesn't have the local folder.

It seems to me that a possible solution would be to be able to 
set a git repository as a dub package supplier. Dub would then 
check out the repository to .dub/packages/<project>/<git hash>, 
and build from there. dub.selections.json could store the commit 
hash for reproducible builds.

Since this would require a hefty amount of work on dub to support 
git remote repos, I wanted to ask upfront: would such a PR 
plausibly be accepted? And in case not, do you have alternate 
proposals for D dependency management that won't force us to tag 
a repo before using it in another project?

Thanks for your consideration!
Jul 31
next sibling parent Jacob Carlborg <doob me.com> writes:
On Wednesday, 31 July 2019 at 13:02:10 UTC, FeepingCreature wrote:
 We (Funkwerk) are currently using rake as a build system. Since 
 rake does not offer package management, we have to list all 
 transitive dependencies of every package, which is obviously 
 not satisfactory. Because of this, we're looking into switching 
 to dub and dub.selection.json as a "source of truth" for the 
 used set of library dependencies.

 We're currently using git submodules for local dependencies. If 
 they were migrated to dub, we'd run into the problem of what to 
 do while working on a dependency. Often changes to a dependency 
 only make sense in the context of the project that needs the 
 dependency, so we have to be able to set dub to use the git 
 branch that we're working on. However, while this can be made 
 to work locally by using dub add-override to point dub at a 
 folder that has the branch checked out, this cannot work on our 
 Jenkins system because obviously that doesn't have the local 
 folder.
A dependency can point to a local directory, including a cloned git submodule. Have a look how I did that in the DLP project. https://github.com/jacob-carlborg/dlp/blob/9982cd75ea347170b4f13fc259789df264bcd7b3/dub.sdl#L9 — /Jacob Carlborg
Jul 31
prev sibling next sibling parent Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 31 July 2019 at 13:02:10 UTC, FeepingCreature wrote:
 We (Funkwerk) are currently using rake as a build system. Since 
 rake does not offer package management, we have to list all 
 transitive dependencies of every package, which is obviously 
 not satisfactory. Because of this, we're looking into switching 
 to dub and dub.selection.json as a "source of truth" for the 
 used set of library dependencies.

 We're currently using git submodules for local dependencies. If 
 they were migrated to dub, we'd run into the problem of what to 
 do while working on a dependency. Often changes to a dependency 
 only make sense in the context of the project that needs the 
 dependency, so we have to be able to set dub to use the git 
 branch that we're working on. However, while this can be made 
 to work locally by using dub add-override to point dub at a 
 folder that has the branch checked out, this cannot work on our 
 Jenkins system because obviously that doesn't have the local 
 folder.

 It seems to me that a possible solution would be to be able to 
 set a git repository as a dub package supplier. Dub would then 
 check out the repository to .dub/packages/<project>/<git hash>, 
 and build from there. dub.selections.json could store the 
 commit hash for reproducible builds.

 Since this would require a hefty amount of work on dub to 
 support git remote repos, I wanted to ask upfront: would such a 
 PR plausibly be accepted? And in case not, do you have 
 alternate proposals for D dependency management that won't 
 force us to tag a repo before using it in another project?

 Thanks for your consideration!
1. You can use path-based dependencies, you won't have to listed dependencies transitively. You _will_ have to: - checkout everything, you can have several package in the same repo for example - "your builds won't be reproducible" aka "you can't regress", because I don't think you can express versions with path-based - dub.selections.json won't be effective for such path-based stuff 2. I think you'd better host your own DUB registry
Jul 31
prev sibling next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Wednesday, 31 July 2019 at 13:02:10 UTC, FeepingCreature wrote:
 We (Funkwerk) are currently using rake as a build system. Since 
 rake does not offer package management, we have to list all 
 transitive dependencies of every package, which is obviously 
 not satisfactory. Because of this, we're looking into switching 
 to dub and dub.selection.json as a "source of truth" for the 
 used set of library dependencies.

 [...]
You know, dub can also retrieve dependencies from Maven? Also the file system repository provider was fixed, enabling you to use any directory/ftp/s3 folder as repository. Regarding you question, yes, such enhancement is heavily welcome. There is also an open pr you can build on. Kind regards Andre
Jul 31
prev sibling parent reply Seb <seb wilzba.ch> writes:
On Wednesday, 31 July 2019 at 13:02:10 UTC, FeepingCreature wrote:
 We (Funkwerk) are currently using rake as a build system. Since 
 rake does not offer package management, we have to list all 
 transitive dependencies of every package, which is obviously 
 not satisfactory. Because of this, we're looking into switching 
 to dub and dub.selection.json as a "source of truth" for the 
 used set of library dependencies.

 [...]
Yes, of course it would be. It's a very commonly requested feature. There even was an old PR for this: https://github.com/dlang/dub/pull/1403
Jul 31
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Wednesday, 31 July 2019 at 14:44:35 UTC, Seb wrote:
 On Wednesday, 31 July 2019 at 13:02:10 UTC, FeepingCreature 
 wrote:
 We (Funkwerk) are currently using rake as a build system. 
 Since rake does not offer package management, we have to list 
 all transitive dependencies of every package, which is 
 obviously not satisfactory. Because of this, we're looking 
 into switching to dub and dub.selection.json as a "source of 
 truth" for the used set of library dependencies.

 [...]
Yes, of course it would be. It's a very commonly requested feature. There even was an old PR for this: https://github.com/dlang/dub/pull/1403
Sweet, thank you! I'm gonna start poking away at this somewhere in the next month, probably. Re everyone else, again, local paths will not work because they'd fail to resolve on our build server. We could, I guess, push branch builds to our dub repo, but that's kind of awkward and would depend on our build architecture in an uncomfortable way. Going directly to the source seems to me the way to go.
Jul 31
parent Andre Pany <andre s-e-a-p.de> writes:
On Wednesday, 31 July 2019 at 15:34:56 UTC, FeepingCreature wrote:
 On Wednesday, 31 July 2019 at 14:44:35 UTC, Seb wrote:
 On Wednesday, 31 July 2019 at 13:02:10 UTC, FeepingCreature 
 wrote:
 We (Funkwerk) are currently using rake as a build system. 
 Since rake does not offer package management, we have to list 
 all transitive dependencies of every package, which is 
 obviously not satisfactory. Because of this, we're looking 
 into switching to dub and dub.selection.json as a "source of 
 truth" for the used set of library dependencies.

 [...]
Yes, of course it would be. It's a very commonly requested feature. There even was an old PR for this: https://github.com/dlang/dub/pull/1403
Sweet, thank you! I'm gonna start poking away at this somewhere in the next month, probably. Re everyone else, again, local paths will not work because they'd fail to resolve on our build server. We could, I guess, push branch builds to our dub repo, but that's kind of awkward and would depend on our build architecture in an uncomfortable way. Going directly to the source seems to me the way to go.
As you have written Jenkins: Please also see the Dub enabled Jenkins pipeline library: https://forum.dlang.org/thread/ljuosdnorythkosdrtyp forum.dlang.org Also the generic issue scanner plugin of Jenkins is now able to read D-Scanner issue json file. The generic code coverage plugin of Jenkins will be in the very next future be able to read LST files ;) Kind regards Andre
Jul 31