www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Dub, Cargo, Go, Gradle, Maven

reply Russel Winder <russel winder.org.uk> writes:
In all the discussion of Dub to date, it hasn't been pointed out that
JVM building merged dependency management and build a long time ago.
Historically:

  Make =E2=86=92 Ant =E2=86=92 Maven =E2=86=92 Gradle

and Gradle can handle C++ as well as JVM language builds.

So the integration of package management and build as seen in Go,
Cargo, and Dub is not a group of outliers. Could it be then that it is
the right thing to do. After all package management is a dependency
management activity and build is a dependency management activity, so
why separate them, just have a single ADG to describe the whole thing.

SCons, CMake, and Meson (also Reggae?) are traditional build tools, but
they assume all package dependency management is handled elsewhere,
i.e. they state what is required for a build but assume some other tool
provides those packages, usually OS package management, but note with
C++ Conan is rapidly becoming a big player. JFrog via Bintray and
Artifactory do appear to be the leaders in repository management and
Gradle works bery well with it.

Rust, Ceylon, and D have to date chosen to eschew systems like Bintray
to create their own language specific versions for whatever reason.
This leads to language specific dependency management and build. Go is
also in this category really except that Git, Mercurial, and Breezy
(n=C3=A9e Bazaar) repositories are the only package storage used.

Then, is a DevOps world, there is deployment, which is usually a
dependency management task. Is a totally new tool doing ADG
manipulation really needed for this?

The lessons from all the tools from SCons to Gradle is that it is all
about ADG manipulation and constraint satisfaction. SCons is really
quite restricted even though it does it very well (*).  Gradle really
tries hard not just to solve the problems of Maven (**), but to do end-
to-end project management well. In a sense it is the antithesis of each
tool does one thing and one thing only model, it is the "there is one
and only one ADG to describe the life of the project". Maven and
Gradle, and to a lesser extent Cargo and Go, emphasise project
management as a wholistic thing, rather than making people deside on
each item of the tool chain. Gradle proves a good plugin system to
allow changes to the default standard project lifecycle.

Gradle uses Groovy scripts or Kotlin scripts for project
specifications. Most projects are easily described by a purely
declarative specification using internal DSLs. However, for those
awkwards bits for some projects a bit of programming in the
specification solves the problem. So this goes against the "a project
specification must be purely declarative so use TOML/JSON/SDL" but is
easy DevOps worth it.=20

Atila's Reggae has already shown how easy it is to use D (or Python,
Ruby, JavaScript, Lua) to define a build in an internal DSL.

Merging ideas from Dub, Gradle, and Reggae, into a project management
tool for D (with C) projects is relatively straightforward of plan
albeit really quite a complicated project. Creating the core ADG
processing is the first requirement. It has to deal with external
dependencies, project build dependencies, and deployment dependencies.
The initial problem is creating an ADG from what is potentially a CDG
and then doing constraint satisfaction to generate actions. SCons and
Gradle have a lot to offer on this.

Having been obsessed by build and project management since about 1976,
I'd be interested in doing some work on this.



(*) The O(N) vs. O(n), SCons vs. Tup thing that T raised in another
thread is important, but actually it is an implementation thing of how
do you detect change, it isn't an algorithmic issue at a system design
level. But it is important.

(**) Which many people ignore because Maven remains the major project
management tool in the JVM-verse.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Feb 12 2018
next sibling parent bachmeier <no spam.net> writes:
On Monday, 12 February 2018 at 10:35:06 UTC, Russel Winder wrote:


 Having been obsessed by build and project management since 
 about 1976, I'd be interested in doing some work on this.


This is interesting to me, though I don't know how helpful I can 
be. At a minimum I will be able to do testing on different 
platforms. I will also write some code if competent to do so.
Feb 12 2018
prev sibling next sibling parent b4siL3 b. <b3.temp gmx.com> writes:
On Monday, 12 February 2018 at 10:35:06 UTC, Russel Winder wrote:

 In all the discussion of Dub to date, it hasn't been pointed 
 out that JVM building merged dependency management and build a 
 long time ago. Historically:

   Make → Ant → Maven → Gradle

 and Gradle can handle C++ as well as JVM language builds.

 So the integration of package management and build as seen in 
 Go, Cargo, and Dub is not a group of outliers. Could it be then 
 that it is the right thing to do. After all package management 
 is a dependency management activity and build is a dependency 
 management activity, so why separate them, just have a single 
 ADG to describe the whole thing.

 SCons, CMake, and Meson (also Reggae?) are traditional build 
 tools, but they assume all package dependency management is 
 handled elsewhere,


Indeed. That's the big problem of these non-specialiazed build 
tools.
Feb 12 2018
prev sibling next sibling parent John Gabriele <jgabriele fastmail.fm> writes:
On Monday, 12 February 2018 at 10:35:06 UTC, Russel Winder wrote:

 In all the discussion of Dub to date, it hasn't been pointed 
 out that JVM building merged dependency management and build a 
 long time ago. Historically:

   Make → Ant → Maven → Gradle

 and Gradle can handle C++ as well as JVM language builds.

 So the integration of package management and build as seen in 
 Go, Cargo, and Dub is not a group of outliers. Could it be then 
 that it is the right thing to do. {snip}


As an additional data point, the main project management tool for 
Clojure is [Leiningen](https://leiningen.org/). I don't 
understand the details under the hood, but it makes use of Maven 
infrastructure and can install packages from the Clojure online 
package repo, as well as from [Maven 
Central](http://search.maven.org/) (the online repo of JVM jars).

Lein installs packages into the same place maven does, into a 
"local repo" in an ~/.m2/repository directory. This makes it very 
easy to `rm -fr` it to start afresh if needed. Lein keeps 
multiple versions of packages when your various projects use 
different versions of the same library.

The lein project config file is itself written in Clojure. See a 
[sample lein config 
file](https://github.com/technomancy/leiningen/blob/master/sample.project.clj)
if interested.

When I first encountered lein I was confused. Was it for project 
creation, dependency management, building and running your app, 
or running tests? Turns out it does all of those, and also 
supports plug-ins to allow it to do all kinds of other 
project-related tasks that aren't built-into the tool (for 
example, generating docs --- see the [full list of lein 
plug-ins](https://github.com/technomancy/leiningen/wiki/Plugins)).
Feb 12 2018
prev sibling next sibling parent Jonathan Marler <johnnymarler gmail.com> writes:
On Monday, 12 February 2018 at 10:35:06 UTC, Russel Winder wrote:

 In all the discussion of Dub to date, it hasn't been pointed 
 out that JVM building merged dependency management and build a 
 long time ago. Historically:

   Make → Ant → Maven → Gradle

 and Gradle can handle C++ as well as JVM language builds.

 So the integration of package management and build as seen in 
 Go, Cargo, and Dub is not a group of outliers. Could it be then 
 that it is the right thing to do. After all package management 
 is a dependency management activity and build is a dependency 
 management activity, so why separate them, just have a single 
 ADG to describe the whole thing.

 SCons, CMake, and Meson (also Reggae?) are traditional build 
 tools, but they assume all package dependency management is 
 handled elsewhere, i.e. they state what is required for a build 
 but assume some other tool provides those packages, usually OS 
 package management, but note with C++ Conan is rapidly becoming 
 a big player. JFrog via Bintray and Artifactory do appear to be 
 the leaders in repository management and Gradle works bery well 
 with it.

 Rust, Ceylon, and D have to date chosen to eschew systems like 
 Bintray to create their own language specific versions for 
 whatever reason. This leads to language specific dependency 
 management and build. Go is also in this category really except 
 that Git, Mercurial, and Breezy (née Bazaar) repositories are 
 the only package storage used.

 Then, is a DevOps world, there is deployment, which is usually 
 a dependency management task. Is a totally new tool doing ADG 
 manipulation really needed for this?

 The lessons from all the tools from SCons to Gradle is that it 
 is all about ADG manipulation and constraint satisfaction. 
 SCons is really quite restricted even though it does it very 
 well (*).  Gradle really tries hard not just to solve the 
 problems of Maven (**), but to do end- to-end project 
 management well. In a sense it is the antithesis of each tool 
 does one thing and one thing only model, it is the "there is 
 one and only one ADG to describe the life of the project". 
 Maven and Gradle, and to a lesser extent Cargo and Go, 
 emphasise project management as a wholistic thing, rather than 
 making people deside on each item of the tool chain. Gradle 
 proves a good plugin system to allow changes to the default 
 standard project lifecycle.

 Gradle uses Groovy scripts or Kotlin scripts for project 
 specifications. Most projects are easily described by a purely 
 declarative specification using internal DSLs. However, for 
 those awkwards bits for some projects a bit of programming in 
 the specification solves the problem. So this goes against the 
 "a project specification must be purely declarative so use 
 TOML/JSON/SDL" but is easy DevOps worth it.

 Atila's Reggae has already shown how easy it is to use D (or 
 Python, Ruby, JavaScript, Lua) to define a build in an internal 
 DSL.

 Merging ideas from Dub, Gradle, and Reggae, into a project 
 management tool for D (with C) projects is relatively 
 straightforward of plan albeit really quite a complicated 
 project. Creating the core ADG processing is the first 
 requirement. It has to deal with external dependencies, project 
 build dependencies, and deployment dependencies. The initial 
 problem is creating an ADG from what is potentially a CDG and 
 then doing constraint satisfaction to generate actions. SCons 
 and Gradle have a lot to offer on this.

 Having been obsessed by build and project management since 
 about 1976, I'd be interested in doing some work on this.



 (*) The O(N) vs. O(n), SCons vs. Tup thing that T raised in 
 another
 thread is important, but actually it is an implementation thing 
 of how
 do you detect change, it isn't an algorithmic issue at a system 
 design
 level. But it is important.

 (**) Which many people ignore because Maven remains the major 
 project management tool in the JVM-verse.


Lot's of stuff here.  I would love for a build/package management 
tool to emerge and take over like git did with source control.  
However, my guess is that this problem is very hard and that's 
why there's so many tools that have their own pros and cons.  But 
it sounds like you're trying to sift through them all and find 
the gems in each tool.

I myself have made attempts to tackle the problem.  I created a 
tool called "dbuild" that allowed you to use D code to describe 
your project and then build it for you (kinda like Raggae I 
suppose).  And my latest project I call "bidmake" which is more 
general.  It implements the model of what I call "build 
contracts".  A project describes a "contract" and various 
"contractors" can be used to fulfill the contract.  You could 
have a "build contractor" that builds a contract, or a 
"documentation contractor" that generates documentation, etc.  
Anyway, I currently use the tool successfully on a few projects 
but I'm still not sure how much potential it has.  Maybe with 
your experience you can take a look at it and see if there's any 
substance to the idea. (https://github.com/marler8997/bidmake)
Feb 12 2018
prev sibling parent reply welkam <wwwelkam gmail.com> writes:
ADG? Google doesnt find anything relevant
Feb 13 2018
parent reply Abdulhaq <alynch4047 gmail.com> writes:
On Tuesday, 13 February 2018 at 10:06:43 UTC, welkam wrote:

 ADG? Google doesnt find anything relevant


Acyclic directed graph
Feb 13 2018
parent reply Graham St Jack <graham.stjack internode.on.net> writes:
On Tue, 13 Feb 2018 10:21:49 +0000, Abdulhaq wrote:


 On Tuesday, 13 February 2018 at 10:06:43 UTC, welkam wrote:

 ADG? Google doesnt find anything relevant


 
 Acyclic directed graph


A.K.A "DAG" - directed acyclic graph.

I am also interested in build systems, and am currently convinced that 
the problem space is complex enough to warrant build systems being 
separate from package management systems.

Maybe a compromise position would be for a package management system to 
define an interface through which it can do things like:

* Discover what the external dependencies are,

* Provide those external dependencies, and

* Invoke a full build.

Then, any number of build systems (and deployment systems?) could be 
adapted to work with the package management system.
Feb 14 2018
parent reply Pjotr Prins <pjotr.public12 thebird.nl> writes:
On Thursday, 15 February 2018 at 04:11:51 UTC, Graham St Jack 
wrote:

 Maybe a compromise position would be for a package management 
 system to define an interface through which it can do things 
 like:

 * Discover what the external dependencies are,

 * Provide those external dependencies, and

 * Invoke a full build.

 Then, any number of build systems (and deployment systems?) 
 could be adapted to work with the package management system.


That is exactly what GNU Guix offers. With support for isolated 
builds, continuous integration testing, and containers thrown in, 
if you want that. People misunderstand Guix somewhat because it 
presents itself as a 'package manager' and even distribution in 
its own right. But actually it is a dependency manager that can 
run on top of any system.

I am writing a BLOG on how to use Guix for Python development 
using its package managers and even dependency injection (say a 
choice of BLAS libraries or LLVM in the case of D). When that is 
done I could do a D writeup if there is enough interest.

I am sure some people roll their eyes when I mention GNU Guix. 
But, hey, if you are an Emacs or gcc user you may be able to 
afford to pay attention to long running GNU projects.

To anticipate real criticism: there are currently two main 
issues: (1) GNU Guix runs on Linux and (2) the default requires a 
running build daemon.

About (1) since this is a libre project the focus is on Linux and 
Hurd. It is actually fairly straightforward to port Guix to other 
OS's. Nix, which shares the build daemon with Guix, runs on OSX 
and Windows.

About (2) the daemon can run unprivileged. I have written 
documents about running Guix without root access (I need it on 
HPC). It is just a little more involved.

Anyway, I don't really care who uses Guix or who uses something 
else. I expect 99.9% of people to ignore these ideas. Just to say 
I simply use it to make my own life easier. The point here is 
that I understand what it took to create Guix and it is 
non-trivial. We can reuse this functionality very easily and take 
control over the dependency graph. You get reproducible builds, 
easy mixing of LLVM versions and many other features. Fixing (1) 
above is much easier than recreating something like Guix from 
scratch. And since Guix is distribution agnostic you can use it 
on any old CentOS, Debian, Ubuntu... you name it. The only thing 
Guix uses is the *running* kernel API. Even glibc and the linked 
library loader come with Guix (and you can easily run multiple 
versions of said libraries). That is a full and deep dependency 
graph.

With Guix I do not need dub or pip or gems. It is trivially easy 
to manage dependencies. Still I can use those package managers if 
I want to. In a controlled fashion.

PS The JVM world has the advantage of being a clear and isolated 
system. Good news is that Guix supports that too and it provides 
an awesome libre bootstrap from source. If you care about the 
free and open in FOSS that is a huge selling point in a world 
that appears to increasingly bootstrap from binary blobs.

It is easy to write-off such ideas. But if you don't try it, you 
don't understand it. Similar to the D vs other language 
discussions. I am not going to say Guix is easy. Similar to the 
fact that D is not easy. But you can gradually get in and learn 
to appreciate the great engineering under the hood. That goes for 
Nix too. Guix and Nix still share the build daemon, though they 
have become completely different systems with different 
characteristics.

If you want to try Guix - I am here to support you.
Feb 14 2018
parent reply John Gabriele <jgabriele fastmail.fm> writes:
On Thursday, 15 February 2018 at 07:21:24 UTC, Pjotr Prins wrote:

 On Thursday, 15 February 2018 at 04:11:51 UTC, Graham St Jack 
 wrote:

 Maybe a compromise position would be for a package management 
 system to define an interface through which it can do things 
 like:

 * Discover what the external dependencies are,

 * Provide those external dependencies, and

 * Invoke a full build.

 Then, any number of build systems (and deployment systems?) 
 could be adapted to work with the package management system.


 That is exactly what GNU Guix offers. With support for isolated 
 builds, continuous integration testing, and containers thrown 
 in, if you want that. People misunderstand Guix somewhat 
 because it presents itself as a 'package manager' and even 
 distribution in its own right. But actually it is a dependency 
 manager that can run on top of any system.


It's a bit confusing since the first thing [the Guix 
webpage](https://www.gnu.org/software/guix/) talks about 
"GuixSD", rather than the Guix tool in its own right.

Wow, I didn't realize how established Guix is. It's [packages 
page](https://www.gnu.org/software/guix/packages/) boasts 6868 
available packages!

The [Guix manual](https://www.gnu.org/software/guix/manual/) 
appears to be comprehensive.

Are any other languages using Guix for their 3rd-party online 
package repo? If not, why?


 I am writing a BLOG on how to use Guix for Python development 
 using its package managers and even dependency injection (say a 
 choice of BLAS libraries or LLVM in the case of D). When that 
 is done I could do a D writeup if there is enough interest.

 I am sure some people roll their eyes when I mention GNU Guix.


Not at all. Seems like an amazing tool!
Feb 15 2018
parent reply Pjotr Prins <pjotr.public12 thebird.nl> writes:
On Thursday, 15 February 2018 at 15:52:41 UTC, John Gabriele 
wrote:

 It's a bit confusing since the first thing [the Guix 
 webpage](https://www.gnu.org/software/guix/) talks about 
 "GuixSD", rather than the Guix tool in its own right.


Yes. We discussed that at the hackathon before FOSDEM. It is 
rather difficult to position a project that has so many use 
cases...


 Wow, I didn't realize how established Guix is. It's [packages 
 page](https://www.gnu.org/software/guix/packages/) boasts 6868 
 available packages!


~2000 added in the last year. I am updating the ldc package to 
1.7.0 this week.


 Are any other languages using Guix for their 3rd-party online 
 package repo? If not, why?


GNU Guile ;). The current state is that everyone is rolling out 
some type of language support. The good news is that Guix allows 
for abstracting existing build systems. If you look at a typical 
python package you can see it leans on pip:

   
https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/python.scm#n1004

The package is written in Guile scheme, but you can see it does 
not take a genius
to package something. In fact Guix can generate the actual 
package definition automatically.

We have not written something similar for dub. But I can do that 
if there is interest.


 Not at all. Seems like an amazing tool!


Agree. Give it a shot. It does not hurt your system because the 
installer only uses /gnu (for the software) and /var/guix (for 
the database). To uninstall simply remove those dirs.
Guix never overwrites system directories.
Feb 15 2018
parent reply John Gabriele <jgabriele fastmail.fm> writes:
On Thursday, 15 February 2018 at 16:47:35 UTC, Pjotr Prins wrote:

 On Thursday, 15 February 2018 at 15:52:41 UTC, John Gabriele 
 wrote:

 It's a bit confusing since the first thing [the Guix 
 webpage](https://www.gnu.org/software/guix/) talks about 
 "GuixSD", rather than the Guix tool in its own right.


 Yes. We discussed that at the hackathon before FOSDEM. It is 
 rather difficult to position a project that has so many use 
 cases...


 Wow, I didn't realize how established Guix is. It's [packages 
 page](https://www.gnu.org/software/guix/packages/) boasts 6868 
 available packages!


 ~2000 added in the last year. I am updating the ldc package to 
 1.7.0 this week.


 Are any other languages using Guix for their 3rd-party online 
 package repo? If not, why?


 GNU Guile ;).


That's excellent. I remember years ago it looked like there was 
action in getting a Guile package repo going but it never panned 
out. Glad they found (and settled on) something! (Though 
<https://www.gnu.org/software/guile/libraries/> doesn't say 
anything about Guix, and I don't see anything about Guix in the 
[Guile 
manual](https://www.gnu.org/software/guile/docs/master/guile.html/)).


 The current state is that everyone is rolling out some type of 
 language support. The good news is that Guix allows for 
 abstracting existing build systems. If you look at a typical 
 python package you can see it leans on pip:

   
 https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/python.scm#n1004

 The package is written in Guile scheme, but you can see it does 
 not take a genius
 to package something. In fact Guix can generate the actual 
 package definition automatically.

 We have not written something similar for dub. But I can do 
 that if there is interest.


I'd love to see an overview article explaining how the pieces 
would fit together. Would Guix work with Dub? If so, what 
specifically (on GNU/Linux) would be required to make that 
happen, and what would using such a solution look like in 
practice?


 Not at all. Seems like an amazing tool!


 Agree. Give it a shot.


I don't yet know enough about D and Dub to even know what giving 
Guix a shot would mean :). I'm on Debian and so use apt for 
installing software. I'll have to spend some time with the Guix 
(and Dub) docs.


 It does not hurt your system because the installer only uses 
 /gnu (for the software) and /var/guix (for the database). To 
 uninstall simply remove those dirs.
 Guix never overwrites system directories.


Nice. Easy removal is most appreciated.
Feb 15 2018
parent Pjotr Prins <pjotr.public12 thebird.nl> writes:
On Thursday, 15 February 2018 at 17:54:57 UTC, John Gabriele 
wrote:

 Are any other languages using Guix for their 3rd-party online 
 package repo? If not, why?


 GNU Guile ;).


 That's excellent. I remember years ago it looked like there was 
 action in getting a Guile package repo going but it never 
 panned out. Glad they found (and settled on) something! (Though 
 <https://www.gnu.org/software/guile/libraries/> doesn't say 
 anything about Guix, and I don't see anything about Guix in the 
 [Guile 
 manual](https://www.gnu.org/software/guile/docs/master/guile.html/)).


The Guile maintainers effectively use Guix for handling the build 
chain and dependencies.
See Andy Wingo's blog https://wingolog.org/ - section 'getting 
the goods' he discusses Guile packaging and Guix. Andy is the 
main compiler writer of Guile. The following is a bit out of 
date, but you get the idea

     
https://wingolog.org/archives/2015/08/04/developing-v8-with-guix.


 I'd love to see an overview article explaining how the pieces 
 would fit together. Would Guix work with Dub? If so, what 
 specifically (on GNU/Linux) would be required to make that 
 happen, and what would using such a solution look like in 
 practice?


I need to write a BLOG. After the Python one.


 I don't yet know enough about D and Dub to even know what 
 giving Guix a shot would mean :). I'm on Debian and so use apt 
 for installing software. I'll have to spend some time with the 
 Guix (and Dub) docs.


I meant you can try Guix anyway. It will be a gain. I run Guix on 
top of Debian myself. I am a real Debian fan - using Debian more 
than 15 years. But when it comes to my development environment 
and my software deployment I use Guix. Debian can not handle that 
well enough.
Feb 15 2018