www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - dub -> meson

reply 9il <ilyayaroshenko gmail.com> writes:
Hi all,

'DUB - call to arms' post by Seb
https://forum.dlang.org/post/unimbbqxosxcmjdhtzch forum.dlang.org

Hope we will move other direction and extend Meson build system 
with better D support and make a new Dlang registry.

Reasons to move forward with Meson:

0. Progressive and large community
1. Has clear python source code: faster to extend, easier to 
maintain.
2. D+Meson already used in production: last year it is a major 
build system for all of Mir and its commercial applications.
3. Meson language is super cool. It is much better than dub for 
commercial projects, for large projects, for library distribution.
4. Ability to compile and configure D code with other languages 
like C/C++.
5. With Meson we don't need to catch up Rust, we can do better 
than Rust with fewer resources.


Best,
Ilya
Apr 15 2019
next sibling parent reply Julian <julian.fondren gmail.com> writes:
On Tuesday, 16 April 2019 at 01:45:26 UTC, 9il wrote:

 Hi all,

 'DUB - call to arms' post by Seb
 https://forum.dlang.org/post/unimbbqxosxcmjdhtzch forum.dlang.org

 Hope we will move other direction and extend Meson build system 
 with better D support and make a new Dlang registry.


Is it just a build system? Isn't a package manager also needed?
Why a *new* registry?


 Reasons to move forward with Meson:

 1. Has clear python source code: faster to extend, easier to 
 maintain.


Oh. Well it was certainly a mistake to try use D for something
like dub. I've thought of D as a kind of competitor to Python
but I sure don't want slow to extend, hard to maintain code.


 4. Ability to compile and configure D code with other languages 
 like C/C++.


That's a surprising oversight. Was it intentional?
Apr 15 2019
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 16/04/2019 2:15 PM, Julian wrote:

 

 4. Ability to compile and configure D code with other languages like 
 C/C++.


 
 That's a surprising oversight. Was it intentional?


Out of scope. Dub was only meant to handle D programs using minimal 
external dependencies and for this purpose it has succeeded.
Apr 15 2019
parent reply 9il <ilyayaroshenko gmail.com> writes:
On Tuesday, 16 April 2019 at 02:30:25 UTC, rikki cattermole wrote:

 On 16/04/2019 2:15 PM, Julian wrote:

 

 4. Ability to compile and configure D code with other 
 languages like C/C++.


 
 That's a surprising oversight. Was it intentional?


 Out of scope. Dub was only meant to handle D programs using 
 minimal external dependencies and for this purpose it has 
 succeeded.


I don't think so. For example, dub does not allow to compile 
selected dependencies in release builds, while the current 
project in debug mode. This is a common case for numeric 
libraries like mir-glas (blas), mir-optim and etc.
So, we need to call dub as prebuild command 
https://github.com/libmir/mir-glas/blob/master/dub.json#L28

For simple system dependencies dub force users to define 
configuration monsters like that for mir-blas:
https://github.com/libmir/mir-blas/blob/master/dub.sdl

Maybe configurations for vibe.d and other packages looks no so 
monstrous, however for my uses cases, it adds a lot of pain. 
Also, this pain in inherited by dependencies:
https://github.com/libmir/mir-lapack/blob/master/dub.sdl
https://github.com/kaleidicassociates/lubeck/blob/master/dub.sdl
Apr 15 2019
parent reply JN <666total wp.pl> writes:
On Tuesday, 16 April 2019 at 05:11:29 UTC, 9il wrote:

 I don't think so. For example, dub does not allow to compile 
 selected dependencies in release builds, while the current 
 project in debug mode. This is a common case for numeric 
 libraries like mir-glas (blas), mir-optim and etc.
 So, we need to call dub as prebuild command 
 https://github.com/libmir/mir-glas/blob/master/dub.json#L28

 For simple system dependencies dub force users to define 
 configuration monsters like that for mir-blas:
 https://github.com/libmir/mir-blas/blob/master/dub.sdl

 Maybe configurations for vibe.d and other packages looks no so 
 monstrous, however for my uses cases, it adds a lot of pain. 
 Also, this pain in inherited by dependencies:
 https://github.com/libmir/mir-lapack/blob/master/dub.sdl
 https://github.com/kaleidicassociates/lubeck/blob/master/dub.sdl


Those "configuration monsters" don't look that bad, compared to 
your average Ant or CMake build script (can't speak for Meson, 
never used it).

The problem with using anything else is that we're back to square 
one. If you push Meson as an official build tool, soon people 
will want to bring CMake, SCons, Gradle, makefiles etc. into the 
mix. Which will be problematic, especially on Windows. Right now 
you can download dub and hope that most of stuff will build, 
without having to chase down dependencies and additional build 
tools. And after a while someone will propose, why do we use all 
these tools, why not create a single build tool, written in D, 
that can support these usecases. And then we'll reinvent dub 
again :)
Apr 15 2019
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Apr 16, 2019 at 06:13:30AM +0000, JN via Digitalmars-d wrote:
[...]

 The problem with using anything else is that we're back to square one.
 If you push Meson as an official build tool, soon people will want to
 bring CMake, SCons, Gradle, makefiles etc. into the mix. Which will be
 problematic, especially on Windows. Right now you can download dub and
 hope that most of stuff will build, without having to chase down
 dependencies and additional build tools. And after a while someone
 will propose, why do we use all these tools, why not create a single
 build tool, written in D, that can support these usecases. And then
 we'll reinvent dub again :)


The problem is that while dub fills a very important role in the D
ecosystem, it also has some unfortunate design decisions that has since
become major limitations as the ecosystem grows. It originally targeted
a very specific, and narrow use case, and the design was fine in that
context.  But as the D ecosystem grows to encompass wider use cases, the
inherent limitations in that design has since become a major issue, esp.
with those of us who need to do stuff beyond just "pull in dependencies,
build everything". For example, for me, supporting D codegen via
external tools (themselves needing to be compiled and run from the same
source tree) and mixed-language codebases (requiring multiple different
toolchains) is absolutely essential.  Since dub has no way of doing
this, it simply isn't even a viable option for me.  Performance is also
a big showstopper -- although it has improved since I last checked, the
code-compile-test turnaround time is still far too slow for everyday
usage.

What we really need is a dub replacement that has a different, more
flexible design that can accomodate things that people want to do, that
the current dub simply cannot due to its design limitations.  There are
some promising ideas in the other thread about improving dub, and I hope
that discussion will eventually lead to a redesign / revamp of dub that
makes it more viable for my use case.


T

-- 
The problem with the world is that everybody else is stupid.
Apr 16 2019
next sibling parent psycha0s <box mail.com> writes:
On Tuesday, 16 April 2019 at 16:42:24 UTC, H. S. Teoh wrote:

 What we really need is a dub replacement that has a different, 
 more flexible design that can accomodate things that people 
 want to do, that the current dub simply cannot due to its 
 design limitations.


I absolutely agree. Trying keep backward compatibility is a good 
intention but sometimes you just have to redo everything from 
scratch using accumulated experience. It can be painful but this 
is evolution. Otherwise it can lead to stagnation.
Apr 16 2019
prev sibling next sibling parent JN <666total wp.pl> writes:
On Tuesday, 16 April 2019 at 16:42:24 UTC, H. S. Teoh wrote:

 On Tue, Apr 16, 2019 at 06:13:30AM +0000, JN via Digitalmars-d 
 wrote: [...]
 What we really need is a dub replacement that has a different, 
 more flexible design that can accomodate things that people 
 want to do, that the current dub simply cannot due to its 
 design limitations.  There are some promising ideas in the 
 other thread about improving dub, and I hope that discussion 
 will eventually lead to a redesign / revamp of dub that makes 
 it more viable for my use case.


 T


Dub replacement doesn't sound good. D ecosystem is fragmented 
enough, no need to add additional split dub/dub replacement. I 
think it'd be best to just start with dub and try to improve on 
it. Cargo could be used as an inspiration, looks to me like in 
Cargo, CMake and Meson and other build systems are added as 
dependencies (https://docs.rs/meson/1.0.0/meson/ 
https://crates.io/crates/cmake ). Perhaps something like this 
could be adapted for dub. Some kind of plugin system, with 
optional cmake and meson plugins.
Apr 16 2019
prev sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Tuesday, 16 April 2019 at 16:42:24 UTC, H. S. Teoh wrote:

 On Tue, Apr 16, 2019 at 06:13:30AM +0000, JN via Digitalmars-d 
 wrote: [...]

 What we really need is a dub replacement that has a different, 
 more flexible design that can accomodate things that people 
 want to do, that the current dub simply cannot due to its 
 design limitations.  There are some promising ideas in the 
 other thread about improving dub, and I hope that discussion 
 will eventually lead to a redesign / revamp of dub that makes 
 it more viable for my use case.


 T


IMO, the best course of action is:
1.0. Create a generic foundation based on DAG for describing 
targets and their relations.
This can start as a completely independent project with zero 
dependencies.

1.1. On top of this core functionality we can add an 
interoperability layer that allows exporting this abstract 
representation to other build system formats.

1.2. We can reuse the existing build systems written in D as a 
build backend.

2.0. Integrate the new build system foundation in Dub 2.0:

2.1 Replace dub's current execution code with this new core while 
keeping backwards compatibility with existing dub projects

2.2 `dub build` consists of two general steps: 1) fetching 
dependencies 2) building each project in the set { 
dependencies..., top-level project } in topological order. (Of 
course there are many other (sub)steps, which I'm mentioning here 
for simplicity.) Now that in 2.1 we replaced Dub's existing 
execution code with the new build foundation we have the option 
to not execute those steps directly, but instead use Dub as a 
meta build-system which can generate build description for other 
build systems to execute.

2.3 Allow describing complex builds based on the foundation 
layer. This can be done either declaratively, directly in 
dub.json / dub.sdl, or dub can call to a D script that generates 
a JSON describing the build dependencies that the core described 
in 1.1 can read.

What do you think? Are you willing to help with the design and/or 
implemention on this initiative?
Apr 16 2019
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 17/04/2019 6:43 PM, Petar Kirov [ZombineDev] wrote:

 On Tuesday, 16 April 2019 at 16:42:24 UTC, H. S. Teoh wrote:

 On Tue, Apr 16, 2019 at 06:13:30AM +0000, JN via Digitalmars-d wrote: 
 [...]

 What we really need is a dub replacement that has a different, more 
 flexible design that can accomodate things that people want to do, 
 that the current dub simply cannot due to its design limitations.  
 There are some promising ideas in the other thread about improving 
 dub, and I hope that discussion will eventually lead to a redesign / 
 revamp of dub that makes it more viable for my use case.


 T


 
 IMO, the best course of action is:
 1.0. Create a generic foundation based on DAG for describing targets and 
 their relations.
 This can start as a completely independent project with zero dependencies.
 
 1.1. On top of this core functionality we can add an interoperability 
 layer that allows exporting this abstract representation to other build 
 system formats.
 
 1.2. We can reuse the existing build systems written in D as a build 
 backend.
 
 2.0. Integrate the new build system foundation in Dub 2.0:
 
 2.1 Replace dub's current execution code with this new core while 
 keeping backwards compatibility with existing dub projects
 
 2.2 `dub build` consists of two general steps: 1) fetching dependencies 
 2) building each project in the set { dependencies..., top-level project 
 } in topological order. (Of course there are many other (sub)steps, 
 which I'm mentioning here for simplicity.) Now that in 2.1 we replaced 
 Dub's existing execution code with the new build foundation we have the 
 option to not execute those steps directly, but instead use Dub as a 
 meta build-system which can generate build description for other build 
 systems to execute.
 
 2.3 Allow describing complex builds based on the foundation layer. This 
 can be done either declaratively, directly in dub.json / dub.sdl, or dub 
 can call to a D script that generates a JSON describing the build 
 dependencies that the core described in 1.1 can read.


That's going to require a ton of work and refactoring.
But it would be well worth funding after "1.0".
Apr 16 2019
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 17 April 2019 at 06:43:35 UTC, Petar Kirov 
[ZombineDev] wrote:

 On Tuesday, 16 April 2019 at 16:42:24 UTC, H. S. Teoh wrote:

 On Tue, Apr 16, 2019 at 06:13:30AM +0000, JN via Digitalmars-d 
 wrote: [...]

 What we really need is a dub replacement that has a different, 
 more flexible design that can accomodate things that people 
 want to do, that the current dub simply cannot due to its 
 design limitations.  There are some promising ideas in the 
 other thread about improving dub, and I hope that discussion 
 will eventually lead to a redesign / revamp of dub that makes 
 it more viable for my use case.


 T


 IMO, the best course of action is:
 1.0. Create a generic foundation based on DAG for describing 
 targets and their relations.
 This can start as a completely independent project with zero 
 dependencies.


I considered this; I even named the project. However, after I 
read this:

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=2ahUKEwjNgZLl3dbhAhXN66YKHXbEBkEQFjAAegQIBBAC&url=https%3A%2F%2Fwww.microsoft.com%2Fen-us%2Fresearch%2Fuploads%2Fprod%2F2018%2F03%2Fbuild-systems-final.pdf&usg=AOvVaw2j71hXjOoEQLNNjvEOp_RQ

I'm not sure just describing a DAG is enough. I've started 
playing about with the same concepts in D here:

https://github.com/atilaneves/rocksteady


 1.1. On top of this core functionality we can add an 
 interoperability layer that allows exporting this abstract 
 representation to other build system formats.


This might not always be possible. First, because of what I said 
above, and secondly, the feature sets sometimes don't match. I 
had to give up trying to express some concepts in the reggae tup 
backend because they're not doable there. And ninja requires 
modifications to the compiler that I solved by writing a compiler 
wrapper.


 2.1 Replace dub's current execution code with this new core 
 while keeping backwards compatibility with existing dub projects


I'm already doing that.


 2.2 `dub build` consists of two general steps: 1) fetching 
 dependencies


Which has nothing to do with building and should be done as a 
separate step.


 2) building each project in the set { dependencies..., 
 top-level project } in topological order.


The builds are independent, they should be done in parallel.


 2.3 Allow describing complex builds based on the foundation 
 layer. This can be done either declaratively, directly in 
 dub.json / dub.sdl, or dub can call to a D script that 
 generates a JSON describing the build dependencies that the 
 core described in 1.1 can read.


This would work for many build requirements, but not all.


 What do you think? Are you willing to help with the design 
 and/or implemention on this initiative?


Already doing it.
Apr 17 2019
parent "Schrom, Brian T" <Brian.Schrom pnnl.gov> writes:
If reimplementing build systems is being investigated, the
jam(perforce)/bjam(b2,boost)/boost.build/rypple searches should dig up a lot of
well thought through issues for very generic and cross platform considerations.
Apr 17 2019
prev sibling parent 9il <ilyayaroshenko gmail.com> writes:
On Tuesday, 16 April 2019 at 02:15:13 UTC, Julian wrote:

 On Tuesday, 16 April 2019 at 01:45:26 UTC, 9il wrote:

 Hi all,

 'DUB - call to arms' post by Seb
 https://forum.dlang.org/post/unimbbqxosxcmjdhtzch forum.dlang.org

 Hope we will move other direction and extend Meson build 
 system with better D support and make a new Dlang registry.


 Is it just a build system? Isn't a package manager also needed?
 Why a *new* registry?


The current registry is based on dub.sdl, dub.json. They can be 
supported as a plugin though.


 Reasons to move forward with Meson:

 1. Has clear python source code: faster to extend, easier to 
 maintain.


 Oh. Well it was certainly a mistake to try use D for something
 like dub. I've thought of D as a kind of competitor to Python
 but I sure don't want slow to extend, hard to maintain code.


 4. Ability to compile and configure D code with other 
 languages like C/C++.


 That's a surprising oversight. Was it intentional?


I mean dub can compile only D sources. Meson can compile 
integrated projects, where some code is C, some C++, some D and 
etc. It is a killer feature to involve new companies.  .
Apr 15 2019
prev sibling next sibling parent reply Andre Pany <andre s-e-a-p.de> writes:
On Tuesday, 16 April 2019 at 01:45:26 UTC, 9il wrote:

 Hi all,

 'DUB - call to arms' post by Seb
 https://forum.dlang.org/post/unimbbqxosxcmjdhtzch forum.dlang.org

 Hope we will move other direction and extend Meson build system 
 with better D support and make a new Dlang registry.

 Reasons to move forward with Meson:

 0. Progressive and large community
 1. Has clear python source code: faster to extend, easier to 
 maintain.
 2. D+Meson already used in production: last year it is a major 
 build system for all of Mir and its commercial applications.
 3. Meson language is super cool. It is much better than dub for 
 commercial projects, for large projects, for library 
 distribution.
 4. Ability to compile and configure D code with other languages 
 like C/C++.
 5. With Meson we don't need to catch up Rust, we can do better 
 than Rust with fewer resources.


 Best,
 Ilya


Don't you think we send following message with this step:
You cannot use the D Programming Language without installing 
Python. Python is able to do things, D is not able to do.


In my opinion this is a rather bad advertising):

Kind regards
Andre
Apr 16 2019
parent 9il <ilyayaroshenko gmail.com> writes:
On Tuesday, 16 April 2019 at 08:25:17 UTC, Andre Pany wrote:

 On Tuesday, 16 April 2019 at 01:45:26 UTC, 9il wrote:

 Don't you think we send following message with this step:
 You cannot use the D Programming Language without installing 
 Python. Python is able to do things, D is not able to do.


 In my opinion this is a rather bad advertising):

 Kind regards
 Andre


Well, bad advertising is advertising, I don't think it is really 
bad. --Ilya
Apr 16 2019
prev sibling next sibling parent drug <drug2004 bk.ru> writes:
On 16.04.2019 4:45, 9il wrote:

 Hi all,
 
 'DUB - call to arms' post by Seb
 https://forum.dlang.org/post/unimbbqxosxcmjdhtzch forum.dlang.org
 
 Hope we will move other direction and extend Meson build system with 
 better D support and make a new Dlang registry.
 
 Reasons to move forward with Meson:
 
 0. Progressive and large community
 1. Has clear python source code: faster to extend, easier to maintain.
 2. D+Meson already used in production: last year it is a major build 
 system for all of Mir and its commercial applications.
 3. Meson language is super cool. It is much better than dub for 
 commercial projects, for large projects, for library distribution.
 4. Ability to compile and configure D code with other languages like C/C++.
 5. With Meson we don't need to catch up Rust, we can do better than Rust 
 with fewer resources.
 
 
 Best,
 Ilya


Dub suppor for Meson (and other ones) is really useful thing, but it 
should be optional advanced feature.
Apr 16 2019
prev sibling next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Tuesday, 16 April 2019 at 01:45:26 UTC, 9il wrote:

 Hi all,

 'DUB - call to arms' post by Seb
 https://forum.dlang.org/post/unimbbqxosxcmjdhtzch forum.dlang.org

 Hope we will move other direction and extend Meson build system 
 with better D support and make a new Dlang registry.

 Reasons to move forward with Meson:

 0. Progressive and large community
 1. Has clear python source code: faster to extend, easier to 
 maintain.
 2. D+Meson already used in production: last year it is a major 
 build system for all of Mir and its commercial applications.
 3. Meson language is super cool. It is much better than dub for 
 commercial projects, for large projects, for library 
 distribution.
 4. Ability to compile and configure D code with other languages 
 like C/C++.
 5. With Meson we don't need to catch up Rust, we can do better 
 than Rust with fewer resources.


 Best,
 Ilya


Just for the records, an alternative registry already exists. You 
can store your dub packages in Maven (Nexus, Artifactory, ...) 
and set the Maven url as Dub registry.

Works like a charme, Dub does here a perfect job.

Kind regards
Andre
Apr 16 2019
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 16 April 2019 at 01:45:26 UTC, 9il wrote:

 Hi all,

 'DUB - call to arms' post by Seb
 https://forum.dlang.org/post/unimbbqxosxcmjdhtzch forum.dlang.org

 Hope we will move other direction and extend Meson build system 
 with better D support and make a new Dlang registry.


Why would we need a new registry? Even if we did, how would we 
convince the existing users that switching would be in their best 
interests?


 Reasons to move forward with Meson:

 0. Progressive and large community
 1. Has clear python source code: faster to extend, easier to 
 maintain.


This wouldn't be too important since said extension should really 
only be done once.


 2. D+Meson already used in production: last year it is a major 
 build system for all of Mir and its commercial applications.


My usage of meson with D revealed how bad its dub integration was.


 3. Meson language is super cool. It is much better than dub for 
 commercial projects, for large projects, for library 
 distribution.


Anything is better than dub as a build system, including raw make.


 4. Ability to compile and configure D code with other languages 
 like C/C++.


AFAIK, reggae, CMake and Premake can all do this as well.


 5. With Meson we don't need to catch up Rust, we can do better 
 than Rust with fewer resources.


I don't know what this means. They seem to be doing fine with 
cargo.
Apr 16 2019
parent Russel Winder <russel winder.org.uk> writes:
On Tue, 2019-04-16 at 12:19 +0000, Atila Neves via Digitalmars-d wrote:
[=E2=80=A6]

=20
 Anything is better than dub as a build system, including raw make.


That's going a bit far, and is fighting talk. But let's not even think
about that argument here. :-)

[=E2=80=A6]

=20
 AFAIK, reggae, CMake and Premake can all do this as well.


SCons, and I believe Meson, are also able to do mixed language builds
very well.

I started a Dub add-in to SCons to handle dependencies but I got
disheartened by the naming strategy of Dub's build of dependencies and
gave up. I'd be happy to restart work on D support for SCons if it had
any chance of being used, and SCons Dub support if there was a
deterministic algorithm for finding Dub builds.

[=E2=80=A6]

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Apr 16 2019