www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - reggae v0.10.0 - The meta build system just got better

reply Atila Neves <atila.neves gmail.com> writes:
https://code.dlang.org/packages/reggae

For those who don't know, reggae is a meta-build system for and 
in D. It's like CMake, if you replace their terrible language 
with D*. Like CMake, it can output make and ninja files. Unlike 
CMake, it can also output tup files and has its own binary 
generator in case the dependency on one of those programs is 
undesired. Also unlike CMake, it supports dub projects 
out-of-the-box by using dub as a library.

TL;DR: you can use to build your dub project and it'll be faster 
and not rebuild the whole thing if you only change one module**.

Highlights of this release:

When dinosaurs roamed the Earth, `dmd -makedeps` didn't work so 
to automatically handle dependencies between D modules reggae had 
to wrap the compiler with a binary named `dcompile`. This would 
call the compiler with `-v`, get the imports from the output, and 
use them to only rebuild what was necessary. Now that `dmd 
-makedeps` got fixed, the extra complication (and time to build 
the extra binary) is gone.

Default dub builds are much faster now. Before, reggae would 
create a reggaefile.d if one didn't exist for dub projects. I got 
tired of waiting for the compiler, and realised that I don't 
actually have to compile D code on-the-fly if I know ahead of 
time what the code is anyway. Now, reggae immediately*** spits 
out the build files for such projects.

Automatically detect when new files are added/deleted: this is 
the main reason that the CMake devs erroneously advise people to 
not use "globs" to just "give me all the source files". The issue 
is when the file list changes, you need to remember to rerun the 
metabuild system to detect the filesystem changes. No more: 
reggae builds now rerun reggae when files are added or deleted. 
Tup, of course, has always done this but it's the only one I know 
of that did.

Before, if the compiler binary changed, then... oops it was if 
you rebuilt in between updating your distro. Now the object files 
depend on the compiler binary.



* Or Python, Ruby, Javascript, or Lua.
** Unless you made every module depend on every other module, in 
which case, don't do that.
** dub can be slow to start up. Slows for me is ~200ms.
Sep 07 2023
next sibling parent reply German Diago <germandiag gmail.com> writes:
On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves wrote:

 https://code.dlang.org/packages/reggae

 For those who don't know, reggae is a meta-build system for and 
 in D. It's like CMake...


How mature is the build system? FWIW, I would like to stay in 
Meson if I can, since I use C++ also and that makes combining 
libs and other niceties easier.
Sep 07 2023
parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 7 September 2023 at 21:25:27 UTC, German Diago wrote:

 On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves 
 wrote:

 https://code.dlang.org/packages/reggae

 For those who don't know, reggae is a meta-build system for 
 and in D. It's like CMake...


 How mature is the build system? FWIW, I would like to stay in 
 Meson if I can, since I use C++ also and that makes combining 
 libs and other niceties easier.


Mature enough that we've been using it at work for years now, but 
YMMV depending on what you use it for. It also supports C++ BTW.
Sep 07 2023
prev sibling next sibling parent Anonymouse <zorael gmail.com> writes:
On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves wrote:

 [...]


Can reggae handle non-trivial dub builds with trees of 
dependencies?

I gave it a quick test, following the examples in the readme, and 
the Makefile it generated looked for sources in the wrong place. 
It also didn't build the dependencies (nor even include their 
source directories as import paths). Maybe I'm doing it wrong.
Sep 09 2023
prev sibling next sibling parent Greggor <Greggor notareal.email> writes:
On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves wrote:

 https://code.dlang.org/packages/reggae

 For those who don't know, reggae is a meta-build system for and 
 in D. It's like CMake, if you replace their terrible language 
 with D*. Like CMake, it can output make and ninja files. Unlike 
 CMake, it can also output tup files and has its own binary 
 generator in case the dependency on one of those programs is 
 undesired. Also unlike CMake, it supports dub projects 
 out-of-the-box by using dub as a library.

 [...]


I spent a some time getting a part of my project to build with 
Reggae & so far I am impressed, The partial compiles are really 
good, I am looking forward to switching!

My largest complaint is that Reggae needs improvements to it's 
documentation, there was enough of it for me to get started and I 
can figure it out, but I do not feel that anyone can just jump 
right into using it. But this is not a unique problem especially 
with build systems, so this is a nitpick.
Sep 11 2023
prev sibling next sibling parent reply Adam D Ruppe <destructionator gmail.com> writes:
I tried reggae today. It did not go well.

http://dpldocs.info/this-week-in-d/Blog.Posted_2023_09_11.html#reggae-editorial
Sep 11 2023
parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 11 September 2023 at 23:15:28 UTC, Adam D Ruppe wrote:

 I tried reggae today. It did not go well.

 http://dpldocs.info/this-week-in-d/Blog.Posted_2023_09_11.html#reggae-editorial


One thing I learned from you is that for small projects `dmd -i` 
is just as fast as incremental compilation, which is rather 
surprising to me. It does mean adding `-I` flags to every 
dependency though, so there's that.

And for larger codebases `dmd -i` can't and won't work.
Sep 12 2023
parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Tuesday, 12 September 2023 at 13:12:29 UTC, Atila Neves wrote:

 It does mean adding `-I` flags to every dependency though, so 
 there's that.


Not if you install them properly.


 And for larger codebases `dmd -i` can't and won't work.


Define "larger".
Sep 12 2023
parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 12 September 2023 at 13:17:50 UTC, Adam D Ruppe wrote:

 On Tuesday, 12 September 2023 at 13:12:29 UTC, Atila Neves 
 wrote:

 It does mean adding `-I` flags to every dependency though, so 
 there's that.


 Not if you install them properly.


That's the job of the package manager does that. An argument 
could be made that it could/should install the dependencies such 
that only one `-I` flag is needed. Even then, I'm not sure how 
it'd handle different versions of the same package unless it did 
something like what Ikey showed at DConf.


 And for larger codebases `dmd -i` can't and won't work.


 Define "larger".


~190k SLOC (not counting the many dub dependencies) killed dmd on 
a system with 64GB RAM + 64GB swap after over a minute. Even if 
it worked, it'd be much, much slower.
Sep 15 2023
parent Adam D Ruppe <destructionator gmail.com> writes:
On Friday, 15 September 2023 at 20:22:50 UTC, Atila Neves wrote:

 An argument could be made that it could/should install the 
 dependencies such that only one `-I` flag is needed.


Indeed, this would be god tier.


 ~190k SLOC (not counting the many dub dependencies) killed dmd 
 on a system with 64GB RAM + 64GB swap after over a minute. Even 
 if it worked, it'd be much, much slower.


What you do with the lines of code is *far* more important than 
how many there are.

The arsd library has about 219,000 lines of text if you delete 
the Windows-only and obsolete modules (doing so just so I can 
actually dmd *.d here on my Linux box). This includes comments 
and such; dscanner --sloc reports about 98,000.

$ wc *.d
<snip>
  218983  870208 7134770 total
$ dscanner --sloc *.d
<snip>
total:  98645

Let's compile it all:

$ /usr/bin/time dmd *.d -L-L/usr/local/pgsql/lib  -unittest 
-L-lX11
5.35user 0.72system 0:06.08elapsed 99%CPU (0avgtext+0avgdata 
1852460maxresident)k
0inputs+70464outputs (0major+536358minor)pagefaults 0swaps

That's a little bit slow, over 5 seconds. About 1.3 of those 
seconds are spent in the linker, the other 4 remain with dmd -c. 
Similarly, that's almost 2 GB of RAM it used, more than it 
probably should, but it worked fine.

My computer btw is a budget model circa 2016. Nothing 
extraordinary about its hardware.

But notice it isn't actually running out of RAM or melting the 
CPU over a period of minutes, despite being about six figures 
lines of code but any measure.


On the other hand, compile:

enum a = () {
    string s;
    foreach(i; 0 .. 20_000_000_000)
      s ~= 'a';
    return s;
}();


Don't actually do it, but you can imagine what will happen. 6 
lines that can spin your cpu and explode your memory. Indeed, 
even just importing this module, even if the build system tried 
not to compile it again, will cause the same problem.

The arsd libs are written - for the most part, there's some 
exceptions - with compile speed in mind. If I see my build slow 
down, I investigate why. Most problems like this can be fixed!

In fact, let's take that snippet and talk about it. I had to 
remove *several* zeroes to make it even work without freezing up 
my computer, but with a 100,000 item loop, it just barely worked. 
Even 200,000 made it OOM.

But ok, a 100,000 item append:

0.53user 1.52system 0:02.17elapsed 95%CPU (0avgtext+0avgdata 
4912656maxresident)k

About 5 GB of RAM devoured by these few lines, taking 2 seconds 
to run. What are some ways we can fix this? The ~= operator is 
actually *awful* at CTFE, its behavior is quadratic (...or worse, 
i didn't confirm this today, but it is obviously bad). So you can 
fix this pretty easily:

enum string a = () {
    // preallocate the buffer instead of append
    char[] s = new char[](100000);
    foreach(ref ch; s)
      ch = 'a';
    return s;
}();

0.17user 0.03system 0:00.21elapsed 98%CPU (0avgtext+0avgdata 
45748maxresident)k 16inputs+1408outputs 
(0major+21995minor)pagefaults 0swaps

Over 10x faster to compile, 1/100th of the RAM, ram result. Real 
world code is frequently doing more than this example and 
rewriting it to work like this might take some real effort.... 
but the results are worth it.

And btw try this: import this module and check your time/memory 
stats. Even if it isn't compiled, since ctfe is run when the 
module is even just imported, you gain *nothing* by separate 
compilation!

...but there are times when you can gain a LOT by separate 
compilation in situations like this, if you can move the ctfe to 
be some private thing not exposed in the interface. This requires 
some work by the lib author too though in most cases. An example 
where you can gain a lot is when something does a lot of internal 
code generation but exposes a small interface, for example a 
scripting language wrapper (though script wrappers can also be 
made to compile reasonably efficiently if you use things like 
preallocation of buffers, keep your generated functions short 
(again, the codegen has quadratic behavior, so many small 
functions work better than a big one, and if you factor the code 
well, you can minimize the amount of generated code and call back 
to generic things, e.g. type erasure), collapse template 
instances, and keep ctfe things ctfe only with a variety of 
techniques, so they are not codegened unless they are actually 
necessary).

My arsd.script and arsd.cgi can wrap large numbers of functions 
and classes reasonably well, but that's why programs using them 
tend to be multi-second builds.... just note that's programs 
using them. Separate compiling the libraries doesn't help. You'd 
have to structure the code to keep those codegen parts internal 
to a package with a minimal interface, then separate compiling 
those internal components might win.

But this is a fairly niche case. Yes, I know there's one major 
commercial D user who do exactly this. But that's the exception, 
not the rule.
Sep 17 2023
prev sibling next sibling parent zjh <fqbqrr 163.com> writes:
On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves wrote:

 https://code.dlang.org/packages/reggae


You can use professional `xmake`,
It's very convenient.
Sep 11 2023
prev sibling parent reply Andrey Zherikov <andrey.zherikov gmail.com> writes:
On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves wrote:

 https://code.dlang.org/packages/reggae

 For those who don't know, reggae is a meta-build system for and 
 in D. It's like CMake, if you replace their terrible language 
 with D*. Like CMake, it can output make and ninja files. Unlike 
 CMake, it can also output tup files and has its own binary 
 generator in case the dependency on one of those programs is 
 undesired. Also unlike CMake, it supports dub projects 
 out-of-the-box by using dub as a library.


Out of curiosity, why do we need one more build tool?

TBH I had the same thought - we should have "good" build tool for 
D that has D as a language. Since I have some experience with 
CMake and agree with a lot of people that it have horrible 
language, I thought it would be beneficial. But then I looked 
round and realized that it will be like this:
![](https://imgs.xkcd.com/comics/standards.png)
So I dropped that idea and think that investing into existing 
build tool to support D will be much more efficient.
Have you looked as Meson, for example - it even has a [section 
about D](https://mesonbuild.com/D.html)?
Sep 20 2023
parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 20 September 2023 at 15:24:52 UTC, Andrey Zherikov 
wrote:

 On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves 
 wrote:

 https://code.dlang.org/packages/reggae

 For those who don't know, reggae is a meta-build system for 
 and in D. It's like CMake, if you replace their terrible 
 language with D*. Like CMake, it can output make and ninja 
 files. Unlike CMake, it can also output tup files and has its 
 own binary generator in case the dependency on one of those 
 programs is undesired. Also unlike CMake, it supports dub 
 projects out-of-the-box by using dub as a library.


 Out of curiosity, why do we need one more build tool?


Because we don't have one now. Using CMake for D is horrible, and 
none of the alternatives are that much better. Hand-written 
Makefiles are a nightmare and will never get dependencies right. 
dub isn't a build system, although it includes a very limited one 
that can't be extended. Want to build D and C++ code in the same 
project? Hah, good luck.


 TBH I had the same thought - we should have "good" build tool 
 for D that has D as a language. Since I have some experience 
 with CMake and agree with a lot of people that it have horrible 
 language, I thought it would be beneficial. But then I looked 
 round and realized that it will be like this:


In practice, this doesn't seem to happen with build systems. 
Instead each language ecosystem ends up using its own: CMake, 
Gradle, Rake, ...


 Have you looked as Meson, for example - it even has a [section 
 about D](https://mesonbuild.com/D.html)?


Yes, and even had to use it for D. I don't know what state it's 
in now, but it was pretty much unusable if one had dub 
dependencies which... is nearly always the case.

Then there's the language: I'd rather use D or Python. If 
anything, Meson's use case could be questioned just as well by 
"why another build language?".
Sep 20 2023
next sibling parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Wednesday, 20 September 2023 at 21:19:22 UTC, Atila Neves 
wrote:


 Then there's the language: I'd rather use D or Python.


Someone, somewhere, almost 20 years ago ...

https://forum.dlang.org/post/40BB3D47.9030007 inwind.it

In the D land, everything always changes, to never really change 
... it's really a big wheel, or better a pendulum! :-P

/Paolo
Sep 21 2023
prev sibling next sibling parent reply Andrey Zherikov <andrey.zherikov gmail.com> writes:
On Wednesday, 20 September 2023 at 21:19:22 UTC, Atila Neves 
wrote:

 Because we don't have one now. Using CMake for D is horrible,


I would say just using CMake is horrible :-D But there are a lot 
of people using it (even myself) and they all are trapped with 
the language.


 Hand-written Makefiles are a nightmare and will never get 
 dependencies right. dub isn't a build system, although it 
 includes a very limited one that can't be extended.


I agree that dub is not a build tool, it's more a package manager.


 TBH I had the same thought - we should have "good" build tool 
 for D that has D as a language. Since I have some experience 
 with CMake and agree with a lot of people that it have 
 horrible language, I thought it would be beneficial. But then 
 I looked round and realized that it will be like this:


 In practice, this doesn't seem to happen with build systems. 
 Instead each language ecosystem ends up using its own: CMake, 
 Gradle, Rake, ...


IMHO this does not prove that D must have its own build system.


 Have you looked as Meson, for example - it even has a [section 
 about D](https://mesonbuild.com/D.html)?


 Yes, and even had to use it for D. I don't know what state it's 
 in now, but it was pretty much unusable if one had dub 
 dependencies which... is nearly always the case.


I believe this is applicable to all build systems atm - they 
don't know what dub is and how to deal with it.


 Then there's the language: I'd rather use D or Python. If 
 anything, Meson's use case could be questioned just as well by 
 "why another build language?".


TBH I thought Meson uses python but just found out it's not. My 
bad.
Just took a quick look at [known build 
tools](https://en.wikipedia.org/wiki/List_of_build_automation_software
Licensing_overview) and noticed that there are (at least) two build tools that
have Python as a build language and support D:
[SCons](https://en.wikipedia.org/wiki/SCons#Major_features) and
[Waf](https://en.wikipedia.org/wiki/Waf_(build_system)#Features). Although I
don't how well they support it.


My whole comment was about the reasoning why we need new build 
system and can't improve existing one. So I would expect more 
wide analysis and comparison 
[here](https://github.com/atilaneves/reggae#why), not just with 
CMake. My main concern is that if we create a brand new build 
tool, years later it might turn out that other existing tool got 
wide adoption for D so this project become abandoned and all 
efforts are thrown away.
Sep 21 2023
parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 21 September 2023 at 13:38:30 UTC, Andrey Zherikov 
wrote:

 On Wednesday, 20 September 2023 at 21:19:22 UTC, Atila Neves 
 wrote:

 Because we don't have one now. Using CMake for D is horrible,


 I would say just using CMake is horrible :-D But there are a 
 lot of people using it (even myself) and they all are trapped 
 with the language.


Not anymore! ;)


 Have you looked as Meson, for example - it even has a 
 [section about D](https://mesonbuild.com/D.html)?


 Yes, and even had to use it for D. I don't know what state 
 it's in now, but it was pretty much unusable if one had dub 
 dependencies which... is nearly always the case.


 I believe this is applicable to all build systems atm - they 
 don't know what dub is and how to deal with it.


Well, no, since reggae *does* know what dub is and uses it as a 
library.


 Then there's the language: I'd rather use D or Python. If 
 anything, Meson's use case could be questioned just as well by 
 "why another build language?".


 TBH I thought Meson uses python but just found out it's not. My 
 bad.
 Just took a quick look at [known build 
 tools](https://en.wikipedia.org/wiki/List_of_build_automation_software
Licensing_overview) and noticed that there are (at least) two build tools that
have Python as a build language and support D:
[SCons](https://en.wikipedia.org/wiki/SCons#Major_features) and
[Waf](https://en.wikipedia.org/wiki/Waf_(build_system)#Features). Although I
don't how well they support it.


I'm aware of both and looked at them many, many, years ago.


 My whole comment was about the reasoning why we need new build 
 system and can't improve existing one.


dub, for starters, and the fact that `dub describe` doesn't work.

Then there's building by D package, which isn't going to be easy 
anywhere else.
Sep 21 2023
parent reply Andrey Zherikov <andrey.zherikov gmail.com> writes:
On Thursday, 21 September 2023 at 15:59:10 UTC, Atila Neves wrote:

 ...


I got your point.

Why does it have multiple languages (front-ends)? Is there anyone 
willing to write build script for C++ on JavaScript or Ruby, for 
example?

Also why is it meta build system? Why can't it just build by 
itself without calling external tools (make, ninja)?

PS I tried to reach you out on Discord and LinkedIn but no 
response so far.
Sep 26 2023
parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 26 September 2023 at 16:17:34 UTC, Andrey Zherikov 
wrote:

 On Thursday, 21 September 2023 at 15:59:10 UTC, Atila Neves 
 wrote:

 ...


 I got your point.

 Why does it have multiple languages (front-ends)? Is there 
 anyone willing to write build script for C++ on JavaScript or 
 Ruby, for example?


I thought that if I built it people would come, but nah.


 Also why is it meta build system?


Because I think that's a superior design. For instance, all the 
start up time dub has to spend on every build disappears.


 Why can't it just build by itself without calling external 
 tools (make, ninja)?


It can: 
https://github.com/atilaneves/reggae/blob/master/bootstrap.sh
Sep 27 2023
parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Wednesday, 27 September 2023 at 15:13:19 UTC, Atila Neves 
wrote:

 On Tuesday, 26 September 2023 at 16:17:34 UTC, Andrey Zherikov 
 wrote:

 On Thursday, 21 September 2023 at 15:59:10 UTC, Atila Neves 
 wrote:

 ...


 I got your point.

 Why does it have multiple languages (front-ends)? Is there 
 anyone willing to write build script for C++ on JavaScript or 
 Ruby, for example?


 I thought that if I built it people would come, but nah.


 Also why is it meta build system?


 Because I think that's a superior design. For instance, all the 
 start up time dub has to spend on every build disappears.


 Why can't it just build by itself without calling external 
 tools (make, ninja)?


 It can: 
 https://github.com/atilaneves/reggae/blob/master/bootstrap.sh


Fwiw I think it look interesting. Will try it
Sep 27 2023
prev sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 20 September 2023 at 21:19:22 UTC, Atila Neves 
wrote:

 On Wednesday, 20 September 2023 at 15:24:52 UTC, Andrey 
 Zherikov wrote:

 On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves 
 wrote:

 https://code.dlang.org/packages/reggae

 For those who don't know, reggae is a meta-build system for 
 and in D. It's like CMake, if you replace their terrible 
 language with D*. Like CMake, it can output make and ninja 
 files. Unlike CMake, it can also output tup files and has its 
 own binary generator in case the dependency on one of those 
 programs is undesired. Also unlike CMake, it supports dub 
 projects out-of-the-box by using dub as a library.


 Out of curiosity, why do we need one more build tool?


 Because we don't have one now. Using CMake for D is horrible, 
 and none of the alternatives are that much better. Hand-written 
 Makefiles are a nightmare and will never get dependencies 
 right. dub isn't a build system, although it includes a very 
 limited one that can't be extended. Want to build D and C++ 
 [snip]


You can call CMake with a pre-build command right? Would it make 
sense to add an example of that?

It also might make sense to include in the tests/projects folder 
an importC example (not sure it makes sense to include a CMake 
example there).
Sep 21 2023
parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 21 September 2023 at 14:19:27 UTC, jmh530 wrote:

 On Wednesday, 20 September 2023 at 21:19:22 UTC, Atila Neves 
 wrote:

 On Wednesday, 20 September 2023 at 15:24:52 UTC, Andrey 
 Zherikov wrote:

 On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves 
 wrote:

 [...]


 Out of curiosity, why do we need one more build tool?


 Because we don't have one now. Using CMake for D is horrible, 
 and none of the alternatives are that much better. 
 Hand-written Makefiles are a nightmare and will never get 
 dependencies right. dub isn't a build system, although it 
 includes a very limited one that can't be extended. Want to 
 build D and C++ [snip]


 You can call CMake with a pre-build command right? Would it 
 make sense to add an example of that?


Yes, and I've done that, but it's not great. I'm going to add 
CMake support to reggae by making CMake export its compilation DB 
and importing that.


 It also might make sense to include in the tests/projects 
 folder an importC example (not sure it makes sense to include a 
 CMake example there).


That's a great idea, thanks!
Sep 21 2023
parent jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 21 September 2023 at 15:54:26 UTC, Atila Neves wrote:

 [snip]

 Yes, and I've done that, but it's not great. I'm going to add 
 CMake support to reggae by making CMake export its compilation 
 DB and importing that.



Interesting.
Sep 21 2023