www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - reggae v0.5.0: new features in the D meta-build system

reply Atila Neves <atila.neves gmail.com> writes:
http://code.dlang.org/my_packages/reggae

What's new:
. API changes: main high-level rules are now called objectFiles, 
link, and scriptlike
. Optional top-level targets: aren't built by default but can be 
built on request
. Phony targets
. staticLibrary rule that does what it says
. unityBuild rule for C/C++ projects that builds a binary as one 
translation unit to speed up compilation
. Support for writing build descriptions in Python, Ruby, Lua and 
JavaScript

That last one might cause scratchy head syndrome. Why would I 
want to enable writing builds in a language other than D? Here's 
why:

1. Interpreting a build description is faster than compiling, 
linking and running it. Not a big deal since the build 
description isn't read often, but it's true
2. No confusion about when to use `enum` and when to use `alias`. 
Again, probably wasn't catching anybody unawares but also true
3. Not having to install a D compiler in order to generate builds 
- I ran into this issue at work the other day trying to use 
reggae on a weird VM with no root access
4. More importantly... to not limit reggae to only D developers

There are quite a few build systems / tools out there in which 
different languages are used. Almost all of them are only used by 
developers of their language niche: Ruby/Rake, Python/SCons/Waf, 
Groovy/Java/Gradle, Haskell/Shake, etc. This is my attempt to 
make reggae appeal to a wider audience.

I wanted to add a Lisp too but then I'd have to pick a dialect 
and an implemetation, only to annoy fans of other dialects. I 
might just do Emacs Lisp to annoy everyone equally! Also, for the 
lulz.

Atila
Sep 16 2015
next sibling parent Leandro Motta Barros via Digitalmars-d-announce writes:
This link should work for everyone: http://code.dlang.org/packages/reggae

(I never tried reggae. Maybe I should, it looks good.)

LMB


On Wed, Sep 16, 2015 at 11:07 AM, Atila Neves via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:

 http://code.dlang.org/my_packages/reggae

 What's new:
 . API changes: main high-level rules are now called objectFiles, link, and
 scriptlike
 . Optional top-level targets: aren't built by default but can be built on
 request
 . Phony targets
 . staticLibrary rule that does what it says
 . unityBuild rule for C/C++ projects that builds a binary as one
 translation unit to speed up compilation
 . Support for writing build descriptions in Python, Ruby, Lua and
 JavaScript

 That last one might cause scratchy head syndrome. Why would I want to
 enable writing builds in a language other than D? Here's why:

 1. Interpreting a build description is faster than compiling, linking and
 running it. Not a big deal since the build description isn't read often,
 but it's true
 2. No confusion about when to use `enum` and when to use `alias`. Again,
 probably wasn't catching anybody unawares but also true
 3. Not having to install a D compiler in order to generate builds - I ran
 into this issue at work the other day trying to use reggae on a weird VM
 with no root access
 4. More importantly... to not limit reggae to only D developers

 There are quite a few build systems / tools out there in which different
 languages are used. Almost all of them are only used by developers of their
 language niche: Ruby/Rake, Python/SCons/Waf, Groovy/Java/Gradle,
 Haskell/Shake, etc. This is my attempt to make reggae appeal to a wider
 audience.

 I wanted to add a Lisp too but then I'd have to pick a dialect and an
 implemetation, only to annoy fans of other dialects. I might just do Emacs
 Lisp to annoy everyone equally! Also, for the lulz.

 Atila
Sep 16 2015
prev sibling next sibling parent Kagamin <spam here.lot> writes:
Ruby link points to reggae-python.
Sep 16 2015
prev sibling next sibling parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Wednesday, 16 September 2015 at 14:07:17 UTC, Atila Neves 
wrote:
 http://code.dlang.org/my_packages/reggae

 What's new:
 Atila
If you want to build a really revolutionary *new* build system you should turn reggae into a client-server-architecture that listens to file moves/copies/notifications via inotify on Linux. I'm not sure how the develop/builder interface would look like then, however. A web-client would be one way. SCons has a very hidden feature called interactive mode via `--interactive` that supports instantaenous incremental builds via a very primitive CLI that basically supports to commands: - build/b TARGET - clean/c TARGET This however does not rebuild the dependency tree between invokations which is a serious limitation. I believe reggae could be even smarter here. I've spent the last two years designing and implementing a SCons-based build system at work. Involving, among others, adding support for Ada and auto-detection of include-paths as you have done in reggae. I you want any advice on this matter please contact me. I'd be glad to be of service. Good luck!
Sep 22 2015
next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Tuesday, 22 September 2015 at 12:39:48 UTC, Per Nordlöw wrote:
 On Wednesday, 16 September 2015 at 14:07:17 UTC, Atila Neves 
 wrote:
 http://code.dlang.org/my_packages/reggae

 What's new:
 Atila
If you want to build a really revolutionary *new* build system you should turn reggae into a client-server-architecture that listens to file moves/copies/notifications via inotify on Linux. I'm not sure how the develop/builder interface would look like then, however. A web-client would be one way.
Like ekam? https://github.com/sandstorm-io/ekam We talked about it when I interviewed Atila: http://arsdnet.net/this-week-in-d/sep-06.html
Sep 22 2015
parent Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Tuesday, 22 September 2015 at 13:22:25 UTC, Joakim wrote:
 Like ekam?
 https://github.com/sandstorm-io/ekam
Sounds very promising!
 We talked about it when I interviewed Atila:
 http://arsdnet.net/this-week-in-d/sep-06.html
Thanks!
Sep 22 2015
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-09-22 14:39, Per Nordlöw wrote:

 SCons has a very hidden feature called interactive mode via
 `--interactive` that supports instantaenous incremental builds via a
 very primitive CLI that basically supports to commands:
Incremental builds in D are currently not reliable. Something about templates not being outputted to all object files it need to. -- /Jacob Carlborg
Sep 22 2015
parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 23 September 2015 at 06:16:08 UTC, Jacob Carlborg 
wrote:
 On 2015-09-22 14:39, Per Nordlöw wrote:

 SCons has a very hidden feature called interactive mode via
 `--interactive` that supports instantaenous incremental builds 
 via a
 very primitive CLI that basically supports to commands:
Incremental builds in D are currently not reliable. Something about templates not being outputted to all object files it need to.
That's not been my experience at all using reggae. I only do incremental builds now and have never run into a problem. Can you give an example? Atila
Sep 24 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-09-24 16:46, Atila Neves wrote:

 That's not been my experience at all using reggae. I only do incremental
 builds now and have never run into a problem. Can you give an example?
Here's one old post [1] that describes the problem. I'm not sure how much of it still applies today. [1] http://forum.dlang.org/thread/h8ddc5$1h67$1 digitalmars.com -- /Jacob Carlborg
Sep 25 2015
parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 25 September 2015 at 12:09:01 UTC, Jacob Carlborg 
wrote:
 On 2015-09-24 16:46, Atila Neves wrote:

 That's not been my experience at all using reggae. I only do 
 incremental
 builds now and have never run into a problem. Can you give an 
 example?
Here's one old post [1] that describes the problem. I'm not sure how much of it still applies today. [1] http://forum.dlang.org/thread/h8ddc5$1h67$1 digitalmars.com
I wasn't around in 2009 so I don't how dmd was then. How does one compile 3 files "at the same time" and generate 3 object files? There was a reference to a -multiobj option in that post but that's not even in the man page. Even if it exists and works as I assume it does, it seems to me to be a silly way to compile files. As Andrei mentioned at DConf this year, the D idiom is to group source files per package when compiling. That's what reggae does by default for D files. It just works. Atila
Sep 25 2015
parent reply anonymous <anonymous example.com> writes:
On Friday 25 September 2015 23:27, Atila Neves wrote:

 How does one compile 3 files "at the same time" and generate 3 
 object files? There was a reference to a -multiobj option in that 
 post but that's not even in the man page.
dmd -c foo.d bar.d baz.d rdmd would probably do this by now, if it weren't for the template instantiation issue. There's a nice, simple example of the issue in the PR discussion: https://github.com/D-Programming-Language/tools/pull/170#issuecomment-112526734
 Even if it exists and works as I assume it does, it seems to me 
 to be a silly way to compile files. As Andrei mentioned at DConf 
 this year, the D idiom is to group source files per package when 
 compiling. That's what reggae does by default for D files. It 
 just works.
Aside from the template instantiation issue, I don't see how doing a package at a time is better than compiling exactly those files that need to be compiled.
Sep 25 2015
parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 25 September 2015 at 22:12:49 UTC, anonymous wrote:
 On Friday 25 September 2015 23:27, Atila Neves wrote:

 How does one compile 3 files "at the same time" and generate 3 
 object files? There was a reference to a -multiobj option in 
 that post but that's not even in the man page.
dmd -c foo.d bar.d baz.d
Huh, I didn't know that. I guess I've always used the `-of` option. Unsurprisingly, so does reggae. Which means that incremental compilation is no longer an issue.
 Aside from the template instantiation issue, I don't see how 
 doing a package at a time is better than compiling exactly 
 those files that need to be compiled.
There have been threads about this before. It turns out that compiling per file is usually slower than compiling the whole package/app at once. It's not intuitive, but it's true (I measured it myself). reggae has an option to compile per-file but I haven't used it since switching to per-package once. Atila
Sep 25 2015
next sibling parent reply anonymous <anonymous example.com> writes:
On Saturday 26 September 2015 01:24, Atila Neves wrote:

 There have been threads about this before. It turns out that 
 compiling per file is usually slower than compiling the whole 
 package/app at once. It's not intuitive, but it's true (I 
 measured it myself). reggae has an option to compile per-file but 
 I haven't used it since switching to per-package once.
Compiling one file at a time is yet another thing. 1) Compile everything into one object file: dmd -c -ofresult.o foo.d bar.d 2) Compile to multiple object files in one run: dmd -c foo.d bar.d 3) Compile to multiple object files in multiple runs: dmd -c foo.d; dmd -c bar.d Since you didn't know about 2, I take it that you measured 1 vs 3. 3 is bound to be slower when the modules share dependencies, because they have to be parsed again and again. 2 may be faster or slower than 1, I don't know, a quick check didn't show a difference. When the template instantiation issue ever gets fixed, I wouldn't be surprised if 2 got slower than 1. Of course, any slowness of 2 must then be weighed against 1 compiling more than necessary.
Sep 25 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-09-26 01:53, anonymous wrote:

 Compiling one file at a time is yet another thing.

 1) Compile everything into one object file:
 dmd -c -ofresult.o foo.d bar.d
 2) Compile to multiple object files in one run:
 dmd -c foo.d bar.d
 3) Compile to multiple object files in multiple runs:
 dmd -c foo.d; dmd -c bar.d

 Since you didn't know about 2, I take it that you measured 1 vs 3. 3 is
 bound to be slower when the modules share dependencies, because they have to
 be parsed again and again.

 2 may be faster or slower than 1, I don't know, a quick check didn't show a
 difference. When the template instantiation issue ever gets fixed, I
 wouldn't be surprised if 2 got slower than 1. Of course, any slowness of 2
 must then be weighed against 1 compiling more than necessary.
In theory 2 should be faster when it comes to recompiling because fewer files need to be recompiled. -- /Jacob Carlborg
Sep 26 2015
prev sibling parent reply Andy Smith <andyrsmith googlemail.com> writes:
HI Atilla,

Dub's looking interesting! Some of the links are broken when 
browsing from code.dlang.org though.

 From http://code.dlang.org/packages/reggae

click on, say, 'detailed documentation'. ( Where I wanted to go!).

Get ..

404 - Not Found

Not Found

Internal error information:
No routes match path '/packages/doc/index.md'


Not sure if that's problem with reggae's layout of the site....

Cheers,

A.
Sep 26 2015
next sibling parent Andy Smith <andyrsmith googlemail.com> writes:
On Saturday, 26 September 2015 at 08:23:46 UTC, Andy Smith wrote:
 HI Atilla,

 Dub's looking interesting! Some of the links are broken when 
 browsing from code.dlang.org though.

 From http://code.dlang.org/packages/reggae

 click on, say, 'detailed documentation'. ( Where I wanted to 
 go!).

 Get ..

 404 - Not Found

 Not Found

 Internal error information:
 No routes match path '/packages/doc/index.md'


 Not sure if that's problem with reggae's layout of the site....

 Cheers,

 A.
^dub^reggae^ :-) Doh! - morning coffee hasn't kicked in yet! Cheers, A.
Sep 26 2015
prev sibling parent =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig rejectedsoftware.com> writes:
Am 26.09.2015 um 10:23 schrieb Andy Smith:
 HI Atilla,

 Dub's looking interesting! Some of the links are broken when browsing
 from code.dlang.org though.

  From http://code.dlang.org/packages/reggae

 click on, say, 'detailed documentation'. ( Where I wanted to go!).

 Get ..

 404 - Not Found

 Not Found

 Internal error information:
 No routes match path '/packages/doc/index.md'


 Not sure if that's problem with reggae's layout of the site....

 Cheers,

 A.
That was due to a missing feature (special handling for relative links) of code.dlang.org. Fixed now.
Sep 26 2015
prev sibling next sibling parent reply NVolcz <niklas.volcz gmail.com> writes:
On Tuesday, 22 September 2015 at 12:39:48 UTC, Per Nordlöw wrote:
 If you want to build a really revolutionary *new* build system 
 you should turn reggae into a client-server-architecture that 
 listens to file moves/copies/notifications via inotify on 
 Linux. I'm not sure how the develop/builder interface would 
 look like then, however. A web-client would be one way.

 </snip>
I heard that SBT does something similar (http://www.se-radio.net/2015/07/se-radio-episode-231-joshua-suereth-and-matthew-farwell-on-sbt-and-software-builds/). From what I understand it is faster due to that you can skip the overhead of startup. I also presume it is a big win for user experience.
Sep 22 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-09-23 08:41, NVolcz wrote:

 I heard that SBT does something similar
 (http://www.se-radio.net/2015/07/se-radio-episode-231-joshua-suereth-and-matthew-farwell-on-sbt-and-software-builds/).


  From what I understand it is faster due to that you can skip the
 overhead of startup.
 I also presume it is a big win for user experience.
You can do both with SBT. But I though the main reason was because of the JVM startup time. -- /Jacob Carlborg
Sep 23 2015
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 22 September 2015 at 12:39:48 UTC, Per Nordlöw wrote:
 On Wednesday, 16 September 2015 at 14:07:17 UTC, Atila Neves 
 wrote:
 http://code.dlang.org/my_packages/reggae

 What's new:
 Atila
If you want to build a really revolutionary *new* build system you should turn reggae into a client-server-architecture that listens to file moves/copies/notifications via inotify on Linux. I'm not sure how the develop/builder interface would look like then, however. A web-client would be one way.
That idea is in the TODO.txt of the project. There's a 99.9% chance I won't bother. I just benchmarked a toy auto-generated project of 30'000 D files, because I wanted to profile and optimise the binary backend against ninja and tup. I'm doing this by measuring the overhead of checking all of these files for modification timestamps, since actually compiling and linking take much longer. The overhead for ninja is 0.18s. Tup does it even faster. My current binary backend spends all of 0.3s, and that's what I was trying to optimize. Make, however... 22s I think. It would seem obvious that using something like inotify would be incredibly fast, but in practice it's irrelevant. Well, it's irrelevant for pretty much every project out there. I'm incredibly sensible to how long my feedback loops are, but if a 30k file project takes under a second to check, I really don't care. Now, you may say that my toy project doesn't accurately represent real life. But I've used ninja on a real C codebase of 40M lines spread over ~14k files. This is much larger than most people will ever deal with. A no-op build took 1s. Make is slow. Just don't use make. The ninja, tup and binary backends of reggae are fast enough. Incidentally, this is why ninja didn't try to get faster. It doesn't need to. I might consider a system where you don't even have to tell the computer to build anything. I'm not even sure that's that useful. Maybe it's just the way I work, I don't know.
 This however does not rebuild the dependency tree between 
 invokations which is a serious limitation. I believe reggae 
 could be even smarter here.
It already is ;)
 I've spent the last two years designing and implementing a 
 SCons-based build system at work. Involving, among others, 
 adding support for Ada and auto-detection of include-paths as 
 you have done in reggae.
Feel free to issue a PR for Ada auto-detection for reggae. :)
 I you want any advice on this matter please contact me. I'd be 
 glad to be of service.
Send me an email, I'm more than happy to waffle away about build systems.
Sep 24 2015
parent =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Thursday, 24 September 2015 at 15:04:49 UTC, Atila Neves wrote:
 I you want any advice on this matter please contact me. I'd be 
 glad to be of service.
Send me an email, I'm more than happy to waffle away about build systems.
BTW. I'm planning on visiting Berlin for DConf 2016. We could have a chat there.
Sep 25 2015
prev sibling parent reply Jason White <54f9byee3t32 gmail.com> writes:
I rarely visit the D forums and even more rarely make a post, but 
this thread caught my eye.

I've been writing a build system in D too: 
https://github.com/jasonwhite/brilliant-build (I'm not very fond 
of the name. Naming is hard!)

It is a general build system with an emphasis on correctness.

It is a work in progress at this point, but I'm very happy with 
how it is turning out. I'm interested in your guys' thoughts on 
it.
Sep 25 2015
parent Atila Neves <atila.neves gmail.com> writes:
On Saturday, 26 September 2015 at 03:51:11 UTC, Jason White wrote:
 I rarely visit the D forums and even more rarely make a post, 
 but this thread caught my eye.

 I've been writing a build system in D too: 
 https://github.com/jasonwhite/brilliant-build (I'm not very 
 fond of the name. Naming is hard!)

 It is a general build system with an emphasis on correctness.

 It is a work in progress at this point, but I'm very happy with 
 how it is turning out. I'm interested in your guys' thoughts on 
 it.
Looks good as a work-in-progress. A lot of the same ideas I used, including scripting languages as a way to describe builds. I'm probably going to use reggae though ;) Atila
Sep 28 2015