www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Scaling rdmd up: build package at a time

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
This is a self-contained and really fun project:

https://issues.dlang.org/show_bug.cgi?id=14654

It requires an understanding of how rdmd currently works (<1KLOC in a 
single module!) and is of huge impact.

Who'd want to get into it? I would if I didn't already have 
std.allocator to have fun with.

(Speaking of which: regions that grow upwards or downwards depending on 
stack growth direction for cache hotness, yum! 
http://erdani.com/d/phobos-prerelease/std_experimental_allocator_region.html)


Andrei
Jun 05 2015
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-06-05 18:15, Andrei Alexandrescu wrote:

 This is a self-contained and really fun project:

 https://issues.dlang.org/show_bug.cgi?id=14654


It looks like you want to support incremental compilation. Two questions:

* Is that possible? I know there has been some problems with this in the 
past, i.e. not all symbols were outputted to all object files

* Why rebuild a whole directory when it could only rebuild a single file?

-- 
/Jacob Carlborg
Jun 06 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/15 6:03 AM, Jacob Carlborg wrote:

 On 2015-06-05 18:15, Andrei Alexandrescu wrote:

 This is a self-contained and really fun project:

 https://issues.dlang.org/show_bug.cgi?id=14654


 It looks like you want to support incremental compilation. Two questions:

 * Is that possible? I know there has been some problems with this in the
 past, i.e. not all symbols were outputted to all object files


Whatever the matters are, we will fix them.


 * Why rebuild a whole directory when it could only rebuild a single file?


Compiling several files at once is faster. The natural boundary is one 
package.


Andrei
Jun 06 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-06-06 17:38, Andrei Alexandrescu wrote:


 Compiling several files at once is faster. The natural boundary is one
 package.


Perhaps I misunderstand something. Why compile files that has not 
changed? I mean that rdmd should compile all files that has changed 
including its dependencies, no more, no less. It should compile all 
these files in one go.

-- 
/Jacob Carlborg
Jun 06 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/15 11:47 AM, Jacob Carlborg wrote:

 On 2015-06-06 17:38, Andrei Alexandrescu wrote:


 Compiling several files at once is faster. The natural boundary is one
 package.


 Perhaps I misunderstand something. Why compile files that has not
 changed? I mean that rdmd should compile all files that has changed
 including its dependencies, no more, no less. It should compile all
 these files in one go.


Yah, that's the traditional C-style module-at-a-time approach. Somewhat 
paradoxically, for D it's faster to compile several files at once, even 
though not all were affected by the change.

So in the package-at-a-time approach if at least one module in a package 
is affected by a change, the entire package gets rebuilt.


Andrei
Jun 06 2015
parent reply "anonymous" <anonymous example.com> writes:
On Saturday, 6 June 2015 at 19:44:15 UTC, Andrei Alexandrescu 
wrote:

 On 6/6/15 11:47 AM, Jacob Carlborg wrote:


[...]

 I mean that rdmd should compile all files that has changed
 including its dependencies, no more, no less. It should 
 compile all
 these files in one go.


 Yah, that's the traditional C-style module-at-a-time approach.


I think the traditional approach would be to compile modules 
separately, not "all [...] in one go".


 Somewhat paradoxically, for D it's faster to compile several 
 files at once, even though not all were affected by the change.

 So in the package-at-a-time approach if at least one module in 
 a package is affected by a change, the entire package gets 
 rebuilt.


This may beat separately compiling modules that needs rebuilding. 
But it doesn't beat compiling all of them at once, does it?

My understanding of things:

Current behaviour: When any module changed, pass all source files 
to one dmd invocation.

The makefile approach: Separately recompile every module that 
needs rebuilding; then link the new object files with the cached 
ones.

Your proposal, variant 1: For every module that needs rebuilding, 
separately recompile its whole package; then link the new object 
files with the cached ones of other packages.

Variant 2: For every module that needs rebuilding, determine what 
package it belongs to; then pass the source files of those 
packages and the cached object files of other packages to one dmd 
invocation.

The seemingly obvious thing to do: Pass the source files that 
need rebuilding and the object files of other modules to one dmd 
invocation.
Jun 06 2015
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-06-06 22:27, anonymous wrote:


 The seemingly obvious thing to do: Pass the source files that need
 rebuilding and the object files of other modules to one dmd invocation.


That's what I've been thinking all along and trying to say.

-- 
/Jacob Carlborg
Jun 06 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Saturday, 6 June 2015 at 21:10:22 UTC, Jacob Carlborg wrote:

 On 2015-06-06 22:27, anonymous wrote:


 The seemingly obvious thing to do: Pass the source files that 
 need
 rebuilding and the object files of other modules to one dmd 
 invocation.


 That's what I've been thinking all along and trying to say.


I don't think I understand. Where would these object files come 
from unless you're doing per-module compilation, C-style?

What I've already implemented is variant 1 mentioned before.

Atila
Jun 06 2015
next sibling parent reply "anonymous" <anonymous example.com> writes:
On Saturday, 6 June 2015 at 21:24:05 UTC, Atila Neves wrote:

 I don't think I understand. Where would these object files come 
 from unless you're doing per-module compilation, C-style?


Huh, I somehow assumed dmd would spit out multiple object files 
when given multiple source files. Since that's not so, variant 2 
and the "seemingly obvious thing" are seemingly not possible.

But if dmd could be made to produce multiple object files ...
(I have no idea if that's doable.)
Jun 06 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/15 2:42 PM, anonymous wrote:

 But if dmd could be made to produce multiple object files ...
 (I have no idea if that's doable.)


Not advantageously. -- Andrei
Jun 06 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-06-06 23:24, Atila Neves wrote:


 I don't think I understand. Where would these object files come from
 unless you're doing per-module compilation, C-style?

 What I've already implemented is variant 1 mentioned before.


If you compile multiple files with DMD without linking it will produce 
multiple object files:

$ ls
bar.d foo.d
$ dmd -c bar.d foo.d
$ ls
bar.d bar.o foo.d foo.o

-- 
/Jacob Carlborg
Jun 07 2015
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/15 1:27 PM, anonymous wrote:

 The seemingly obvious thing to do: Pass the source files that need
 rebuilding and the object files of other modules to one dmd invocation.


Doesn't work because no separate object file per source is being 
produced. -- Andrei
Jun 06 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-06-06 23:41, Andrei Alexandrescu wrote:


 Doesn't work because no separate object file per source is being
 produced. -- Andrei


They're produced if you don't link.

-- 
/Jacob Carlborg
Jun 07 2015
prev sibling parent "anonymous" <anonymous example.com> writes:
On Saturday, 6 June 2015 at 20:27:12 UTC, anonymous wrote:

 The seemingly obvious thing to do: Pass the source files that 
 need rebuilding and the object files of other modules to one 
 dmd invocation.


I implemented this. Preliminary pull request:
https://github.com/D-Programming-Language/tools/pull/170
Jun 14 2015
prev sibling next sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Friday, 5 June 2015 at 16:15:21 UTC, Andrei Alexandrescu wrote:

 This is a self-contained and really fun project:

 https://issues.dlang.org/show_bug.cgi?id=14654

 It requires an understanding of how rdmd currently works 
 (<1KLOC in a single module!) and is of huge impact.

 Who'd want to get into it? I would if I didn't already have 
 std.allocator to have fun with.

 (Speaking of which: regions that grow upwards or downwards 
 depending on stack growth direction for cache hotness, yum! 
 http://erdani.com/d/phobos-prerelease/std_experimental_allocator_region.html)


My $0.02: any project large enough to care about if rdmd builds 
per package or not is likely to need a real build system anyway.

Atila
Jun 06 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/15 8:51 AM, Atila Neves wrote:

 My $0.02: any project large enough to care about if rdmd builds per
 package or not is likely to need a real build system anyway.


Integrating the strategy within rdmd will help make it popular and 
widespread. I recall Liran has had issues with build times but hadn't 
tried per-package builds because they weren't a common idiom. -- Andrei
Jun 06 2015
parent "Atila Neves" <atila.neves gmail.com> writes:
On Saturday, 6 June 2015 at 15:54:39 UTC, Andrei Alexandrescu 
wrote:

 On 6/6/15 8:51 AM, Atila Neves wrote:

 My $0.02: any project large enough to care about if rdmd 
 builds per
 package or not is likely to need a real build system anyway.


 Integrating the strategy within rdmd will help make it popular 
 and widespread. I recall Liran has had issues with build times 
 but hadn't tried per-package builds because they weren't a 
 common idiom. -- Andrei


 From the conversation I had with him, he wouldn't be able to use 
rdmd anyway. I understand the "the default should be per-package 
and encouraged" argument though. In which case I'd tell people to 
try reggae ;)

Atila
Jun 06 2015
prev sibling parent reply "Shammah Chancellor" <S S.com> writes:
On Friday, 5 June 2015 at 16:15:21 UTC, Andrei Alexandrescu wrote:

 This is a self-contained and really fun project:

 https://issues.dlang.org/show_bug.cgi?id=14654

 It requires an understanding of how rdmd currently works 
 (<1KLOC in a single module!) and is of huge impact.

 Who'd want to get into it? I would if I didn't already have 
 std.allocator to have fun with.

 (Speaking of which: regions that grow upwards or downwards 
 depending on stack growth direction for cache hotness, yum! 
 http://erdani.com/d/phobos-prerelease/std_experimental_allocator_region.html)


 Andrei


How can any of this work without a special object format that is
template aware?
Jun 06 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/15 12:15 PM, Shammah Chancellor wrote:

 On Friday, 5 June 2015 at 16:15:21 UTC, Andrei Alexandrescu wrote:

 This is a self-contained and really fun project:

 https://issues.dlang.org/show_bug.cgi?id=14654

 It requires an understanding of how rdmd currently works (<1KLOC in a
 single module!) and is of huge impact.

 Who'd want to get into it? I would if I didn't already have
 std.allocator to have fun with.

 (Speaking of which: regions that grow upwards or downwards depending
 on stack growth direction for cache hotness, yum!
 http://erdani.com/d/phobos-prerelease/std_experimental_allocator_region.html)



 Andrei


 How can any of this work without a special object format that is
 template aware?


It works; these are distinct concerns. -- Andrei
Jun 06 2015