www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Compilation strategy

reply Russel Winder <russel winder.org.uk> writes:
A quick straw poll.  Do people prefer to have all sources compiled in a
single compiler call, or (more like C++) separate compilation of each
object followed by a link call.=20

Thanks.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Dec 15 2012
next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder 
wrote:

 Do people prefer to have all sources compiled in a
 single compiler call


I prefer the single call.
Dec 15 2012
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/15/12 11:55 AM, Russel Winder wrote:

 A quick straw poll.  Do people prefer to have all sources compiled in a
 single compiler call, or (more like C++) separate compilation of each
 object followed by a link call.


In phobos we use a single call for building the library. Then (at least 
on Posix) we use multiple calls for running unittests.

Andrei
Dec 15 2012
parent "David Nadlinger" <see klickverbot.at> writes:
On Saturday, 15 December 2012 at 17:02:08 UTC, Andrei 
Alexandrescu wrote:

 In phobos we use a single call for building the library. Then 
 (at least on Posix) we use multiple calls for running unittests.


This highlights the problem with giving a single answer to the 
question: Building a large project in one call is often 
impractical. It only works for Phobos library builds because many 
templates don't even get instantiated.

David
Dec 15 2012
prev sibling next sibling parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder 
wrote:

 A quick straw poll.  Do people prefer to have all sources 
 compiled in a
 single compiler call, or (more like C++) separate compilation 
 of each
 object followed by a link call.


Single compiler call is easier for small projects, but I worry 
about compile times for larger projects...
Dec 15 2012
next sibling parent reply "Jakob Bornecrantz" <wallbraker gmail.com> writes:
On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander 
wrote:

 On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder 
 wrote:

 A quick straw poll.  Do people prefer to have all sources 
 compiled in a
 single compiler call, or (more like C++) separate compilation 
 of each
 object followed by a link call.


 Single compiler call is easier for small projects, but I worry 
 about compile times for larger projects...



As evident by Phobos and my own project[1], for larger projects 
multiple concurrent calls is the only way to go. Rebuilding 
everything does take a bit, but a bit of thought behind the 
layout of the project and things work much faster when working on 
specific areas.

Cheers, Jakob.

[1] http://github.com/Charged/Miners
Dec 15 2012
parent "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Saturday, 15 December 2012 at 17:27:38 UTC, Jakob Bornecrantz 
wrote:

 On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander

 Single compiler call is easier for small projects, but I worry 
 about compile times for larger projects...


 As evident by Phobos and my own project[1], for larger projects 
 multiple concurrent calls is the only way to go. Rebuilding 
 everything does take a bit, but a bit of thought behind the 
 layout of the project and things work much faster when working 
 on specific areas.


Phobos is only around 200kloc. I'm worrying about the really 
large projects (multi-million lines of code).
Dec 15 2012
prev sibling parent reply "RenatoUtsch" <renatoutsch gmail.com> writes:
On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander 
wrote:

 On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder 
 wrote:

 A quick straw poll.  Do people prefer to have all sources 
 compiled in a
 single compiler call, or (more like C++) separate compilation 
 of each
 object followed by a link call.


 Single compiler call is easier for small projects, but I worry 
 about compile times for larger projects...


Yes, I'm writing a build system for D (that will be pretty damn 
good, I think, it has some interesting new concepts), and 
compiling each source separately to an object, and then linking 
everything will allow easily to make the build parallel, dividing 
the sources to compile in various threads. Or the compiler 
already does that if I pass all source files in one call?

-- Renato Utsch
Dec 15 2012
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Dec 15, 2012 at 06:31:17PM +0100, RenatoUtsch wrote:

 On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander
 wrote:

On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder
wrote:

A quick straw poll.  Do people prefer to have all sources compiled
in a single compiler call, or (more like C++) separate compilation
of each object followed by a link call.


Single compiler call is easier for small projects, but I worry
about compile times for larger projects...


 
 Yes, I'm writing a build system for D (that will be pretty damn
 good, I think, it has some interesting new concepts), and compiling
 each source separately to an object, and then linking everything
 will allow easily to make the build parallel, dividing the sources
 to compile in various threads. Or the compiler already does that if
 I pass all source files in one call?


[...]

I find that the current front-end (common to dmd, gdc, ldc) tends to
work better when passed multiple source files at once. It tends to be
faster, presumably because it only has to parse commonly-imported files
once, and also produces smaller object/executable sizes -- maybe due to
fewer duplicated template instantiations? I'm not sure of the exact
reasons, but this behaviour appears consistent throughout dmd and gdc,
and I presume also ldc (I didn't test that). So based on this, I'd lean
toward compiling multiple files at once.

However, in very large project, clearly this won't work very well. If it
takes half an hour to build the entire system, it makes the code -
compile - test cycle very slow, which reduces programmer productivity.

So perhaps one possible middle ground would be to link packages
separately, but compile all the sources within a single package at once.
Presumably, if the project is properly organized, recompiling a single
package won't take too long, and has the perk of optimizing for size
within packages. This will probably also map to SCons easily, since
SCons builds per-directory.


T

-- 
They pretend to pay us, and we pretend to work. -- Russian saying
Dec 15 2012
parent reply "RenatoUtsch" <renatoutsch gmail.com> writes:
On Saturday, 15 December 2012 at 18:00:58 UTC, H. S. Teoh wrote:

 On Sat, Dec 15, 2012 at 06:31:17PM +0100, RenatoUtsch wrote:

 On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander
 wrote:

On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder
wrote:

A quick straw poll.  Do people prefer to have all sources 
compiled
in a single compiler call, or (more like C++) separate 
compilation
of each object followed by a link call.


Single compiler call is easier for small projects, but I worry
about compile times for larger projects...


 
 Yes, I'm writing a build system for D (that will be pretty damn
 good, I think, it has some interesting new concepts), and 
 compiling
 each source separately to an object, and then linking 
 everything
 will allow easily to make the build parallel, dividing the 
 sources
 to compile in various threads. Or the compiler already does 
 that if
 I pass all source files in one call?


 [...]

 I find that the current front-end (common to dmd, gdc, ldc) 
 tends to
 work better when passed multiple source files at once. It tends 
 to be
 faster, presumably because it only has to parse 
 commonly-imported files
 once, and also produces smaller object/executable sizes -- 
 maybe due to
 fewer duplicated template instantiations? I'm not sure of the 
 exact
 reasons, but this behaviour appears consistent throughout dmd 
 and gdc,
 and I presume also ldc (I didn't test that). So based on this, 
 I'd lean
 toward compiling multiple files at once.


Yeah, I did read about this somewhee.


 However, in very large project, clearly this won't work very 
 well. If it
 takes half an hour to build the entire system, it makes the 
 code -
 compile - test cycle very slow, which reduces programmer 
 productivity.

 So perhaps one possible middle ground would be to link packages
 separately, but compile all the sources within a single package 
 at once.
 Presumably, if the project is properly organized, recompiling a 
 single
 package won't take too long, and has the perk of optimizing for 
 size
 within packages. This will probably also map to SCons easily, 
 since
 SCons builds per-directory.


 T


Well, the idea is good. Small projects usually don't have much 
packages, so there will be just a few compiler calls. And 
compiling files concurrently will only have a meaningful efect if 
the project is large, and a large project will have a lot of 
packages.

Maybe adding an option to choose between compiling all sources at 
once, per package, or per source. For example, in development and 
debug builds the compilation is per file or package, but in 
release builds all sources are compiled at once, or various 
packages at once.

This way release builds will take advantage of this behavior that 
the frontend has, but developers won't have productivity issues. 
And, of couse, the behaviour will not be fixed, the devs that are 
using the build system will choose that.
Dec 15 2012
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Dec 15, 2012 at 07:30:52PM +0100, RenatoUtsch wrote:

 On Saturday, 15 December 2012 at 18:00:58 UTC, H. S. Teoh wrote:


[...]

So perhaps one possible middle ground would be to link packages
separately, but compile all the sources within a single package at
once.  Presumably, if the project is properly organized, recompiling
a single package won't take too long, and has the perk of optimizing
for size within packages. This will probably also map to SCons
easily, since SCons builds per-directory.




[...]

 Well, the idea is good. Small projects usually don't have much
 packages, so there will be just a few compiler calls. And compiling
 files concurrently will only have a meaningful efect if the project
 is large, and a large project will have a lot of packages.


Yes, that's the idea behind it.



 Maybe adding an option to choose between compiling all sources at
 once, per package, or per source. For example, in development and
 debug builds the compilation is per file or package, but in release
 builds all sources are compiled at once, or various packages at once.
 
 This way release builds will take advantage of this behavior that
 the frontend has, but developers won't have productivity issues.
 And, of couse, the behaviour will not be fixed, the devs that are
 using the build system will choose that.


I forgot to mention also, that passing too many source files to the
compiler may sometimes cause memory consumption issues, as the compiler
has to hold everything in memory. This may not be practical for very
large project, where you can't fit everything into RAM.


T

-- 
Stop staring at me like that! You'll offend... no, you'll hurt your eyes!
Dec 15 2012
parent "RenatoUtsch" <renatoutsch gmail.com> writes:
On Saturday, 15 December 2012 at 18:44:35 UTC, H. S. Teoh wrote:

 On Sat, Dec 15, 2012 at 07:30:52PM +0100, RenatoUtsch wrote:

 On Saturday, 15 December 2012 at 18:00:58 UTC, H. S. Teoh 
 wrote:


 [...]

So perhaps one possible middle ground would be to link 
packages
separately, but compile all the sources within a single 
package at
once.  Presumably, if the project is properly organized, 
recompiling
a single package won't take too long, and has the perk of 
optimizing
for size within packages. This will probably also map to SCons
easily, since SCons builds per-directory.




 [...]

 Well, the idea is good. Small projects usually don't have much
 packages, so there will be just a few compiler calls. And 
 compiling
 files concurrently will only have a meaningful efect if the 
 project
 is large, and a large project will have a lot of packages.


 Yes, that's the idea behind it.



 Maybe adding an option to choose between compiling all sources 
 at
 once, per package, or per source. For example, in development 
 and
 debug builds the compilation is per file or package, but in 
 release
 builds all sources are compiled at once, or various packages 
 at once.
 
 This way release builds will take advantage of this behavior 
 that
 the frontend has, but developers won't have productivity 
 issues.
 And, of couse, the behaviour will not be fixed, the devs that 
 are
 using the build system will choose that.


 I forgot to mention also, that passing too many source files to 
 the
 compiler may sometimes cause memory consumption issues, as the 
 compiler
 has to hold everything in memory. This may not be practical for 
 very
 large project, where you can't fit everything into RAM.


 T


Well, so compiling by packages seem to be the best approach. When 
I return home I will do some tests to see what I can do.

-- Renato
Dec 15 2012
prev sibling parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 12/15/2012 10:30 AM, RenatoUtsch wrote:

 Well, the idea is good. Small projects usually don't have much packages,
 so there will be just a few compiler calls. And compiling files
 concurrently will only have a meaningful efect if the project is large,
 and a large project will have a lot of packages.

 Maybe adding an option to choose between compiling all sources at once,
 per package, or per source. For example, in development and debug builds
 the compilation is per file or package, but in release builds all
 sources are compiled at once, or various packages at once.


I always thought it would be cool if my build tool could automatically 
determine the dependency graph of my d code and use it to intelligently 
break my project into small enough semi-independent chunks to compile.
Dec 16 2012
prev sibling next sibling parent reply "jerro" <a a.com> writes:
On Saturday, 15 December 2012 at 17:31:19 UTC, RenatoUtsch wrote:

 On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander 
 wrote:

 On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder 
 wrote:

 A quick straw poll.  Do people prefer to have all sources 
 compiled in a
 single compiler call, or (more like C++) separate compilation 
 of each
 object followed by a link call.


 Single compiler call is easier for small projects, but I worry 
 about compile times for larger projects...


 Yes, I'm writing a build system for D (that will be pretty damn 
 good, I think, it has some interesting new concepts)


I took a look at your github project, there isn't any code yet, 
but I like the concept. I was actually planing to do something 
similar, but since you are already doing it, I think my time 
would be better spent contributing to your project. Will there be 
some publicly available code in the near future?
Dec 15 2012
parent "RenatoUtsch" <renatoutsch gmail.com> writes:
On Saturday, 15 December 2012 at 18:24:50 UTC, jerro wrote:

 On Saturday, 15 December 2012 at 17:31:19 UTC, RenatoUtsch 
 wrote:

 On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander 
 wrote:

 On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder 
 wrote:

 A quick straw poll.  Do people prefer to have all sources 
 compiled in a
 single compiler call, or (more like C++) separate 
 compilation of each
 object followed by a link call.


 Single compiler call is easier for small projects, but I 
 worry about compile times for larger projects...


 Yes, I'm writing a build system for D (that will be pretty 
 damn good, I think, it has some interesting new concepts)


 I took a look at your github project, there isn't any code yet, 
 but I like the concept. I was actually planing to do something 
 similar, but since you are already doing it, I think my time 
 would be better spent contributing to your project. Will there 
 be some publicly available code in the near future?


I expect to release a first alpha version in about 15~30 days, 
maybe less, it depends on how much time I will have on the rest 
of this month.
Dec 15 2012
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/15/2012 9:31 AM, RenatoUtsch wrote:

 Yes, I'm writing a build system for D (that will be pretty damn good, I think,
 it has some interesting new concepts), and compiling each source separately to
 an object, and then linking everything will allow easily to make the build
 parallel, dividing the sources to compile in various threads. Or the compiler
 already does that if I pass all source files in one call?


The compiler does a little multithreading, but not enough to make a difference. 
I've certainly thought about various schemes to parallelize it, though.
Dec 15 2012
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/15/2012 8:55 AM, Russel Winder wrote:

 A quick straw poll.  Do people prefer to have all sources compiled in a
 single compiler call, or (more like C++) separate compilation of each
 object followed by a link call.


Both are needed, and are suitable for different purposes. It's like asking if 
you prefer a standard or a philips screwdriver.
Dec 15 2012
prev sibling next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 15.12.2012 17:55, schrieb Russel Winder:

 A quick straw poll.  Do people prefer to have all sources compiled in a
 single compiler call, or (more like C++) separate compilation of each
 object followed by a link call.

 Thanks.


I prefer to compile by package, like in any module supporting language.

--
Paulo
Dec 16 2012
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder 
wrote:

 A quick straw poll.  Do people prefer to have all sources 
 compiled in a
 single compiler call, or (more like C++) separate compilation 
 of each
 object followed by a link call.

 Thanks.


I currently use full compilation. I have no choice because the 
compiler isn't emitting all symbols I need and get linking errors 
when doing separate compilation. Compiling my code require more 
than 2.5Gb of RAM now and is quite slow.

To avoid multiple instantiation of the same template by the 
compiler, I guess the best option to me would be to compile on a 
by package basis.
Dec 16 2012