www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Study: build times for D programs

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Hello,


I was talking to Walter on how to define a good study of D's compilation 
speed. We figured that we clearly need a good baseline, otherwise 
numbers have little meaning.

One idea would be to take a real, non-trivial application, written in 
both D and another compiled language. We then can measure build times 
for both applications, and also measure the relative speeds of the 
generated executables.

Although it sounds daunting to write the same nontrivial program twice, 
it turns out such an application does exist: dmdscript, a Javascript 
engine written by Walter in both C++ and D. It has over 40KLOC so it's 
of a good size to play with.

What we need is a volunteer who dusts off the codebase (e.g. the D 
source is in D1 and should be adjusted to compile with D2), run careful 
measurements, and show the results. Is anyone interested?


Thanks,

Andrei
Jul 24 2012
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 24-Jul-12 18:34, Andrei Alexandrescu wrote:
 Hello,
 Although it sounds daunting to write the same nontrivial program twice,
 it turns out such an application does exist: dmdscript, a Javascript
 engine written by Walter in both C++ and D. It has over 40KLOC so it's
 of a good size to play with.

 What we need is a volunteer who dusts off the codebase (e.g. the D
 source is in D1 and should be adjusted to compile with D2), run careful
 measurements, and show the results. Is anyone interested?
Well, I'd rather pass this to someone else but I have DMDScript repo that could be built with DMD ~ 2.056. I'll upload it to github, been meaning to do this for ages. -- Dmitry Olshansky
Jul 24 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/24/12 10:53 AM, Dmitry Olshansky wrote:
 On 24-Jul-12 18:34, Andrei Alexandrescu wrote:
 Hello,
 Although it sounds daunting to write the same nontrivial program twice,
 it turns out such an application does exist: dmdscript, a Javascript
 engine written by Walter in both C++ and D. It has over 40KLOC so it's
 of a good size to play with.

 What we need is a volunteer who dusts off the codebase (e.g. the D
 source is in D1 and should be adjusted to compile with D2), run careful
 measurements, and show the results. Is anyone interested?
Well, I'd rather pass this to someone else but I have DMDScript repo that could be built with DMD ~ 2.056. I'll upload it to github, been meaning to do this for ages.
Excellent, thanks! Andrei
Jul 24 2012
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 24-Jul-12 18:54, Andrei Alexandrescu wrote:
 On 7/24/12 10:53 AM, Dmitry Olshansky wrote:
 On 24-Jul-12 18:34, Andrei Alexandrescu wrote:
 Hello,
 Although it sounds daunting to write the same nontrivial program twice,
 it turns out such an application does exist: dmdscript, a Javascript
 engine written by Walter in both C++ and D. It has over 40KLOC so it's
 of a good size to play with.

 What we need is a volunteer who dusts off the codebase (e.g. the D
 source is in D1 and should be adjusted to compile with D2), run careful
 measurements, and show the results. Is anyone interested?
Well, I'd rather pass this to someone else but I have DMDScript repo that could be built with DMD ~ 2.056. I'll upload it to github, been meaning to do this for ages.
Excellent, thanks! Andrei
Done: https://github.com/blackwhale/DMDScript An awful lot of stuff got deprecated, e.g. it still uses class A: public B{...} syntax. To those taking this task - ready your shovels ;) -- Dmitry Olshansky
Jul 24 2012
prev sibling next sibling parent reply "Roman D. Boiko" <rb d-coding.com> writes:
On Tuesday, 24 July 2012 at 14:34:58 UTC, Andrei Alexandrescu 
wrote:
 the D source is in D1 and should be adjusted to compile with 
 D2),
That would provide performance (compilation and run-time) for D1 only (with D2 compiler). Performance of a typical D2 app would likely be different.
Jul 24 2012
next sibling parent reply "Paulo Pinto" <pjmlp progtools.org> writes:
"Roman D. Boiko"  wrote in message 
news:hpibxcqsmlpmgyngjzwp forum.dlang.org...
On Tuesday, 24 July 2012 at 14:34:58 UTC, Andrei Alexandrescu wrote:
 the D source is in D1 and should be adjusted to compile with D2),
That would provide performance (compilation and run-time) for D1 only (with D2 compiler). Performance of a typical D2 app would likely be different.
Still, is a good starting point.
Jul 24 2012
parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/24/2012 7:58 AM, Paulo Pinto wrote:
 "Roman D. Boiko"  wrote in message news:hpibxcqsmlpmgyngjzwp forum.dlang.org...
 On Tuesday, 24 July 2012 at 14:34:58 UTC, Andrei Alexandrescu wrote:
 the D source is in D1 and should be adjusted to compile with D2),
That would provide performance (compilation and run-time) for D1 only (with D2 compiler). Performance of a typical D2 app would likely be different.
Still, is a good starting point.
The reality is, no matter what such benchmark is chosen, it will be attacked as being biased. There is no such thing as a perfect apples-apples comparison between languages, and there'll be no shortage of criticism of any shortcomings, valid and invalid. That doesn't mean we shouldn't do it. Heck, I've even been accused of "sabotaging" the Digital Mars C++ compiler in order to make D look good!
Jul 24 2012
prev sibling next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 24-Jul-12 18:54, Roman D. Boiko wrote:
 On Tuesday, 24 July 2012 at 14:34:58 UTC, Andrei Alexandrescu wrote:
 the D source is in D1 and should be adjusted to compile with D2),
That would provide performance (compilation and run-time) for D1 only (with D2 compiler). Performance of a typical D2 app would likely be different.
In fact it's rather D2-ified. But yeah, no template heavy code in sight. It doesn't even use std.range/std.algorithm IRC. -- Dmitry Olshansky
Jul 24 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/24/12 10:54 AM, Roman D. Boiko wrote:
 On Tuesday, 24 July 2012 at 14:34:58 UTC, Andrei Alexandrescu wrote:
 the D source is in D1 and should be adjusted to compile with D2),
That would provide performance (compilation and run-time) for D1 only (with D2 compiler). Performance of a typical D2 app would likely be different.
Ehm. There's any number of arguments that can be made to question the validity of the study: * the coding style does not use the entire language in either or both implementations * the application domain favors one language or the other * the application's use of libraries is too low/too high * the translation is too literal * the translation changes the size of the code (which is the case here, as the D version is actually shorter) Nevertheless, I think there is value in the study. We're looking at a real nontrivial application that wasn't written for a study, but for actual use, and that implements the same design and same functionality in both languages. Andrei
Jul 24 2012
next sibling parent "Roman D. Boiko" <rb d-coding.com> writes:
On Tuesday, 24 July 2012 at 15:06:58 UTC, Andrei Alexandrescu 
wrote:
 Nevertheless, I think there is value in the study. We're 
 looking at a real nontrivial application that wasn't written 
 for a study, but for actual use, and that implements the same 
 design and same functionality in both languages.
OK. And it could serve as a basis for further variations: * introduce some feature (e.g., ranges), measure impact * measure impact of multiple features alone and in combination Of course, trivial changes would unlikely yield anything useful, but I believe there is a way to produce valuable data in a controlled research.
Jul 24 2012
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/24/2012 8:06 AM, Andrei Alexandrescu wrote:
 Nevertheless, I think there is value in the study. We're looking at a real
 nontrivial application that wasn't written for a study, but for actual use, and
 that implements the same design and same functionality in both languages.
The translation is also just that, a line-by-line translation that started by copying the .c source files to .d. It's probably as good as you're going to get in comparing compile speed.
Jul 24 2012
prev sibling next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/24/12, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 snip
I've got a codebase where it takes DMD 15 seconds to output an error message to stdout. The error message is 3000 lines long. (and people thought C++ errors were bad!). It's all thanks to this bug: http://d.puremagic.com/issues/show_bug.cgi?id=8082 The codebase isn't public yet so I can't help you with comparisons. Non-release full builds take 16 seconds for a template-heavy ~12k codebase (without counting lines of external dependencies). I use a lot of static foreach loops btw. Personally I think full builds are very fast compared to C++, although the transition from a small codebase which takes less than a second to compile to a bigger codebase which takes over a dozen seconds to compile is an unpleasant experience. I'd love to see DMD speed up its compile-time features like templates, mixins, static foreach, etc.
Jul 24 2012
parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Tue, 24 Jul 2012 18:53:25 +0200
Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:

 On 7/24/12, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 snip
I've got a codebase where it takes DMD 15 seconds to output an error message to stdout. The error message is 3000 lines long. (and people thought C++ errors were bad!). It's all thanks to this bug: http://d.puremagic.com/issues/show_bug.cgi?id=8082 The codebase isn't public yet so I can't help you with comparisons. Non-release full builds take 16 seconds for a template-heavy ~12k codebase (without counting lines of external dependencies). I use a lot of static foreach loops btw. Personally I think full builds are very fast compared to C++, although the transition from a small codebase which takes less than a second to compile to a bigger codebase which takes over a dozen seconds to compile is an unpleasant experience. I'd love to see DMD speed up its compile-time features like templates, mixins, static foreach, etc.
Yea. Programs using Goldie ( semitwist.com/goldie ) take a long time to compile (by D standards, not by C++ standards). I tried to benchmark it a while back, and was never really confident in the results I was getting or my understanding of the DMD source, so I never brought it up before. But it *seemed* to be template matching that was the big bottleneck (ie, IIUC, determining which template to instantiate, and I think the function was actually called "match" or something like that). Goldie does make use of a *lot* of that sort of thing.
Jul 24 2012
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Tuesday, July 24, 2012 15:49:38 Nick Sabalausky wrote:
 Yea. Programs using Goldie ( semitwist.com/goldie ) take a long time to
 compile (by D standards, not by C++ standards). I tried to benchmark
 it a while back, and was never really confident in the results I was
 getting or my understanding of the DMD source, so I never brought it up
 before. But it *seemed* to be template matching that was the big
 bottleneck (ie, IIUC, determining which template to instantiate,
 and I think the function was actually called "match" or something like
 that). Goldie does make use of a *lot* of that sort of thing.
I don't have any hard evidence for it, but I've always gotten the impression that it was templates, mixins, and CTFE which really slowed down compilation. Certainly, they increase the memory consumption of the compiler by quite a bit. My guess would be that if we were looking to improve the compiler's performance, that's where we'd need to focus. But we'd have to actually profile the compiler on a variety of projects to be sure of that (which is at least partially related to what Andrei is suggesting). - Jonathan M Davis
Jul 24 2012
next sibling parent "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Tuesday, 24 July 2012 at 22:19:07 UTC, Jonathan M Davis wrote:
 On Tuesday, July 24, 2012 15:49:38 Nick Sabalausky wrote:
 Yea. Programs using Goldie ( semitwist.com/goldie ) take a 
 long time to
 compile (by D standards, not by C++ standards). I tried to 
 benchmark
 it a while back, and was never really confident in the results 
 I was
 getting or my understanding of the DMD source, so I never 
 brought it up
 before. But it *seemed* to be template matching that was the 
 big
 bottleneck (ie, IIUC, determining which template to 
 instantiate,
 and I think the function was actually called "match" or 
 something like
 that). Goldie does make use of a *lot* of that sort of thing.
I don't have any hard evidence for it, but I've always gotten the impression that it was templates, mixins, and CTFE which really slowed down compilation. Certainly, they increase the memory consumption of the compiler by quite a bit. My guess would be that if we were looking to improve the compiler's performance, that's where we'd need to focus. But we'd have to actually profile the compiler on a variety of projects to be sure of that (which is at least partially related to what Andrei is suggesting). - Jonathan M Davis
There's also the nasty O(n^2) optimiser issue. http://d.puremagic.com/issues/show_bug.cgi?id=7157
Jul 24 2012
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/24/2012 3:18 PM, Jonathan M Davis wrote:
 But we'd have to actually profile
 the compiler on a variety of projects to be sure of that (which is at least
 partially related to what Andrei is suggesting).
I wouldn't be a bit surprised to find that there are some O(n*n) or worse algorithms embedded in the compiler that can be triggered by some types of code. Profiling is the way to root them out.
Jul 24 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-07-25 00:18, Jonathan M Davis wrote:

 I don't have any hard evidence for it, but I've always gotten the impression
 that it was templates, mixins, and CTFE which really slowed down compilation.
 Certainly, they increase the memory consumption of the compiler by quite a
 bit. My guess would be that if we were looking to improve the compiler's
 performance, that's where we'd need to focus. But we'd have to actually profile
 the compiler on a variety of projects to be sure of that (which is at least
 partially related to what Andrei is suggesting).
We did some profiling on derelict in the process of adding support for D2. This was mostly testing string mixins, the result was: It's a lot faster to use few string mixins containing a lot of code then using many string mixins containing very little code. -- /Jacob Carlborg
Jul 24 2012
prev sibling next sibling parent reply Guillaume Chatelet <chatelet.guillaume gmail.com> writes:
On 07/24/12 16:34, Andrei Alexandrescu wrote:
 I was talking to Walter on how to define a good study of D's compilation
 speed. We figured that we clearly need a good baseline, otherwise
 numbers have little meaning.
I agree.
 One idea would be to take a real, non-trivial application, written in
 both D and another compiled language. We then can measure build times
 for both applications, and also measure the relative speeds of the
 generated executables.
Well I kind of did exactly that. I was planning to start a Blog ("you know the blog you should really really start but can't find time to do so") with such a comparison. I started it a few months ago and can't finish the post so it's still there, lying half finished. But as the subject pops out of the NG it would be stupid not to talk about it. I intended to add relevant numbers and go from deterministic measurable facts to more subjective remarks ( was it fun ? is it more maintainable ? ) but I really just did a bit of the the first part :( Anyway, so for people interested in my "findings" here is the half finished post : http://goo.gl/16Yrb This could serve as a basis of do's and don'ts for a more relevant comparison as Andrei proposed. For instance it could be interesting to compare the performance of several C++ and D compilers to get a measure of the performance standard deviation expected within the language. Also I think the D code could have been more idiomatic and optimized further more : it was just a quick test ( yet quite time consuming ). Both projects are opensource, one is endorsed by the company I'm working for (https://github.com/mikrosimage/sequenceparser), the other one is just a personal project for the purpose of the comparison ( https://github.com/gchatelet/d_sequence_parser ) By the way, it reminds me of the 'Computer Language Benchmarks Game' (http://shootout.alioth.debian.org/). I know D is not welcome aboard but couldn't we try do run the game for ourself so to have some more data ? -- Guillaume
Jul 24 2012
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/24/2012 11:02 AM, Guillaume Chatelet wrote:
 By the way, it reminds me of the 'Computer Language Benchmarks Game'
 (http://shootout.alioth.debian.org/). I know D is not welcome aboard but
 couldn't we try do run the game for ourself so to have some more data ?
Small programs are completely inadequate for getting any reasonable measure of compiler speed. Even worse, they can be terribly wrong. (Back in the olden days, when men were men and and the sun revolved about the earth, everyone raved about Borland's compilation speed. In tests I ran myself, I found that it was fast, right up until you hit a certain size of source code, maybe about 5000 lines. Then, it fell off a cliff, and compile speed was terrible. But hey, it looked great in those tiny benchmarks.) The people who care about compile speed are compiling gigantic programs, and smallish ones can and do exhibit a very different performance profile. DMDScript is a medium sized program, not a gigantic one, but it's the best we've got for comparison.
Jul 24 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/24/12 8:20 PM, Walter Bright wrote:
 On 7/24/2012 11:02 AM, Guillaume Chatelet wrote:
 By the way, it reminds me of the 'Computer Language Benchmarks Game'
 (http://shootout.alioth.debian.org/). I know D is not welcome aboard but
 couldn't we try do run the game for ourself so to have some more data ?
Small programs are completely inadequate for getting any reasonable measure of compiler speed. Even worse, they can be terribly wrong.
Nevertheless there's value in the shootout. Yes, if someone is up for it that would be great. I also think if we have the setup ready we could convince the site maintainer to integrate D into the suite. Andrei
Jul 24 2012
parent Isaac Gouy <igouy2 yahoo.com> writes:
Andrei Alexandrescu Wrote:

-snip-
 Nevertheless there's value in the shootout. Yes, if someone is up for it 
 that would be great.
The Python measurement scripts are here -- http://shootout.alioth.debian.org/download/bencher.zip The whole ball of wax is available for download as a nightly snapshot, but if it was me I'd take the time to select particular programs from public CVS folders -- http://anonscm.debian.org/viewvc/shootout/shootout/bench/
 I also think if we have the setup ready we could 
 convince the site maintainer to integrate D into the suite.
I don't think so ;-)
Jul 25 2012
prev sibling parent Isaac Gouy <igouy2 yahoo.com> writes:
Walter Bright Wrote:

-snip-
 (Back in the olden days, when men were men and and the sun revolved about the 
 earth, everyone raved about Borland's compilation speed. In tests I ran
myself, 
 I found that it was fast, right up until you hit a certain size of source
code, 
 maybe about 5000 lines. Then, it fell off a cliff, and compile speed was 
 terrible. But hey, it looked great in those tiny benchmarks.)
-snip- Back in the olden days... "[Wirth] used the compiler's self-compilation speed as a measure of the compiler's quality." http://shootout.alioth.debian.org/dont-jump-to-conclusions.php#app
Jul 25 2012
prev sibling next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 24/07/12 15:34, Andrei Alexandrescu wrote:
 One idea would be to take a real, non-trivial application, written in both D
and
 another compiled language. We then can measure build times for both
 applications, and also measure the relative speeds of the generated
executables.
Suggest that this gets done with all 3 of the main D compilers, not just DMD. I'd like to see the tradeoff between compilation speed and executable speed that one gets between them. I do have some pretty much equivalent simulation code written in both D and C++. For a rough comparison: Language Compiler Compile time (s) Runtime (s) D GDC 1.5 25.3 D DMD 0.4 52.1 C++ g++ 2.3 21.8 C++ Clang++ 1.8 27.6 DMD used is a fairly recent pull from GitHub; GDC is the 4.6.3 package found in Ubuntu 12.04. I don't have a working LDC2 compiler on my system. :-( The C++ has a template-based policy class design, while the D code uses template mixins to similar effect. The D code can be found here: https://github.com/WebDrake/Dregs While I'm happy to also share the C++ code, I confess I'm shy to do so given that it probably represents a travesty of the beautiful ideas Andrei developed on policy class design ... :-) Best wishes, -- Joe
Jul 24 2012
next sibling parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 07/24/2012 04:54 PM, Joseph Rushton Wakeling wrote:

 Language Compiler Compile time (s) Runtime (s)
 D GDC 1.5 25.3
 D DMD 0.4 52.1
 C++ g++ 2.3 21.8
 C++ Clang++ 1.8 27.6
Those C++ builds have very few C++ source files, right? In my experience each source file takes a few seconds each, except the most trivial ones, because the standard library headers are compiled over and over again. :/ Ali
Jul 24 2012
prev sibling next sibling parent reply "David Nadlinger" <see klickverbot.at> writes:
On Tuesday, 24 July 2012 at 23:55:03 UTC, Joseph Rushton Wakeling 
wrote:
  For a rough comparison: […]
Even for a rough comparison of compile times, you need to include compiler switches used. For example, the difference between Clang -O0 vs. Clang -O3 is usually huge. David
Jul 25 2012
next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 25/07/12 09:37, David Nadlinger wrote:
 On Tuesday, 24 July 2012 at 23:55:03 UTC, Joseph Rushton Wakeling wrote:
  For a rough comparison: […]
Even for a rough comparison of compile times, you need to include compiler switches used. For example, the difference between Clang -O0 vs. Clang -O3 is usually huge.
C++ compiler and library flags: -ansi -pedantic -Wall -O3 -march=native -mtune=native -I. -DHAVE_INLINE -lm -lgsl -lgslcblas -lgrsl dmd and gdmd flags: -O -release -inline (which for gdmd corresponds to -O3 -fweb -frelease -finline-functions -I /usr/local/include/d2/). And yes, as Ali observed, this is a very small codebase (D is 3 files, 374 lines total; C++ is 18 files, 1266 lines -- so the comparison isn't 100% fair; but on the other hand, that's testament to how D can be used for more elegant code....).
Jul 25 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/25/12 4:37 AM, David Nadlinger wrote:
 On Tuesday, 24 July 2012 at 23:55:03 UTC, Joseph Rushton Wakeling wrote:
 For a rough comparison: […]
Even for a rough comparison of compile times, you need to include compiler switches used. For example, the difference between Clang -O0 vs. Clang -O3 is usually huge.
Yes, and both debug and release build times are important. Andrei
Jul 25 2012
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/25/2012 8:13 AM, Andrei Alexandrescu wrote:
 Yes, and both debug and release build times are important.
Optimized build time comparisons are less relevant - are you really willing to trade off faster optimization times for less optimization? I think it's more the time of the edit-compile-debug loop, which would be the unoptimized build times.
Jul 25 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/25/12 1:24 PM, Walter Bright wrote:
 On 7/25/2012 8:13 AM, Andrei Alexandrescu wrote:
 Yes, and both debug and release build times are important.
Optimized build time comparisons are less relevant - are you really willing to trade off faster optimization times for less optimization? I think it's more the time of the edit-compile-debug loop, which would be the unoptimized build times.
There are systems that only work in release mode (e.g. performance is part of the acceptability criteria) and for which debugging means watching logs. So the problem is not faster optimization times for less optimization (though that's possible, too), but instead build times for a given level of optimization. Andrei
Jul 25 2012
parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/25/2012 10:50 AM, Andrei Alexandrescu wrote:
 On 7/25/12 1:24 PM, Walter Bright wrote:
 On 7/25/2012 8:13 AM, Andrei Alexandrescu wrote:
 Yes, and both debug and release build times are important.
Optimized build time comparisons are less relevant - are you really willing to trade off faster optimization times for less optimization? I think it's more the time of the edit-compile-debug loop, which would be the unoptimized build times.
There are systems that only work in release mode (e.g. performance is part of the acceptability criteria) and for which debugging means watching logs. So the problem is not faster optimization times for less optimization (though that's possible, too), but instead build times for a given level of optimization.
The easy way to improve optimized build times is to do less optimization. I'm saying be careful what you ask for - you might get it!
Jul 25 2012
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 25.07.2012 19:24, Walter Bright wrote:
 On 7/25/2012 8:13 AM, Andrei Alexandrescu wrote:
 Yes, and both debug and release build times are important.
Optimized build time comparisons are less relevant - are you really willing to trade off faster optimization times for less optimization? I think it's more the time of the edit-compile-debug loop, which would be the unoptimized build times.
The "edit-compile-debug loop" is a use case where the D module system does not shine so well. Compare build times when only editing a single source file: With the help of incremental linking, building a large C++ project only takes seconds. In contrast, the D project usually recompiles everything from scratch with every little change.
Jul 25 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/25/12 4:53 PM, Rainer Schuetze wrote:
 On 25.07.2012 19:24, Walter Bright wrote:
 On 7/25/2012 8:13 AM, Andrei Alexandrescu wrote:
 Yes, and both debug and release build times are important.
Optimized build time comparisons are less relevant - are you really willing to trade off faster optimization times for less optimization? I think it's more the time of the edit-compile-debug loop, which would be the unoptimized build times.
The "edit-compile-debug loop" is a use case where the D module system does not shine so well. Compare build times when only editing a single source file: With the help of incremental linking, building a large C++ project only takes seconds. In contrast, the D project usually recompiles everything from scratch with every little change.
The same dependency management techniques can be applied to large D projects, as to large C++ projects. (And of course there are a few new ones.) What am I missing? Andrei
Jul 25 2012
next sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Wed, 25 Jul 2012 17:31:10 -0400
Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:

 On 7/25/12 4:53 PM, Rainer Schuetze wrote:
 The "edit-compile-debug loop" is a use case where the D module
 system does not shine so well. Compare build times when only
 editing a single source file:
 With the help of incremental linking, building a large C++ project
 only takes seconds.
 In contrast, the D project usually recompiles everything from
 scratch with every little change.
The same dependency management techniques can be applied to large D projects, as to large C++ projects. (And of course there are a few new ones.) What am I missing?
Aren't there still issues with what object files DMD chooses to store instantiated templates into? Or has that all been fixed? The xfbuild developers wrestled a lot with this and AIUI eventually gave up. The symptoms are that you'll eventually start getting linker errors related to template instantiations, which will be fixed when you then do a complete rebuild.
Jul 25 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-07-26 00:17, Nick Sabalausky wrote:

 Aren't there still issues with what object files DMD chooses to store
 instantiated templates into? Or has that all been fixed?

 The xfbuild developers wrestled a lot with this and AIUI eventually
 gave up. The symptoms are that you'll eventually start getting linker
 errors related to template instantiations, which will be
 fixed when you then do a complete rebuild.
I'm pretty sure nothing has changed. But Walter said if you use the -lib flag it will output the templates to all object files. That will complicate things a bit but still possible to make it work. -- /Jacob Carlborg
Jul 26 2012
prev sibling parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 25.07.2012 23:31, Andrei Alexandrescu wrote:
 On 7/25/12 4:53 PM, Rainer Schuetze wrote:
 On 25.07.2012 19:24, Walter Bright wrote:
 On 7/25/2012 8:13 AM, Andrei Alexandrescu wrote:
 Yes, and both debug and release build times are important.
Optimized build time comparisons are less relevant - are you really willing to trade off faster optimization times for less optimization? I think it's more the time of the edit-compile-debug loop, which would be the unoptimized build times.
The "edit-compile-debug loop" is a use case where the D module system does not shine so well. Compare build times when only editing a single source file: With the help of incremental linking, building a large C++ project only takes seconds. In contrast, the D project usually recompiles everything from scratch with every little change.
The same dependency management techniques can be applied to large D projects, as to large C++ projects. (And of course there are a few new ones.) What am I missing?
Incremental compilation does not work so well because - with combined declaration and implementation in the source, you also get the full dependencies if you just need a short declaration - even with di-files imports are viral: you must be very careful if you try to remove them from di-files because you might break runtime initialization order. - di-file generation has other known problems (e.g. missing implementation for CTFE) I thought about implementing incremental builds for Visual D, but soon gave up when I noticed that a single file compilation in a medium sized project (Visual D itself) almost takes as long as recompiling the whole thing. I suspect the problem is that dmd fully analyzes all the imported files and only skips the code generation for these. It could be much faster if it would do the analysis lazily (though this might slightly change evaluation order and skip error messages in unused code blocks).
Jul 25 2012
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, July 25, 2012 22:53:08 Rainer Schuetze wrote:
 On 25.07.2012 19:24, Walter Bright wrote:
 On 7/25/2012 8:13 AM, Andrei Alexandrescu wrote:
 Yes, and both debug and release build times are important.
Optimized build time comparisons are less relevant - are you really willing to trade off faster optimization times for less optimization? I think it's more the time of the edit-compile-debug loop, which would be the unoptimized build times.
The "edit-compile-debug loop" is a use case where the D module system does not shine so well. Compare build times when only editing a single source file: With the help of incremental linking, building a large C++ project only takes seconds. In contrast, the D project usually recompiles everything from scratch with every little change.
D should actually compile _faster_ if you compile everything at once - certainly for smaller projects - since it then only has to lex and parse each module once. Incremental builds avoid having to fully compile each module every time, but there's still plenty of extra lexing and parsing which goes on. I don't know how much it shifts with large projects (maybe incremental builds actually end up being better then, because you have enough files which aren't related to one another that the amount of code which needs to be relexed a reparsed is minimal in comparison to the number of files), but you can do incremental building with dmd if you want to. It's just more typical to do it all at once, because for most projects, that's faster. So, I don't see how there's an complaint against D here. - Jonathan M Davis
Jul 25 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-07-25 23:56, Jonathan M Davis wrote:

 D should actually compile _faster_ if you compile everything at once -
 certainly for smaller projects - since it then only has to lex and parse each
 module once. Incremental builds avoid having to fully compile each module
 every time, but there's still plenty of extra lexing and parsing which goes
 on.

 I don't know how much it shifts with large projects (maybe incremental builds
 actually end up being better then, because you have enough files which aren't
 related to one another that the amount of code which needs to be relexed a
 reparsed is minimal in comparison to the number of files), but you can do
 incremental building with dmd if you want to. It's just more typical to do it
 all at once, because for most projects, that's faster. So, I don't see how
 there's an complaint against D here.
Incremental builds don't have to mean "pass a single file to the compiler". You can start by passing all the files at once to the compiler and then later you just pass all the files that have changed, at once. But I don't know how much difference it will be to recompiling the whole project. -- /Jacob Carlborg
Jul 26 2012
parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 07/26/2012 02:28 AM, Jacob Carlborg wrote:
 On 2012-07-25 23:56, Jonathan M Davis wrote:
 Incremental builds don't have to mean "pass a single file to the
 compiler". You can start by passing all the files at once to the
 compiler and then later you just pass all the files that have changed,
 at once.
GNU make has the special $? prerequisite that may help with the above: "The names of all the prerequisites that are newer than the target, with spaces between them. " http://www.gnu.org/software/make/manual/make.html#index-g_t_0024_003f-944 Ali
Jul 26 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-07-26 19:54, Ali Çehreli wrote:

 GNU make has the special $? prerequisite that may help with the above:
 "The names of all the prerequisites that are newer than the target, with
 spaces between them. "


 http://www.gnu.org/software/make/manual/make.html#index-g_t_0024_003f-944

 Ali
I'm trying to avoid "make" as much as possible. -- /Jacob Carlborg
Jul 26 2012
parent "SomeDude" <lovelydear mailmetrash.com> writes:
On Thursday, 26 July 2012 at 18:57:30 UTC, Jacob Carlborg wrote:
 On 2012-07-26 19:54, Ali Çehreli wrote:

 GNU make has the special $? prerequisite that may help with 
 the above:
 "The names of all the prerequisites that are newer than the 
 target, with
 spaces between them. "


 http://www.gnu.org/software/make/manual/make.html#index-g_t_0024_003f-944

 Ali
I'm trying to avoid "make" as much as possible.
+1
Jul 28 2012
prev sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Thu, 26 Jul 2012 10:54:07 -0700
Ali =C7ehreli <acehreli yahoo.com> wrote:

 On 07/26/2012 02:28 AM, Jacob Carlborg wrote:
  > On 2012-07-25 23:56, Jonathan M Davis wrote:
=20
  > Incremental builds don't have to mean "pass a single file to the
  > compiler". You can start by passing all the files at once to the
  > compiler and then later you just pass all the files that have
  > changed, at once.
=20
 GNU make has the special $? prerequisite that may help with the
 above: "The names of all the prerequisites that are newer than the
 target, with spaces between them. "
=20
    http://www.gnu.org/software/make/manual/make.html#index-g_t_0024_003f-=
944
=20
So in other words, it'll completely crap out when a path contains spaces? (What is this, 1994?)
Jul 26 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/25/12, Jonathan M Davis <jmdavisProg gmx.com> wrote:
 D should actually compile _faster_ if you compile everything at once -
 certainly for smaller projects - since it then only has to lex and parse
 each
 module once. Incremental builds avoid having to fully compile each module
 every time, but there's still plenty of extra lexing and parsing which goes on.
That's assuming that the lexing/parsing is the bottleneck for DMD. For example: a full build of WindowsAPI takes 14.6 seconds on my machine. But when compiling one module at a time and using parallelism it takes 7 seconds instead. And all it takes is a simple parallel loop.
Jul 25 2012
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/25/2012 1:53 PM, Rainer Schuetze wrote:
 The "edit-compile-debug loop" is a use case where the D module system does not
 shine so well. Compare build times when only editing a single source file:
 With the help of incremental linking, building a large C++ project only takes
 seconds.
 In contrast, the D project usually recompiles everything from scratch with
every
 little change.
I suspect that's one of two possibilities: 1. everything is passed on one command line to dmd. This, of course, requires dmd to recompile everything. 2. modules are not separated into .d and .di files. Hence every module that imports a .d file has to, at least, parse and semantically analyze the whole thing, although it won't optimize or generate code for it. As for incremental linking, optlink has always been faster at doing a full link than the Microsoft linker does for an incremental link.
Jul 25 2012
parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 26.07.2012 03:48, Walter Bright wrote:
 On 7/25/2012 1:53 PM, Rainer Schuetze wrote:
 The "edit-compile-debug loop" is a use case where the D module system
 does not
 shine so well. Compare build times when only editing a single source
 file:
 With the help of incremental linking, building a large C++ project
 only takes
 seconds.
 In contrast, the D project usually recompiles everything from scratch
 with every
 little change.
I suspect that's one of two possibilities: 1. everything is passed on one command line to dmd. This, of course, requires dmd to recompile everything. 2. modules are not separated into .d and .di files. Hence every module that imports a .d file has to, at least, parse and semantically analyze the whole thing, although it won't optimize or generate code for it.
I think working with di-files is too painful. A lot of the analysis in imported files could be skipped.
 As for incremental linking, optlink has always been faster at doing a
 full link than the Microsoft linker does for an incremental link.
Agreed, incremental linking is just a work-around for the linkers slowness.
Jul 25 2012
prev sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 25/07/12 16:13, Andrei Alexandrescu wrote:
 Yes, and both debug and release build times are important.
If you can advise some flag combinations (for D and C++) you'd like to see tested, I'll happily do them.
Jul 26 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/26/12 4:15 AM, Joseph Rushton Wakeling wrote:
 On 25/07/12 16:13, Andrei Alexandrescu wrote:
 Yes, and both debug and release build times are important.
If you can advise some flag combinations (for D and C++) you'd like to see tested, I'll happily do them.
The classic to ones are: (a) no flags at all, (b) -O -release -inline, (c) -O -release -inline -noboundscheck. You can skip the latter as it won't impact build time. Andrei
Jul 26 2012
next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 26/07/12 15:42, Andrei Alexandrescu wrote:
 If you can advise some flag combinations (for D and C++) you'd like to
 see tested, I'll happily do them.
The classic to ones are: (a) no flags at all, (b) -O -release -inline, (c) -O -release -inline -noboundscheck.
Here's a little table of DMD to GDC comparisons for the Dregs codebase: ----------DMD---------- ----------GDC---------- compiler flags compile time runtime compile time runtime -O -release -inline 0.43s 52s 1.51s 25s -O -release 0.35s 47s 1.50s 25s -O -noboundscheck 0.35s 56s 1.66s 25s -O -inline 0.47s 1m 5s 1.94s 45s -O 0.36s 1m 5s 1.98s 45s -release -inline 0.31s 1m 3s 0.63s 1m 3s -release 0.29s 1m 3s 0.63s 1m 3s -inline 0.32s 1m 24s 0.70s 1m 26s -noboundscheck 0.29s 1m 10s 0.666s 1m 9s [none] 0.29s 1m 24s 0.72s 1m 26s -debug 0.30s 1m 24s 0.70s 1m 26s -unittest 0.42s 1m 25s 0.75s 1m 26s -debug -unittest 0.42s 1m 25s 0.78s 1m 26s
Jul 26 2012
prev sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, July 26, 2012 18:00:21 Joseph Rushton Wakeling wrote:
 On 26/07/12 15:42, Andrei Alexandrescu wrote:
 If you can advise some flag combinations (for D and C++) you'd like to
 see tested, I'll happily do them.
The classic to ones are: (a) no flags at all, (b) -O -release -inline, (c) -O -release -inline -noboundscheck.
Here's a little table of DMD to GDC comparisons for the Dregs codebase: ----------DMD---------- ----------GDC---------- compiler flags compile time runtime compile time runtime -O -release -inline 0.43s 52s 1.51s 25s -O -release 0.35s 47s 1.50s 25s -O -noboundscheck 0.35s 56s 1.66s 25s -O -inline 0.47s 1m 5s 1.94s 45s -O 0.36s 1m 5s 1.98s 45s -release -inline 0.31s 1m 3s 0.63s 1m 3s -release 0.29s 1m 3s 0.63s 1m 3s -inline 0.32s 1m 24s 0.70s 1m 26s -noboundscheck 0.29s 1m 10s 0.666s 1m 9s [none] 0.29s 1m 24s 0.72s 1m 26s -debug 0.30s 1m 24s 0.70s 1m 26s -unittest 0.42s 1m 25s 0.75s 1m 26s -debug -unittest 0.42s 1m 25s 0.78s 1m 26s
Clearly -O is where the big runtime speed difference is at between dmd and gdc, which _is_ a bit obvious, but I'm surprised that -inline had no differences, since dmd is generally accused at being poor at inlining. That probably just indicates that it's a frontend issue (which I suppose makes sense when I think about it). I guess that the way to go if you want to maxize both your efficiency and the code's efficiency is to do most of the coding with dmd but generate the final program with gdc (though obviously building and testing with both the whole way is probably necessary to ensure stability; still much of that could be automated and not affect programmer efficiency). - Jonathan M Davis
Jul 26 2012
next sibling parent reply "David Nadlinger" <see klickverbot.at> writes:
On Thursday, 26 July 2012 at 18:59:14 UTC, Jonathan M Davis wrote:
 Clearly -O is where the big runtime speed difference is at 
 between dmd and gdc,
 which _is_ a bit obvious, but I'm surprised that -inline had no 
 differences,
 since dmd is generally accused at being poor at inlining. That 
 probably just
 indicates that it's a frontend issue (which I suppose makes 
 sense when I think
 about it).
GDC probably performs inlining by default on -O2/-O3, just like LDC does. Also note that for the -release case (any performance measurements without it are most probably not worth it due to all the extra _d_invariant, etc. calls), -inline seems to increase the runtime of the DMD-produced executable by 10%. For inlining, you inevitably have to rely on heuristics, and there will always be cases where it slows down execution (worst case: the slight improvement in code size causes cache thrashing in a hot path), but 10% in a fairly standard application seems to be quite a lot. David
Jul 26 2012
next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 26/07/12 20:27, David Nadlinger wrote:
 GDC probably performs inlining by default on -O2/-O3, just like LDC does.
I was surprised that using -inline alone (without any optimization option) doesn't produce any meaningful improvement. It cuts maybe 1s off the DMD-compiled runtime, but it's not clear to me that actually corresponds to a reliable difference. Perhaps GDC just ignores the -inline flag ... ? I suppose it's possible that this is code that does not respond well to inlining, although I'd have thought the obvious optimization would be to inline many of the object methods that are only called internally and that are called in a tight loop: do { userDivergence(ratings); userReputation(ratings); reputationObjectOld_[] = reputationObject_[]; objectReputation(ratings); diff = 0; foreach(size_t o, Reputation rep; reputationObject_) { auto aux = rep - reputationObjectOld_[o]; diff += aux*aux; } ++iterations; } while (diff > convergence_); I might tweak it manually so that userDivergence(), userReputation() and objectReputation() are inline, and see if it makes any difference.
Jul 26 2012
prev sibling next sibling parent Iain Buclaw <ibuclaw ubuntu.com> writes:
On 26 July 2012 23:07, Joseph Rushton Wakeling
<joseph.wakeling webdrake.net> wrote:
 On 26/07/12 20:27, David Nadlinger wrote:
 GDC probably performs inlining by default on -O2/-O3, just like LDC does.
I was surprised that using -inline alone (without any optimization option) doesn't produce any meaningful improvement. It cuts maybe 1s off the DMD-compiled runtime, but it's not clear to me that actually corresponds to a reliable difference. Perhaps GDC just ignores the -inline flag ... ?
-inline is mapped to -finline-functions in GDC. Inlining is possibly done, but only in the backend. Some extra notes to bear in mind about GDC. 1) All methods and function literals are marked as 'inline' by default. 2) Cross module inlining does not occur if you are compiling one-at-a-time. Regards -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0';
Jul 26 2012
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 27/07/12 07:29, Iain Buclaw wrote:
 -inline is mapped to -finline-functions in GDC.  Inlining is possibly
 done, but only in the backend.

 Some extra notes to bear in mind about GDC.
 1) All methods and function literals are marked as 'inline' by default.
 2) Cross module inlining does not occur if you are compiling one-at-a-time.
Good to know. In this case it's all compiled together in one go: DC = gdmd DFLAGS = -O -release -inline DREGSRC = dregs/core.d dregs/codetermine.d all: test test: test.d $(DREGSRC) $(DC) $(DFLAGS) -of$ $^ .PHONY: clean clean: rm -f test *.o I'm just surprised that using -inline produces no measurable difference at all in performance for GDC, whether or not any other optimization flags are used. As I said, maybe I'll test some manual inlining and see what difference it might make ...
Jul 27 2012
prev sibling parent Iain Buclaw <ibuclaw ubuntu.com> writes:
On 27 July 2012 09:09, Joseph Rushton Wakeling
<joseph.wakeling webdrake.net> wrote:
 On 27/07/12 07:29, Iain Buclaw wrote:
 -inline is mapped to -finline-functions in GDC.  Inlining is possibly
 done, but only in the backend.

 Some extra notes to bear in mind about GDC.
 1) All methods and function literals are marked as 'inline' by default.
 2) Cross module inlining does not occur if you are compiling
 one-at-a-time.
Good to know. In this case it's all compiled together in one go: DC = gdmd DFLAGS = -O -release -inline DREGSRC = dregs/core.d dregs/codetermine.d all: test test: test.d $(DREGSRC) $(DC) $(DFLAGS) -of$ $^ .PHONY: clean clean: rm -f test *.o I'm just surprised that using -inline produces no measurable difference at all in performance for GDC, whether or not any other optimization flags are used. As I said, maybe I'll test some manual inlining and see what difference it might make ...
My best assumption would be it may say something more about the way the program was written itself rather than the compiler. Regards -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0';
Jul 27 2012
prev sibling parent reply "David Nadlinger" <see klickverbot.at> writes:
On Thursday, 26 July 2012 at 18:59:14 UTC, Jonathan M Davis wrote:
 […] That probably just
 indicates that it's a frontend issue (which I suppose makes 
 sense when I think
 about it).
Oh, and I don't know what exactly you are referring to here, but any difference between DMD and GDC is likely not a frontend issue, as GDC uses the DMD frontend, with only minor modifications. David
Jul 26 2012
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, July 26, 2012 21:29:57 David Nadlinger wrote:
 On Thursday, 26 July 2012 at 18:59:14 UTC, Jonathan M Davis wrote:
 [=E2=80=A6] That probably just
 indicates that it's a frontend issue (which I suppose makes
 sense when I think
 about it).
=20 Oh, and I don't know what exactly you are referring to here, but any difference between DMD and GDC is likely not a frontend issue, as GDC uses the DMD frontend, with only minor modifications.
That was my point. -inline seems to be pretty much identical between th= e two=20 compilers, and if the inlining is done in the frontend, then that makes= sense.=20 Thinking on it, it makes sense to me that it would be in the frontend, = but I=20 don't know where it actually is. - Jonathan M Davis
Jul 26 2012
parent "David Nadlinger" <see klickverbot.at> writes:
On Thursday, 26 July 2012 at 19:36:51 UTC, Jonathan M Davis wrote:
 That was my point. -inline seems to be pretty much identical 
 between the two
 compilers, and if the inlining is done in the frontend, then 
 that makes sense.
 Thinking on it, it makes sense to me that it would be in the 
 frontend, but I
 don't know where it actually is.
Ah, okay, I see what you meant. But no, as far as I'm aware, GDC doesn't use DMD's inliner, but rather relies on the GCC one. LDC does the same, we entirely disable DMD's inlining code, it turned out to just be not worth it. David
Jul 26 2012
prev sibling parent reply "ixid" <nuaccount gmail.com> writes:
beautiful ideas Andrei developed on policy class design
Where would one find these ideas?
Jul 25 2012
next sibling parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 07/25/2012 09:46 AM, ixid wrote:
 beautiful ideas Andrei developed on policy class design
Where would one find these ideas?
There are some papers at Andrei's site: http://erdani.com/index.php/articles/ Search for "policy" there. Policy based design is the main topic in Andrei's book, "Modern C++ Design": http://www.amazon.com/Modern-Design-Generic-Programming-Patterns/dp/0201704315 That book covers a publicly-available library, Loki, again by Andrei: http://loki-lib.sourceforge.net/ Ali
Jul 25 2012
prev sibling parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On Wed, 25 Jul 2012 18:46:58 +0200, ixid <nuaccount gmail.com> wrote:

 beautiful ideas Andrei developed on policy class design
Where would one find these ideas?
http://www.amazon.com/Modern-Design-Generic-Programming-Patterns/dp/0201704315 -- Simen
Jul 25 2012
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, July 26, 2012 00:34:07 Andrej Mitrovic wrote:
 On 7/25/12, Jonathan M Davis <jmdavisProg gmx.com> wrote:
 D should actually compile _faster_ if you compile everything at once -
 certainly for smaller projects - since it then only has to lex and parse
 each
 module once. Incremental builds avoid having to fully compile each module
 every time, but there's still plenty of extra lexing and parsing which
 goes on.
That's assuming that the lexing/parsing is the bottleneck for DMD.
Not necessarily. The point is that there's extra work that has to be done when compiling separately. So, whether it takes more or less time depends on how much other work you're avoiding by doing an incremental build. Certainly, I'd expect a full incremental build from scratch to take longer than one which was not incremental.
 For example: a full build of  WindowsAPI takes 14.6 seconds on my machine.
 But when compiling one module at a time and using parallelism it takes
 7 seconds instead. And all it takes is a simple parallel loop.
Parallelism? How on earth do you manage that? dmd has no support for running on multiple threads AFAIK. Do you run multiple copies of dmd at once? Certainly, compiling files in parallel changes things. You've got multiple cores working on it at that point, so the equation is completely different. - Jonathan M Davis
Jul 25 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-07-26 00:42, Jonathan M Davis wrote:

 I'd expect a full incremental build from scratch to take longer than one which
was
 not incremental.
Why? Just pass all the files to the compiler at once. Nothing says an incremental build needs to pass a single file to the compiler. -- /Jacob Carlborg
Jul 26 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/26/12, Jonathan M Davis <jmdavisProg gmx.com> wrote:
 Parallelism? How on earth do you manage that? dmd has no support for running
 on multiple threads AFAIK.
 You've got multiple
 cores working on it at that point, so the equation is completely different.
That's exactly my point, you can take advantage of parallelism externally if you compile module-by-module simply by invoking multiple DMD processes. And who doesn't own a multicore machine these days?
Jul 25 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/26/12, Jonathan M Davis <jmdavisProg gmx.com> wrote:
 Certainly, I'd expect a full incremental build from scratch to take longer
than one which was not incremental.
Well that would probably only be done once. With full builds you do it every time.
Jul 25 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, July 26, 2012 00:44:14 Andrej Mitrovic wrote:
 On 7/26/12, Jonathan M Davis <jmdavisProg gmx.com> wrote:
 Parallelism? How on earth do you manage that? dmd has no support for
 running on multiple threads AFAIK.
 You've got multiple
 cores working on it at that point, so the equation is completely
 different.
That's exactly my point, you can take advantage of parallelism externally if you compile module-by-module simply by invoking multiple DMD processes. And who doesn't own a multicore machine these days?
Well, regardless, my and Andrei's point was that C++ has nothing on us here. We can do incremental just fine. The fact that most people just build the whole program from scratch every time is irrelevant. That just means that the build times are fast enough for most people not to care about doing incremental builds, not that they can't do them. - Jonathan M Davis
Jul 25 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 24 Jul 2012 10:34:57 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 Hello,


 I was talking to Walter on how to define a good study of D's compilation  
 speed. We figured that we clearly need a good baseline, otherwise  
 numbers have little meaning.
Might I draw attention again to this bug: http://d.puremagic.com/issues/show_bug.cgi?id=4900 Granted, this is really freaking old. A re-application of profiling should be done. But in general, what I have observed from DMD compiling is that the number and size (string size) of symbols is what really bogs it down. Most of the time, it's lightning fast. The reason the dcollections unit test taxes it so much is because I'm instantiating 15 objects for each container type, and each one has humongous symbols, a consequence of so many template arguments. -Steve
Aug 24 2012
prev sibling parent "d_follower" <d_follower fakemail.com> writes:
On Tuesday, 24 July 2012 at 14:34:58 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 I was talking to Walter on how to define a good study of D's 
 compilation speed. We figured that we clearly need a good 
 baseline, otherwise numbers have little meaning.

 One idea would be to take a real, non-trivial application, 
 written in both D and another compiled language. We then can 
 measure build times for both applications, and also measure the 
 relative speeds of the generated executables.

 Although it sounds daunting to write the same nontrivial 
 program twice, it turns out such an application does exist: 
 dmdscript, a Javascript engine written by Walter in both C++ 
 and D. It has over 40KLOC so it's of a good size to play with.

 What we need is a volunteer who dusts off the codebase (e.g. 
 the D source is in D1 and should be adjusted to compile with 
 D2), run careful measurements, and show the results. Is anyone 
 interested?


 Thanks,

 Andrei
You can try testing DMD (written in C++) against DDMD (written in D). I don't think you can find more fair comparison (both projects are in sync - though dated - and project size is fairly large).
Aug 24 2012