www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D needs to publicize its speed of compilation

reply Joakim <dlang joakim.fea.st> writes:
This one of the main strengths of D, it is what Walter focuses 
on, yet I have seen almost nothing on the D blog talking about 
this.  What brought me to emphasize this today is this recent 
post about how long it takes to compile the mostly-C++ Chromium 
web browser and the reddit discussion about it:

https://lobste.rs/s/iri1te/chromium_has_compilation_time_problem
https://www.reddit.com/r/programming/comments/7ktzog/chromium_has_a_compilation_time_problem/

I'm tempted to call BS on that 6-7 hour build time, as I used to 
contribute to the Chromium project, and I think it used to take 
me 20 minutes for a release build in a FreeBSD jail on a fairly 
weak, 2008-vintage mini-desktop, a dual-core Celeron with 2 GBs 
of RAM (don't hold me to that build time, just a vague 
recollection, but probably in the ballpark).  Of course, the last 
time I built Chromium was more than 5 years ago, and a lot has 
likely changed since then, such as now using LTO to speed up the 
browser apparently, and maybe the cross-compilation toolchain for 
ARM is slower, though others note similar times for native x64 
compilation also.

That still implies a slowdown of 2-3 orders of magnitude over the 
last 5 years, given the much more powerful hardware he's using, 
which is nuts.

D really needs the community to write blog posts talking about 
how fast it is, publicizing that there is an alternative to these 
glacial build times: who wants to do this?  It doesn't need to be 
on the D blog, could just be on your personal blog, but it is a 
message that really needs to be spread.
Dec 22 2017
next sibling parent reply user1234 <user1234 12.nl> writes:
On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:

 This one of the main strengths of D, it is what Walter focuses 
 on, yet I have seen almost nothing on the D blog talking about 
 this.  What brought me to emphasize this today is this recent 
 post about how long it takes to compile the mostly-C++ Chromium 
 web browser and the reddit discussion about it:

 https://lobste.rs/s/iri1te/chromium_has_compilation_time_problem
 https://www.reddit.com/r/programming/comments/7ktzog/chromium_has_a_compilation_time_problem/

 I'm tempted to call BS on that 6-7 hour build time, as I used 
 to contribute to the Chromium project, and I think it used to 
 take me 20 minutes for a release build in a FreeBSD jail on a 
 fairly weak, 2008-vintage mini-desktop, a dual-core Celeron 
 with 2 GBs of RAM (don't hold me to that build time, just a 
 vague recollection, but probably in the ballpark).  Of course, 
 the last time I built Chromium was more than 5 years ago, and a 
 lot has likely changed since then, such as now using LTO to 
 speed up the browser apparently, and maybe the 
 cross-compilation toolchain for ARM is slower, though others 
 note similar times for native x64 compilation also.

 That still implies a slowdown of 2-3 orders of magnitude over 
 the last 5 years, given the much more powerful hardware he's 
 using, which is nuts.

 D really needs the community to write blog posts talking about 
 how fast it is, publicizing that there is an alternative to 
 these glacial build times: who wants to do this?  It doesn't 
 need to be on the D blog, could just be on your personal blog, 
 but it is a message that really needs to be spread.


If this ever happens, what will be published will have to be 
based on LDC2.
If benchmarks based on DMD are published, the article will be 
subject to the criticism that is that the shorter build time is 
due to the optimization pass, since it's known not to be super 
deep in DMD backend.
Dec 22 2017
next sibling parent thedeemon <dlang thedeemon.com> writes:
On Friday, 22 December 2017 at 11:39:48 UTC, user1234 wrote:


 If benchmarks based on DMD are published, the article will be 
 subject to the criticism that is that the shorter build time is 
 due to the optimization pass, since it's known not to be super 
 deep in DMD backend.


Well, Go folks don't shy away from praising their compilation 
speed, but the situation is very similar there: Go compiles fast 
because it doesn't optimize the code. Last time I checked their 
compiler had zero options mentioning optimization, it's like 
"always debug build".
Dec 22 2017
prev sibling parent Joakim <dlang joakim.fea.st> writes:
On Friday, 22 December 2017 at 11:39:48 UTC, user1234 wrote:

 If this ever happens, what will be published will have to be 
 based on LDC2.
 If benchmarks based on DMD are published, the article will be 
 subject to the criticism that is that the shorter build time is 
 due to the optimization pass, since it's known not to be super 
 deep in DMD backend.


Sure, ldc is slower but not significantly so, and will still beat 
the pants off clang, Rust, or Swift.

On Friday, 22 December 2017 at 11:46:49 UTC, Chris wrote:

 D does not do itself any favors when it keeps accepting 
 mediocre results in benchmarks.


These are all runtime codegen benchmarks, whereas I was talking 
about build speed.


 https://www.techempower.com/benchmarks/previews/round15/

 Some years it does not complete the tests, in others it shows 
 results that are below mediocre.

 Vibe.d ( and other D web frameworks ) benching twice slower 
 then scripting languages like PHP/Ruby. That does not advertise 
 D.


Sonke and others have noted that there are issues that need to be 
worked out with that benchmark submission.


 How about:

 http://benchmarksgame.alioth.debian.org/

 No D there? Performance must be bad because its not listed at 
 all ( for a language that exist 20 years )?


It was on there for many years but was tossed out by a subsequent 
maintainer, who doesn't want it for some reason, as thedeemon 
pointed out:

http://forum.dlang.org/post/lspybogtpqorauausmmv forum.dlang.org


 Another one:

 https://github.com/costajob/app-servers

 Single threaded ( look at CPU total ), losing to other 
 languages again.


I think vibe.d is meant to be run in multiple process instances 
instead, and it does second-best to Rust on a single core (maybe 
beat out by Crystal too?), which is pretty good.


 Impressions are everything when there is a wealth of languages 
 to pick from. Anybody stumbling over these results think: Well, 
 i am better going with Go, Rust, Crystal, ... for a web hosting 
 as they show more consistent high speed results.


I wouldn't say they're "everything," but they intrigue people 
enough to dig deeper into the language or just to move on without 
bothering.  Of course, you may get great benchmarks with Rust but 
once you try coding with it, you still may not like using it.

On Friday, 22 December 2017 at 14:48:22 UTC, thedeemon wrote:

 On Friday, 22 December 2017 at 11:46:49 UTC, Chris wrote:

 http://benchmarksgame.alioth.debian.org/

 No D there? Performance must be bad because its not listed at 
 all ( for a language that exist 20 years )?


 D is not there for the only reason of that benchmark maintainer 
 unwilling to include D. Technically you can take any C solution 
 there, translate it to D (mostly by renaming from .c to .d) and 
 with LDC get the same speed as C shows.


Better than C actually, if you use D libraries instead:

http://forum.dlang.org/post/olelrjiaiwdxhjhilagv forum.dlang.org

Getting back to the original topic of build speed, highlighting 
that the dmd compiler itself now builds in seconds after the 
C++->D translation would seem to be a layup:

http://forum.dlang.org/thread/oknuas$2lfp$1 digitalmars.com
Dec 23 2017
prev sibling next sibling parent reply Chris <Chris c.com> writes:
On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:

 This one of the main strengths of D, it is what Walter focuses 
 on, yet I have seen almost nothing on the D blog talking about 
 this.  What brought me to emphasize this today is this recent 
 post about how long it takes to compile the mostly-C++ Chromium 
 web browser and the reddit discussion about it:

 https://lobste.rs/s/iri1te/chromium_has_compilation_time_problem
 https://www.reddit.com/r/programming/comments/7ktzog/chromium_has_a_compilation_time_problem/

 I'm tempted to call BS on that 6-7 hour build time, as I used 
 to contribute to the Chromium project, and I think it used to 
 take me 20 minutes for a release build in a FreeBSD jail on a 
 fairly weak, 2008-vintage mini-desktop, a dual-core Celeron 
 with 2 GBs of RAM (don't hold me to that build time, just a 
 vague recollection, but probably in the ballpark).  Of course, 
 the last time I built Chromium was more than 5 years ago, and a 
 lot has likely changed since then, such as now using LTO to 
 speed up the browser apparently, and maybe the 
 cross-compilation toolchain for ARM is slower, though others 
 note similar times for native x64 compilation also.

 That still implies a slowdown of 2-3 orders of magnitude over 
 the last 5 years, given the much more powerful hardware he's 
 using, which is nuts.

 D really needs the community to write blog posts talking about 
 how fast it is, publicizing that there is an alternative to 
 these glacial build times: who wants to do this?  It doesn't 
 need to be on the D blog, could just be on your personal blog, 
 but it is a message that really needs to be spread.


D does not do itself any favors when it keeps accepting mediocre 
results in benchmarks.

https://www.techempower.com/benchmarks/previews/round15/

Some years it does not complete the tests, in others it shows 
results that are below mediocre.

Vibe.d ( and other D web frameworks ) benching twice slower then 
scripting languages like PHP/Ruby. That does not advertise D.

How about:

http://benchmarksgame.alioth.debian.org/

No D there? Performance must be bad because its not listed at all 
( for a language that exist 20 years )?

Another one:

https://github.com/costajob/app-servers

Single threaded ( look at CPU total ), losing to other languages 
again.

Impressions are everything when there is a wealth of languages to 
pick from. Anybody stumbling over these results think: Well, i am 
better going with Go, Rust, Crystal, ... for a web hosting as 
they show more consistent high speed results.
Dec 22 2017
parent reply thedeemon <dlang thedeemon.com> writes:
On Friday, 22 December 2017 at 11:46:49 UTC, Chris wrote:

 http://benchmarksgame.alioth.debian.org/

 No D there? Performance must be bad because its not listed at 
 all ( for a language that exist 20 years )?


D is not there for the only reason of that benchmark maintainer 
unwilling to include D. Technically you can take any C solution 
there, translate it to D (mostly by renaming from .c to .d) and 
with LDC get the same speed as C shows.
Dec 22 2017
parent Russel Winder <russel winder.org.uk> writes:
On Fri, 2017-12-22 at 14:48 +0000, thedeemon via Digitalmars-d wrote:

 [=E2=80=A6]
=20
 D is not there for the only reason of that benchmark maintainer=20
 unwilling to include D. Technically you can take any C solution=20
 there, translate it to D (mostly by renaming from .c to .d) and=20
 with LDC get the same speed as C shows.


For performance related stuff Go people use gccgo, just as D people use
gdc (or more usually ldc).

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Dec 22 2017
prev sibling next sibling parent reply aberba <karabutaworld gmail.com> writes:
On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:

 This one of the main strengths of D, it is what Walter focuses 
 on, yet I have seen almost nothing on the D blog talking about 
 this.  What brought me to emphasize this today is this recent 
 post about how long it takes to compile the mostly-C++ Chromium 
 web browser and the reddit discussion about it:

 [...]


Memory consumption is a deal breaker.
Dec 24 2017
parent reply Joakim <dlang joakim.fea.st> writes:
On Sunday, 24 December 2017 at 22:55:12 UTC, aberba wrote:

 On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:

 This one of the main strengths of D, it is what Walter focuses 
 on, yet I have seen almost nothing on the D blog talking about 
 this.  What brought me to emphasize this today is this recent 
 post about how long it takes to compile the mostly-C++ 
 Chromium web browser and the reddit discussion about it:

 [...]


 Memory consumption is a deal breaker.


The dmd frontend prioritized speed over memory, which can cause 
problems with large or CTFE-heavy projects on low-memory systems:

http://www.drdobbs.com/cpp/increasing-compiler-speed-by-over-75/240158941

On Wednesday, 27 December 2017 at 21:23:15 UTC, Atila Neves wrote:

 I thought D did market itself that way.


It has historically been a selling point, but perhaps it has been 
taken for granted.  I'm saying it needs a marketing push now.

I just went and checked the front page of the website, and I see 
a new marketing slogan that was added a couple weeks ago, "write 
fast, read fast, and run fast.  Fast code, fast:"

https://github.com/dlang/dlang.org/pull/1965

However, "build fast" is not included in that list.  You could 
argue it's implied by "write fast" or the last "code, fast," but 
it's not mentioned in the list under "write fast" below.  So even 
the front page doesn't highlight build speed, one of the core 
advantages of D.


 However, my experience has been that D has fast builds from 
 scratch, but is really really slow for incremental builds. 
 Here's the thing: personally I don't care about anything except 
 incremental builds. It's great that Phobos compiles in seconds 
 when I do a git fetch after months. Win! Oh wait, I'm working 
 on a bug, changed one character and it still takes the same 
 amount of time...

 Given that I write D code every day now, this incremental 
 compilation situation is driving me nuts, and I'm definitely 
 going to do something about it.

 It's getting so bad that the day at work I had to write C++ I 
 was happy. Because the edit-compile-link-test cycle was 
 _faster_. Think about that for a bit - I enjoyed writing C++ 
 (!!! either it or Scala are the slowest languages to compile 
 ever) because I got faster feedback. Epic fail.

 It's also possible I use templates and CTFE more that the 
 regular D programmer. But... they're some of the main reasons I 
 use D in the first place! I don't want to write Java in D 
 syntax.


What do you plan to do about it?
Dec 27 2017
parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 28 December 2017 at 07:05:25 UTC, Joakim wrote:

 On Sunday, 24 December 2017 at 22:55:12 UTC, aberba wrote:

 On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:


 What do you plan to do about it?


I have an idea or two. I just need time/mental energy to do this.

Atila
Dec 28 2017
parent reply Seb <seb wilzba.ch> writes:
On Thursday, 28 December 2017 at 13:20:34 UTC, Atila Neves wrote:

 On Thursday, 28 December 2017 at 07:05:25 UTC, Joakim wrote:

 On Sunday, 24 December 2017 at 22:55:12 UTC, aberba wrote:

 On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:


 What do you plan to do about it?


 I have an idea or two. I just need time/mental energy to do 
 this.

 Atila


Are you aware of this great work: 
https://github.com/dlang/dmd/pull/7239?
Dec 28 2017
parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 28 December 2017 at 13:26:55 UTC, Seb wrote:

 On Thursday, 28 December 2017 at 13:20:34 UTC, Atila Neves 
 wrote:

 On Thursday, 28 December 2017 at 07:05:25 UTC, Joakim wrote:

 On Sunday, 24 December 2017 at 22:55:12 UTC, aberba wrote:

 On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:


 What do you plan to do about it?


 I have an idea or two. I just need time/mental energy to do 
 this.

 Atila


 Are you aware of this great work: 
 https://github.com/dlang/dmd/pull/7239?


I wasn't, thanks for pointing it out. Looks interesting, but all 
the compile times were still over a second, which is my pain 
threshold.

Atila
Dec 29 2017
prev sibling next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:

 This one of the main strengths of D, it is what Walter focuses 
 on, yet I have seen almost nothing on the D blog talking about 
 this.  What brought me to emphasize this today is this recent 
 post about how long it takes to compile the mostly-C++ Chromium 
 web browser and the reddit discussion about it:

 https://lobste.rs/s/iri1te/chromium_has_compilation_time_problem
 https://www.reddit.com/r/programming/comments/7ktzog/chromium_has_a_compilation_time_problem/

 I'm tempted to call BS on that 6-7 hour build time, as I used 
 to contribute to the Chromium project, and I think it used to 
 take me 20 minutes for a release build in a FreeBSD jail on a 
 fairly weak, 2008-vintage mini-desktop, a dual-core Celeron 
 with 2 GBs of RAM (don't hold me to that build time, just a 
 vague recollection, but probably in the ballpark).  Of course, 
 the last time I built Chromium was more than 5 years ago, and a 
 lot has likely changed since then, such as now using LTO to 
 speed up the browser apparently, and maybe the 
 cross-compilation toolchain for ARM is slower, though others 
 note similar times for native x64 compilation also.

 That still implies a slowdown of 2-3 orders of magnitude over 
 the last 5 years, given the much more powerful hardware he's 
 using, which is nuts.

 D really needs the community to write blog posts talking about 
 how fast it is, publicizing that there is an alternative to 
 these glacial build times: who wants to do this?  It doesn't 
 need to be on the D blog, could just be on your personal blog, 
 but it is a message that really needs to be spread.


I thought D did market itself that way.

However, my experience has been that D has fast builds from 
scratch, but is really really slow for incremental builds. Here's 
the thing: personally I don't care about anything except 
incremental builds. It's great that Phobos compiles in seconds 
when I do a git fetch after months. Win! Oh wait, I'm working on 
a bug, changed one character and it still takes the same amount 
of time...

Given that I write D code every day now, this incremental 
compilation situation is driving me nuts, and I'm definitely 
going to do something about it.

It's getting so bad that the day at work I had to write C++ I was 
happy. Because the edit-compile-link-test cycle was _faster_. 
Think about that for a bit - I enjoyed writing C++ (!!! either it 
or Scala are the slowest languages to compile ever) because I got 
faster feedback. Epic fail.

It's also possible I use templates and CTFE more that the regular 
D programmer. But... they're some of the main reasons I use D in 
the first place! I don't want to write Java in D syntax.

Atila

Atila
Dec 27 2017
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/27/2017 1:23 PM, Atila Neves wrote:

 However, my experience has been that D has fast builds from scratch, but is 
 really really slow for incremental builds.


You can do fast incremental builds with D by not putting all the files on the 
command line. Just put one.
Dec 27 2017
parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 28 December 2017 at 07:21:28 UTC, Walter Bright 
wrote:

 On 12/27/2017 1:23 PM, Atila Neves wrote:

 However, my experience has been that D has fast builds from 
 scratch, but is really really slow for incremental builds.


 You can do fast incremental builds with D by not putting all 
 the files on the command line. Just put one.


I don't build the compiler command-line myself, nor do I want to. 
Even then, recompiling one file isn't useful to me, what is is to 
recompile what's needed and run the tests. Emacs/flycheck already 
highlights compilation errors in just one file.

Atila
Dec 28 2017
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Dec 28, 2017 at 01:22:13PM +0000, Atila Neves via Digitalmars-d wrote:

 On Thursday, 28 December 2017 at 07:21:28 UTC, Walter Bright wrote:

 On 12/27/2017 1:23 PM, Atila Neves wrote:

 However, my experience has been that D has fast builds from
 scratch, but is really really slow for incremental builds.


 You can do fast incremental builds with D by not putting all the
 files on the command line. Just put one.


 
 I don't build the compiler command-line myself, nor do I want to. Even
 then, recompiling one file isn't useful to me, what is is to recompile
 what's needed and run the tests.


[...]

Isn't that what a build system is for?


T

-- 
GEEK = Gatherer of Extremely Enlightening Knowledge
Dec 28 2017
parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 28 December 2017 at 15:54:03 UTC, H. S. Teoh wrote:

 On Thu, Dec 28, 2017 at 01:22:13PM +0000, Atila Neves via 
 Digitalmars-d wrote:

 On Thursday, 28 December 2017 at 07:21:28 UTC, Walter Bright 
 wrote:

 On 12/27/2017 1:23 PM, Atila Neves wrote:

 However, my experience has been that D has fast builds 
 from scratch, but is really really slow for incremental 
 builds.


 You can do fast incremental builds with D by not putting all 
 the files on the command line. Just put one.


 
 I don't build the compiler command-line myself, nor do I want 
 to. Even then, recompiling one file isn't useful to me, what 
 is is to recompile what's needed and run the tests.


 [...]

 Isn't that what a build system is for?


 T


Yes, and I wrote one. It turns out that I misunderstood what 
Walter said though.

Atila
Dec 29 2017
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/28/2017 5:22 AM, Atila Neves wrote:

 On Thursday, 28 December 2017 at 07:21:28 UTC, Walter Bright wrote:

 On 12/27/2017 1:23 PM, Atila Neves wrote:

 However, my experience has been that D has fast builds from scratch, but is 
 really really slow for incremental builds.


 You can do fast incremental builds with D by not putting all the files on the 
 command line. Just put one.


 
 I don't build the compiler command-line myself, nor do I want to. Even then, 
 recompiling one file isn't useful to me, what is is to recompile what's needed 
 and run the tests. Emacs/flycheck already highlights compilation errors in
just 
 one file.


I don't understand. C++ compiles files one at a time. You can do the same in D. 
How can this model be useful in C++ but not with D?
Dec 28 2017
next sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 28 December 2017 at 22:58, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 12/28/2017 5:22 AM, Atila Neves wrote:

 On Thursday, 28 December 2017 at 07:21:28 UTC, Walter Bright wrote:

 On 12/27/2017 1:23 PM, Atila Neves wrote:

 However, my experience has been that D has fast builds from scratch, but
 is really really slow for incremental builds.


 You can do fast incremental builds with D by not putting all the files on
 the command line. Just put one.



 I don't build the compiler command-line myself, nor do I want to. Even
 then, recompiling one file isn't useful to me, what is is to recompile
 what's needed and run the tests. Emacs/flycheck already highlights
 compilation errors in just one file.



 I don't understand. C++ compiles files one at a time. You can do the same in
 D. How can this model be useful in C++ but not with D?


D modules are like C++ headers.  Change one, and you need to recompile
everything that imports it.
Dec 28 2017
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/28/2017 2:16 PM, Iain Buclaw wrote:

 D modules are like C++ headers.  Change one, and you need to recompile
 everything that imports it.


That's right. That doesn't mean that D should incrementally compile slower than
C++.
Dec 29 2017
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 28 December 2017 at 21:58:07 UTC, Walter Bright 
wrote:

 On 12/28/2017 5:22 AM, Atila Neves wrote:

 On Thursday, 28 December 2017 at 07:21:28 UTC, Walter Bright 
 wrote:

 On 12/27/2017 1:23 PM, Atila Neves wrote:

 However, my experience has been that D has fast builds from 
 scratch, but is really really slow for incremental builds.


 You can do fast incremental builds with D by not putting all 
 the files on the command line. Just put one.


 
 I don't build the compiler command-line myself, nor do I want 
 to. Even then, recompiling one file isn't useful to me, what 
 is is to recompile what's needed and run the tests. 
 Emacs/flycheck already highlights compilation errors in just 
 one file.


 I don't understand. C++ compiles files one at a time. You can 
 do the same in D. How can this model be useful in C++ but not 
 with D?


Because:

. As Iain pointed out independently, D modules are like C/C++ 
headers. All dependencies must also be recompiled. If I edit a 
C++ source file, only that source file has to be recompiled. I 
can use .di files, and they can even be generated for me (nice!) 
but there's no build system to do this properly. This is one of 
the ideas I alluded to earlier - I want to put this sort of thing 
in reggae. And now that dmd is a dub package I can even just use 
the actual compiler code. i.e. Generate the .di, if it's the same 
as before don't overwrite the old one to not mess with the 
timestamp.

. C++ doesn't have built-in unit tests. Which means a framework 
has to be used and since nobody wants to put their tests in 
#ifdef "blocks" they live in separate files. Changing the 
implementation doesn't require the tests to be rebuilt. Yes, I 
can separate them in D too, but then I lose DDoc benefits and 
it's actually nicer to see tests next to the code. The fact that 
all D files are effectively headers means editing a unittest 
means recompiling all modules that depend on the current one.

. Related to the previous point: compiling separately in D is 
actually slower overall than by package, because of all of the 
dependencies (I measured). Yes, I can compile by package, but I 
don't want to do this by hand. To the extent of my knowledge the 
only way to get per-package compilation by default or otherwise 
in the D ecosystem is reggae. But:

. Until recently, it wasn't possible to use 
__traits(getUnitTests) with separate compilation. I've run into 
other issues with when doing so (I can't remember what they 
were). Even with the bug fix editing a test entails compiling 
several modules and it's slower than just one C++ file. In C++ 
editing a test source file always means recompiling exactly one 
file.

. Speaking of dependencies, there's no easy way currently to know 
what they are. _If_ I want to build per package/module and only 
build what's needed, then I need to either hand-roll a Makefile 
(points gun to head) or use reggae. If there's any other 
alternative, I don't know about it. Even Phobos recompiles the 
world. That's not a good example to set.

. C++ test framework's hacky usage of global variable assignment 
to register tests seems to be faster to compile than D's 
compile-time reflection to achieve the same result. I'd have to 
benchmark to be sure though.

. If one has dub dependencies (and in all likelihood one has), 
then either one is a reggae early adopter (I keep mentioning it 
to highlight why I wrote it in the first place), or in the 99.9% 
of other users typing either `dub build` or `dub test`. There is 
no per-package build option. There's a per-module option, but 
it's single-threaded, therefore slow. Also, no dependency 
tracking, so let's recompile the whole dub package! In one 
thread. Zzzzzzzzz.

What I really want is to hit F5 in emacs, rebuild the current 
file _and_ all of its dependencies, cache the dependency build 
per file so that if they haven't change don't rebuild, link and 
run the unit tests. But build _only_ what's needed to avoid 
linker errors in order to be able to run the unit tests for _one_ 
file.

I spent an afternoon writing the necessary elisp to get this done 
and it worked! Then I tried a different project and it failed 
miserably. So did the 3rd one. I think I just remembered one of 
the separate compilation bugs I mentioned above. I know, I know, 
if I don't file it it won't get fixed (even if I file it I have a 
sneaky suspicion it'll be me fixing it anyway), but figuring out 
what went wrong takes time and energy and I haven't had enough of 
either to take this off my TODO list.

To summarise:

. Dependencies.
. There's no standard build system that isn't called dub.
. There's only one alternative build system that can do any of 
what's needed.
. That alternative build system hasn't implemented my .di idea 
yet.
. Separate compilation bugs. That nobody ever hits because dub 
builds everything at once.



Atila
Dec 29 2017
prev sibling next sibling parent reply I Love Stuffing <ilovestuffing gmail.com> writes:
On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:

 This one of the main strengths of D, it is what Walter focuses 
 on, yet I have seen almost nothing on the D blog talking about 
 this.  What brought me to emphasize this today is this recent 
 post about how long it takes to compile the mostly-C++ Chromium 
 web browser and the reddit discussion about it:

 https://lobste.rs/s/iri1te/chromium_has_compilation_time_problem
 https://www.reddit.com/r/programming/comments/7ktzog/chromium_has_a_compilation_time_problem/

 I'm tempted to call BS on that 6-7 hour build time, as I used 
 to contribute to the Chromium project, and I think it used to 
 take me 20 minutes for a release build in a FreeBSD jail on a 
 fairly weak, 2008-vintage mini-desktop, a dual-core Celeron 
 with 2 GBs of RAM (don't hold me to that build time, just a 
 vague recollection, but probably in the ballpark).  Of course, 
 the last time I built Chromium was more than 5 years ago, and a 
 lot has likely changed since then, such as now using LTO to 
 speed up the browser apparently, and maybe the 
 cross-compilation toolchain for ARM is slower, though others 
 note similar times for native x64 compilation also.

 That still implies a slowdown of 2-3 orders of magnitude over 
 the last 5 years, given the much more powerful hardware he's 
 using, which is nuts.

 D really needs the community to write blog posts talking about 
 how fast it is, publicizing that there is an alternative to 
 these glacial build times: who wants to do this?  It doesn't 
 need to be on the D blog, could just be on your personal blog, 
 but it is a message that really needs to be spread.


To be fair, you are basing this off C++. C++ has an obnoxiously 
slow build system at times and everyone knows this, have known 
this, and have even addressed some parts of it. Yet they still 
use it. I don't think build times are that big of a deal. It's a 
nice to have, but as long as a fairly sizable project compiles in 
a less than a minute, you won't stand out in a bad way. If 
anything, D should leverage the fast compile times to take 
advantage of the leeway it gives so the compiler can do a bit of 
extra work to give better error messages.
Dec 29 2017
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, December 29, 2017 22:37:56 I Love Stuffing via Digitalmars-d 
wrote:

 To be fair, you are basing this off C++. C++ has an obnoxiously
 slow build system at times and everyone knows this, have known
 this, and have even addressed some parts of it. Yet they still
 use it. I don't think build times are that big of a deal. It's a
 nice to have, but as long as a fairly sizable project compiles in
 a less than a minute, you won't stand out in a bad way.


So much about build times and how much they matter seems to be about
perception. For instance, I used to work at a company whose main project
took over 3 - 4 hours to do a clean build. After the project was redone in
Java, the build times were about 10 minutes, and folks complained about how
long they were even though they were worlds better than the C++ build times
had been.

It also depends on your personality. Personally, while I like builds being
near instantaneous when dealing with unit tests, if they take a few seconds,
it's not a big deal, but I know folks who get seriously annoyed if they can
notice much of a difference time-wise between hitting enter to run the build
command and hitting enter with no command. Either way though, having build
times be more than a few seconds when dealing with unit tests tends to cause
serious problems even though having a build for an entire project take a few
minutes usually isn't a big deal unless you're really impatient. Having the
delay between making changes and testing them be very long can _really_ slow
down development, and so that's where build times really matter.

- Jonathan M Davis
Dec 29 2017
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 29, 2017 at 03:49:07PM -0700, Jonathan M Davis via Digitalmars-d
wrote:
[...]

 It also depends on your personality. Personally, while I like builds
 being near instantaneous when dealing with unit tests, if they take a
 few seconds, it's not a big deal, but I know folks who get seriously
 annoyed if they can notice much of a difference time-wise between
 hitting enter to run the build command and hitting enter with no
 command. Either way though, having build times be more than a few
 seconds when dealing with unit tests tends to cause serious problems
 even though having a build for an entire project take a few minutes
 usually isn't a big deal unless you're really impatient. Having the
 delay between making changes and testing them be very long can
 _really_ slow down development, and so that's where build times really
 matter.


[...]

If you think a few minutes is bad... I've been on the receiving end of
builds that take 30-45 *minutes* for 1-line code changes
(*ahem*cough*C++ header files*cough*ahem*).  It put a serious damper on
my ability to make any progress with the code. It's *possible*, yes, but
it's not something I'd wish on anyone.


T

-- 
MSDOS = MicroSoft's Denial Of Service
Dec 29 2017
prev sibling next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, December 29, 2017 15:51:53 H. S. Teoh via Digitalmars-d wrote:

 On Fri, Dec 29, 2017 at 03:49:07PM -0700, Jonathan M Davis via
 Digitalmars-d wrote: [...]


 It also depends on your personality. Personally, while I like builds
 being near instantaneous when dealing with unit tests, if they take a
 few seconds, it's not a big deal, but I know folks who get seriously
 annoyed if they can notice much of a difference time-wise between
 hitting enter to run the build command and hitting enter with no
 command. Either way though, having build times be more than a few
 seconds when dealing with unit tests tends to cause serious problems
 even though having a build for an entire project take a few minutes
 usually isn't a big deal unless you're really impatient. Having the
 delay between making changes and testing them be very long can
 _really_ slow down development, and so that's where build times really
 matter.


 [...]

 If you think a few minutes is bad... I've been on the receiving end of
 builds that take 30-45 *minutes* for 1-line code changes
 (*ahem*cough*C++ header files*cough*ahem*).  It put a serious damper on
 my ability to make any progress with the code. It's *possible*, yes, but
 it's not something I'd wish on anyone.


Oh, certainly. But my point is that once it starts taking minutes to be able
to build and run the unit tests for what you're currently doing, it becomes
infeasible to edit-test-edit-etc. in any kind of reasonable cycle. It
doesn't have to get anywhere near the 30 minute mark for it to be a serious
problem. So, of course, 30+ minutes is an utter disaster if you're looking
to test stuff as you go.

- Jonathan M Davis
Dec 29 2017
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 29, 2017 at 05:52:36PM -0700, Jonathan M Davis via Digitalmars-d
wrote:
[...]

 Oh, certainly. But my point is that once it starts taking minutes to
 be able to build and run the unit tests for what you're currently
 doing, it becomes infeasible to edit-test-edit-etc. in any kind of
 reasonable cycle. It doesn't have to get anywhere near the 30 minute
 mark for it to be a serious problem. So, of course, 30+ minutes is an
 utter disaster if you're looking to test stuff as you go.


[...]

Yeah... recently I've been resorting to `dmd -unittest -main module.d`
on single modules for faster turnaround time, because in spite of D's
famed compilation speeds, some things still make it go far too slow,
e.g. if you use template-heavy / CTFE-heavy code, or there are just too
many files to compile.

This has also been encouraging me to write code in a far more modular
fashion than even traditional modularists would do, in the sense that
where possible, even dependent modules won't be imported, but instead
left to a template parameter, to be instantiated by the main program
with the dependent module. Contrived example:

-----
void printSomething(alias stdio = selfImport!"std.stdio", Args...)(string fmt,
Args args)
{
	stdio.writefln(fmt, args);
}

unittest
{
	struct TestHarness {
		static void writefln() { /* fake implementation here */ }
	}
	printSomething!TestHarness("%s", "blah");
}
-----

The idea is that importing std.stdio won't even happen when compiling
with -unittest -main. This approach lets you completely detach a module
from its dependencies, so you never have to worry about what else it
depends on when compiling with -unittest -main, and also conveniently
allows you to substitute complicated types with cheaper substitutes that
suffice for testing, e.g., instead of writing to stdout or actually
causing side-effects visible outside the program, have a substitute
module (or just struct!) that redirects output or filesystem changes to
a log that's verified by the unittest to be the correct sequence of
operations. The substitute type doesn't even have to implement the
complete API of the module/object in question; just enough of it for the
unittest to do what it needs to do.

Being able to substitute a struct in place of a module inside a unittest
is one of the awesome things about D that make me love it so. :-D

And a side perk is that you'll get automatic attribute inference for all
such functions, if the original function would otherwise be just a
non-template function. (On that note, nowadays I'm tempted to just stick
empty compile-time parameters on every function I can; it enables
attribute inference and also reduces executable bloat if the function is
never actually called from anywhere. The only bad thing is the compiler
won't run semantic on potentially broken code... but then again, if you
never use it, it doesn't matter, and if it ought to matter, there should
be a unittest for it anyway.)


T

-- 
"You are a very disagreeable person." "NO."
Dec 29 2017
parent reply Atila Neves <atila.neves gmail.com> writes:
On Saturday, 30 December 2017 at 01:09:59 UTC, H. S. Teoh wrote:

 On Fri, Dec 29, 2017 at 05:52:36PM -0700, Jonathan M Davis via 
 Digitalmars-d wrote: [...]

 [...]


 [...]

 Yeah... recently I've been resorting to `dmd -unittest -main 
 module.d` on single modules for faster turnaround time, because 
 in spite of D's famed compilation speeds, some things still 
 make it go far too slow, e.g. if you use template-heavy / 
 CTFE-heavy code, or there are just too many files to compile.

 [...]


This is pretty interesting. Unfortunately I can see it getting 
messy with multiple dependent modules and/or if the function has 
"real" template parameters on top. I have to think about how to 
use your strategy in my own code.

Atila
Jan 02 2018
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Jan 02, 2018 at 04:00:11PM +0000, Atila Neves via Digitalmars-d wrote:

 On Saturday, 30 December 2017 at 01:09:59 UTC, H. S. Teoh wrote:


[...]

 Yeah... recently I've been resorting to `dmd -unittest -main
 module.d` on single modules for faster turnaround time, because in
 spite of D's famed compilation speeds, some things still make it go
 far too slow, e.g. if you use template-heavy / CTFE-heavy code, or
 there are just too many files to compile.
 
 [...]


 
 This is pretty interesting. Unfortunately I can see it getting messy
 with multiple dependent modules and/or if the function has "real"
 template parameters on top. I have to think about how to use your
 strategy in my own code.


[...]

Yeah, when you have dependent modules it can get pretty messy.  Which is
why I've been moving towards modularising my code beyond what people
would usually do, i.e., turn even symbols in dependent modules into
template parameters so that you can make the module completely
independent.  Here's one actual code example I'm particularly proud of:

------
/**
 * Expands ' '-directives in a range of strings.
 *
 * Returns: A range of strings with lines that begin with ' ' substituted with
 * the contents of the file named by the rest of the line.
 */
auto expandFileDirectives(File = std.stdio.File, R)(R args)
    if (isInputRange!R && is(ElementType!R : const(char)[]))
{
    import std.algorithm.iteration : joiner, map;
    import std.algorithm.searching : startsWith;
    import std.range : only;
    import std.range.interfaces : InputRange, inputRangeObject;
    import std.typecons : No;

    return args.map!(arg => arg.startsWith(' ') ?
                    cast(InputRange!string) inputRangeObject(
                        File(arg[1 .. $]).byLineCopy(No.keepTerminator)) :
                    cast(InputRange!string) inputRangeObject(only(arg)))
               .joiner;
}

unittest
{
    static struct FakeFile
    {
        static string[][string] files;

        string[] contents;
        this(string filename) { contents = files[filename]; }
        auto byLineCopy(KeepTerminator dummy) { return contents; }
    }

    FakeFile.files["file1"] = [
        "x=1",
        "y=2"
    ];
    FakeFile.files["file2"] = [
        "z=3"
    ];

    import std.algorithm.comparison : equal;
    assert([
        "p=abc",
        " file1",
        "q=def",
        " file2",
        "e=ghi"
    ].expandFileDirectives!FakeFile.equal([
        "p=abc",
        "x=1",
        "y=2",
        "q=def",
        "z=3",
        "e=ghi"
    ]));
}
------

As you can see, I made File into a template parameter so that the
unittest can simulate a virtual filesystem with a struct, without ever
needing to deal with the messiness of actually creating and managing
temporary files on a real filesystem. This is (mostly) invisible to user
code (i.e., other code that uses this function) because if you don't
specify what File should be, it defaults to the usual std.stdio.File.

Note also that not all of File's API needed to be implemented; since the
function only uses .byLineCopy, that's all I needed to implement in my
surrogate FakeFile struct.

Not all code can be cut up in this way, though.  I've only achieved this
in a few low-level modules; for other modules I'm still facing the
problem of having to specify dependent modules on the command-line. But
in trying to attain to this goal, it really forces you to think about
structuring your code in a better way than just throwing in haphazard
dependencies that may not strictly be necessary.


T

-- 
Just because you can, doesn't mean you should.
Jan 02 2018
parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 01/02/2018 10:13 AM, H. S. Teoh wrote:


 As you can see, I made File into a template parameter so that the
 unittest can simulate a virtual filesystem with a struct


All I'm hearing is "blah blah blog." ;)

Ali
Jan 02 2018