www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - LDC: Speed up incremental builds with object file caching

reply Johan Engelen <j j.nl> writes:
I just finished another post about LDC:
https://johanengelen.github.io/ldc/2016/09/17/LDC-object-file-caching.html

Thanks in advance for letting me know about any bugs you find, in 
the text or in the code :)

-Johan
Sep 17 2016
next sibling parent reply Daniel Kozak via Digitalmars-d-announce writes:
Dne 17.9.2016 v 20:46 Johan Engelen via Digitalmars-d-announce napsal(a):


 I just finished another post about LDC:
 https://johanengelen.github.io/ldc/2016/09/17/LDC-object-file-caching.html 


 Thanks in advance for letting me know about any bugs you find, in the 
 text or in the code :)

 -Johan


Nice article, however font on this site is too small to read, I have to 
zoom in :(
Sep 17 2016
parent reply Johan Engelen <j j.nl> writes:
On Saturday, 17 September 2016 at 19:17:44 UTC, Daniel Kozak 
wrote:

 
 Nice article, however font on this site is too small to read, I 
 have to zoom in :(


Do more people have this problem? For me, the font size is 
similar to, say, Github.
Sep 18 2016
parent reply Daniel Kozak via Digitalmars-d-announce writes:
Dne 18.9.2016 v 10:07 Johan Engelen via Digitalmars-d-announce napsal(a):


 On Saturday, 17 September 2016 at 19:17:44 UTC, Daniel Kozak wrote:

 Nice article, however font on this site is too small to read, I have 
 to zoom in :(


 Do more people have this problem? For me, the font size is similar to, 
 say, Github.


Maybe it is a font type (it is light) this is how it looks on my display:
http://imgur.com/a/r6pX8
Sep 18 2016
parent reply Daniel Kozak <kozzi11 gmail.com> writes:
On Sunday, 18 September 2016 at 10:21:41 UTC, Daniel Kozak wrote:

 Dne 18.9.2016 v 10:07 Johan Engelen via Digitalmars-d-announce 
 napsal(a):


 On Saturday, 17 September 2016 at 19:17:44 UTC, Daniel Kozak 
 wrote:

 Nice article, however font on this site is too small to read, 
 I have to zoom in :(


 Do more people have this problem? For me, the font size is 
 similar to, say, Github.


 Maybe it is a font type (it is light) this is how it looks on 
 my display:
 http://imgur.com/a/r6pX8


But on my android phone it is OK. So maybe I should bought some 
2k or 4k monitor ;)
Sep 18 2016
parent Johan Engelen <j j.nl> writes:
On Sunday, 18 September 2016 at 12:15:12 UTC, Daniel Kozak wrote:

 But on my android phone it is OK. So maybe I should bought some 
 2k or 4k monitor ;)


Definitely one of the best upgrades I ever did (4k monitor). 
Warning: there's no way back.
Sep 18 2016
prev sibling next sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Saturday, 17 September 2016 at 18:46:26 UTC, Johan Engelen 
wrote:

 I just finished another post about LDC:
 https://johanengelen.github.io/ldc/2016/09/17/LDC-object-file-caching.html

 Thanks in advance for letting me know about any bugs you find, 
 in the text or in the code :)

 -Johan


Reddit: 
https://www.reddit.com/r/programming/comments/539wpv/ldc_speed_up_incremental_builds_of_d_programs/
Sep 17 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/17/2016 08:16 PM, Mike Parker wrote:

 On Saturday, 17 September 2016 at 18:46:26 UTC, Johan Engelen wrote:

 I just finished another post about LDC:
 https://johanengelen.github.io/ldc/2016/09/17/LDC-object-file-caching.html


 Thanks in advance for letting me know about any bugs you find, in the
 text or in the code :)

 -Johan


 Reddit:
 https://www.reddit.com/r/programming/comments/539wpv/ldc_speed_up_incremental_builds_of_d_programs/


Forgot to mention: please hold off any announcements during the weekend. 
All social news outlets experience lows during the weekend, and reddit 
is no exception. See e.g. 
https://www.redditlater.com/analysis#/r/programming

Best timed announcement: Monday at 9 AM EST (noon Pacific Time).


Thanks,

Andrei
Sep 18 2016
parent reply Chris Wright <dhasenan gmail.com> writes:
On Sun, 18 Sep 2016 11:25:51 -0400, Andrei Alexandrescu wrote:

 Best timed announcement: Monday at 9 AM EST (noon Pacific Time).


You mean 6am Pacific? Or has the sun reversed directions without me 
realizing it?
Sep 18 2016
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/18/2016 11:31 AM, Chris Wright wrote:

 On Sun, 18 Sep 2016 11:25:51 -0400, Andrei Alexandrescu wrote:

 Best timed announcement: Monday at 9 AM EST (noon Pacific Time).


 You mean 6am Pacific? Or has the sun reversed directions without me
 realizing it?


Sorry, 6 am Pacific. -- Andrei
Sep 18 2016
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Saturday, 17 September 2016 at 18:46:26 UTC, Johan Engelen 
wrote:

 I just finished another post about LDC:
 https://johanengelen.github.io/ldc/2016/09/17/LDC-object-file-caching.html

 Thanks in advance for letting me know about any bugs you find, 
 in the text or in the code :)


Interesting approach to speed up compilation without running into 
dmd's problems of template instance emission when compiling 
multiple modules to multiple objects.

Many people use noatime or relatime when mounting their 
filesystems, so access time isn't the best eviction strategy 
unless you touch the files.
Sep 18 2016
parent reply Johan Engelen <j j.nl> writes:
On Sunday, 18 September 2016 at 08:14:47 UTC, Martin Nowak wrote:

 Interesting approach to speed up compilation without running 
 into dmd's problems of template instance emission when 
 compiling multiple modules to multiple objects.


I think LDC has the same problems (template instances are emitted 
in the first module not the one that needed the instantiation?). 
Knowing this, you may be able to set up a case where things 
break, but I think it would have to involve recompilation with a 
different set of sources than the first compile. E.g. `ldc2 -c 
a.d b.d c.d` first, then `ldc2 -c b.d c.d`, and then trying to 
link. Which probably doesn't work well without caching either... 
We should deprecate non-singleobj compiles.


 Many people use noatime or relatime when mounting their 
 filesystems, so access time isn't the best eviction strategy 
 unless you touch the files.


I forgot to mention this in the article, thanks! I read that a 
file's last access timestamp  may also not be updated on Windows 
(default Windows 7 setting), so I already had LDC `touch` the 
cached file upon a cache hit. Added it as a footnote in the 
article.
Sep 18 2016
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2016-09-18 11:10, Johan Engelen wrote:


 I think LDC has the same problems (template instances are emitted in the
 first module not the one that needed the instantiation?).


I thought that LDC did not have that problem, or there was a flag change 
the behavior.

-- 
/Jacob Carlborg
Sep 18 2016
prev sibling parent Martin Nowak <code dawg.eu> writes:
On Sunday, 18 September 2016 at 09:10:41 UTC, Johan Engelen wrote:

 I think LDC has the same problems (template instances are 
 emitted in the first module not the one that needed the 
 instantiation?). Knowing this, you may be able to set up a case 
 where things break, but I think it would have to involve 
 recompilation with a different set of sources than the first 
 compile. E.g. `ldc2 -c a.d b.d c.d` first, then `ldc2 -c b.d 
 c.d`, and then trying to link. Which probably doesn't work well 
 without caching either... We should deprecate non-singleobj 
 compiles.


Yes, it's difficult to make that work correctly when the object 
cache is outside of the compiler. Maybe archives would provide a 
good way to build a better incremental rebuild support into the 
compiler.
At least with dmd's multiobj archives, template instances become 
their own object file.
It's definitely worth to put some more thought into how we can 
leverage parallel builds and incremental rebuilds.

https://trello.com/c/dBfiJHEM/72-spec-better-parallel-incremental-build-strategies-for-bigger-projects
Sep 19 2016
prev sibling parent =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Saturday, 17 September 2016 at 18:46:26 UTC, Johan Engelen 
wrote:

 I just finished another post about LDC:
 https://johanengelen.github.io/ldc/2016/09/17/LDC-object-file-caching.html


Great innovative work!

One thing: I'm missing an option for only doing time-stamp 
checking of the dependencies similar to how SCons checks if 
cached files are obselete.

See the Decider function documented here:

http://www.scons.org/doc/HTML/scons-user/ch06.html#idp1416061700

It can be either

- Decider('MD5')
- Decider('timestamp-newer')
- Decider('timestamp-match')
- Decider('MD5-timestamp')

As I see it the LDC-enhancement always does content hashing, that 
is, it is similar to SCons' `Decider('MD5')`. If so, it would be 
very worthwhile checking how much faster an approach similar to 
`Decider('MD5-timestamp')` would be. This approach only hashes 
content of dependencies if they have been modified since last 
hashing during last build of that object with the same compiler 
flags was done.
Sep 18 2016