www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - dmdcache

reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
A colleague of mine has written dmdcache which may be very useful for 
some projects:

   https://github.com/seeraven/dmdcache

It drops our build time

   from 8 minutes
   to 45 seconds

on a particular build environment for about half a dozen D programs, one 
of which ends up being a 2G executable! WAT! :) And the total cache size 
is 5.5G. Wow!

This build is with dmd 2.084.1 and that one particular application uses 
tons of template instantiations most of which are in generated source 
code. If I remember correctly, 2.084.1 does not contain template symbol 
name improvements and that may be the reason for the large size.

Enjoy!

Ali
Apr 25 2020
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 25 April 2020 at 10:17:50 UTC, Ali Çehreli wrote:

 A colleague of mine has written dmdcache which may be very 
 useful for some projects:

   https://github.com/seeraven/dmdcache

 It drops our build time

   from 8 minutes
   to 45 seconds

 on a particular build environment for about half a dozen D 
 programs, one of which ends up being a 2G executable! WAT! :) 
 And the total cache size is 5.5G. Wow!

 This build is with dmd 2.084.1 and that one particular 
 application uses tons of template instantiations most of which 
 are in generated source code. If I remember correctly, 2.084.1 
 does not contain template symbol name improvements and that may 
 be the reason for the large size.

 Enjoy!

 Ali


The main problem with this is that it does not take string 
imports into account, (or does it ???, I don't see how it could 
....)
Also the compilers output can depend on the timestamp at which 
the compilation was done.
Apr 25 2020
parent reply bauss <jj_1337 live.dk> writes:
On Saturday, 25 April 2020 at 10:35:49 UTC, Stefan Koch wrote:

 On Saturday, 25 April 2020 at 10:17:50 UTC, Ali Çehreli wrote:

 A colleague of mine has written dmdcache which may be very 
 useful for some projects:

   https://github.com/seeraven/dmdcache

 It drops our build time

   from 8 minutes
   to 45 seconds

 on a particular build environment for about half a dozen D 
 programs, one of which ends up being a 2G executable! WAT! :) 
 And the total cache size is 5.5G. Wow!

 This build is with dmd 2.084.1 and that one particular 
 application uses tons of template instantiations most of which 
 are in generated source code. If I remember correctly, 2.084.1 
 does not contain template symbol name improvements and that 
 may be the reason for the large size.

 Enjoy!

 Ali


 The main problem with this is that it does not take string 
 imports into account, (or does it ???, I don't see how it could 
 ....)
 Also the compilers output can depend on the timestamp at which 
 the compilation was done.


Yeah, doesn't look like it which means it might not be useful in 
projects that does a lot of compile-time stuff.

There is no way to determine the import statements either as 
those themselves can be generated at compile-time.

I can see this being useful for large projects that does not use 
CTFE but other than that I think it might just create subtle bugs 
because you won't immediately know that some part of your code 
didn't update and isn't working as intended because the code for 
it was imported from another file etc.
Apr 25 2020
next sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 4/25/20 9:01 AM, bauss wrote:


 On Saturday, 25 April 2020 at 10:35:49 UTC, Stefan Koch wrote:



 The main problem with this is that it does not take string imports
 into account




[...]


 Yeah, doesn't look like it which means it might not be useful in
 projects that does a lot of compile-time stuff.

 There is no way to determine the import statements either as those
 themselves can be generated at compile-time.


Luckily, dmd gives all of that information with both -v and -X command 
line switches.

However, currently dmdcache doesn't seem to look for string imports; 
opportunity for improvement. :)

Ali
Apr 26 2020
parent rikki cattermole <rikki cattermole.co.nz> writes:
On 27/04/2020 1:52 AM, Ali Çehreli wrote:

 However, currently dmdcache doesn't seem to look for string imports; 
 opportunity for improvement. :)


That should not be necessary if you base it on -J instead.
Apr 26 2020
prev sibling parent sighoya <sighoya gmail.com> writes:
On Saturday, 25 April 2020 at 16:01:03 UTC, bauss wrote:

There is no way to determine the import statements either as 
those themselves can be generated at compile-time.



Question: is then DMD capable to add generated source code in 
separate files to the compilation process when we can just import 
code with arbitrary string paths?
Dec 29 2020
prev sibling next sibling parent reply Johan <j j.nl> writes:
On Saturday, 25 April 2020 at 10:17:50 UTC, Ali Çehreli wrote:

 A colleague of mine has written dmdcache which may be very 
 useful for some projects:

   https://github.com/seeraven/dmdcache

 It drops our build time

   from 8 minutes
   to 45 seconds


Hey Ali,
   Did you also try with LDC's built-in object file caching (asm 
codegen caching) ?

Cheers,
   Johan
Apr 25 2020
parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 4/25/20 4:39 AM, Johan wrote:

 On Saturday, 25 April 2020 at 10:17:50 UTC, Ali =C3=87ehreli wrote:

 A colleague of mine has written dmdcache which may be very useful for =





 some projects:

 =C2=A0 https://github.com/seeraven/dmdcache

 It drops our build time

 =C2=A0 from 8 minutes
 =C2=A0 to 45 seconds


=20
 Hey Ali,
  =C2=A0 Did you also try with LDC's built-in object file caching (asm c=


odegen=20

 caching) ?
=20
 Cheers,
  =C2=A0 Johan
=20


That sounds very promising. Time to introduce ldc to the project.

Ali
Apr 26 2020
prev sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Saturday, 25 April 2020 at 10:17:50 UTC, Ali Çehreli wrote:

 A colleague of mine has written dmdcache which may be very 
 useful for some projects:

   https://github.com/seeraven/dmdcache

 It drops our build time

   from 8 minutes
   to 45 seconds

 on a particular build environment for about half a dozen D 
 programs, one of which ends up being a 2G executable! WAT! :) 
 And the total cache size is 5.5G. Wow!

 This build is with dmd 2.084.1 and that one particular 
 application uses tons of template instantiations most of which 
 are in generated source code. If I remember correctly, 2.084.1 
 does not contain template symbol name improvements and that may 
 be the reason for the large size.

 Enjoy!

 Ali


Perhaps I'm being very dumb, but how does this differ from just 
using make?
Apr 25 2020
next sibling parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 4/25/20 5:30 AM, John Colvin wrote:


 how does this differ from just using make?


make is great and I love it (really) but it works at a coarser level. 
There is no way for it to know that a particular command will produce 
the same output.

As I understand it, dmdcache is supposed to be similar to ccache, which 
we already use with make to speed up our C++ compilations:

   https://ccache.dev/

Ali
Apr 26 2020
prev sibling parent Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Saturday, 25 April 2020 at 12:30:17 UTC, John Colvin wrote:

 But how does this differ from just using make?


make doesn't support total-dependency-content-digest keying of 
cached artifacts similar to what, for instance, Bazel does.
Dec 29 2020