www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Can compilation times not be sped up any further?

reply Brett <Brett gmail.com> writes:
I have a huge code base... it seems D likes to recompile 
everything every time from scratch. I realize there are ways to 
fix this but can't D go a step further?

Much of the code does not change. D has to compile a file per 
file basis.

Could D not just has the file in such a way as to reuse parts of 
previous compilations?

e.g., each function in the file could be hashed and if no changes 
are made then previous code generation is used.

There should be absolutely no reason a function, specially pure 
functions have to be regenerated every compilation.

By proper hashing design one should be able to almost limit 
compilation to only the changes that matter.

If a file, module, package does not change it should not be 
recompiled but reused. If a file's internals do not change in 
specific ways then they too generally can be used.

This would probably not be too complex of a problem if integrated 
in with D's parser. Each node in the AST is associated with a 
hash of the text(or it could be the AST node itself in some way 
but the text should do).

All code blocks need to determine what escapes and the object 
code should express those escapes in some way so they can be 
fixed up(which is basically global references and potential meta 
changes). If the compiler monitors and records this stuff proper 
then most of the work is already done.

Then it is just a matter of reading in the object code at the 
right time to replace the old code.

While we can compile to individual object files this requires 
user interaction. I am thinking of D having one large file that 
contains all the info that it can use to reduce compilation times 
by reusing information rather than regenerating. Files that do 
not change text and have do not escape references do not have to 
be recompiled. The compiler itself should handle this, not the 
user. For individual files that do change, a more granular 
approach is used. This requires every object file to also include 
the granularity though for maximum speed.
Oct 19 2019
next sibling parent Seb <seb wilzba.ch> writes:
On Saturday, 19 October 2019 at 21:56:44 UTC, Brett wrote:

 I have a huge code base... it seems D likes to recompile 
 everything every time from scratch. I realize there are ways to 
 fix this but can't D go a step further?

 [...]


I think this might answer some of your questions:

https://forum.dlang.org/post/mailman.484.1570820437.8294.dlang-internal puremagic.com

tl;dr: there's a PR ...
Oct 19 2019
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Oct 19, 2019 at 09:56:44PM +0000, Brett via Digitalmars-d wrote:

 I have a huge code base... it seems D likes to recompile everything
 every time from scratch. I realize there are ways to fix this but
 can't D go a step further?


Are you using dub?  If you are, you might want to consider using a
different build tool instead. One of the reasons I gave up dub long ago
was because of its compulsion to rebuild everything from scratch every
single time, which is unbearably slow. Now that I have my own build
system setup with SCons, I'm much happier.

Running dmd without dub is actually incredibly fast (compared to, say,
C++) that you generally only need separate compilation for packages.
Compiling all modules in a single package at once is generally fast
enough that there's no need to go to the per-file level of granularity.


T

-- 
"I suspect the best way to deal with procrastination is to put off the
procrastination itself until later. I've been meaning to try this, but haven't
gotten around to it yet. " -- swr
Oct 19 2019
parent Andre Pany <andre s-e-a-p.de> writes:
On Sunday, 20 October 2019 at 05:44:14 UTC, H. S. Teoh wrote:

 On Sat, Oct 19, 2019 at 09:56:44PM +0000, Brett via 
 Digitalmars-d wrote:

 I have a huge code base... it seems D likes to recompile 
 everything every time from scratch. I realize there are ways 
 to fix this but can't D go a step further?


 Are you using dub?  If you are, you might want to consider 
 using a different build tool instead. One of the reasons I gave 
 up dub long ago was because of its compulsion to rebuild 
 everything from scratch every single time, which is unbearably 
 slow. Now that I have my own build system setup with SCons, I'm 
 much happier.

 Running dmd without dub is actually incredibly fast (compared 
 to, say, C++) that you generally only need separate compilation 
 for packages. Compiling all modules in a single package at once 
 is generally fast enough that there's no need to go to the 
 per-file level of granularity.


 T


In a dub package you can also organize your modules into sub 
packages. These sub packages are only compiled when a source 
module of it is touched.

This might already solve the problem. But yes, there are a lot of 
potential in Dub to make it faster.

Kind regards
Andre
Oct 20 2019