www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Good build system?

reply burt <invalid_email_address cab.abc> writes:
Hello,

I have a project that is taking very long to compile (I'm on 
Windows).

It consists of a bunch of precompiled library (.lib) files and 
some .d source files inside of my "source" directory. Currently I 
am using DUB and the .lib files are added to "sourceFiles" in 
dub.json, which works, but if I edit a single letter it will 
recompile everything.

One alternative would be to split up my package into 
"subPackages" to avoid recompiling everything, but that would 
mean splitting up my current directory structure into a structure 
that is harder to navigate.

Another alternative I tried is Reggae, but it doesn't work on 
Windows:
1. When calling .reggae/dcompile, it fails, because the Windows 
shell (intuitively as always) splits of the ".reggae" from 
"/dcompile" and treats them as separate arguments (managed to fix 
this by editing the Reggae source slightly, in 
payload\reggae\build.d : expandCmd).
2. Either it takes FOREVER to compile and link the thing, or it 
gets stuck in some kind of infinite loop.

Anyway, it doesn't work.

What kind of build system could I use to speed up compilation 
when editing very little, or could I configure DUB so it 
recompiles per file instead of per package?

Thank you.
Aug 15 2020
next sibling parent reply Avrina <avrina12309412342 gmail.com> writes:
On Saturday, 15 August 2020 at 14:22:44 UTC, burt wrote:

 Hello,

 I have a project that is taking very long to compile (I'm on 
 Windows).

 It consists of a bunch of precompiled library (.lib) files and 
 some .d source files inside of my "source" directory. Currently 
 I am using DUB and the .lib files are added to "sourceFiles" in 
 dub.json, which works, but if I edit a single letter it will 
 recompile everything.

 One alternative would be to split up my package into 
 "subPackages" to avoid recompiling everything, but that would 
 mean splitting up my current directory structure into a 
 structure that is harder to navigate.

 Another alternative I tried is Reggae, but it doesn't work on 
 Windows:
 1. When calling .reggae/dcompile, it fails, because the Windows 
 shell (intuitively as always) splits of the ".reggae" from 
 "/dcompile" and treats them as separate arguments (managed to 
 fix this by editing the Reggae source slightly, in 
 payload\reggae\build.d : expandCmd).
 2. Either it takes FOREVER to compile and link the thing, or it 
 gets stuck in some kind of infinite loop.

 Anyway, it doesn't work.

 What kind of build system could I use to speed up compilation 
 when editing very little, or could I configure DUB so it 
 recompiles per file instead of per package?

 Thank you.


What project are you compiling? It would help to diagnose the 
problem, as well as what the specs of the computer you are using 
are.
Aug 15 2020
parent burt <invalid_email_address cab.abc> writes:
On Saturday, 15 August 2020 at 15:45:44 UTC, Avrina wrote:

 What project are you compiling? It would help to diagnose the 
 problem, as well as what the specs of the computer you are 
 using are.


It is basically an implementation for a DSL, which has to parse, 
process and interpret it. The interpretation part has already 
been implemented and doesn't have to be recompiled (.lib), but it 
does have to link with it (which takes up quite a bit of time as 
well, because it has to redo it every time).

On Saturday, 15 August 2020 at 16:15:08 UTC, WebFreak001 wrote:

 [...]

 to speed up dub I recommend

 1) split up your code base into multiple subpackages (isolated 
 packages of code as seen in vibe.d)
 2a) try using --build-mode=allAtOnce which will compile the 
 entire project in one go (note that this will increase memory 
 usage)
 2b) try using --build-mode=singleFile which will compile each 
 file separately and link them all together (note that this 
 eliminates any possible caching by the compiler, but reduces 
 memory usage which could also be a bottle neck)

 Of these steps 1) will definitely help while developing as it 
 will not recompile the packages which didn't change. 2a) or 2b) 
 can be useful if you are on a very beefy machine or have memory 
 constraints on a low level machine


Splitting it up into subpackages helped indeed. 
--build-mode=singleFile did not speed it up much, but it did show 
me where the bottleneck was: the compiler was taking FOREVER on 
parser.d (I mean minutes, for a 3000-line file), so it sure 
helped to split that one off. Maybe there's a ton of hidden 
template instantiations going on? Or there's a compiler bug 
somewhere? I don't know.

Linking is still taking very long; I'm compiling for 64-bit with 
the MS linker. Could using another linker or some other linker 
technique speed it up?

Thank you for your response.
Aug 15 2020
prev sibling next sibling parent WebFreak001 <d.forum webfreak.org> writes:
On Saturday, 15 August 2020 at 14:22:44 UTC, burt wrote:

 Hello,

 I have a project that is taking very long to compile (I'm on 
 Windows).

 It consists of a bunch of precompiled library (.lib) files and 
 some .d source files inside of my "source" directory. Currently 
 I am using DUB and the .lib files are added to "sourceFiles" in 
 dub.json, which works, but if I edit a single letter it will 
 recompile everything.

 One alternative would be to split up my package into 
 "subPackages" to avoid recompiling everything, but that would 
 mean splitting up my current directory structure into a 
 structure that is harder to navigate.

 Another alternative I tried is Reggae, but it doesn't work on 
 Windows:
 1. When calling .reggae/dcompile, it fails, because the Windows 
 shell (intuitively as always) splits of the ".reggae" from 
 "/dcompile" and treats them as separate arguments (managed to 
 fix this by editing the Reggae source slightly, in 
 payload\reggae\build.d : expandCmd).
 2. Either it takes FOREVER to compile and link the thing, or it 
 gets stuck in some kind of infinite loop.

 Anyway, it doesn't work.

 What kind of build system could I use to speed up compilation 
 when editing very little, or could I configure DUB so it 
 recompiles per file instead of per package?

 Thank you.


to speed up dub I recommend

1) split up your code base into multiple subpackages (isolated 
packages of code as seen in vibe.d)
2a) try using --build-mode=allAtOnce which will compile the 
entire project in one go (note that this will increase memory 
usage)
2b) try using --build-mode=singleFile which will compile each 
file separately and link them all together (note that this 
eliminates any possible caching by the compiler, but reduces 
memory usage which could also be a bottle neck)

Of these steps 1) will definitely help while developing as it 
will not recompile the packages which didn't change. 2a) or 2b) 
can be useful if you are on a very beefy machine or have memory 
constraints on a low level machine
Aug 15 2020
prev sibling next sibling parent reply Andre Pany <andre s-e-a-p.de> writes:
On Saturday, 15 August 2020 at 14:22:44 UTC, burt wrote:

 Hello,

 I have a project that is taking very long to compile (I'm on 
 Windows).

 [...]


As side information, it is possible to introduce sub packages 
without changing the folder structure. That said, I do not 
recommend it. Example 
https://www.github.com/adamdruppe/arsd/tree/master/

Kind regards
Andre
Aug 15 2020
parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 15 August 2020 at 18:24:15 UTC, Andre Pany wrote:

 As side information, it is possible to introduce sub packages 
 without changing the folder structure. That said, I do not 
 recommend it. Example 
 https://www.github.com/adamdruppe/arsd/tree/master/


The problem is that since all sub packages still have the same 
import path, any change to any sub package triggers a rebuild of 
all sub packages since they may import modules from each other, 
so you don't gain compile time there.
Aug 15 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 15 August 2020 at 18:33:06 UTC, Dennis wrote:

 The problem is that since all sub packages still have the same 
 import path, any change to any sub package triggers a rebuild 
 of all sub packages since they may import modules from each 
 other, so you don't gain compile time there.


It shouldn't do that; if it does, I'd call it an implementation 
bug. You can list out the specific files that make up a 
subpackage.
Aug 15 2020
parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 15 August 2020 at 19:20:39 UTC, Adam D. Ruppe wrote:

 It shouldn't do that; if it does, I'd call it an implementation 
 bug. You can list out the specific files that make up a 
 subpackage.


But an enum / template in a different module could still affect 
the subpackage even if it's not linked in, so dub conservatively 
assumes it invalidated the build. The same applies to  
stringImportPaths. Maybe if dub supported also specifying 
`importFiles` instead of `importPaths` it could assume those 
other modules are not imported.
Aug 15 2020
parent Andre Pany <andre s-e-a-p.de> writes:
On Saturday, 15 August 2020 at 19:48:51 UTC, Dennis wrote:

 On Saturday, 15 August 2020 at 19:20:39 UTC, Adam D. Ruppe 
 wrote:

 It shouldn't do that; if it does, I'd call it an 
 implementation bug. You can list out the specific files that 
 make up a subpackage.


 But an enum / template in a different module could still affect 
 the subpackage even if it's not linked in, so dub 
 conservatively assumes it invalidated the build. The same 
 applies to  stringImportPaths. Maybe if dub supported also 
 specifying `importFiles` instead of `importPaths` it could 
 assume those other modules are not imported.


If files from stringImportPaths are changed, this won't cause a 
rebuild of the dub package as the change is not detected by dub. 
For this reason a new attribute extraDependencyFiles were added. 
This attribute lists files which should cause a dub rebuild if 
file content changes.

Kind regards
Andre
Aug 15 2020
prev sibling next sibling parent reply user1234 <user1234 12.de> writes:
On Saturday, 15 August 2020 at 14:22:44 UTC, burt wrote:

 Hello,

 I have a project that is taking very long to compile (I'm on 
 Windows).


Your project should take 10 secs to compile even with 
--build=release.
It's possible that, in case you would use DMD, you hit a problem 
of counter performance of the inliner/optimizer. If this is the 
case try ldc2. In some **rare occasions**, such the inler problem 
I mentioned, a release build can be faster with ldc2.

Side question, would your parser would be pegged based ?
Aug 15 2020
parent burt <invalid_email_address cab.abc> writes:
On Saturday, 15 August 2020 at 18:45:28 UTC, user1234 wrote:

 [...]

 Your project should take 10 secs to compile even with 
 --build=release.
 It's possible that, in case you would use DMD, you hit a 
 problem of counter performance of the inliner/optimizer. If 
 this is the case try ldc2. In some **rare occasions**, such the 
 inler problem I mentioned, a release build can be faster with 
 ldc2.


I’ll try ldc.


 Side question, would your parser would be pegged based ?


No, it’s a “manually” made parser.

Thank you
Aug 16 2020
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Saturday, 15 August 2020 at 14:22:44 UTC, burt wrote:

 Hello,

 I have a project that is taking very long to compile (I'm on 
 Windows).

 [...]


Yeah, sorry about that, getting reggae to work on Windows has 
been on my to-do list for far too long. It's going to happen 
because we need it to at work.
Aug 24 2020