www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.ide - Speed up compilation in Visual D

reply Alex <AJ gmail.com> writes:
Is there any way to speed of compilation in Visual D? Seems 
everything is being recompiled from scratch even with minute 
changes. I'd think compiling to object files might help but maybe 
not?
Apr 22 2019
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 23/04/2019 02:50, Alex wrote:

 Is there any way to speed of compilation in Visual D? Seems everything
 is being recompiled from scratch even with minute changes. I'd think
 compiling to object files might help but maybe not?


With visualdproj project files, you can select between building all at
once (which would cause what you describe unless you separate your code
into multiple libraries) and single file compilation which will only
recompile what (transitively) imports the file with the changes, but
that's often only faster if there are only few files to update.

The vcproj projects can compile per package, i.e. there is some kind of
automatic separation into libraries. In addition, the latest version
allows you to enable parallel compilation on multiple cores.
Apr 23 2019
parent reply Alex <AJ gmail.com> writes:
On Tuesday, 23 April 2019 at 07:03:36 UTC, Rainer Schuetze wrote:

 On 23/04/2019 02:50, Alex wrote:

 Is there any way to speed of compilation in Visual D? Seems 
 everything is being recompiled from scratch even with minute 
 changes. I'd think compiling to object files might help but 
 maybe not?


 With visualdproj project files, you can select between building 
 all at once (which would cause what you describe unless you 
 separate your code into multiple libraries) and single file 
 compilation which will only recompile what (transitively) 
 imports the file with the changes, but that's often only faster 
 if there are only few files to update.


How do I enable this? There are probably 10 thousand files and I 
updating just a few.

I tried some of the different compilation options[combined 
compile and link and separate compile and link] and they all took 
about the same time except with separate compile I'd get the 
warning:

x64\Debug DMD\_.obj : warning LNK4042: object specified more than 
once; extras ignored

x4


 The vcproj projects can compile per package, i.e. there is some 
 kind of automatic separation into libraries. In addition, the 
 latest version allows you to enable parallel compilation on 
 multiple cores.


Is there a way to move a visualdproj to a vcproj?

What it sounds like is that you are saying vcproj is better?
Apr 23 2019
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 23/04/2019 18:03, Alex wrote:

 On Tuesday, 23 April 2019 at 07:03:36 UTC, Rainer Schuetze wrote:

 On 23/04/2019 02:50, Alex wrote:

 Is there any way to speed of compilation in Visual D? Seems
 everything is being recompiled from scratch even with minute changes.
 I'd think compiling to object files might help but maybe not?


 With visualdproj project files, you can select between building all at
 once (which would cause what you describe unless you separate your
 code into multiple libraries) and single file compilation which will
 only recompile what (transitively) imports the file with the changes,
 but that's often only faster if there are only few files to update.


 
 How do I enable this? There are probably 10 thousand files and I
 updating just a few.
 
 I tried some of the different compilation options[combined compile and
 link and separate compile and link] and they all took about the same
 time except with separate compile I'd get the warning:
 
 x64\Debug DMD\_.obj : warning LNK4042: object specified more than once;
 extras ignored
 
 x4


I guess you have multiple file _.d in different directories? You should
enable "keep path from source" on the output page so these are not
written to the same object files.

"combined" and "separate" compile and link only have subtle differences
that won't have an effect on large projects. "combined" will try to let
dmd also do the link in the same invocation, but that is often
prohibited by other options.

You might want to try "single file compilation" with "keep path from
source" instead.


 

 The vcproj projects can compile per package, i.e. there is some kind
 of automatic separation into libraries. In addition, the latest
 version allows you to enable parallel compilation on multiple cores.


 
 Is there a way to move a visualdproj to a vcproj?


Unfortunately, no. You can include all files from a directory
recursively, as used by dmd
(https://github.com/dlang/dmd/blob/master/src/vcbuild/dmd.vcxproj#L214),
but it won't recreate the folder structure in the project. VC projects
also don't let you drop directories into the solution, so folders have
to be created manually.


 
 What it sounds like is that you are saying vcproj is better?


It's probably the future, but there are a couple of features that
haven't made it to the msbuild based projects, e.g. OMF support, private
phobos build.
Apr 23 2019
parent reply Alex <AJ gmail.com> writes:
On Wednesday, 24 April 2019 at 05:34:10 UTC, Rainer Schuetze 
wrote:

 On 23/04/2019 18:03, Alex wrote:

 On Tuesday, 23 April 2019 at 07:03:36 UTC, Rainer Schuetze 
 wrote:

 On 23/04/2019 02:50, Alex wrote:

 Is there any way to speed of compilation in Visual D? Seems 
 everything is being recompiled from scratch even with minute 
 changes. I'd think compiling to object files might help but 
 maybe not?


 With visualdproj project files, you can select between 
 building all at once (which would cause what you describe 
 unless you separate your code into multiple libraries) and 
 single file compilation which will only recompile what 
 (transitively) imports the file with the changes, but that's 
 often only faster if there are only few files to update.


 
 How do I enable this? There are probably 10 thousand files and 
 I updating just a few.
 
 I tried some of the different compilation options[combined 
 compile and link and separate compile and link] and they all 
 took about the same time except with separate compile I'd get 
 the warning:
 
 x64\Debug DMD\_.obj : warning LNK4042: object specified more 
 than once; extras ignored
 
 x4


 I guess you have multiple file _.d in different directories? 
 You should enable "keep path from source" on the output page so 
 these are not written to the same object files.

 "combined" and "separate" compile and link only have subtle 
 differences that won't have an effect on large projects. 
 "combined" will try to let dmd also do the link in the same 
 invocation, but that is often prohibited by other options.

 You might want to try "single file compilation" with "keep path 
 from source" instead.


When I try single file compilation I get errors about models not 
being found. the problem doesn't exist in combined compilation.

I think, but not sure, it's because I have some files in 
different directories that access files across "realms".

So I'm having to add include's for all these directories just to 
get things to compile ;/

Since I use a hierarchy of folders, this is a real pain in the 
ass ;/ D doesn't seem to let one recursively search included 
subdirectories for the includes ;/

It wants a flat list of files to work well, it seems, which is 
asinine given that it is very hard to maintain projects that way.

Why it works on combined compile I don't know. The build log is 
vastly different and seems to be figuring out the import paths 
and doing something with rsp files.

I won't be able to use single file compilation the way it is 
working since I'd have to include about 100 directories ;/

....

Ok, so I went ahead and did it, it wasn't 100 but about 20. The 
copy full path came in handy.

Now I get a compiler crash!

It seems because it is due to the fact that I have some code that 
reads modules and imports their text, which requires a -J for 
every directory, and I have to add it for every new file path I 
end up using(One thing I hate D about because it forces me to 
keep track of all this mess rather than just working).

The thing is, I have all the proper -J's under command line. But 
they are not showing up on the command line but they are when I 
do combined compile.

So possibly this is a bug in Visual D where single compile mode 
is ignoring the additional command line options?

As if when generating the command line for single file 
compilation you forgot to append the additional options string to 
it.




May I make a suggestion?


Allow visual D to automatically generate paths for the command 
line by using wild cards.

This might require a special input box like

-J [                         ] [+]

Where one can then add paths like one can do with imports, 
versions, etc

but one can use wild cards like

C:\Project\*;C:\WaltersSecureInfoToHack\*

and visual D will parse all the sub directories and add them 
manually.

This will alleviate some major headaches with D's security but it 
should be valid since it's used inside an IDE. (since no one 
passes around visual D project files anyways.

The same could be done for import paths too.

It would simply allow me to use one import such as C:\Projects\*. 
Since all ones I added are hard coded, if I ever move things 
around I have to modify all the paths(and remember to do so).


Thanks.
Apr 25 2019
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 26/04/2019 06:50, Alex wrote:

 You might want to try "single file compilation" with "keep path from
 source" instead.


 When I try single file compilation I get errors about models not being
 found. the problem doesn't exist in combined compilation.
 
 I think, but not sure, it's because I have some files in different
 directories that access files across "realms".

 So I'm having to add include's for all these directories just to get


 things to compile ;/
 
 Since I use a hierarchy of folders, this is a real pain in the ass ;/ D
 doesn't seem to let one recursively search included subdirectories for
 the includes ;/
 
 It wants a flat list of files to work well, it seems, which is asinine
 given that it is very hard to maintain projects that way.
 
 Why it works on combined compile I don't know. The build log is vastly
 different and seems to be figuring out the import paths and doing
 something with rsp files.


If the modules are passed on the command line, the compiler already
knows about them by the module statements and doesn't have to search any
directories.


 
 I won't be able to use single file compilation the way it is working
 since I'd have to include about 100 directories ;/
 
 ....
 
 Ok, so I went ahead and did it, it wasn't 100 but about 20. The copy
 full path came in handy.
 
 Now I get a compiler crash!
 
 It seems because it is due to the fact that I have some code that reads
 modules and imports their text, which requires a -J for every directory,
 and I have to add it for every new file path I end up using(One thing I
 hate D about because it forces me to keep track of all this mess rather
 than just working).
 
 The thing is, I have all the proper -J's under command line. But they
 are not showing up on the command line but they are when I do combined
 compile.


You can also add them to the "String import paths" option.


 So possibly this is a bug in Visual D where single compile mode is
 ignoring the additional command line options?



 As if when generating the command line for single file compilation you
 forgot to append the additional options string to it.


Yes, looks like a bug in Visual D: it adds the additional command line
options to the link, but not to the single file compiler invocations.


 
 
 
 May I make a suggestion?
 
 
 Allow visual D to automatically generate paths for the command line by
 using wild cards.
 
 This might require a special input box like
 
 -J [                         ] [+]
 
 Where one can then add paths like one can do with imports, versions, etc
 
 but one can use wild cards like
 
 C:\Project\*;C:\WaltersSecureInfoToHack\*
 
 and visual D will parse all the sub directories and add them manually.
 
 This will alleviate some major headaches with D's security but it should
 be valid since it's used inside an IDE. (since no one passes around
 visual D project files anyways.
 
 The same could be done for import paths too.
 
 It would simply allow me to use one import such as C:\Projects\*. Since
 all ones I added are hard coded, if I ever move things around I have to
 modify all the paths(and remember to do so).


I'm not so sure this is a recommended workflow.

If the modules in the different folders depend on each other, you might
start the module hierarchy a bit further up, e.g. make c:\projects the
root of the package hierarchy. Then only a single import path is needed.

If you have independent packages in your projects folder, you could
create separate projects building static libraries. Enabling "Add import
paths of dependent projects" can automatically add the folders to the
main project (but it might miss string imports ATM). This would also
help the initial problem with only recompiling parts that have changed.

You could also generate the import path options to a response file and
add that to the additional command lines as " import_options.rsp", but
it needs the bug above to be fixed.
Apr 26 2019
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 26/04/2019 09:52, Rainer Schuetze wrote:

 As if when generating the command line for single file compilation you
 forgot to append the additional options string to it.


 Yes, looks like a bug in Visual D: it adds the additional command line
 options to the link, but not to the single file compiler invocations.
 


fixed in https://github.com/dlang/visuald/releases/tag/v0.49.2
Apr 28 2019
parent reply Alex <AJ gmail.com> writes:
On Sunday, 28 April 2019 at 08:25:18 UTC, Rainer Schuetze wrote:

 On 26/04/2019 09:52, Rainer Schuetze wrote:

 As if when generating the command line for single file 
 compilation you forgot to append the additional options 
 string to it.


 Yes, looks like a bug in Visual D: it adds the additional 
 command line options to the link, but not to the single file 
 compiler invocations.
 


 fixed in https://github.com/dlang/visuald/releases/tag/v0.49.2


Man, it's really slow compiling!

It took over 3 minutes to compile the same project that compiles 
in 10 seconds with combined compile.

Even after compilation of all the object files it takes 10 
seconds just to recompile 1 file.

There is absolutely no speed up.

Also, I always get

x64\Debug DMD\S.exe not up to date: d:\repos\s\s\x64\debug 
dmd\s.exe older than d:\repos\s\s\$(outdir)\g\d\set.obj

and so it has to build this one obj file every time.
I'm not sure what is going on here set.d is not being changed as 
far as I know.

In any case, unless there is a huge bug in single file 
compilation I don't see it being at all faster than combined 
compile.
Apr 28 2019
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 28/04/2019 13:47, Alex wrote:

 On Sunday, 28 April 2019 at 08:25:18 UTC, Rainer Schuetze wrote:

 On 26/04/2019 09:52, Rainer Schuetze wrote:

 As if when generating the command line for single file compilation
 you forgot to append the additional options string to it.


 Yes, looks like a bug in Visual D: it adds the additional command
 line options to the link, but not to the single file compiler
 invocations.


 fixed in https://github.com/dlang/visuald/releases/tag/v0.49.2


 
 Man, it's really slow compiling!
 
 It took over 3 minutes to compile the same project that compiles in 10
 seconds with combined compile.


Yes, it's usually that bad if you have a good number of dependencies
(imports).


 
 Even after compilation of all the object files it takes 10 seconds just
 to recompile 1 file.
 


If a single file needs 10 sec, there are probably quite a few imports.
You can check the dep-files in the intermediate directory to see
detected dependencies.


 There is absolutely no speed up.
 
 Also, I always get
 
 x64\Debug DMD\S.exe not up to date: d:\repos\s\s\x64\debug dmd\s.exe
 older than d:\repos\s\s\$(outdir)\g\d\set.obj


That sounds like it is only linking again, though "$(outdir)" in the
name looks wrong. Is that really your build directory?


 and so it has to build this one obj file every time.
 I'm not sure what is going on here set.d is not being changed as far as
 I know.
 
 In any case, unless there is a huge bug in single file compilation I
 don't see it being at all faster than combined compile.
 


Combined compilation is preferred in most cases. If you can separate
into non-dependent libraries that could help.
Apr 28 2019
next sibling parent Alex <AJ gmail.com> writes:
On Sunday, 28 April 2019 at 12:22:15 UTC, Rainer Schuetze wrote:

 On 28/04/2019 13:47, Alex wrote:

 On Sunday, 28 April 2019 at 08:25:18 UTC, Rainer Schuetze 
 wrote:

 On 26/04/2019 09:52, Rainer Schuetze wrote:

 As if when generating the command line for single file 
 compilation you forgot to append the additional options 
 string to it.


 Yes, looks like a bug in Visual D: it adds the additional 
 command line options to the link, but not to the single file 
 compiler invocations.


 fixed in https://github.com/dlang/visuald/releases/tag/v0.49.2


 
 Man, it's really slow compiling!
 
 It took over 3 minutes to compile the same project that 
 compiles in 10 seconds with combined compile.


 Yes, it's usually that bad if you have a good number of 
 dependencies
 (imports).


So I removed most of the gtkd import modules I wasn't using and 
it went from 810 to 540. Didn't speed anything up but maybe 1/2 a 
second.

I guess the compiler has no self profiling to figure out where it 
is spending most of it's time?
Apr 28 2019
prev sibling parent Alex <AJ gmail.com> writes:
On Sunday, 28 April 2019 at 12:22:15 UTC, Rainer Schuetze wrote:

 On 28/04/2019 13:47, Alex wrote:

 On Sunday, 28 April 2019 at 08:25:18 UTC, Rainer Schuetze 
 wrote:

 On 26/04/2019 09:52, Rainer Schuetze wrote:

 As if when generating the command line for single file 
 compilation you forgot to append the additional options 
 string to it.


 Yes, looks like a bug in Visual D: it adds the additional 
 command line options to the link, but not to the single file 
 compiler invocations.


 fixed in https://github.com/dlang/visuald/releases/tag/v0.49.2


 
 Man, it's really slow compiling!
 
 It took over 3 minutes to compile the same project that 
 compiles in 10 seconds with combined compile.


 Yes, it's usually that bad if you have a good number of 
 dependencies
 (imports).


 
 Even after compilation of all the object files it takes 10 
 seconds just to recompile 1 file.
 


 If a single file needs 10 sec, there are probably quite a few 
 imports. You can check the dep-files in the intermediate 
 directory to see detected dependencies.


The .dep file has 800+ lines.


 There is absolutely no speed up.
 
 Also, I always get
 
 x64\Debug DMD\S.exe not up to date: d:\repos\s\s\x64\debug 
 dmd\s.exe older than d:\repos\s\s\$(outdir)\g\d\set.obj


 That sounds like it is only linking again, though "$(outdir)" 
 in the name looks wrong. Is that really your build directory?


Yes, the $outdir is there. I assume it is the x64 or win32 
directory... the message is not expanding the token for some 
reason. I thought it was odd too which is why I posted it.

I simply shortened the directory names since the specific names 
are not important and just add unnecessary characters.


 and so it has to build this one obj file every time.
 I'm not sure what is going on here set.d is not being changed 
 as far as
 I know.
 
 In any case, unless there is a huge bug in single file 
 compilation I don't see it being at all faster than combined 
 compile.
 


 Combined compilation is preferred in most cases. If you can 
 separate into non-dependent libraries that could help.


The main problem I see with this is a lot of stuff I do is based 
on templates so I don't think it would help? The compiler will 
still have to access the files in the same way and load them to 
parse?

If you think it will significantly help I would try it. But 
shaving off a few % really isn't my goal.

Most of the dependencies are from phobos and gtkD.

about 600 are from gtk and 150 from phobos.

gtkD already uses a dll.

A lot of the modules I'm loading from gtkd are not needed I 
suppose... Do you know of a util that can determine all the 
actual necessary modules a project needs? I could then remove 
those from gtk that are not used. I add them just so I wouldn't 
have to add them later if I needed them but maybe it is slowing 
things down quite a bit.
Apr 28 2019