www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Visual D: Settings to Improve compil and link process

reply "ParticlePeter" <ParticlePeter gmx.de> writes:
Hello Community,

I thought there's a separate forum for VisualD. It did exist when 
VisualD was on DSource, so why not add it here as well? Or am I 
to blind to see?

Anyway, this thread is an addition to my previous one in this 
forum:
http://forum.dlang.org/thread/wkkuvzkzeupyfdpweals forum.dlang.org

I noticed increasing compile times in my projects. As I use only 
VisualD I would like to understand how I would shorten compile 
time with this tool.
Brief description to my setup, more details can be fond in the 
other post.
I have one Lib, called MyLib consisting of around 15 modules, one 
class per module, zero to two template methods per class with one 
type parameter each.
All my projects use MyLib, and call the template methods with 
different types. Most of the compile time of any of theses 
projects is spent in rebuilding MyLib.
I am not sure why and where so much time is spent, but is there a 
way to profile my COMPILE time with VisualD?

There are several VisualD project properties which I do not 
understand fully, but hope that they might help, namely:

Configuration Properties - General - Files to clean
Configuration Properties - Compiler - Output - Multiple Object 
Files
Configuration Properties - Compiler - Output - Keep Path From 
Source File

Could not cleaning some files improve compilation ?
Could Multiple Object Files be used to separately compile 
non-template and template code blocks ?
What does Keep Path From Source File do at all ?

It is possible to remove the template methods from my classes, 
create free functions instead and use them in a UFCS way.
Unfortunately I have not figured out UFCS properly, as my 
approaches do not work ( guess due to UFCS restrictions ).

Anyway, would it help ( and is it possible at all ) to put the 
template functions in a separate module so that only the 
corresponding object file requires a recompile ?
I am asking in the context of the Multiple Object Files setting.

I tried to not build MyLib at all, and use the sources directly, 
but the project setup is kind of confusing. Again, there is a 
setting which I seem to not understand fully. So one project, 
lets say MyProject, has only its own module files added to the 
VisualD project. I want the compiler to also use the MyLib source 
files, but do not want to add them to MyProject, reasoning 
bellow. There is one entry in the project properties, which 
should make this behavior possible, but it doses not. Why ?

Configuration Properties - Compiler - General - Additional Imports

As far as I understand this setting, I am supposed to enter 
module search paths. But I do get linker errors when I do not add 
either MyLib.lib to the linker settings or all the source files 
of MyLib to MyProject.

Reasoning why I do not want to do this: I have one solution file 
with the MyLib project and ten projects like MyProject using 
MyLib. I would need to add all the source files to all the 
projects, they would be reachable and editable at 11 different 
location within my solution file. This does not not seem to be a 
clean way to set it up.

Any advice to any or all the thoughts and issues ?

Thanks in advance.

Cheers, ParticlePeter
Jul 05 2014
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 05.07.2014 16:05, ParticlePeter wrote:

 Hello Community,

 I thought there's a separate forum for VisualD. It did exist when
 VisualD was on DSource, so why not add it here as well? Or am I to blind
 to see?


The forum digitalmars.D.ide is probably the best fit.


 Anyway, this thread is an addition to my previous one in this forum:
 http://forum.dlang.org/thread/wkkuvzkzeupyfdpweals forum.dlang.org

 I noticed increasing compile times in my projects. As I use only VisualD
 I would like to understand how I would shorten compile time with this tool.
 Brief description to my setup, more details can be fond in the other post.
 I have one Lib, called MyLib consisting of around 15 modules, one class
 per module, zero to two template methods per class with one type
 parameter each.
 All my projects use MyLib, and call the template methods with different
 types. Most of the compile time of any of theses projects is spent in
 rebuilding MyLib.
 I am not sure why and where so much time is spent, but is there a way to
 profile my COMPILE time with VisualD?


The longer compile time is very likely caused by the compiler. You could 
check "Verbose compile" (command line option -v) to see what templates 
are expanded and what functions are compiled. This might give you a hint 
where the compiler spends the time.


 There are several VisualD project properties which I do not understand
 fully, but hope that they might help, namely:

 Configuration Properties - General - Files to clean
 Configuration Properties - Compiler - Output - Multiple Object Files
 Configuration Properties - Compiler - Output - Keep Path From Source File

 Could not cleaning some files improve compilation ?


No.


 Could Multiple Object Files be used to separately compile non-template
 and template code blocks ?


No. This option exposes an undocumented command line option that splits 
a module into multiple object files as if dmd creates a library. Do not 
use it.


 What does Keep Path From Source File do at all ?


This instructs dmd to not strip the path components from module source 
file name when generating the object file name. This avoids troubles if 
you build to multiple object files, but there are modules with identical 
names in different packages.


 It is possible to remove the template methods from my classes, create
 free functions instead and use them in a UFCS way.
 Unfortunately I have not figured out UFCS properly, as my approaches do
 not work ( guess due to UFCS restrictions ).


That would not help, templates are instantiated and compiled when they 
are used, not when they are declared.


 Anyway, would it help ( and is it possible at all ) to put the template
 functions in a separate module so that only the corresponding object
 file requires a recompile ?
 I am asking in the context of the Multiple Object Files setting.


Incremental building of single object files is not well supported by 
dmd, there are too many troubles with instantiating templates in 
different object files.


 I tried to not build MyLib at all, and use the sources directly, but the
 project setup is kind of confusing. Again, there is a setting which I
 seem to not understand fully. So one project, lets say MyProject, has
 only its own module files added to the VisualD project. I want the
 compiler to also use the MyLib source files, but do not want to add them
 to MyProject, reasoning bellow. There is one entry in the project
 properties, which should make this behavior possible, but it doses not.
 Why ?

 Configuration Properties - Compiler - General - Additional Imports

 As far as I understand this setting, I am supposed to enter module
 search paths.


This is correct.


 But I do get linker errors when I do not add either
 MyLib.lib to the linker settings or all the source files of MyLib to
 MyProject.


You should add a dependency in the "Project -> Project Dependencies" 
dialog. This will ensure proper rebuilds and add the library to the 
command line of dependent projects.



 Reasoning why I do not want to do this: I have one solution file with
 the MyLib project and ten projects like MyProject using MyLib. I would
 need to add all the source files to all the projects, they would be
 reachable and editable at 11 different location within my solution file.
 This does not not seem to be a clean way to set it up.

 Any advice to any or all the thoughts and issues ?

 Thanks in advance.

 Cheers, ParticlePeter
Jul 06 2014
parent reply "ParticlePeter" <ParticlePeter gmx.de> writes:
On Sunday, 6 July 2014 at 08:09:07 UTC, Rainer Schuetze wrote:

 On 05.07.2014 16:05, ParticlePeter wrote:


...

 It is possible to remove the template methods from my classes, 
 create
 free functions instead and use them in a UFCS way.
 Unfortunately I have not figured out UFCS properly, as my 
 approaches do
 not work ( guess due to UFCS restrictions ).


 That would not help, templates are instantiated and compiled 
 when they are used, not when they are declared.


Hence the idea of splitting the class. After the split, the class 
has only non-template methods and is part of MyLib. Template UFCS 
functions working with this class are in another module, and not 
part of the lib. With this approach MyLib does not need to 
recompile, only the UFCS functions module needs to recompile due 
to different instantiations. UFCS works now for me btw.



 I tried to not build MyLib at all, and use the sources 
 directly, but the
 project setup is kind of confusing. Again, there is a setting 
 which I
 seem to not understand fully. So one project, lets say 
 MyProject, has
 only its own module files added to the VisualD project. I want 
 the
 compiler to also use the MyLib source files, but do not want 
 to add them
 to MyProject, reasoning bellow. There is one entry in the 
 project
 properties, which should make this behavior possible, but it 
 doses not.
 Why ?

 Configuration Properties - Compiler - General - Additional 
 Imports

 As far as I understand this setting, I am supposed to enter 
 module
 search paths.


 This is correct.


 But I do get linker errors when I do not add either
 MyLib.lib to the linker settings or all the source files of 
 MyLib to
 MyProject.


 You should add a dependency in the "Project -> Project 
 Dependencies" dialog. This will ensure proper rebuilds and add 
 the library to the command line of dependent projects.


The idea is to NOT create a lib, but just use the source code 
from MyLib in MyProject as additional includes. No project 
dependencies. I was hoping that some .obj files are still 
available from last builds, and do not need to recompile, as they 
are up to date ( reason for question about not cleaning files ).
The modules form MyProject do import the MyLib modules properly, 
I do not get compiler errors. However, the compiler should create 
Object files from MyLib modules, and the linker should link them. 
But he does not.
On the other hand, when I add MyLib modules to MyProject ( 
Rightclick MyProject - add - existing item... MyLib source files 
) then linking works. I do not understand why the later step is 
necessary.
Jul 06 2014
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 06.07.2014 19:51, ParticlePeter wrote:

 On Sunday, 6 July 2014 at 08:09:07 UTC, Rainer Schuetze wrote:

 On 05.07.2014 16:05, ParticlePeter wrote:


 ...

 It is possible to remove the template methods from my classes, create
 free functions instead and use them in a UFCS way.
 Unfortunately I have not figured out UFCS properly, as my approaches do
 not work ( guess due to UFCS restrictions ).


 That would not help, templates are instantiated and compiled when they
 are used, not when they are declared.


 Hence the idea of splitting the class. After the split, the class has
 only non-template methods and is part of MyLib. Template UFCS functions
 working with this class are in another module, and not part of the lib.
 With this approach MyLib does not need to recompile, only the UFCS
 functions module needs to recompile due to different instantiations.
 UFCS works now for me btw.


Ok, that allows separate compilation of the class, but templates are 
still compiled with the rest of the program. I thought the templates 
were the part that cause the slow compilation.



 I tried to not build MyLib at all, and use the sources directly, but the
 project setup is kind of confusing. Again, there is a setting which I
 seem to not understand fully. So one project, lets say MyProject, has
 only its own module files added to the VisualD project. I want the
 compiler to also use the MyLib source files, but do not want to add them
 to MyProject, reasoning bellow. There is one entry in the project
 properties, which should make this behavior possible, but it doses not.
 Why ?

 Configuration Properties - Compiler - General - Additional Imports

 As far as I understand this setting, I am supposed to enter module
 search paths.


 This is correct.


 But I do get linker errors when I do not add either
 MyLib.lib to the linker settings or all the source files of MyLib to
 MyProject.


 You should add a dependency in the "Project -> Project Dependencies"
 dialog. This will ensure proper rebuilds and add the library to the
 command line of dependent projects.


 The idea is to NOT create a lib, but just use the source code from MyLib
 in MyProject as additional includes. No project dependencies. I was
 hoping that some .obj files are still available from last builds, and do
 not need to recompile, as they are up to date ( reason for question
 about not cleaning files ).


These object files are in the library ;-) That means manual selection, 
though, as incremental builds to multiple object files don't work with 
dmd, and single file compilation is painfully slow.


 The modules form MyProject do import the MyLib modules properly, I do
 not get compiler errors. However, the compiler should create Object
 files from MyLib modules, and the linker should link them. But he does not.
 On the other hand, when I add MyLib modules to MyProject ( Rightclick
 MyProject - add - existing item... MyLib source files ) then linking
 works. I do not understand why the later step is necessary.


dmd does not compile imported modules, but rdmd does.
Jul 06 2014
parent reply "ParticlePeter" <ParticlePeter gmx.de> writes:
On Sunday, 6 July 2014 at 19:27:38 UTC, Rainer Schuetze wrote:


 Ok, that allows separate compilation of the class, but 
 templates are still compiled with the rest of the program. I 
 thought the templates were the part that cause the slow 
 compilation.


I had no chance to profile till now, but I don't think that the 
templates are slow. They just extract type information from 
primitive or struct arrays, and pass the type info and void array 
on. I have the impression that the whole rebuild-relink process 
of the library itself was taking so long, so I wanted ensure that 
the non-templte code ( the majority ) does not have to be 
compiled over and over again.


 These object files are in the library ;-) That means manual 
 selection, though, as incremental builds to multiple object 
 files don't work with dmd, and single file compilation is 
 painfully slow.


Not sure if I am getting this right, so when one object file has 
to be recompiled all other object files, even if up to date, 
would be recompiled ?


 The modules form MyProject do import the MyLib modules 
 properly, I do
 not get compiler errors. However, the compiler should create 
 Object
 files from MyLib modules, and the linker should link them. But 
 he does not.
 On the other hand, when I add MyLib modules to MyProject ( 
 Rightclick
 MyProject - add - existing item... MyLib source files ) then 
 linking
 works. I do not understand why the later step is necessary.


 dmd does not compile imported modules, but rdmd does.


Ähm ... not seeing the connection here either, why is this 
significant ?

I feel that I could not explain my problem properly, so one 
example:
Importing phobos modules. I do not have to define any import path 
or lib file in the project settings, I just need to import 
std.somthing. That's because the import path for phobos modules 
are stored in the dmd sc.ini file.
When I want to import my modules which are somewhere on my 
hard-drive and not added to my project I need to tell the 
compiler where these modules can be found, using the additional 
import path project setting. That's fine, doing this.

But result is, std.somthing works, my modules in a path known by 
the compiler don't work, giving me linker errors. Why ? ( I do 
not create a lib, I just want to import the module. )
Jul 07 2014
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 07.07.2014 12:46, ParticlePeter wrote:

 On Sunday, 6 July 2014 at 19:27:38 UTC, Rainer Schuetze wrote:


 These object files are in the library ;-) That means manual selection,
 though, as incremental builds to multiple object files don't work with
 dmd, and single file compilation is painfully slow.


 Not sure if I am getting this right, so when one object file has to be
 recompiled all other object files, even if up to date, would be
 recompiled ?


That's how it is currently done if you don't use "single file 
compilation". Compiling only modified and dependent modules in one step 
could work incrementally, but especially template instantiations make 
this hard to do correctly.


 The modules form MyProject do import the MyLib modules properly, I do
 not get compiler errors. However, the compiler should create Object
 files from MyLib modules, and the linker should link them. But he
 does not.
 On the other hand, when I add MyLib modules to MyProject ( Rightclick
 MyProject - add - existing item... MyLib source files ) then linking
 works. I do not understand why the later step is necessary.


 dmd does not compile imported modules, but rdmd does.


 Ähm ... not seeing the connection here either, why is this significant ?


dmd just compiles the files given on the command line. rdmd makes two 
passes, one to collect imported files, and another to compile all the 
collected files. So rdmd works the way you want dmd to work (if I 
understand you correctly).


 I feel that I could not explain my problem properly, so one example:
 Importing phobos modules. I do not have to define any import path or lib
 file in the project settings, I just need to import std.somthing. That's
 because the import path for phobos modules are stored in the dmd sc.ini
 file.
 When I want to import my modules which are somewhere on my hard-drive
 and not added to my project I need to tell the compiler where these
 modules can be found, using the additional import path project setting.
 That's fine, doing this.

 But result is, std.somthing works, my modules in a path known by the
 compiler don't work, giving me linker errors. Why ? ( I do not create a
 lib, I just want to import the module. )


phobos is precompiled to a library and is automatically included in the 
link. If you want your custom modules to work the same way, you have to 
compile them to a library.
Jul 07 2014
parent "ParticlePerter" <ParticlePeter gmx.de> writes:
On Monday, 7 July 2014 at 22:00:51 UTC, Rainer Schuetze wrote:

 On 07.07.2014 12:46, ParticlePeter wrote:

 On Sunday, 6 July 2014 at 19:27:38 UTC, Rainer Schuetze wrote:


 These object files are in the library ;-) That means manual 
 selection,
 though, as incremental builds to multiple object files don't 
 work with
 dmd, and single file compilation is painfully slow.


 Not sure if I am getting this right, so when one object file 
 has to be
 recompiled all other object files, even if up to date, would be
 recompiled ?


 That's how it is currently done if you don't use "single file 
 compilation". Compiling only modified and dependent modules in 
 one step could work incrementally, but especially template 
 instantiations make this hard to do correctly.


 The modules form MyProject do import the MyLib modules 
 properly, I do
 not get compiler errors. However, the compiler should create 
 Object
 files from MyLib modules, and the linker should link them. 
 But he
 does not.
 On the other hand, when I add MyLib modules to MyProject ( 
 Rightclick
 MyProject - add - existing item... MyLib source files ) then 
 linking
 works. I do not understand why the later step is necessary.


 dmd does not compile imported modules, but rdmd does.


 Ähm ... not seeing the connection here either, why is this 
 significant ?


 dmd just compiles the files given on the command line. rdmd 
 makes two passes, one to collect imported files, and another to 
 compile all the collected files. So rdmd works the way you want 
 dmd to work (if I understand you correctly).


 I feel that I could not explain my problem properly, so one 
 example:
 Importing phobos modules. I do not have to define any import 
 path or lib
 file in the project settings, I just need to import 
 std.somthing. That's
 because the import path for phobos modules are stored in the 
 dmd sc.ini
 file.
 When I want to import my modules which are somewhere on my 
 hard-drive
 and not added to my project I need to tell the compiler where 
 these
 modules can be found, using the additional import path project 
 setting.
 That's fine, doing this.

 But result is, std.somthing works, my modules in a path known 
 by the
 compiler don't work, giving me linker errors. Why ? ( I do not 
 create a
 lib, I just want to import the module. )


 phobos is precompiled to a library and is automatically 
 included in the link. If you want your custom modules to work 
 the same way, you have to compile them to a library.


Thanks for claryfying all the above. If the std.library is 
rebuild whenever a template is used, than my assumption that 
MyLib compiles slow due to usage of ( very simple ) templates 
must be wrong. I will dive deeper into profiling. Thanks a lot.

Cheers, ParticlePeter
Jul 09 2014