www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Linking large D programms

reply Torsten Sommerfeld <Globe13.Trotter17 gmx.de> writes:
Hello,

I have a problem creating executables from more than 100 d modules (under
linux). DMD compiles even 300 files in seconds but the linking step takes
several minutes. It is clear that the linker has do handle an enormous amount
of symbols so it needs the time.
However it is hardly acceptable on the long run.

Are there some alternative linkers to replace ld? Is there an incremental
linker which can be used?

If I cannot find a solution for this problem I fear I have to convert my
project to Java (my D code looks almost like Java code) wher this "limitation"
is not there.
I do not like Java much and converting is a painful lot of work but as it is I
cannot continue with D.

Does anyone have any ideas?

- Torsten
May 01 2010
next sibling parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 01/05/10 09:48, Torsten Sommerfeld wrote:

 Hello,

 I have a problem creating executables from more than 100 d modules (under
 linux). DMD compiles even 300 files in seconds but the linking step takes
 several minutes. It is clear that the linker has do handle an enormous amount
 of symbols so it needs the time.
 However it is hardly acceptable on the long run.

 Are there some alternative linkers to replace ld? Is there an incremental
 linker which can be used?

 If I cannot find a solution for this problem I fear I have to convert my
 project to Java (my D code looks almost like Java code) wher this "limitation"
 is not there.
 I do not like Java much and converting is a painful lot of work but as it is I
 cannot continue with D.

 Does anyone have any ideas?

 - Torsten


I hear GOLD is destined to replace ld, and is supposedly far faster. 
Your distro might have a repository for it, if not you'll have to 
compile it from source. It might also be worth profiling (go)ld while 
it's linking, and seeing where it's spending the most time. If it's 
spending a lot of time reading object files then the solution would be 
to combine some object files into one before invoking the linker, eg 
using dmd's -lib switch or passing all the files at once and using -of 
foo.o so only one object file is produced.
May 01 2010
parent Walter Bright <newshound1 digitalmars.com> writes:
Robert Clipsham wrote:

 I hear GOLD is destined to replace ld, and is supposedly far faster. 
 Your distro might have a repository for it, if not you'll have to 
 compile it from source. It might also be worth profiling (go)ld while 
 it's linking, and seeing where it's spending the most time. If it's 
 spending a lot of time reading object files then the solution would be 
 to combine some object files into one before invoking the linker, eg 
 using dmd's -lib switch or passing all the files at once and using -of 
 foo.o so only one object file is produced.


Robert's right, passing as many as you can on one command line with -offoo.o 
will result in one .o file, this should greatly speed linking.
May 01 2010
prev sibling parent reply =?UTF-8?B?IkrDqXLDtG1lIE0uIEJlcmdlciI=?= <jeberger free.fr> writes:
Torsten Sommerfeld wrote:

 Hello,
=20
 I have a problem creating executables from more than 100 d modules (und=


er

 linux). DMD compiles even 300 files in seconds but the linking step tak=


es

 several minutes. It is clear that the linker has do handle an enormous =


amount

 of symbols so it needs the time.
 However it is hardly acceptable on the long run.
=20
 Are there some alternative linkers to replace ld? Is there an increment=


al

 linker which can be used?
=20
 If I cannot find a solution for this problem I fear I have to convert m=


y

 project to Java (my D code looks almost like Java code) wher this "limi=


tation"

 is not there.
 I do not like Java much and converting is a painful lot of work but as =


it is I

 cannot continue with D.
=20
 Does anyone have any ideas?
=20


	You might want to look at the "-r" option for ld. I allows to do
partial linking by combining several object files into an object
file, resolving everything that it can resolve. So if you have:

package1/module1
        /module2
package2/module1
        /module2

	You can do:
ld -o package1.o -r package1/module1.o package1/module2.o
ld -o package2.o -r package2/module1.o package2/module2.o
ld -o program package1.o package2.o

	Of course, the first time you do it, it will still be long, but if
you only modify something in package2, then you won't need to relink
package1 so it should speed up the development cycle.

		Jerome
--=20
mailto:jeberger free.fr
http://jeberger.free.fr
Jabber: jeberger jabber.fr
May 01 2010
parent Torsten Sommerfeld <Globe13.Trotter17 gmx.de> writes:
Thank you for your answers.
I have really managed to improve the linking speed:

First I have tried to use the GOLD linker. Because I am using ubuntu, the
package
was available in the sources. But now, because I do not know the binutils and I
could not find a useful documentation I do not know if I am using it now or
not. I
guess yes, because ld-- version gives "GNU gold (GNU Binutils for Ubuntu 2.20)
1.9"
Nevertheless it did not help to reduce linking time.

You have mentioned the -of command flag. For dmd there is no documentation about
it in the help so I could not make it work.

However during my try and error to use it I figured out that my guess how dmd
links was not correct.

It seems, that if the -c option is used (no linking) then for each .d file an .o
file is created. However if the -c flag is not giving, only one .o file is
created.

I got to recognize that the output speed (-v flag) became extremely slow when -c
was not used I wondered how this can be, because linking was not supposed to be
done at that particular time anyway.

After I have managed to separate compiling and linking (calling dmd with -c and
then calling gcc with all object files and libraries) I could see that linking
250
Object files took 1 second! (Compared to 20 min if dmd does the job).

So my guess is, that creating the one object file (not using -c) during
compiling
is very inefficient - and in consequence I have unfairly blamed the the linker
for it.

Torsten
May 01 2010