www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - dmd's linking order

reply Jonathan M Davis <jmdavisProg gmx.com> writes:
This stackoverflow question raises an interesting issue with linking order and 
dmd:

http://stackoverflow.com/questions/10095150/std-net-curl-linker-errors-in-linux

I don't know if it's this way on Linux machines in general, but the OP had to 
link his program manually to be able to use std.curl. dmd appears to put the -
L argument before any of its own linker arguments, and in this case, the 
linking argument for curl needs to go on the end.

So, my question is whether dmd should be changed to put any -L arguments 
passed to it after the arguments that it passes to the linker itself. I'm far 
from an expert on this and am quite surprised that the order of arguments to 
the linker matters, but since it does, it seems to me that we should find the 
optimal order for dmd to use if we can, since it's not terribly user friendly 
to force people to call gcc or ld directly rather than using dmd to link just 
because they want to pass an argument to the linker. That's what the -L flag is 
supposed to be for.

- Jonathan M Davis
Apr 10 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-04-11 03:46, Jonathan M Davis wrote:

 This stackoverflow question raises an interesting issue with linking order and
 dmd:

 http://stackoverflow.com/questions/10095150/std-net-curl-linker-errors-in-linux

 I don't know if it's this way on Linux machines in general, but the OP had to
 link his program manually to be able to use std.curl. dmd appears to put the -
 L argument before any of its own linker arguments, and in this case, the
 linking argument for curl needs to go on the end.

 So, my question is whether dmd should be changed to put any -L arguments
 passed to it after the arguments that it passes to the linker itself. I'm far
 from an expert on this and am quite surprised that the order of arguments to
 the linker matters, but since it does, it seems to me that we should find the
 optimal order for dmd to use if we can, since it's not terribly user friendly
 to force people to call gcc or ld directly rather than using dmd to link just
 because they want to pass an argument to the linker. That's what the -L flag is
 supposed to be for.

 - Jonathan M Davis


Could it happen that the linker arguments need to be placed first 
sometimes ?

-- 
/Jacob Carlborg
Apr 11 2012
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, April 11, 2012 10:25:37 Jacob Carlborg wrote:

 On 2012-04-11 03:46, Jonathan M Davis wrote:

 This stackoverflow question raises an interesting issue with linking order
 and dmd:
 
 http://stackoverflow.com/questions/10095150/std-net-curl-linker-errors-in-
 linux
 
 I don't know if it's this way on Linux machines in general, but the OP had
 to link his program manually to be able to use std.curl. dmd appears to
 put the - L argument before any of its own linker arguments, and in this
 case, the linking argument for curl needs to go on the end.
 
 So, my question is whether dmd should be changed to put any -L arguments
 passed to it after the arguments that it passes to the linker itself. I'm
 far from an expert on this and am quite surprised that the order of
 arguments to the linker matters, but since it does, it seems to me that
 we should find the optimal order for dmd to use if we can, since it's not
 terribly user friendly to force people to call gcc or ld directly rather
 than using dmd to link just because they want to pass an argument to the
 linker. That's what the -L flag is supposed to be for.
 
 - Jonathan M Davis


 
 Could it happen that the linker arguments need to be placed first
 sometimes ?


I have no idea. If they always need to be in a particular order, then we can 
just make dmd put the user provide ones in the correct spot, but if it varies 
depending on the flags, then either dmd is going to have to know which flags go 
in which order (assuming that it can), or there's no way to solve the problem 
without introducing new flags to dmd to give you more control over linking 
order or just using the linker directly.

Unfortunately, I have no idea why the linking order even matters in the first 
place, so I can't really say what we need to do here. Hopefully, someone else 
around here _does_ know. But the issue does seem to need to be brought up.

- Jonathan M Davis
Apr 11 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-04-11 10:37, Jonathan M Davis wrote:


 I have no idea. If they always need to be in a particular order, then we can
 just make dmd put the user provide ones in the correct spot, but if it varies
 depending on the flags, then either dmd is going to have to know which flags go
 in which order (assuming that it can), or there's no way to solve the problem
 without introducing new flags to dmd to give you more control over linking
 order or just using the linker directly.

 Unfortunately, I have no idea why the linking order even matters in the first
 place, so I can't really say what we need to do here. Hopefully, someone else
 around here _does_ know. But the issue does seem to need to be brought up.


I also finds it strange that linking order should matter.

-- 
/Jacob Carlborg
Apr 11 2012
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 11 Apr 2012 04:47:19 -0400, Jacob Carlborg <doob me.com> wrote:


 On 2012-04-11 10:37, Jonathan M Davis wrote:


 I have no idea. If they always need to be in a particular order, then  
 we can
 just make dmd put the user provide ones in the correct spot, but if it  
 varies
 depending on the flags, then either dmd is going to have to know which  
 flags go
 in which order (assuming that it can), or there's no way to solve the  
 problem
 without introducing new flags to dmd to give you more control over  
 linking
 order or just using the linker directly.

 Unfortunately, I have no idea why the linking order even matters in the  
 first
 place, so I can't really say what we need to do here. Hopefully,  
 someone else
 around here _does_ know. But the issue does seem to need to be brought  
 up.


 I also finds it strange that linking order should matter.


It was a recent change in ld, it happened to me when I upgraded to ubutu  
11.10.

See this bug:

http://d.puremagic.com/issues/show_bug.cgi?id=6822

I remember this behavior from old systems I used to work on (really old,  
like SunOS 4).

I'm not a linker expert, but I found this page which describes the changes  
and the reasoning:

See this post: https://wiki.ubuntu.com/NattyNarwhal/ToolchainTransition

My suggestion is to allow specifying in the dmd.conf file the specific  
location of phobos/druntime in the link order, then leave the -L options  
at the front.

-Steve
Apr 11 2012
parent reply "Nick Sabalausky" <SeeWebsiteToContactMe semitwist.com> writes:
"Steven Schveighoffer" <schveiguy yahoo.com> wrote in message 
news:op.wcln1rh9eav7ka localhost.localdomain...

 I'm not a linker expert, but I found this page which describes the changes 
 and the reasoning:

 See this post: https://wiki.ubuntu.com/NattyNarwhal/ToolchainTransition


That says the order-dependent  behavior is caused by --as-needed being 
default. I'm no linker expert either, but the description of the purpose 
of --as-needed sounds...goofy and pointless (although maybe it makes sense 
for C's lack of a proper module system?).

It can be disabled, apperently, by using --no-as-needed. That page tries to 
discourage people from doing that though, but the reason it gives seems 
vague. Personally, I'd wonder whether going along with --as-needed is really 
even worth doing. (Of course, if it turns out to be easy to fix the 
ordering, then we may as well.)
Apr 11 2012
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 11 Apr 2012 14:51:14 -0400
schrieb "Nick Sabalausky" <SeeWebsiteToContactMe semitwist.com>:


 "Steven Schveighoffer" <schveiguy yahoo.com> wrote in message 
 news:op.wcln1rh9eav7ka localhost.localdomain...

 I'm not a linker expert, but I found this page which describes the changes 
 and the reasoning:

 See this post: https://wiki.ubuntu.com/NattyNarwhal/ToolchainTransition


 
 That says the order-dependent  behavior is caused by --as-needed being 
 default. I'm no linker expert either, but the description of the purpose 
 of --as-needed sounds...goofy and pointless (although maybe it makes sense 
 for C's lack of a proper module system?).
 
 It can be disabled, apperently, by using --no-as-needed. That page tries to 
 discourage people from doing that though, but the reason it gives seems 
 vague. Personally, I'd wonder whether going along with --as-needed is really 
 even worth doing. (Of course, if it turns out to be easy to fix the 
 ordering, then we may as well.)


I didn't know that --as-needed causes so much trouble. We've been using it on
Gentoo a while before Ubuntu made that step and I know it took a while (~a
year) until all packages would compile with it.
The problem that this solved was that libraries and applications are linked to
other libraries they don't actually call into. On a long stream of dependencies
an application would dynamically load way more libraries than it actually uses,
increasing the start time. So the linker lacked the intelligence to filter out
unnecessary libraries before.
Since there are some cases, where you must link to a library anyway, you can
put --no-as-needed in front of them. Look at the positive side: You can now by
default link against all libraries on your system without bloating the
executable! :)

-- 
Marco
Apr 11 2012
parent "Nick Sabalausky" <SeeWebsiteToContactMe semitwist.com> writes:
"Marco Leise" <Marco.Leise gmx.de> wrote in message 
news:20120412074942.0c6bb2cd marco-leise...

 Am Wed, 11 Apr 2012 14:51:14 -0400
 schrieb "Nick Sabalausky" <SeeWebsiteToContactMe semitwist.com>:


 "Steven Schveighoffer" <schveiguy yahoo.com> wrote in message
 news:op.wcln1rh9eav7ka localhost.localdomain...

 I'm not a linker expert, but I found this page which describes the 
 changes
 and the reasoning:

 See this post: https://wiki.ubuntu.com/NattyNarwhal/ToolchainTransition


 That says the order-dependent  behavior is caused by --as-needed being
 default. I'm no linker expert either, but the description of the purpose
 of --as-needed sounds...goofy and pointless (although maybe it makes 
 sense
 for C's lack of a proper module system?).

 It can be disabled, apperently, by using --no-as-needed. That page tries 
 to
 discourage people from doing that though, but the reason it gives seems
 vague. Personally, I'd wonder whether going along with --as-needed is 
 really
 even worth doing. (Of course, if it turns out to be easy to fix the
 ordering, then we may as well.)


 I didn't know that --as-needed causes so much trouble. We've been using it 
 on Gentoo a while before Ubuntu made that step and I know it took a while 
 (~a year) until all packages would compile with it.
 The problem that this solved was that libraries and applications are 
 linked to other libraries they don't actually call into. On a long stream 
 of dependencies an application would dynamically load way more libraries 
 than it actually uses, increasing the start time. So the linker lacked the 
 intelligence to filter out unnecessary libraries before.
 Since there are some cases, where you must link to a library anyway, you 
 can put --no-as-needed in front of them. Look at the positive side: You 
 can now by default link against all libraries on your system without 
 bloating the executable! :)


I was under the impression ld already stripped out unused symbols anyway?
Apr 11 2012
prev sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
 Unfortunately, I have no idea why the linking order even matters in the  
 first
 place, so I can't really say what we need to do here. Hopefully, someone  
 else
 around here _does_ know. But the issue does seem to need to be brought  
 up.


The linker will only use succeeding archives to resolve undefined symbols,
i.e. phobos2 needs to precede curl. It seems to work when linking against
shared libraries though.
Apr 11 2012
prev sibling next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
On Apr 11, 2012, at 1:25 AM, Jacob Carlborg <doob me.com> wrote:


 Could it happen that the linker arguments need to be placed first sometime=


s ?

If it's a user-created library then maybe. The general rule on Unix is that d=
ependent objects need to be listed before the object they depend on.  I thin=
k the linker only does a single pass. Optlink doesn't have this problem--it'=
s way nicer in this regard.=20=
Apr 11 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-04-11 16:59, Sean Kelly wrote:

 On Apr 11, 2012, at 1:25 AM, Jacob Carlborg<doob me.com>  wrote:


 Could it happen that the linker arguments need to be placed first sometimes ?


 If it's a user-created library then maybe. The general rule on Unix is that
dependent objects need to be listed before the object they depend on.  I think
the linker only does a single pass. Optlink doesn't have this problem--it's way
nicer in this regard.


That would be the only case where optlink is nicer :)

-- 
/Jacob Carlborg
Apr 11 2012
parent Artur Skawina <art.08.09 gmail.com> writes:
On 04/11/12 20:30, Jacob Carlborg wrote:

 On 2012-04-11 16:59, Sean Kelly wrote:

 On Apr 11, 2012, at 1:25 AM, Jacob Carlborg<doob me.com>  wrote:


 Could it happen that the linker arguments need to be placed first sometimes ?


 If it's a user-created library then maybe. The general rule on Unix is that
dependent objects need to be listed before the object they depend on.  I think
the linker only does a single pass. Optlink doesn't have this problem--it's way
nicer in this regard.


 
 That would be the only case where optlink is nicer :)
 


Umm, "-( -llib1 -llib2 -)". 
But using the correct order would be the right solution.

artur
Apr 11 2012
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Apr 11, 2012 at 01:37:48AM -0700, Jonathan M Davis wrote:
[...]

 Unfortunately, I have no idea why the linking order even matters in
 the first place, so I can't really say what we need to do here.
 Hopefully, someone else around here _does_ know. But the issue does
 seem to need to be brought up.


[...]

This is because many linkers (including *nix ld, IIRC) resolve symbols
in the order the libraries were specified on the command-line. I don't
remember the exact reason for this, but it probably has to do with
improving the performance of the symbol resolution algorithm. It's a
legacy from the early days of linker technology.


T

-- 
VI = Visual Irritation
Apr 11 2012
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 11 Apr 2012 13:04:59 -0400, H. S. Teoh <hsteoh quickfur.ath.cx>  
wrote:


 On Wed, Apr 11, 2012 at 01:37:48AM -0700, Jonathan M Davis wrote:
 [...]

 Unfortunately, I have no idea why the linking order even matters in
 the first place, so I can't really say what we need to do here.
 Hopefully, someone else around here _does_ know. But the issue does
 seem to need to be brought up.


 [...]

 This is because many linkers (including *nix ld, IIRC) resolve symbols
 in the order the libraries were specified on the command-line. I don't
 remember the exact reason for this, but it probably has to do with
 improving the performance of the symbol resolution algorithm. It's a
 legacy from the early days of linker technology.


Not exactly.  For example, Ubuntu 10 was perfectly happy accepting  
libraries in any order.  Only with Ubuntu 11 did this "revert" to the old  
way.

I'm not sure that the reasoning was that it's "simpler", because clearly  
it's possible (and implemented!)

Look for my other post for Ubuntu's explanation why, I don't really  
understand it.

-Steve
Apr 11 2012
parent reply "eles" <eles eles.com> writes:
 Not exactly.  For example, Ubuntu 10 was perfectly happy 
 accepting libraries in any order.  Only with Ubuntu 11 did this 
 "revert" to the old way.


Actually, it looked like that. What happenned behind the doors it 
was that the linking proceeded with --as-needed and the expansion 
was on the spot.

Assume that you had libA depending on libB (but also on some 
other library that you do not use, namely libC).

if you tried to link:

gcc my_program.o -lB -lA (so, in reverse order)

this looked a bit like:

gcc my_program.o -lA -lB

BUT what really happened was that the true link command was now:

gcc my_program.o -lA -lB -lC -lB

where the "-lA -lB -lC" is the expansion of the previous "-lA".
Apr 11 2012
parent reply "eles" <eles eles.com> writes:
 if you tried to link:

 gcc my_program.o -lB -lA (so, in reverse order)


read:

gcc my_program.o -lB -lA (so, in unnatural order)


 this looked a bit like:

 gcc my_program.o -lA -lB


read:

  "it looked like the order is arbitrary"


 BUT what really happened was that the true link command was now:

 gcc my_program.o -lA -lB -lC -lB


read:

gcc my_program.o -lB -lA -lB -lC (so that the symbols of libA 
*were* resolved)


 where the "-lA -lB -lC" is the expansion of the previous "-lA".
Apr 11 2012
parent reply Mike Wey <mike-wey example.com> writes:
On 04/11/2012 09:54 PM, eles wrote:

 if you tried to link:

 gcc my_program.o -lB -lA (so, in reverse order)


 read:

 gcc my_program.o -lB -lA (so, in unnatural order)


 this looked a bit like:

 gcc my_program.o -lA -lB


 read:

 "it looked like the order is arbitrary"


 BUT what really happened was that the true link command was now:

 gcc my_program.o -lA -lB -lC -lB


 read:

 gcc my_program.o -lB -lA -lB -lC (so that the symbols of libA *were*
 resolved)


 where the "-lA -lB -lC" is the expansion of the previous "-lA".




And to tell ld to not depend on the order in which the libs are 
specifies, you can use --start-group and --end-group.

 From the ld man page:

--start-group archives --end-group
     The archives should be a list of archive files.  They may be either
     explicit file names, or -l options.

     The specified archives are searched repeatedly until no new
     undefined references are created.  Normally, an archive is searched
     only once in the order that it is specified on the command line.
     If a symbol in that archive is needed to resolve an undefined
     symbol referred to by an object in an archive that appears later on
     the command line, the linker would not be able to resolve that
     reference.  By grouping the archives, they all be searched
     repeatedly until all possible references are resolved.

     Using this option has a significant performance cost.  It is best
     to use it only when there are unavoidable circular references
     between two or more archives.

-- 
Mike Wey
Apr 11 2012
parent "eles" <eles eles.com> writes:
 And to tell ld to not depend on the order in which the libs are 
 specifies, you can use --start-group and --end-group.

 From the ld man page:

 --start-group archives --end-group
     The archives should be a list of archive files.  They may 
 be either
     explicit file names, or -l options.

     The specified archives are searched repeatedly until no new
     undefined references are created.


Thanks. I knew that possibility exists, I just did not remember 
the parameters.

Thank you for documenting this here.
Apr 12 2012