www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - duplicate symbol linker errors, my fault or D's?

reply "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
I'm not sure if the linker errors I'm getting are my fault or D's.

I'm trying to do incremental compilation since the size of my
program is starting to break my computer, slowing compilations
down by four or five times.

Suppose I have two libraries:
lib1.a
lib2.a

I build lib1 with a bunch of files. Then I build lib2 by linking
with lib1:

dmd -lib -oflib2.a ... file1.d file2.d lib1.a

Now file1 and file2 actually import a lot of the files built into
lib1, but neither is in lib1.

When I try to compile file3.d:

dmd file3.d -L-llib2
ld: duplicate symbol
_D53/Users/zach/source/dmd2/src/phobos/std/array.d.237__arrayZ in
./lib/lib2.a(array_17_e15.o) and ./lib/lib2.a(array_17_e82.o)
collect2: ld returned 1 exit status

I fear I'm not going to be able to use incremental compilation
because D is duplicating some symbols I can't track because I'm
not declaring them myself.

There's a lot of stuff I don't know about programming. Therefore
I hope I'm making a beginner's blunder instead.

What's your opinion?

Zach
Mar 05 2012
next sibling parent reply "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
Reading the documentation about compiler options and flags here:
http://dlang.org/dmd-osx.html

led me to believe that building libraries was the right way to do 
incremental compilation. But I thought, well, can I just build 
file1.o object files instead?

So I've started doing that and I've got no problems so far. I 
guess I don't need all that fancy .lib and -L-lmyLib stuff after 
all. You can pack all sorts of stuff into a file1.o file instead.

I guess I solved my problem for now. Hopefully I'm making sense. 
:-)
Mar 05 2012
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Tuesday, March 06, 2012 01:53:02 Zach the Mystic wrote:

 Reading the documentation about compiler options and flags here:
 http://dlang.org/dmd-osx.html
 
 led me to believe that building libraries was the right way to do
 incremental compilation. But I thought, well, can I just build
 file1.o object files instead?
 
 So I've started doing that and I've got no problems so far. I
 guess I don't need all that fancy .lib and -L-lmyLib stuff after
 all. You can pack all sorts of stuff into a file1.o file instead.
 
 I guess I solved my problem for now. Hopefully I'm making sense.


Libraries are not intented for incremental compilation. They are for 
distributing code in a unit which can be used by programs. And in the case of 
a shared library, it gives the added benefit of reducing the amount of 
duplicate code you get in binaries (saving both memory and disk space).

If you want to do incremental compilation, then use -c to generate object files 
that you link together when you create the actual executable.

- Jonathan M Davis
Mar 05 2012
next sibling parent reply "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
 Libraries are not intented for incremental compilation. They 
 are for
 distributing code in a unit which can be used by programs. And 
 in the case of
 a shared library, it gives the added benefit of reducing the 
 amount of
 duplicate code you get in binaries (saving both memory and disk 
 space).

 If you want to do incremental compilation, then use -c to 
 generate object files
 that you link together when you create the actual executable.

 - Jonathan M Davis


Thanks for taking the time to answer. I simply didn't realize 
that you could pack like 60 modules in to one ".o" object file. 
But I do now!
Mar 05 2012
parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, March 06, 2012 05:26:34 Zach the Mystic wrote:

 Libraries are not intented for incremental compilation. They
 are for
 distributing code in a unit which can be used by programs. And
 in the case of
 a shared library, it gives the added benefit of reducing the
 amount of
 duplicate code you get in binaries (saving both memory and disk
 space).
 
 If you want to do incremental compilation, then use -c to
 generate object files
 that you link together when you create the actual executable.
 
 - Jonathan M Davis


 
 Thanks for taking the time to answer. I simply didn't realize
 that you could pack like 60 modules in to one ".o" object file.
 But I do now!


You could put your entire program in a .o file if you wanted to, though I don't 
know why you would, since if you're generating only one object file, you might 
as well generate an executable and save yourself the extra linking step. I 
don't believe that there's any limit to what you can put in an object file 
beyond what you can put in an executable.

http://en.wikipedia.org/wiki/Object_file

- Jonathan M Davis
Mar 05 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-03-06 02:21, Jonathan M Davis wrote:

 On Tuesday, March 06, 2012 01:53:02 Zach the Mystic wrote:

 Reading the documentation about compiler options and flags here:
 http://dlang.org/dmd-osx.html

 led me to believe that building libraries was the right way to do
 incremental compilation. But I thought, well, can I just build
 file1.o object files instead?

 So I've started doing that and I've got no problems so far. I
 guess I don't need all that fancy .lib and -L-lmyLib stuff after
 all. You can pack all sorts of stuff into a file1.o file instead.

 I guess I solved my problem for now. Hopefully I'm making sense.


 Libraries are not intented for incremental compilation. They are for
 distributing code in a unit which can be used by programs. And in the case of
 a shared library, it gives the added benefit of reducing the amount of
 duplicate code you get in binaries (saving both memory and disk space).

 If you want to do incremental compilation, then use -c to generate object files
 that you link together when you create the actual executable.

 - Jonathan M Davis


Actually the -lib switch might be the answer to one of the incremental 
compilation problems DMD suffers from. That DMD usually does not output 
all symbols to all object files which can result in missing symbols when 
doing incremental compilation. I've heard that the when using the -lib 
flag DMD will output all symbols to all object files.

-- 
/Jacob Carlborg
Mar 05 2012
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, March 06, 2012 08:29:53 Jacob Carlborg wrote:

 On 2012-03-06 02:21, Jonathan M Davis wrote:

 On Tuesday, March 06, 2012 01:53:02 Zach the Mystic wrote:

 Reading the documentation about compiler options and flags here:
 http://dlang.org/dmd-osx.html
 
 led me to believe that building libraries was the right way to do
 incremental compilation. But I thought, well, can I just build
 file1.o object files instead?
 
 So I've started doing that and I've got no problems so far. I
 guess I don't need all that fancy .lib and -L-lmyLib stuff after
 all. You can pack all sorts of stuff into a file1.o file instead.
 
 I guess I solved my problem for now. Hopefully I'm making sense.


 
 Libraries are not intented for incremental compilation. They are for
 distributing code in a unit which can be used by programs. And in the case
 of a shared library, it gives the added benefit of reducing the amount of
 duplicate code you get in binaries (saving both memory and disk space).
 
 If you want to do incremental compilation, then use -c to generate object
 files that you link together when you create the actual executable.
 
 - Jonathan M Davis


 
 Actually the -lib switch might be the answer to one of the incremental
 compilation problems DMD suffers from. That DMD usually does not output
 all symbols to all object files which can result in missing symbols when
 doing incremental compilation. I've heard that the when using the -lib
 flag DMD will output all symbols to all object files.


If so, it's an implementation issue. In principle, libraries have nothing to 
do with incremental compilation. They're about modularizing code for 
reusability.

- Jonathan M Davis
Mar 05 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-03-06 08:56, Jonathan M Davis wrote:

 On Tuesday, March 06, 2012 08:29:53 Jacob Carlborg wrote:

 On 2012-03-06 02:21, Jonathan M Davis wrote:

 On Tuesday, March 06, 2012 01:53:02 Zach the Mystic wrote:

 Reading the documentation about compiler options and flags here:
 http://dlang.org/dmd-osx.html

 led me to believe that building libraries was the right way to do
 incremental compilation. But I thought, well, can I just build
 file1.o object files instead?

 So I've started doing that and I've got no problems so far. I
 guess I don't need all that fancy .lib and -L-lmyLib stuff after
 all. You can pack all sorts of stuff into a file1.o file instead.

 I guess I solved my problem for now. Hopefully I'm making sense.


 Libraries are not intented for incremental compilation. They are for
 distributing code in a unit which can be used by programs. And in the case
 of a shared library, it gives the added benefit of reducing the amount of
 duplicate code you get in binaries (saving both memory and disk space).

 If you want to do incremental compilation, then use -c to generate object
 files that you link together when you create the actual executable.

 - Jonathan M Davis


 Actually the -lib switch might be the answer to one of the incremental
 compilation problems DMD suffers from. That DMD usually does not output
 all symbols to all object files which can result in missing symbols when
 doing incremental compilation. I've heard that the when using the -lib
 flag DMD will output all symbols to all object files.


 If so, it's an implementation issue. In principle, libraries have nothing to
 do with incremental compilation. They're about modularizing code for
 reusability.

 - Jonathan M Davis


Yes, but Walter doesn't seem to want to fix that issue.

-- 
/Jacob Carlborg
Mar 06 2012
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Tuesday, March 06, 2012 11:31:40 Jacob Carlborg wrote:

 On 2012-03-06 08:56, Jonathan M Davis wrote:

 On Tuesday, March 06, 2012 08:29:53 Jacob Carlborg wrote:

 On 2012-03-06 02:21, Jonathan M Davis wrote:

 On Tuesday, March 06, 2012 01:53:02 Zach the Mystic wrote:

 Reading the documentation about compiler options and flags here:
 http://dlang.org/dmd-osx.html
 
 led me to believe that building libraries was the right way to do
 incremental compilation. But I thought, well, can I just build
 file1.o object files instead?
 
 So I've started doing that and I've got no problems so far. I
 guess I don't need all that fancy .lib and -L-lmyLib stuff after
 all. You can pack all sorts of stuff into a file1.o file instead.
 
 I guess I solved my problem for now. Hopefully I'm making sense.


 
 Libraries are not intented for incremental compilation. They are for
 distributing code in a unit which can be used by programs. And in the
 case
 of a shared library, it gives the added benefit of reducing the amount
 of
 duplicate code you get in binaries (saving both memory and disk space).
 
 If you want to do incremental compilation, then use -c to generate
 object
 files that you link together when you create the actual executable.
 
 - Jonathan M Davis


 
 Actually the -lib switch might be the answer to one of the incremental
 compilation problems DMD suffers from. That DMD usually does not output
 all symbols to all object files which can result in missing symbols when
 doing incremental compilation. I've heard that the when using the -lib
 flag DMD will output all symbols to all object files.


 
 If so, it's an implementation issue. In principle, libraries have nothing
 to do with incremental compilation. They're about modularizing code for
 reusability.
 
 - Jonathan M Davis


 
 Yes, but Walter doesn't seem to want to fix that issue.


Well, from the sounds of it, he's going to need to be convinced one way or 
another - which can be frustratingly difficult at times. Sometimes, he's right 
even when pretty much no one agrees with him, but other times, he just doesn't 
seem to get it. He's hard to convince in either case though.

- Jonathan M Davis
Mar 06 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-03-07 01:47, Jonathan M Davis wrote:

 On Tuesday, March 06, 2012 11:31:40 Jacob Carlborg wrote:

 On 2012-03-06 08:56, Jonathan M Davis wrote:

 On Tuesday, March 06, 2012 08:29:53 Jacob Carlborg wrote:

 On 2012-03-06 02:21, Jonathan M Davis wrote:

 On Tuesday, March 06, 2012 01:53:02 Zach the Mystic wrote:

 Reading the documentation about compiler options and flags here:
 http://dlang.org/dmd-osx.html

 led me to believe that building libraries was the right way to do
 incremental compilation. But I thought, well, can I just build
 file1.o object files instead?

 So I've started doing that and I've got no problems so far. I
 guess I don't need all that fancy .lib and -L-lmyLib stuff after
 all. You can pack all sorts of stuff into a file1.o file instead.

 I guess I solved my problem for now. Hopefully I'm making sense.


 Libraries are not intented for incremental compilation. They are for
 distributing code in a unit which can be used by programs. And in the
 case
 of a shared library, it gives the added benefit of reducing the amount
 of
 duplicate code you get in binaries (saving both memory and disk space).

 If you want to do incremental compilation, then use -c to generate
 object
 files that you link together when you create the actual executable.

 - Jonathan M Davis


 Actually the -lib switch might be the answer to one of the incremental
 compilation problems DMD suffers from. That DMD usually does not output
 all symbols to all object files which can result in missing symbols when
 doing incremental compilation. I've heard that the when using the -lib
 flag DMD will output all symbols to all object files.


 If so, it's an implementation issue. In principle, libraries have nothing
 to do with incremental compilation. They're about modularizing code for
 reusability.

 - Jonathan M Davis


 Yes, but Walter doesn't seem to want to fix that issue.


 Well, from the sounds of it, he's going to need to be convinced one way or
 another - which can be frustratingly difficult at times. Sometimes, he's right
 even when pretty much no one agrees with him, but other times, he just doesn't
 seem to get it. He's hard to convince in either case though.

 - Jonathan M Davis


If I recall correctly, he mentioned some issues with COMDAT. Here is the 
full discussion:

http://www.digitalmars.com/d/archives/digitalmars/D/Incremental_compilation_with_DMD_96138.html

Mentions something about COMDAT: 
http://www.digitalmars.com/d/archives/digitalmars/D/Incremental_compilation_with_DMD_96138.html#N96197

-- 
/Jacob Carlborg
Mar 06 2012
prev sibling parent "Zach the Mystic" <reachMINUSTHISzachgmail dot.com> writes:
On Tuesday, 6 March 2012 at 07:29:54 UTC, Jacob Carlborg wrote:

 Actually the -lib switch might be the answer to one of the 
 incremental compilation problems DMD suffers from. That DMD 
 usually does not output all symbols to all object files which 
 can result in missing symbols when doing incremental 
 compilation. I've heard that the when using the -lib flag DMD 
 will output all symbols to all object files.


Hey you know, this could also be a problem because of my original 
post. Instead of missing symbols, with -lib, you could start to 
suffer from unintended duplicates, too.
Mar 07 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-03-06 01:04, Zach the Mystic wrote:

 I'm not sure if the linker errors I'm getting are my fault or D's.

 I'm trying to do incremental compilation since the size of my
 program is starting to break my computer, slowing compilations
 down by four or five times.


I just want to give you a warning about incremental compilation. 
Currently it doesn't work out that well in D due to various reasons. You 
can search the newsgroups archive for extended posts about this. But to 
mention some of the reasons:

DMD doesn't output all symbols to all object files resulting in missing 
symbols when doing incremental compilation. I think it's mostly template 
symbols that cause problems. I've heard that using the -lib flag, DMD 
will output all symbols to all object files.

Since header/import files are rarely used in D you often need to 
recompile quite many files even if you only do localized changes, 
specially compared to C/C++ using header files.

-- 
/Jacob Carlborg
Mar 05 2012