www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Hair-pulling, D, and Optlink

reply John Reimer <brk_6502 yahoo.com> writes:
Perhaps some kind soul can help me here.  I've spent many hours trying 
to make these procedures work to no avail. I imagine I could get bald soon.

I'm trying to use Derek's build utility to create an executable.  I 
would be asking him about this problem, but I'm beginning to think that 
the problem isn't in build: it's more likely something wrong with the 
Digitalmars link.exe or, perhaps, my head. (That said, if you have any 
idea's here Derek, please tell.  We can discuss it more on dsource if 
need be.)

My platform is Windows XP.
My compiler is dmd 0.123.
I'm using build version 2.07

What I've done:

Using the build tools facility for creating library files, I've managed 
to put together three lib files (from Antonio's projects):  dool.lib, 
dui.lib, and dantfw.lib.

Both dui.lib and dantfw.lib use dool quite liberally.

Building these libraries has been quite easy.  I do something like this 
(using dui as an example):



The build.d is a file at the root of the project directory tree that 
imports all appropriate modules needed to form the library; it's kind of 
like an All.d for the build utility.  This directs build to collect the 
appropriate modules for compilation.  Furthermore, build.d contains a 
pragma(tartget,"dui.lib") to indicate the name of the output file.  A 
library file is automatically created because no "main" or "winmain" 
exists in any of the dui modules.  All object files in the project are 
included in the library via the -allobj switch.

To create dool.lib and dantfw.lib, I follow the same procedure.  I then 
put the three library files into the \dmd\lib directory for future use.


THE PROBLEM:

The next step is to build an actual application using dui.lib, dool.lib, 
and dantfw.lib.  This should be a piece of cake!  The application is 
leds, Antonio's D IDE, which I want to successfully compile on Windows 
using the new dynamic DUI interface.

First of all, I should make it clear that I've succeed in compiling leds 
with Derkek's build.  It's easy if I don't use any pre-compiled library 
files.  Just aim build at the project, give it enough information so 
that it knows where to find the dui, dool, and dantfw source modules and 
fire away.  It compiles and links all the required object files for leds 
in a snap.

But I want to be able to use the library files I created priorly. That 
is not so easy, I found out.

I try a command like so:


-I\Projects\D\dool\trunk\src -I\Projects\D\dantfw\trunk\src dool.lib 
dantfw.lib dui.lib

This doesn't work.  During the link phase it fails:

#leds\Scintilla.obj(Scintilla)


#leds\CodeView.obj(CodeView)












This is from dool's Character.d file which contains a template like this:







The last three aliases end the file.  The template is filled with 
several static methods like toUpper, toLower, isAlnum, isAlpha, etc.

So how do I fix the above problem?  Simple, I do this:


-I\Projects\D\dool\trunk\src -I\Projects\D\dantfw\trunk\src dool.lib 
dantfw.lib dui.lib dool\Character.obj

Notice, all I did is tacked on the Character.obj.  Everything compiles 
successfully.  But no matter what I do, I cannot get those symbol errors 
eliminated by including dool.lib alone which is supposed to contain 
Character.obj already!  And it does indeed from the checks I've made:




And in both cases find that the symbols defined above do indeed exist. 
Dool.lib has them in there!  So why doesn't optlink see them?!

Why does linking with Character.obj work while linking with dool.lib 
does not?  Furthermore, it appears that Character.obj is the only file 
in dool.lib that seems to be causing the trouble.  No other object files 
needed to be added to the command line to get the program to link.  Is 
this some sort problem with storing templates in a library file?  I 
thought library files were extremely simple structures that merely 
archived object files.  Is there something more to them that is tricky?

I really want this to work.  It's the only way to have build do it's 
intended task for me.  If I can't make use of the libraries I build, 
what good are libraries in D?

Thanks for listening!

- JJR
May 16 2005
next sibling parent reply Derek Parnell <derek psych.ward> writes:
On Mon, 16 May 2005 04:43:05 -0700, John Reimer wrote:


 Perhaps some kind soul can help me here.  I've spent many hours trying 
 to make these procedures work to no avail. I imagine I could get bald soon.


[snip]

I'm thinking about your issue now ... hang on, it might take awhile as
you've got lots of things going on here...

Firstly, the compiler needs .d files to *compile* applications. This means
that if you import a file, its a ".d" file that is imported (used by the
compiler) and not the .obj file or any library. 

Secondly, the linker uses .obj and .lib files and never used .d files.

This means, from the point of view of the source file being compiled, that
you source code *must* have all the imported ".d" source files available
too. These can be either the full implementation of modules or just their
stubs - it doesn't matter. 

So, if you are compiling a file called "xyzzy.d", and it imports "abc.d"
and that imports "ghj.d", then at the time of compiling xyzzy.d, you must
have both abc.d and ghj.d available and visible to the compiler. This will
create an xyzzy.obj (only). 

To link this, with the modules 'abc' and 'ghj', which are stored in the
library 'qwerty.lib', you only need to supply 'xyzzy.obj' and
'qwerty'.lib'. 

If you are using Build, and do not want to use any of the 'excess' object
files, then place in those source files (eg. abc.d and ghj.d in this
example), the Build pragma(nolink).

eg.

   version(build) pragma(nolink);

This will cause Build to make sure that their object files are not passed
on to the linker. The net result is that 'dmd' will get the source file for
'xyzzy.d' and not get the 'abc.obj' and 'ghj.obj' files.

You will most likely use the dmd lib pragma or the build link pragma ...

eg.  

  version(build) pragma(link, "dool.lib", "dui.lib", "dantfw.lib");

The -X switch on Build command line will ensure that Build doesn't try to
compile any files in these packages.

Hope this helps. Let me know how it goes.  Use DSource forum if you like.
-- 
Derek Parnell
Melbourne, Australia
16/05/2005 11:01:22 PM
May 16 2005
next sibling parent John Reimer <brk_6502 yahoo.com> writes:
Thanks for taking the time to answer, Derek.

I need to get some sleep before I go at this again.  I'll study your
answer and keep you posted on further issues/questions.  I'm going to try
this on Linux and see if there's any difference.  If linux works, I can
imagine it might be a optlink problem.

I still think something is weird with the linker. :-(  But I could be in
over my head with build. :-P

-JJR

On Mon, 16 May 2005 23:20:15 +1000, Derek Parnell wrote:


 On Mon, 16 May 2005 04:43:05 -0700, John Reimer wrote:
 

 Perhaps some kind soul can help me here.  I've spent many hours trying 
 to make these procedures work to no avail. I imagine I could get bald soon.


 
 [snip]
 
 I'm thinking about your issue now ... hang on, it might take awhile as
 you've got lots of things going on here...
 
 Firstly, the compiler needs .d files to *compile* applications. This means
 that if you import a file, its a ".d" file that is imported (used by the
 compiler) and not the .obj file or any library. 
 
 Secondly, the linker uses .obj and .lib files and never used .d files.
 
 This means, from the point of view of the source file being compiled, that
 you source code *must* have all the imported ".d" source files available
 too. These can be either the full implementation of modules or just their
 stubs - it doesn't matter. 
 
 So, if you are compiling a file called "xyzzy.d", and it imports "abc.d"
 and that imports "ghj.d", then at the time of compiling xyzzy.d, you must
 have both abc.d and ghj.d available and visible to the compiler. This will
 create an xyzzy.obj (only). 
 
 To link this, with the modules 'abc' and 'ghj', which are stored in the
 library 'qwerty.lib', you only need to supply 'xyzzy.obj' and
 'qwerty'.lib'. 
 
 If you are using Build, and do not want to use any of the 'excess' object
 files, then place in those source files (eg. abc.d and ghj.d in this
 example), the Build pragma(nolink).
 
 eg.
 
    version(build) pragma(nolink);
 
 This will cause Build to make sure that their object files are not passed
 on to the linker. The net result is that 'dmd' will get the source file for
 'xyzzy.d' and not get the 'abc.obj' and 'ghj.obj' files.
 
 You will most likely use the dmd lib pragma or the build link pragma ...
 
 eg.  
 
   version(build) pragma(link, "dool.lib", "dui.lib", "dantfw.lib");
 
 The -X switch on Build command line will ensure that Build doesn't try to
 compile any files in these packages.
 
 Hope this helps. Let me know how it goes.  Use DSource forum if you like.
May 16 2005
prev sibling next sibling parent John Reimer <brk_6502 yahoo.com> writes:
On Mon, 16 May 2005 23:20:15 +1000, Derek Parnell wrote:


 
 Firstly, the compiler needs .d files to *compile* applications. This means
 that if you import a file, its a ".d" file that is imported (used by the
 compiler) and not the .obj file or any library. 


 

 Secondly, the linker uses .obj and .lib files and never used .d files.


 

 This means, from the point of view of the source file being compiled, that
 you source code *must* have all the imported ".d" source files available
 too. These can be either the full implementation of modules or just their
 stubs - it doesn't matter. 
 
 So, if you are compiling a file called "xyzzy.d", and it imports "abc.d"
 and that imports "ghj.d", then at the time of compiling xyzzy.d, you must
 have both abc.d and ghj.d available and visible to the compiler. This will
 create an xyzzy.obj (only). 
 
 To link this, with the modules 'abc' and 'ghj', which are stored in the
 library 'qwerty.lib', you only need to supply 'xyzzy.obj' and
 'qwerty'.lib'. 



Very Good explanations for what's going on.  I understand all this. :-)



 If you are using Build, and do not want to use any of the 'excess' object
 files, then place in those source files (eg. abc.d and ghj.d in this
 example), the Build pragma(nolink).
 
 eg.
 
    version(build) pragma(nolink);
 
 This will cause Build to make sure that their object files are not passed
 on to the linker. The net result is that 'dmd' will get the source file for
 'xyzzy.d' and not get the 'abc.obj' and 'ghj.obj' files.
 
 You will most likely use the dmd lib pragma or the build link pragma ...
 
 eg.  
 
   version(build) pragma(link, "dool.lib", "dui.lib", "dantfw.lib");
 
 The -X switch on Build command line will ensure that Build doesn't try to
 compile any files in these packages.
 
 Hope this helps. Let me know how it goes.  Use DSource forum if you like.


Even more good descriptions.  I think this might be my problem:  I've been
using switches on build with the assumption that I new what they were
doing.  If I re-attack the problem with these clarified workings of build,
I think I still have a chance.  I'll let you know how it goes.  For some
reason, I thought -X meant "don't include object files in the final
executable/lib"; your explanation help clarify why I was having trouble
with another issue.  I thought build wasn't working right, but it was
my misunderstanding of what build was doing.

Thanks for the help,

John
May 16 2005
prev sibling parent John Reimer <brk_6502 yahoo.com> writes:
Not too surprisingly, the issue turned out to be non existent on Linux. 
Performing the same steps results in a proper executable.  

Gentoo Linux 2.6 based kernel
gcc 3.4.3
dmd 0.123

Thus, I had no trouble including just libdool.a, libdantfw.a, and
libdui.a as I would have expected.  The Build tool worked like a champ.

I think I really don't like optlinker on windows. :-(

-John
May 16 2005
prev sibling next sibling parent reply Ant <Ant_member pathlink.com> writes:
In article <d6a11p$1kpq$1 digitaldaemon.com>, John Reimer says...

Perhaps some kind soul can help me here.  I've spent many hours trying 
to make these procedures work to no avail. I imagine I could get bald soon.


[...]


#leds\Scintilla.obj(Scintilla)




Ah! CharacterT...
I had the same problem.
I just made CharacterT(T) into 
Character
{
   alias char T;
}
and forget the others types.

So I do have the same problem.

Ant
May 16 2005
parent John Reimer <brk_6502 yahoo.com> writes:
On Mon, 16 May 2005 13:46:16 +0000, Ant wrote:


#leds\Scintilla.obj(Scintilla)




 
 Ah! CharacterT...
 I had the same problem.
 I just made CharacterT(T) into 
 Character
 {
    alias char T;
 }
 and forget the others types.
 
 So I do have the same problem.
 
 Ant


The strange thing, Ant, is that it compiles if I don't use the library
format. Build just pulls in all necessary objects and makes one large
file.  That's the only way I can get to work.  And I don't want to have to
continually pull in object files every time I use "build."

-JJR
May 16 2005
prev sibling parent reply "Walter" <newshound digitalmars.com> writes:
The problem is that when templates are instantiated, they are inserted into
an object file record called a COMDAT. COMDATs have the necessary feature
that if multiple object files have a COMDAT with the same name, the
duplicates are discarded.

This feature is necessary for template instantiations, since the compiler
compiling one module doesn't know about the same templates being
instantiated by another module.

An unfortunate side effect is that the one cannot pull object files out of a
library by referencing COMDATs alone, one must reference something else in
that object file, too.

The dmdscript.lib library has the same problem with protoerror.d. I solved
it with the kludge of inserting:

    int foo;

into the file, and then referencing it in dobject.d with:

    int* pfoo = &dmdscript.protoerror.foo;

Not too pretty, but it works.
May 16 2005
next sibling parent reply Nick <Nick_member pathlink.com> writes:
Interesting. But would it be possible to make the compiler auto detect these
cases and insert a hidden reference "behind the scenes"? It seems like a
straight forward workaround, and straight forward workarounds like to be
automated :)

Nick

In article <d6am2c$28v5$1 digitaldaemon.com>, Walter says...

The problem is that when templates are instantiated, they are inserted into
an object file record called a COMDAT. COMDATs have the necessary feature
that if multiple object files have a COMDAT with the same name, the
duplicates are discarded.

This feature is necessary for template instantiations, since the compiler
compiling one module doesn't know about the same templates being
instantiated by another module.

An unfortunate side effect is that the one cannot pull object files out of a
library by referencing COMDATs alone, one must reference something else in
that object file, too.

The dmdscript.lib library has the same problem with protoerror.d. I solved
it with the kludge of inserting:

    int foo;

into the file, and then referencing it in dobject.d with:

    int* pfoo = &dmdscript.protoerror.foo;

Not too pretty, but it works.
May 16 2005
parent reply "Walter" <newshound digitalmars.com> writes:
I'm not sure how that could work.

"Nick" <Nick_member pathlink.com> wrote in message
news:d6aq0f$2chg$1 digitaldaemon.com...

 Interesting. But would it be possible to make the compiler auto detect


these

 cases and insert a hidden reference "behind the scenes"? It seems like a
 straight forward workaround, and straight forward workarounds like to be
 automated :)
May 16 2005
parent reply John Reimer <brk_6502 yahoo.com> writes:
Walter wrote:

 I'm not sure how that could work.
 
 "Nick" <Nick_member pathlink.com> wrote in message
 news:d6aq0f$2chg$1 digitaldaemon.com...
 

Interesting. But would it be possible to make the compiler auto detect


 
 these
 

cases and insert a hidden reference "behind the scenes"? It seems like a
straight forward workaround, and straight forward workarounds like to be
automated :)


 
 
 


This has the potential for being a quite troublsome problem.  Dool has 
several template-only object files in the lib.  Problems with the other 
objects in the lib haven't shown up because they are being used yet in a 
project. :-(  Linux is free of this problem, thankfully.

So there is no possibility of a fix for this?  Is it quite complicated 
to repair? Perhaps it's a problem with the legacy OMF format.  Your 
technique fixed the problem like you said, but I'd have to start adding 
it to several object files in the library; dool has minimum coupling, so 
in order to fix the problems I'd have to reverse this fact and start 
making spaghetti imports in the library.

Thank you very much for the solution, though.  I can get by for now.

-JJR
May 16 2005
parent reply "Walter" <newshound digitalmars.com> writes:
"John Reimer" <brk_6502 yahoo.com> wrote in message
news:d6bg0g$2ufn$1 digitaldaemon.com...

 This has the potential for being a quite troublsome problem.  Dool has
 several template-only object files in the lib.  Problems with the other
 objects in the lib haven't shown up because they are being used yet in a
 project. :-(  Linux is free of this problem, thankfully.

 So there is no possibility of a fix for this?  Is it quite complicated
 to repair? Perhaps it's a problem with the legacy OMF format.  Your
 technique fixed the problem like you said, but I'd have to start adding
 it to several object files in the library; dool has minimum coupling, so
 in order to fix the problems I'd have to reverse this fact and start
 making spaghetti imports in the library.

 Thank you very much for the solution, though.  I can get by for now.


It's a problem with the OMF format. It isn't easilly fixable.
May 16 2005
parent reply Sean Kelly <sean f4.ca> writes:
Walter wrote:

 "John Reimer" <brk_6502 yahoo.com> wrote in message
 news:d6bg0g$2ufn$1 digitaldaemon.com...

 This has the potential for being a quite troublsome problem.  Dool has
 several template-only object files in the lib.  Problems with the other
 objects in the lib haven't shown up because they are being used yet in a
 project. :-(  Linux is free of this problem, thankfully.




I just ran into this while merging some additional Mango code into Ares. 
  Fortunately, this thread was fairly easy to track down, but 
implementing the fix has required changing almost 10 files so far, and 
Ares includes a very minimal portion of Mango.  I have no idea why Mango 
itself doesn't have this problem, but it seems I'm stuck with it.  And 
while this is merely annoying so far, I can see this becoming a real 
maintenance problem down the road.


 So there is no possibility of a fix for this?  Is it quite complicated
 to repair? Perhaps it's a problem with the legacy OMF format.  Your
 technique fixed the problem like you said, but I'd have to start adding
 it to several object files in the library; dool has minimum coupling, so
 in order to fix the problems I'd have to reverse this fact and start
 making spaghetti imports in the library.

 Thank you very much for the solution, though.  I can get by for now.


 
 It's a problem with the OMF format. It isn't easilly fixable.


Why isn't this a problem with C++ template code?  A module is roughly 
the same as a C++ translation unit, so I imagine the same problem must 
occur there as well.

Here's Walter's description of the problem for reference:
----------

The problem is that when templates are instantiated, they are inserted into
an object file record called a COMDAT. COMDATs have the necessary feature
that if multiple object files have a COMDAT with the same name, the
duplicates are discarded.

This feature is necessary for template instantiations, since the compiler
compiling one module doesn't know about the same templates being
instantiated by another module.

An unfortunate side effect is that the one cannot pull object files out of a
library by referencing COMDATs alone, one must reference something else in
that object file, too.

The dmdscript.lib library has the same problem with protoerror.d. I solved
it with the kludge of inserting:

     int foo;

into the file, and then referencing it in dobject.d with:

     int* pfoo = &dmdscript.protoerror.foo;

Not too pretty, but it works.
Apr 05 2006
parent reply kris <foo bar.com> writes:
This problem is not limited to OMF files -- it shows up on linux too, 
where ELF format is used.

Sean asked why this issue does not crop up with DMC, or other C++ 
compilers supporting templates? I'd like to understand that too. Would 
be nice if this (serious) difficulty were to get some lovin'

- Kris


Sean Kelly wrote:

 Walter wrote:
 

 "John Reimer" <brk_6502 yahoo.com> wrote in message
 news:d6bg0g$2ufn$1 digitaldaemon.com...


 This has the potential for being a quite troublsome problem.  Dool has
 several template-only object files in the lib.  Problems with the other
 objects in the lib haven't shown up because they are being used yet in a
 project. :-(  Linux is free of this problem, thankfully.




 
 
 I just ran into this while merging some additional Mango code into Ares. 
  Fortunately, this thread was fairly easy to track down, but 
 implementing the fix has required changing almost 10 files so far, and 
 Ares includes a very minimal portion of Mango.  I have no idea why Mango 
 itself doesn't have this problem, but it seems I'm stuck with it.  And 
 while this is merely annoying so far, I can see this becoming a real 
 maintenance problem down the road.
 

 So there is no possibility of a fix for this?  Is it quite complicated
 to repair? Perhaps it's a problem with the legacy OMF format.  Your
 technique fixed the problem like you said, but I'd have to start adding
 it to several object files in the library; dool has minimum coupling, so
 in order to fix the problems I'd have to reverse this fact and start
 making spaghetti imports in the library.

 Thank you very much for the solution, though.  I can get by for now.



 It's a problem with the OMF format. It isn't easilly fixable.


 
 
 Why isn't this a problem with C++ template code?  A module is roughly 
 the same as a C++ translation unit, so I imagine the same problem must 
 occur there as well.
 
 Here's Walter's description of the problem for reference:
 ----------
 
 The problem is that when templates are instantiated, they are inserted into
 an object file record called a COMDAT. COMDATs have the necessary feature
 that if multiple object files have a COMDAT with the same name, the
 duplicates are discarded.
 
 This feature is necessary for template instantiations, since the compiler
 compiling one module doesn't know about the same templates being
 instantiated by another module.
 
 An unfortunate side effect is that the one cannot pull object files out 
 of a
 library by referencing COMDATs alone, one must reference something else in
 that object file, too.
 
 The dmdscript.lib library has the same problem with protoerror.d. I solved
 it with the kludge of inserting:
 
     int foo;
 
 into the file, and then referencing it in dobject.d with:
 
     int* pfoo = &dmdscript.protoerror.foo;
 
 Not too pretty, but it works.
Apr 21 2006
parent reply Lars Ivar Igesund <larsivar igesund.net> writes:
kris wrote:


 This problem is not limited to OMF files -- it shows up on linux too,
 where ELF format is used.
 
 Sean asked why this issue does not crop up with DMC, or other C++
 compilers supporting templates? I'd like to understand that too. Would
 be nice if this (serious) difficulty were to get some lovin'
 
 - Kris


Note that this has been reported before:

http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D.bugs/5951

Walter promised to check out it, but I haven't noticed any replies. Sitting
around #D, I've gotten the impression that Mr Chancellor and others
consider D's templates to be thouroughly broken due to these issues.
Apr 21 2006
parent reply Walter Bright <newshound digitalmars.com> writes:
Lars Ivar Igesund wrote:

 kris wrote:
 

 This problem is not limited to OMF files -- it shows up on linux too,
 where ELF format is used.

 Sean asked why this issue does not crop up with DMC, or other C++
 compilers supporting templates? I'd like to understand that too. Would
 be nice if this (serious) difficulty were to get some lovin'

 - Kris


 
 Note that this has been reported before:
 
 http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D.bugs/5951
 
 Walter promised to check out it, but I haven't noticed any replies. Sitting
 around #D, I've gotten the impression that Mr Chancellor and others
 consider D's templates to be thouroughly broken due to these issues.


It "works" in C++ because the compiler regenerates the template 
instantiation in *every* object file where it is used. The C++ compiler 
doesn't know about other object files.

In D, when multiple sources are given on the command line, the compiler 
doesn't generate redundant template instantiations.
Apr 21 2006
parent reply Sean Kelly <sean f4.ca> writes:
Walter Bright wrote:

 Lars Ivar Igesund wrote:

 kris wrote:


 This problem is not limited to OMF files -- it shows up on linux too,
 where ELF format is used.

 Sean asked why this issue does not crop up with DMC, or other C++
 compilers supporting templates? I'd like to understand that too. Would
 be nice if this (serious) difficulty were to get some lovin'


 Note that this has been reported before:

 http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D.bugs/5951

 Walter promised to check out it, but I haven't noticed any replies. 
 Sitting
 around #D, I've gotten the impression that Mr Chancellor and others
 consider D's templates to be thouroughly broken due to these issues.


 
 It "works" in C++ because the compiler regenerates the template 
 instantiation in *every* object file where it is used. The C++ compiler 
 doesn't know about other object files.
 
 In D, when multiple sources are given on the command line, the compiler 
 doesn't generate redundant template instantiations.


But Ares is built one object at a time via a makefile, so there are no 
"multiple sources given on the command line."  Did you mean, perhaps, 
that this is somehow related to how DMD treats modules vs. how DMC 
treats source files?  ie. that the #include model effectively 
copy/pastes code while importing modules does not?


Sean
Apr 21 2006
parent reply Sean Kelly <sean f4.ca> writes:
Sean Kelly wrote:

 Walter Bright wrote:

 Lars Ivar Igesund wrote:

 kris wrote:


 This problem is not limited to OMF files -- it shows up on linux too,
 where ELF format is used.

 Sean asked why this issue does not crop up with DMC, or other C++
 compilers supporting templates? I'd like to understand that too. Would
 be nice if this (serious) difficulty were to get some lovin'


 Note that this has been reported before:

 http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D.bugs/5951

 Walter promised to check out it, but I haven't noticed any replies. 
 Sitting
 around #D, I've gotten the impression that Mr Chancellor and others
 consider D's templates to be thouroughly broken due to these issues.


 It "works" in C++ because the compiler regenerates the template 
 instantiation in *every* object file where it is used. The C++ 
 compiler doesn't know about other object files.

 In D, when multiple sources are given on the command line, the 
 compiler doesn't generate redundant template instantiations.


 
 But Ares is built one object at a time via a makefile, so there are no 
 "multiple sources given on the command line."  Did you mean, perhaps, 
 that this is somehow related to how DMD treats modules vs. how DMC 
 treats source files?  ie. that the #include model effectively 
 copy/pastes code while importing modules does not?


Let me clarify a bit further.  The problem I ran into when importing 
Mango code into Ares was that when modules with non-template code 
instantiated template code also defined in the library, then I'd get a 
link error when attempting to use that non-template library code in an 
application.  I speculated that perhaps the template code simply wasn't 
being generated for some reason so I ran obj2asm on the library object 
files and found I was wrong--so far as I could tell, the template code 
had been instantiated into the object file of every module using the 
code.  But that's as far as I got.  I wondered if perhaps the code was 
perhaps being thrown out by the 'lib' tool, but didn't make much 
progress inspecting the library code itself.  Further, it seems kind of 
odd that the same problem seems to occur on Linux, which has a different 
object file format AFAIK.

Aside from the workaround you suggest a while back--adding 'fake' 
non-template data and referencing it in the appropriate places--are 
there any other options for library writers?  Your proposed solution 
works (or it did for me at any rate), but it's a nasty hack that I don't 
see being well-accepted in the long term.  Implementation complexity 
aside, I would be happy with even "broken" code generation similar to 
how C++ object files are created if it would actually solve this problem 
without introducing a host of new ones.  But I still don't entirely 
understand what's going on.  Is the problem truly in code generation or 
does it occur later during library assembly or during linking?  And are 
there any alternatives such as different object file formats that might 
correct this as well?  From your discussion of COMDATs I don't 
understand why this would be a problem on Linux as well, unless perhaps 
the object files have a similarly bad design?


Sean
Apr 21 2006
parent reply Walter Bright <newshound digitalmars.com> writes:
Sean Kelly wrote:

 Let me clarify a bit further.  The problem I ran into when importing 
 Mango code into Ares was that when modules with non-template code 
 instantiated template code also defined in the library, then I'd get a 
 link error when attempting to use that non-template library code in an 
 application.  I speculated that perhaps the template code simply wasn't 
 being generated for some reason so I ran obj2asm on the library object 
 files and found I was wrong--so far as I could tell, the template code 
 had been instantiated into the object file of every module using the 
 code.  But that's as far as I got.  I wondered if perhaps the code was 
 perhaps being thrown out by the 'lib' tool, but didn't make much 
 progress inspecting the library code itself.  Further, it seems kind of 
 odd that the same problem seems to occur on Linux, which has a different 
 object file format AFAIK.


If the template instantiation occurs in an object file that is put into 
a library, the name associated with it is *not* inserted into the 
library's symbol table. Thus, the linker won't find it when searching a 
library for the name, even if the name is defined in every object file 
in that library. This exact same thing happens with C++, except you 
don't notice it because the template instantiation is *also* inserted 
into every object file that references that template instantiation, 
including the non-library application code. If just one of those gets 
linked in by either linking in the object file directly or by having 
another reference to an object file that instantiates the template, then 
  the linker will see it.


 Aside from the workaround you suggest a while back--adding 'fake' 
 non-template data and referencing it in the appropriate places--are 
 there any other options for library writers?  Your proposed solution 
 works (or it did for me at any rate), but it's a nasty hack that I don't 
 see being well-accepted in the long term.  Implementation complexity 
 aside, I would be happy with even "broken" code generation similar to 
 how C++ object files are created if it would actually solve this problem 
 without introducing a host of new ones.


The C++ solution sucks because it (often) creates massive object files 
and slow compilation times.


 But I still don't entirely 
 understand what's going on.  Is the problem truly in code generation or 
 does it occur later during library assembly or during linking?


The problem is the object file format. There is no way to specify "if 
multiple instances of this name occur, discard all but one" along with 
"include this name in the library dictionary."



 And are 
 there any alternatives such as different object file formats that might 
 correct this as well?  From your discussion of COMDATs I don't 
 understand why this would be a problem on Linux as well, unless perhaps 
 the object files have a similarly bad design?


ELF does things the same braindamaged way. Sheesh, ELF doesn't even 
support "discard this name if nobody references it", a feature commonly 
found on DOS object formats 25 years ago.
Apr 21 2006
parent Sean Kelly <sean f4.ca> writes:
Thanks for the clear explanation.  I now feel like I have a reasonable 
understanding of the problem, even if I'm no closer to a solution :-)


Sean
Apr 21 2006
prev sibling next sibling parent John Reimer <brk_6502 yahoo.com> writes:
Walter wrote:

 The problem is that when templates are instantiated, they are inserted into
 an object file record called a COMDAT. COMDATs have the necessary feature
 that if multiple object files have a COMDAT with the same name, the
 duplicates are discarded.
 
 This feature is necessary for template instantiations, since the compiler
 compiling one module doesn't know about the same templates being
 instantiated by another module.
 
 An unfortunate side effect is that the one cannot pull object files out of a
 library by referencing COMDATs alone, one must reference something else in
 that object file, too.
 
 The dmdscript.lib library has the same problem with protoerror.d. I solved
 it with the kludge of inserting:
 
     int foo;
 
 into the file, and then referencing it in dobject.d with:
 
     int* pfoo = &dmdscript.protoerror.foo;
 
 Not too pretty, but it works.
 
 


I really appreciate this explanation, Walter.  I figured you would know 
what was going on!  You may have saved me a few hairs on my scalp.

- JJR
May 16 2005
prev sibling parent Carlos Santander <csantander619 gmail.com> writes:
Walter escribió:

 The problem is that when templates are instantiated, they are inserted into
 an object file record called a COMDAT. COMDATs have the necessary feature
 that if multiple object files have a COMDAT with the same name, the
 duplicates are discarded.
 
 This feature is necessary for template instantiations, since the compiler
 compiling one module doesn't know about the same templates being
 instantiated by another module.
 
 An unfortunate side effect is that the one cannot pull object files out of a
 library by referencing COMDATs alone, one must reference something else in
 that object file, too.
 
 The dmdscript.lib library has the same problem with protoerror.d. I solved
 it with the kludge of inserting:
 
     int foo;
 
 into the file, and then referencing it in dobject.d with:
 
     int* pfoo = &dmdscript.protoerror.foo;
 
 Not too pretty, but it works.
 
 


Could you add it somewhere in the docs? Something like, "if you have a template 
like this and use it like this, then do this to make it work".

While I've experienced that problem (and I think Ben too), I have no idea what 
you're talking about. Maybe checking dmdscript might help.

-- 
Carlos Santander Bernal
May 16 2005