www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP45: fixing the dllimport/dllexport issue

reply Benjamin Thaut <code benjamin-thaut.de> writes:
The current behaviour of export is not sufficient to create a shared 
runtime on windows. Its alos not working very well in a few other cases. 
For background information please read the links provided in the DIP. 
This DIP tries to solve the problem by imitating the proven preprocessor 
techniques used in C/C++ to create windows dlls.

Destroy.

Kind Regards
Benjamin Thaut
Aug 27 2013
next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Well, a link would have been great:

http://wiki.dlang.org/DIP45
Aug 27 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 27 August 2013 at 10:16:24 UTC, Benjamin Thaut wrote:
 Well, a link would have been great:

 http://wiki.dlang.org/DIP45
I don't think that is the way to go. We need a way to mark a module as being an "header" import for a shared lib. With that solution, you'll ends up duplicating all modules with import and export, and putting the export(foobar) all over the place in a module.
Aug 27 2013
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 27.08.2013 13:28, schrieb deadalnix:
 On Tuesday, 27 August 2013 at 10:16:24 UTC, Benjamin Thaut wrote:
 Well, a link would have been great:

 http://wiki.dlang.org/DIP45
I don't think that is the way to go. We need a way to mark a module as being an "header" import for a shared lib. With that solution, you'll ends up duplicating all modules with import and export, and putting the export(foobar) all over the place in a module.
Well ideally it should be enough to put one export(foobar): at the top of the file. Also I don't see why there is any duplication of modules involved? What exactly do you mean? And how should your header import for a shared lib work, if you link against exactly that lib statically? Kind Regards Benjamin Thaut
Aug 27 2013
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 8/27/13, Benjamin Thaut <code benjamin-thaut.de> wrote:
 Well, a link would have been great:

 http://wiki.dlang.org/DIP45
Using export(identifier) is not going to be reliable, since chances of clashes are high. E.g. if libFoo uses export(Foo) and libBar also uses export(Foo), you won't be able to distinguish between the two. Instead I think 'identifier' should be a module name. However, I have a better idea. Why not only introduce compiler switches that are based on module names rather than having to annotate what export does in code? For example: dmd -m64 -export libB.* -import libA.* -of"libB.dll" dllmain.d libB.d -L/DLL -L/IMPLIB:"libB.lib" -LlibA.lib "-export libB.*" means all modules and subpackages of the libB package should be exported, whereas "-import libA.*" means the opposite for libA. This way you don't have to edit any existing code.
Aug 27 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 27.08.2013 15:33, schrieb Andrej Mitrovic:
 On 8/27/13, Benjamin Thaut <code benjamin-thaut.de> wrote:
 Well, a link would have been great:

 http://wiki.dlang.org/DIP45
Using export(identifier) is not going to be reliable, since chances of clashes are high. E.g. if libFoo uses export(Foo) and libBar also uses export(Foo), you won't be able to distinguish between the two. Instead I think 'identifier' should be a module name. However, I have a better idea. Why not only introduce compiler switches that are based on module names rather than having to annotate what export does in code? For example: dmd -m64 -export libB.* -import libA.* -of"libB.dll" dllmain.d libB.d -L/DLL -L/IMPLIB:"libB.lib" -LlibA.lib "-export libB.*" means all modules and subpackages of the libB package should be exported, whereas "-import libA.*" means the opposite for libA. This way you don't have to edit any existing code.
I thought about something along those lines aswell. But wouldn't that create a very long command line if you do it for druntime?
Aug 27 2013
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 27.08.2013 15:40, Benjamin Thaut wrote:
 Am 27.08.2013 15:33, schrieb Andrej Mitrovic:
 On 8/27/13, Benjamin Thaut <code benjamin-thaut.de> wrote:
 Well, a link would have been great:

 http://wiki.dlang.org/DIP45
Using export(identifier) is not going to be reliable, since chances of clashes are high. E.g. if libFoo uses export(Foo) and libBar also uses export(Foo), you won't be able to distinguish between the two. Instead I think 'identifier' should be a module name. However, I have a better idea. Why not only introduce compiler switches that are based on module names rather than having to annotate what export does in code? For example: dmd -m64 -export libB.* -import libA.* -of"libB.dll" dllmain.d libB.d -L/DLL -L/IMPLIB:"libB.lib" -LlibA.lib "-export libB.*" means all modules and subpackages of the libB package should be exported, whereas "-import libA.*" means the opposite for libA. This way you don't have to edit any existing code.
I thought about something along those lines aswell. But wouldn't that create a very long command line if you do it for druntime?
I think export (at module granularity) can be handled by building the modules with symbols to export using a command line switch "-exportall", and all other modules without. I'd prefer if the compilation against the imported library would be agnostic whether it is later linked against a static or dynamic library. How is this done on linux right now? It does not need "export"/"import" to build against a shared phobos library. Is "import" assumed for any data access and later removed by some magic in the linker?
Aug 28 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-08-29 08:43, Rainer Schuetze wrote:

 How is this done on linux right now? It does not need "export"/"import"
 to build against a shared phobos library. Is "import" assumed for any
 data access and later removed by some magic in the linker?
"export" is noop on Posix. Every symbol is accessible. -- /Jacob Carlborg
Aug 28 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 29 August 2013 at 06:54:46 UTC, Jacob Carlborg wrote:
 On 2013-08-29 08:43, Rainer Schuetze wrote:

 How is this done on linux right now? It does not need 
 "export"/"import"
 to build against a shared phobos library. Is "import" assumed 
 for any
 data access and later removed by some magic in the linker?
"export" is noop on Posix. Every symbol is accessible.
C++ is doing its best to move away from it. We should probably don't follow that convention.
Aug 29 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 29.08.2013 09:24, schrieb deadalnix:
 On Thursday, 29 August 2013 at 06:54:46 UTC, Jacob Carlborg wrote:
 On 2013-08-29 08:43, Rainer Schuetze wrote:

 How is this done on linux right now? It does not need "export"/"import"
 to build against a shared phobos library. Is "import" assumed for any
 data access and later removed by some magic in the linker?
"export" is noop on Posix. Every symbol is accessible.
C++ is doing its best to move away from it. We should probably don't follow that convention.
Moving away from what exactly? -- Kind Regards Benjamin Thaut
Aug 29 2013
parent reply "David Nadlinger" <code klickverbot.at> writes:
On Thursday, 29 August 2013 at 08:36:03 UTC, Benjamin Thaut wrote:
 Am 29.08.2013 09:24, schrieb deadalnix:
 C++ is doing its best to move away from it. We should probably 
 don't
 follow that convention.
Moving away from what exactly?
Everything being visible by default.
Aug 29 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 29.08.2013 10:46, schrieb David Nadlinger:
 On Thursday, 29 August 2013 at 08:36:03 UTC, Benjamin Thaut wrote:
 Am 29.08.2013 09:24, schrieb deadalnix:
 C++ is doing its best to move away from it. We should probably don't
 follow that convention.
Moving away from what exactly?
Everything being visible by default.
So what should be do? So far I like Andrej Mitrovic's solution best. But I think we should omit the placement holder so that instead of "-export std.*" we use "-export std" which basically means the same thing. Otherwise people might geht the idea stuff like this will work "-export *.somesubmodule". Personally I'm ok with any working solution. I wanted to use shared dlls since I started using D 3 years ago, and nothing has happend in that regard since then. Kind Regards Benjamin Thaut
Aug 29 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 29 August 2013 at 09:13:02 UTC, Benjamin Thaut wrote:
 So what should be do? So far I like Andrej Mitrovic's solution 
 best. But I think we should omit the placement holder so that 
 instead of "-export std.*" we use "-export std" which basically 
 means the same thing. Otherwise people might geht the idea 
 stuff like this will work "-export *.somesubmodule".

 Personally I'm ok with any working solution. I wanted to use 
 shared dlls since I started using D 3 years ago, and nothing 
 has happend in that regard since then.

 Kind Regards
 Benjamin Thaut
Here is my proposal. Everything is NOT exported on unixes system, just like on windows. It is consistent accross system and provide opportunities for optimizations and faster linkage. symobol marked as export are understood as ddlexport (or publicly visible on unixes) import declaration marked as export as understood as importing a module compiled in a shared object. This time, declaration in the imported mopdule marked as export are understood as ddlimport.
Aug 29 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 29.08.2013 11:42, schrieb deadalnix:
 import declaration marked as export as understood as importing a module
 compiled in a shared object. This time, declaration in the imported
 mopdule marked as export are understood as ddlimport.
But what if you import a module that is linked statically? That would mean export would be treated as dllimport and it will fail to link because the _imp_ symbols are missing when linking statically?
Aug 29 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 08/29/2013 12:03 PM, Benjamin Thaut wrote:
 But what if you import a module that is linked statically? That would
 mean export would be treated as dllimport and it will fail to link
 because the _imp_ symbols are missing when linking statically?
Could we create alias symbols?
Aug 29 2013
next sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 29.08.2013 21:12, schrieb Martin Nowak:
 On 08/29/2013 12:03 PM, Benjamin Thaut wrote:
 But what if you import a module that is linked statically? That would
 mean export would be treated as dllimport and it will fail to link
 because the _imp_ symbols are missing when linking statically?
Could we create alias symbols?
Yes, but that would add a indirection to all data accesses even if you link the library in statically. That would basically remove the benefit from linking statically. If we want control over what gets exported and what not we will need export anyway, and in my eyes a additional compiler switch for windows is not bad enough to justify adding a level of indirction to data accesses into static libraries. Kind Regards Benjamin Thaut
Aug 29 2013
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On 08/29/2013 09:12 PM, Martin Nowak wrote:
 On 08/29/2013 12:03 PM, Benjamin Thaut wrote:
 But what if you import a module that is linked statically? That would
 mean export would be treated as dllimport and it will fail to link
 because the _imp_ symbols are missing when linking statically?
Could we create alias symbols?
Indeed this seems to work. OMF has an ALIAS record (http://www.azillionmonkeys.com/qed/Omfg.pdf) and COFF has weak externals (http://blog.omega-prime.co.uk/?p=121). So we could add _imp_* aliases for every exported symbol. When someone links against the import library they are never used, when someone links against a static library they redirect to the actual definitions. Would that work?
Aug 29 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 29.08.2013 21:20, schrieb Martin Nowak:
 On 08/29/2013 09:12 PM, Martin Nowak wrote:
 On 08/29/2013 12:03 PM, Benjamin Thaut wrote:
 But what if you import a module that is linked statically? That would
 mean export would be treated as dllimport and it will fail to link
 because the _imp_ symbols are missing when linking statically?
Could we create alias symbols?
Indeed this seems to work. OMF has an ALIAS record (http://www.azillionmonkeys.com/qed/Omfg.pdf) and COFF has weak externals (http://blog.omega-prime.co.uk/?p=121). So we could add _imp_* aliases for every exported symbol. When someone links against the import library they are never used, when someone links against a static library they redirect to the actual definitions. Would that work?
Well no its not that simple. You can't just alias it. You still have to create a import table. That would mean ALL data accesses to data symbols would have another level of indirection. Always. The _imp_ symbols don't refer to the data directly, they refer to the location the data is stored at. As explained in the article I linked (http://blog.omega-prime.co.uk/?p=115) this requires different assembly to be generated. To make this work we would always have to generate assembly with one level of indirection added for every data symbol access, because we can not know if the symbol might be imported from a DLL or not. As D tries to achive near C++ performance I would arther want to avoid that.
Aug 29 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 08/29/2013 09:28 PM, Benjamin Thaut wrote:
 Well no its not that simple. You can't just alias it. You still have to
 create a import table. That would mean ALL data accesses to data symbols
 would have another level of indirection. Always. The _imp_ symbols don't
 refer to the data directly, they refer to the location the data is
 stored at. As explained in the article I linked
 (http://blog.omega-prime.co.uk/?p=115) this requires different assembly
 to be generated. To make this work we would always have to generate
 assembly with one level of indirection added for every data symbol
 access, because we can not know if the symbol might be imported from a
 DLL or not. As D tries to achive near C++ performance I would arther
 want to avoid that.
So the alias trick would only work for functions. For data the _imp_* symbol would need to be a pointer to the data? How about LTO, when statically linking it should be possible to optimize away the indirection. Also are there special relocations? module libA; export int var; int* _imp_var = &var; // created by compiler module foo; import libA; void bar() { auto val = var; // creates val = *_imp_var; }
Aug 29 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 29.08.2013 22:04, schrieb Martin Nowak:
 So the alias trick would only work for functions.
yes
 For data the _imp_* symbol would need to be a pointer to the data?
yes
 How about LTO, when statically linking it should be possible to optimize
 away the indirection.
Rainer Schuetze stated that some linkers are capable of doing this optimizations. But I don't know aynthing further about this topic.
 module libA;
 export int var;
 int* _imp_var = &var; // created by compiler

 module foo;
 import libA;

 void bar()
 {
      auto val = var; // creates val = *_imp_var;
 }
Yes that would work. Still there must be a reason why microsoft doesn't do stuff like that in their C++ toolchain. Its certanly going to cost performance.
Aug 29 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 08/29/2013 10:17 PM, Benjamin Thaut wrote:
 How about LTO, when statically linking it should be possible to optimize
 away the indirection.
Rainer Schuetze stated that some linkers are capable of doing this optimizations. But I don't know aynthing further about this topic.
 module libA;
 export int var;
 int* _imp_var = &var; // created by compiler

 module foo;
 import libA;

 void bar()
 {
      auto val = var; // creates val = *_imp_var;
 }
Yes that would work. Still there must be a reason why microsoft doesn't do stuff like that in their C++ toolchain. Its certanly going to cost performance.
I just tested it and it works. lib.c int var = 0xdeadbeaf; int* _imp_var = &var; main.c #include <stdio.h> extern int* _imp_var; void main() { printf("%d\n", *_imp_var); } cl /c /O2 /GL lib.c cl /O2 /GL main.c lib.obj get objconv from http://www.agner.org/optimize/ objconv -fasm main.exe Search for deadbeaf in main.asm to get the symbol number (?_1176). It directly loads the variable. mov edx, dword ptr [?_1176]
Aug 29 2013
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 29 August 2013 at 21:40:08 UTC, Martin Nowak wrote:
 On 08/29/2013 10:17 PM, Benjamin Thaut wrote:
 How about LTO, when statically linking it should be possible 
 to optimize
 away the indirection.
Rainer Schuetze stated that some linkers are capable of doing this optimizations. But I don't know aynthing further about this topic.
 module libA;
 export int var;
 int* _imp_var = &var; // created by compiler

 module foo;
 import libA;

 void bar()
 {
     auto val = var; // creates val = *_imp_var;
 }
Yes that would work. Still there must be a reason why microsoft doesn't do stuff like that in their C++ toolchain. Its certanly going to cost performance.
I just tested it and it works. lib.c int var = 0xdeadbeaf; int* _imp_var = &var; main.c #include <stdio.h> extern int* _imp_var; void main() { printf("%d\n", *_imp_var); } cl /c /O2 /GL lib.c cl /O2 /GL main.c lib.obj get objconv from http://www.agner.org/optimize/ objconv -fasm main.exe Search for deadbeaf in main.asm to get the symbol number (?_1176). It directly loads the variable. mov edx, dword ptr [?_1176]
I was doubting my idea, but you conviced me :P
Aug 29 2013
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 29.08.2013 23:40, Martin Nowak wrote:
 On 08/29/2013 10:17 PM, Benjamin Thaut wrote:
 How about LTO, when statically linking it should be possible to optimize
 away the indirection.
Rainer Schuetze stated that some linkers are capable of doing this optimizations. But I don't know aynthing further about this topic.
 module libA;
 export int var;
 int* _imp_var = &var; // created by compiler

 module foo;
 import libA;

 void bar()
 {
      auto val = var; // creates val = *_imp_var;
 }
Yes that would work. Still there must be a reason why microsoft doesn't do stuff like that in their C++ toolchain. Its certanly going to cost performance.
I just tested it and it works. lib.c int var = 0xdeadbeaf; int* _imp_var = &var; main.c #include <stdio.h> extern int* _imp_var; void main() { printf("%d\n", *_imp_var); } cl /c /O2 /GL lib.c cl /O2 /GL main.c lib.obj get objconv from http://www.agner.org/optimize/ objconv -fasm main.exe Search for deadbeaf in main.asm to get the symbol number (?_1176). It directly loads the variable. mov edx, dword ptr [?_1176]
It's a bit easier to see the code by adding debug symbols and using dumpbin /disasm main.exe. Unfortnately, this won't help us a lot, because the intermediate object files have some unknown format and in fact are just transferring some code representation to the linker that then invokes the compiler and optimizer on the full source code. This is very specific to the C/C++ toolchain and I don't think we can take advantage of it.
Aug 29 2013
parent Martin Nowak <code dawg.eu> writes:
On 08/30/2013 08:20 AM, Rainer Schuetze wrote:
 It's a bit easier to see the code by adding debug symbols and using
 dumpbin /disasm main.exe.

 Unfortnately, this won't help us a lot, because the intermediate object
 files have some unknown format and in fact are just transferring some
 code representation to the linker that then invokes the compiler and
 optimizer on the full source code. This is very specific to the C/C++
 toolchain and I don't think we can take advantage of it.
This is sadly true for the dmd->COFF->VC link toolchain. But we'll likely see this from GDC or LDC in the future. Despite that I don't think this is a huge performance issue at all we could put all the import data pointers in one section so that they are all packed and allow efficient use of the CPU cache.
Aug 30 2013
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 29.08.2013 08:54, Jacob Carlborg wrote:
 On 2013-08-29 08:43, Rainer Schuetze wrote:

 How is this done on linux right now? It does not need "export"/"import"
 to build against a shared phobos library. Is "import" assumed for any
 data access and later removed by some magic in the linker?
"export" is noop on Posix. Every symbol is accessible.
I meant the import part. How are accesses or data references to data inside another shared library implemented?
Aug 29 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 08/29/2013 08:30 PM, Rainer Schuetze wrote:
 I meant the import part. How are accesses or data references to data
 inside another shared library implemented?
References from the data segment use absolute relocations. References from PIC code use the GOT.
Aug 29 2013
next sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 29.08.2013 21:09, Martin Nowak wrote:
 On 08/29/2013 08:30 PM, Rainer Schuetze wrote:
 I meant the import part. How are accesses or data references to data
 inside another shared library implemented?
References from the data segment use absolute relocations. References from PIC code use the GOT.
So an indirection through the GOT is added to every access to a non-static global if you compile with -fPIC?
Aug 29 2013
parent "Wyatt" <wyatt.epp gmail.com> writes:
On Friday, 30 August 2013 at 06:23:02 UTC, Rainer Schuetze wrote:
 On 29.08.2013 21:09, Martin Nowak wrote:
 References from the data segment use absolute relocations.
 References from PIC code use the GOT.
So an indirection through the GOT is added to every access to a non-static global if you compile with -fPIC?
My understanding was references using the GOT have the same number of indirections as in the PLT -- that is, a one-time cost for the dynamic linker to dereference and replace an entry with the absolute address -- and the major difference is PLT is done lazily, while GOT is fixed up in its entirety at runtime. Having said that, now I wonder if I'm misunderstanding the question? -Wyatt
Aug 30 2013
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 29.08.2013 21:09, Martin Nowak wrote:
 On 08/29/2013 08:30 PM, Rainer Schuetze wrote:
 I meant the import part. How are accesses or data references to data
 inside another shared library implemented?
References from the data segment use absolute relocations.
Just remembered that we don't have that in Windows, so if we want to mimick what C++ does, we'll have to generate initialization-code to be run at startup to fill in the proper pointers from the import table.
Aug 30 2013
next sibling parent Martin Nowak <code dawg.eu> writes:
On 08/30/2013 12:50 PM, Rainer Schuetze wrote:
 References from the data segment use absolute relocations.
Just remembered that we don't have that in Windows, so if we want to mimick what C++ does, we'll have to generate initialization-code to be run at startup to fill in the proper pointers from the import table.
This is also done at runtime by the loader. The additional startup time is one reason why exporting data in ABIs is bad. The other is copy relocations for writeable data (http://docs.oracle.com/cd/E19082-01/819-0690/chapter4-84604/index.html).
Aug 30 2013
prev sibling parent reply "Kagamin" <spam here.lot> writes:
On Friday, 30 August 2013 at 10:50:40 UTC, Rainer Schuetze wrote:
 Just remembered that we don't have that in Windows
Why? What's the difference between code and data symbols? I thought, they're just addresses.
Sep 02 2013
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 03.09.2013 08:17, Kagamin wrote:
 On Friday, 30 August 2013 at 10:50:40 UTC, Rainer Schuetze wrote:
 Just remembered that we don't have that in Windows
Why? What's the difference between code and data symbols? I thought, they're just addresses.
Inside an executable/DLL image, all relocations are relative to the image itself, only the import table has references to other DLLs. To initialize data with a pointer into another DLL, you cannot do that with a direct relocation, you have to run some code to copy the value from the import table into the data. Accessing data or functions in another DLL from code already has this extra code compiled in.
Sep 03 2013
parent "Kagamin" <spam here.lot> writes:
On Tuesday, 3 September 2013 at 07:02:33 UTC, Rainer Schuetze 
wrote:
 To initialize data with a pointer into another DLL, you cannot 
 do that with a direct relocation, you have to run some code to 
 copy the value from the import table into the data.
Well, yes, static initialization is problematic, but you can have statically initialized pointer to an imported function, which is the same problem.
Sep 11 2013
prev sibling parent reply "Kagamin" <spam here.lot> writes:
On Thursday, 29 August 2013 at 06:43:39 UTC, Rainer Schuetze 
wrote:
 How is this done on linux right now? It does not need 
 "export"/"import" to build against a shared phobos library. Is 
 "import" assumed for any data access and later removed by some 
 magic in the linker?
I guess, it adds missing functions which jump to the actual implementation in so. This is done on windows too in import libraries, but export qualifier makes code a little faster because you make one indirect call instead of a direct call + indirect jump (it can be direct, but for this loader should be able to write to the code section to relocate the jump). You also only need an import table for such call.
Aug 29 2013
parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 29.08.2013 11:33, Kagamin wrote:
 On Thursday, 29 August 2013 at 06:43:39 UTC, Rainer Schuetze wrote:
 How is this done on linux right now? It does not need
 "export"/"import" to build against a shared phobos library. Is
 "import" assumed for any data access and later removed by some magic
 in the linker?
I guess, it adds missing functions which jump to the actual implementation in so. This is done on windows too in import libraries, but export qualifier makes code a little faster because you make one indirect call instead of a direct call + indirect jump (it can be direct, but for this loader should be able to write to the code section to relocate the jump). You also only need an import table for such call.
That's how it works for functions, but what about accesses to data in another DLL? Suppose phobos being built as a shared library, how do you access typeid(Object) from another DLL, both from code or inside data, e.g. as the base class member of a class info struct?
Aug 29 2013
prev sibling next sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
I found a very good articel on the dllimport/dllexport issue on windows 
explaining both dynamic linking of function symbols and data symbols:

http://blog.omega-prime.co.uk/?p=115
Aug 29 2013
prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
So let me summarize the currently discussed solutions:

1) Use alias symbols
Pros:
  - Better usability (no additional command line parameters needed when 
compiling / linking against DLLs)

Cons:
  - Less efficient code for cross DLL function calls (one additional 
call instruction)
  - Less efficient code for all accesses to global data, no matter if it 
will end up in a DLL or not. That means __gshared variables, global 
shared variables, hidden global data like module info, type info, 
vtables, etc (one additional level of indirection). It might be possible 
to avoid this in most cases using link time optimization. But it is 
unlikely that we can get LTO to work easily.
  - Additional level of indirection might confuse current debuggers

2) Use additional command line
Pros:
  - More efficient code

Cons:
  - Additional command line parameters needed when compiling / linking 
against a DLL. (can be hidden away inside sc.ini for phobos / druntime)
Aug 30 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 30 August 2013 at 08:23:20 UTC, Benjamin Thaut wrote:
 So let me summarize the currently discussed solutions:

 1) Use alias symbols
 Pros:
  - Better usability (no additional command line parameters 
 needed when compiling / linking against DLLs)

 Cons:
  - Less efficient code for cross DLL function calls (one 
 additional call instruction)
  - Less efficient code for all accesses to global data, no 
 matter if it will end up in a DLL or not. That means __gshared 
 variables, global shared variables, hidden global data like 
 module info, type info, vtables, etc (one additional level of 
 indirection). It might be possible to avoid this in most cases 
 using link time optimization. But it is unlikely that we can 
 get LTO to work easily.
  - Additional level of indirection might confuse current 
 debuggers
My understanding is that it can be optimized away. Or can it ?
 2) Use additional command line
 Pros:
  - More efficient code

 Cons:
  - Additional command line parameters needed when compiling / 
 linking against a DLL. (can be hidden away inside sc.ini for 
 phobos / druntime)
Aug 30 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 30.08.2013 11:17, schrieb deadalnix:
 My understanding is that it can be optimized away. Or can it ?
It can in C/C++ because microsoft has really good LTO (link time optimization). As Rainer Schuetze explained the format of the object files used for LTO is propriatary and unkown to the public. So we can not easily use that, and its unlikely we can use it at all. Also for 32-bit windows it can not be optimized away because optlink does not do LTO. In theory it could be optimized away, but in pratice I doubt we are going to see this kind of optimization in dmd any time soon. And as dmd is the only viable option on windows right now I would prefer the better performing second solution to this problem
Aug 30 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 30 August 2013 at 09:33:25 UTC, Benjamin Thaut wrote:
 Am 30.08.2013 11:17, schrieb deadalnix:
 My understanding is that it can be optimized away. Or can it ?
It can in C/C++ because microsoft has really good LTO (link time optimization). As Rainer Schuetze explained the format of the object files used for LTO is propriatary and unkown to the public. So we can not easily use that, and its unlikely we can use it at all. Also for 32-bit windows it can not be optimized away because optlink does not do LTO.
win32 is kind of a dead end anyway. It is still very alive today, but we want to aim for the future. And it WILL disappear. The object format is kind of anoying.
 In theory it could be optimized away, but in pratice I doubt we 
 are going to see this kind of optimization in dmd any time 
 soon. And as dmd is the only viable option on windows right now 
 I would prefer the better performing second solution to this 
 problem
I understand. But sacrificing good design for immediate reward is bound to cost way more in the long run if D is successful. And if D fails, then it doesn't matter.
Aug 30 2013
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 30.08.2013 11:52, schrieb deadalnix:
 On Friday, 30 August 2013 at 09:33:25 UTC, Benjamin Thaut wrote:
 Am 30.08.2013 11:17, schrieb deadalnix:
 My understanding is that it can be optimized away. Or can it ?
It can in C/C++ because microsoft has really good LTO (link time optimization). As Rainer Schuetze explained the format of the object files used for LTO is propriatary and unkown to the public. So we can not easily use that, and its unlikely we can use it at all. Also for 32-bit windows it can not be optimized away because optlink does not do LTO.
win32 is kind of a dead end anyway. It is still very alive today, but we want to aim for the future. And it WILL disappear. The object format is kind of anoying.
 In theory it could be optimized away, but in pratice I doubt we are
 going to see this kind of optimization in dmd any time soon. And as
 dmd is the only viable option on windows right now I would prefer the
 better performing second solution to this problem
I understand. But sacrificing good design for immediate reward is bound to cost way more in the long run if D is successful. And if D fails, then it doesn't matter.
I don't consider good it good design to add lots of indirections just to remove a command line parameter to the compiler. Its not like its going to make any difference in source code.
Aug 30 2013
prev sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 30.08.2013 10:23, schrieb Benjamin Thaut:
 1) Use alias symbols
I'm convinced now that Martin Nowaks aliasing solution is the better solution as there will be less runtime overhead then I originally thought. Also the usability will be better. It also leaves room for further optimizations by doing some clever pointer patching, in case we find a solution that works on x64. If no one does a veto I'm going to update the DIP with this aliasing solution and all other discussed changes. (e.g. making export a attribute) Kind Regards Benjamin Thaut -- Kind Regards Benjamin Thaut
Aug 30 2013
prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
I updated the DIP with all discussed content. Feedback is welcome.

http://wiki.dlang.org/DIP45

Kind Regards
Benjamin Thaut
Sep 01 2013
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 01.09.2013 11:49, Benjamin Thaut wrote:
 I updated the DIP with all discussed content. Feedback is welcome.

 http://wiki.dlang.org/DIP45

 Kind Regards
 Benjamin Thaut
LGTM. What about the current use case of just exporting some functions for being loaded by a C application? It should still work using a def-file (which you have to use anyway most of the time due to http://d.puremagic.com/issues/show_bug.cgi?id=3956) Maybe already deep into implementation but how are we going to deal with the initialization of data? module sharedlib; __gshared int var; module exe; import sharedlib; __gshared int* pvar = &var; (This kind of relocation also has to be done for stuff like TypeInfo_Class). It needs initialization code to fill pvar with the correct pointer. Do we support the same for pointers in TLS? int* tlsvar = &var; This might need initialization per thread, or patching the original TLS segment.
Sep 01 2013
next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 01.09.2013 17:26, schrieb Rainer Schuetze:
 On 01.09.2013 11:49, Benjamin Thaut wrote:
 I updated the DIP with all discussed content. Feedback is welcome.

 http://wiki.dlang.org/DIP45

 Kind Regards
 Benjamin Thaut
LGTM.
Lets get this merged? ^^ First someone has to implement it ;-)
 What about the current use case of just exporting some functions for
 being loaded by a C application? It should still work using a def-file
 (which you have to use anyway most of the time due to
 http://d.puremagic.com/issues/show_bug.cgi?id=3956)
Well you can still use a .def file of course. Also no one stops you just annotating simple extern(C) functions with the export attribute. I don't see any problem there.
 Maybe already deep into implementation but how are we going to deal with
 the initialization of data?

 module sharedlib;
 __gshared int var;

 module exe;
 import sharedlib;
 __gshared int* pvar = &var;

 (This kind of relocation also has to be done for stuff like
 TypeInfo_Class). It needs initialization code to fill pvar with the
 correct pointer.
Yes, you are absolutely correct. I don't know enough about the internals of a compiler (dmd in perticular) though to answer this question. Can't we use the same mechanism C++ uses to run all its initializers?
 Do we support the same for pointers in TLS?

 int* tlsvar = &var;

 This might need initialization per thread, or patching the original TLS
 segment.
The DIP has a seperate section on TLS variables. Did you miss to read that? Kind Regards Benjamin Thaut
Sep 01 2013
next sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 01.09.2013 18:43, Benjamin Thaut wrote:
 Am 01.09.2013 17:26, schrieb Rainer Schuetze:
 On 01.09.2013 11:49, Benjamin Thaut wrote:
 I updated the DIP with all discussed content. Feedback is welcome.

 http://wiki.dlang.org/DIP45

 Kind Regards
 Benjamin Thaut
LGTM.
Lets get this merged? ^^ First someone has to implement it ;-)
 What about the current use case of just exporting some functions for
 being loaded by a C application? It should still work using a def-file
 (which you have to use anyway most of the time due to
 http://d.puremagic.com/issues/show_bug.cgi?id=3956)
Well you can still use a .def file of course. Also no one stops you just annotating simple extern(C) functions with the export attribute. I don't see any problem there.
It would export the ModuleInfo and related symbols.
 Maybe already deep into implementation but how are we going to deal with
 the initialization of data?

 module sharedlib;
 __gshared int var;

 module exe;
 import sharedlib;
 __gshared int* pvar = &var;

 (This kind of relocation also has to be done for stuff like
 TypeInfo_Class). It needs initialization code to fill pvar with the
 correct pointer.
Yes, you are absolutely correct. I don't know enough about the internals of a compiler (dmd in perticular) though to answer this question. Can't we use the same mechanism C++ uses to run all its initializers?
Yes, it would be an option to generate init functions to be run with the C initializers. But that might introduce a dependency on initialization order. We can write relocation info into another segment and do the relocations before runtime init.
 Do we support the same for pointers in TLS?

 int* tlsvar = &var;

 This might need initialization per thread, or patching the original TLS
 segment.
The DIP has a seperate section on TLS variables. Did you miss to read that?
No. I meant thread global variables that are preinitialized. The appropriate relocation doesn't exist, the __acccess_tls_var function has to be called in a thread local init function. Rainer
Sep 01 2013
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 01.09.2013 19:58, schrieb Rainer Schuetze:
 On 01.09.2013 18:43, Benjamin Thaut wrote:
 Am 01.09.2013 17:26, schrieb Rainer Schuetze:
 On 01.09.2013 11:49, Benjamin Thaut wrote:
 I updated the DIP with all discussed content. Feedback is welcome.

 http://wiki.dlang.org/DIP45

 Kind Regards
 Benjamin Thaut
LGTM.
Lets get this merged? ^^ First someone has to implement it ;-)
 What about the current use case of just exporting some functions for
 being loaded by a C application? It should still work using a def-file
 (which you have to use anyway most of the time due to
 http://d.puremagic.com/issues/show_bug.cgi?id=3956)
Well you can still use a .def file of course. Also no one stops you just annotating simple extern(C) functions with the export attribute. I don't see any problem there.
It would export the ModuleInfo and related symbols.
Is this a issue? There just would be some unused exports in the dll, wouldn't it? If this really is a problem we could add a compiler switch to stop implict exporting of symbols.
 Maybe already deep into implementation but how are we going to deal with
 the initialization of data?

 module sharedlib;
 __gshared int var;

 module exe;
 import sharedlib;
 __gshared int* pvar = &var;

 (This kind of relocation also has to be done for stuff like
 TypeInfo_Class). It needs initialization code to fill pvar with the
 correct pointer.
Yes, you are absolutely correct. I don't know enough about the internals of a compiler (dmd in perticular) though to answer this question. Can't we use the same mechanism C++ uses to run all its initializers?
Yes, it would be an option to generate init functions to be run with the C initializers. But that might introduce a dependency on initialization order. We can write relocation info into another segment and do the relocations before runtime init.
I would say whatever works best for the plattform at hand. No need to force one way or the other.
 Do we support the same for pointers in TLS?

 int* tlsvar = &var;

 This might need initialization per thread, or patching the original TLS
 segment.
The DIP has a seperate section on TLS variables. Did you miss to read that?
No. I meant thread global variables that are preinitialized. The appropriate relocation doesn't exist, the __acccess_tls_var function has to be called in a thread local init function.
Can't we initialize these right before running the shared module constructors? To make the implementation less work we could also make this a error for now and maybe implement it later. What would you prefer?
 Rainer
Sep 01 2013
prev sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Sunday, 1 September 2013 at 16:43:14 UTC, Benjamin Thaut wrote:
 The DIP has a seperate section on TLS variables. Did you miss 
 to read that?
Can't we just use the normal (g)libc facilities for handling TLS in shared objects on Linux? The TLS data from the "DLL" is assigned to another tls module, and everything is fine, or am I missing something? David
Sep 01 2013
parent reply Iain Buclaw <ibuclaw ubuntu.com> writes:
On 1 September 2013 19:16, David Nadlinger <code klickverbot.at> wrote:
 On Sunday, 1 September 2013 at 16:43:14 UTC, Benjamin Thaut wrote:
 The DIP has a seperate section on TLS variables. Did you miss to read
 that?
Can't we just use the normal (g)libc facilities for handling TLS in shared objects on Linux? The TLS data from the "DLL" is assigned to another tls module, and everything is fine, or am I missing something? David
Yes, no need to invent a new cog to go on top of what the base runtime already offers. -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0';
Sep 01 2013
next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 01.09.2013 21:19, schrieb Iain Buclaw:
 On 1 September 2013 19:16, David Nadlinger <code klickverbot.at> wrote:
 On Sunday, 1 September 2013 at 16:43:14 UTC, Benjamin Thaut wrote:
 The DIP has a seperate section on TLS variables. Did you miss to read
 that?
Can't we just use the normal (g)libc facilities for handling TLS in shared objects on Linux? The TLS data from the "DLL" is assigned to another tls module, and everything is fine, or am I missing something? David
Yes, no need to invent a new cog to go on top of what the base runtime already offers.
I didn't know linux has a facility for accessing TLS data across shared library boundaries. I will put the implementation suggestion into the windows specific section then.
Sep 01 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 09/01/2013 09:33 PM, Benjamin Thaut wrote:
 I didn't know linux has a facility for accessing TLS data across shared
 library boundaries. I will put the implementation suggestion into the
 windows specific section then.
Yes, it has a bunch of special relocations and runtime linker support for this. It would be nice if someone could figure out and summarize all the TLS index and TIB business on Windows. http://msdn.microsoft.com/en-us/library/windows/desktop/ms686997(v=vs.85).aspx http://en.wikipedia.org/wiki/Thread-local_storage#Windows_implementation http://en.wikipedia.org/wiki/Win32_Thread_Information_Block
Sep 05 2013
next sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 06.09.2013 04:00, schrieb Martin Nowak:
 On 09/01/2013 09:33 PM, Benjamin Thaut wrote:
 I didn't know linux has a facility for accessing TLS data across shared
 library boundaries. I will put the implementation suggestion into the
 windows specific section then.
Yes, it has a bunch of special relocations and runtime linker support for this. It would be nice if someone could figure out and summarize all the TLS index and TIB business on Windows. http://msdn.microsoft.com/en-us/library/windows/desktop/ms686997(v=vs.85).aspx http://en.wikipedia.org/wiki/Thread-local_storage#Windows_implementation http://en.wikipedia.org/wiki/Win32_Thread_Information_Block
What do we need this information for? -- Kind Regards Benjamin Thaut
Sep 06 2013
prev sibling parent "David Nadlinger" <code klickverbot.at> writes:
On Friday, 6 September 2013 at 02:00:22 UTC, Martin Nowak wrote:
 Yes, it has a bunch of special relocations and runtime linker 
 support for this.
 It would be nice if someone could figure out and summarize all 
 the TLS index and TIB business on Windows.

 http://msdn.microsoft.com/en-us/library/windows/desktop/ms686997(v=vs.85).aspx
 http://en.wikipedia.org/wiki/Thread-local_storage#Windows_implementation
 http://en.wikipedia.org/wiki/Win32_Thread_Information_Block
Best material I found on Windows TLS so far: http://www.nynaeve.net/?s=tls Haven't really looked into the interaction between multiple DLLs at all so far, though. David
Sep 07 2013
prev sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 01.09.2013 21:19, schrieb Iain Buclaw:
 On 1 September 2013 19:16, David Nadlinger <code klickverbot.at> wrote:
 On Sunday, 1 September 2013 at 16:43:14 UTC, Benjamin Thaut wrote:
 The DIP has a seperate section on TLS variables. Did you miss to read
 that?
Can't we just use the normal (g)libc facilities for handling TLS in shared objects on Linux? The TLS data from the "DLL" is assigned to another tls module, and everything is fine, or am I missing something? David
Yes, no need to invent a new cog to go on top of what the base runtime already offers.
Basically I thought whatever is inside the Description would go into the spec. Everything inside the implementation details is just a suggestion for implementing proper dllexport / dllimport on windows plattforms. We shouldn't hard wire these implementation suggestions into the spec.
Sep 01 2013
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On 09/01/2013 05:26 PM, Rainer Schuetze wrote:
 Maybe already deep into implementation but how are we going to deal with
 the initialization of data?

 module sharedlib;
 __gshared int var;

 module exe;
 import sharedlib;
 __gshared int* pvar = &var;

 (This kind of relocation also has to be done for stuff like
 TypeInfo_Class). It needs initialization code to fill pvar with the
 correct pointer.
Based on your saying that there are no absolute data relocations for this on windows we'd have to perform runtime initialization. At runtime it would simply copy _imp_var into pvar, right?
 Do we support the same for pointers in TLS?

 int* tlsvar = &var;

 This might need initialization per thread, or patching the original TLS
 segment.
No you can't take the address of a TLS var at compile time.
Sep 05 2013
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 06.09.2013 03:51, schrieb Martin Nowak:
 On 09/01/2013 05:26 PM, Rainer Schuetze wrote:
 Maybe already deep into implementation but how are we going to deal with
 the initialization of data?

 module sharedlib;
 __gshared int var;

 module exe;
 import sharedlib;
 __gshared int* pvar = &var;

 (This kind of relocation also has to be done for stuff like
 TypeInfo_Class). It needs initialization code to fill pvar with the
 correct pointer.
Based on your saying that there are no absolute data relocations for this on windows we'd have to perform runtime initialization. At runtime it would simply copy _imp_var into pvar, right?
Yes. As Rainer already mentioned we can either do this by running some initialiers before initializing druntime, or we could add additional information in a section of the dll and run some generic dll initialization code.
 Do we support the same for pointers in TLS?

 int* tlsvar = &var;

 This might need initialization per thread, or patching the original TLS
 segment.
No you can't take the address of a TLS var at compile time.
Its not taking a address of a TLS variable. It is initializing a TLS variable with the address of a __gshared variable. -- Kind Regards Benjamin Thaut
Sep 06 2013
prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Ok there is a new problem. Consider the following example:

module lib1;

export int lib1Func()
{
   return 5;
}


module lib2;
import lib1;

export int lib2Func()
{
   return lib1Func() + 1;
}


If you compile lib1 into a static library and then copmpile lib2 into a 
DLL which statically links against lib1 both symbols lib1Func and 
lib2Func will be exported from the dll because lib1Func will get marked 
as dllexport when compiled into a static library and the static library 
transports that information into the linking process of lib2. This 
happens both on win32 and win64. But this is something you really don't 
want. Lets imagine you have a DLL that statically links against phobos. 
Suddenly you will have all phobos exported symbols in your dll. This can 
also lead to double defined symbols in case a user links against your 
dll and against the shared phobos dll. So we have to fix this somehow.

I propose that we add a command-line-paramter to the compiler (windows 
only) which actually enables dllexport. So all behavior described in the 
DIP will be enabled by default, but to actually make it mark symbols 
with dllexport you have to specifiy a additional command-line-parameter. 
We could call it "-dll" or something.

In case you want to try it out yourself, here is the repro case: 
http://stuff.benjamin-thaut.de/D/dll2.zip

Kind Regards
Benjamin Thaut
Sep 07 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 7 September 2013 at 12:47:19 UTC, Benjamin Thaut 
wrote:
 I propose that we add a command-line-paramter to the compiler 
 (windows only) which actually enables dllexport. So all 
 behavior described in the DIP will be enabled by default, but 
 to actually make it mark symbols with dllexport you have to 
 specifiy a additional command-line-parameter. We could call it 
 "-dll" or something.
I have to think about it. But I'd like to avoid any special behavior for windows language side. I also kind of feel that we need to export lib1 as well, as you may not know what lib2 does with what it gets from lib1 (passing objects around for instance). So certainly at least some part of lib1 need to be exported.
Sep 07 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 07.09.2013 20:42, schrieb deadalnix:
 On Saturday, 7 September 2013 at 12:47:19 UTC, Benjamin Thaut wrote:
 I propose that we add a command-line-paramter to the compiler (windows
 only) which actually enables dllexport. So all behavior described in
 the DIP will be enabled by default, but to actually make it mark
 symbols with dllexport you have to specifiy a additional
 command-line-parameter. We could call it "-dll" or something.
I have to think about it. But I'd like to avoid any special behavior for windows language side. I also kind of feel that we need to export lib1 as well, as you may not know what lib2 does with what it gets from lib1 (passing objects around for instance). So certainly at least some part of lib1 need to be exported.
I don't agree if that statement. If you have three libs. lib1, lib2 and lib3. And lib2 and lib3 link statically against lib1 you are going to get linker errors because both lib2 and lib3 contain the symbols of lib1. And you don't have any options to avoid that (unless you got into the source of lib1 and remove all export attributes, which might not even be possible because you don't have the source)
Sep 07 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 7 September 2013 at 21:41:01 UTC, Benjamin Thaut 
wrote:
 I don't agree if that statement. If you have three libs. lib1, 
 lib2 and lib3. And lib2 and lib3 link statically against lib1 
 you are going to get linker errors because both lib2 and lib3 
 contain the symbols of lib1. And you don't have any options to 
 avoid that (unless you got into the source of lib1 and remove 
 all export attributes, which might not even be possible because 
 you don't have the source)
And you should have an error, especially is you pass object from one side to the other. typeid won't match (which will create a mess in the runtime), and worse, struct layout may not match (hello memory corruption). If the problem is that big, we can still have a -noexport flag or something, but that doesn't seems really safe to me.
Sep 07 2013
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 08.09.2013 06:19, schrieb deadalnix:
 On Saturday, 7 September 2013 at 21:41:01 UTC, Benjamin Thaut wrote:
 I don't agree if that statement. If you have three libs. lib1, lib2
 and lib3. And lib2 and lib3 link statically against lib1 you are going
 to get linker errors because both lib2 and lib3 contain the symbols of
 lib1. And you don't have any options to avoid that (unless you got
 into the source of lib1 and remove all export attributes, which might
 not even be possible because you don't have the source)
And you should have an error, especially is you pass object from one side to the other. typeid won't match (which will create a mess in the runtime), and worse, struct layout may not match (hello memory corruption). If the problem is that big, we can still have a -noexport flag or something, but that doesn't seems really safe to me.
But only assuming that you are passing around objects of that type. I see a dll as a module of encapsulation. If the library is only unsed internally nothing of its dependencies should get exported. Because if they are your users might rely on the fact that they are, and from that point on you have to garantuee that all those additional exported functions from your dependencies stay in your library, even if you don't need the dependency anymore.
Sep 08 2013
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
 I also kind of feel that we need to export lib1 as well, as you may not
 know what lib2 does with what it gets from lib1 (passing objects around
 for instance). So certainly at least some part of lib1 need to be
 exported.
For sure lib2 might forward to some lib1 symbols. Not sure what might accidentally require linkage.
 I don't agree if that statement. If you have three libs. lib1, lib2 and
 lib3. And lib2 and lib3 link statically against lib1 you are going to
 get linker errors because both lib2 and lib3 contain the symbols of lib1.
A: Why do two DLLs which export the same symbols cause linker errors? Because the corresponding import libraries contain the same symbols? B: You shouldn't mix static with dynamic linking, ODR issues are what you get in return. C: What would happen if you statically link against lib2 and lib3?
Sep 09 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 10.09.2013 05:39, schrieb Martin Nowak:
 I also kind of feel that we need to export lib1 as well, as you may not
 know what lib2 does with what it gets from lib1 (passing objects around
 for instance). So certainly at least some part of lib1 need to be
 exported.
For sure lib2 might forward to some lib1 symbols. Not sure what might accidentally require linkage.
 I don't agree if that statement. If you have three libs. lib1, lib2 and
 lib3. And lib2 and lib3 link statically against lib1 you are going to
 get linker errors because both lib2 and lib3 contain the symbols of lib1.
A: Why do two DLLs which export the same symbols cause linker errors?
Yes
 Because the corresponding import libraries contain the same symbols?
 B: You shouldn't mix static with dynamic linking, ODR issues are what
 you get in return.
Well you should not. But it will happen. Espeically if you don't have any control over what others due. E.g. when using third party libraries.
 C: What would happen if you statically link against lib2 and lib3?
Well you would get a linker error as well. But the reasoing behind this is as follows. You use a third party library that comes as a shared library. That third party library uses say libcurl internally. Now you use libcurl too. You link against libcurl. But because the third party library you link to also uses libcurl you will get linker errors. Also you might not want to use the same version of libcurl then the thirdparty library you use. This is going to become a problem especially in bigger projects, where you are not able to compile the third party libraries yourself because you don't have the source code. In such a case it can easily happen that you get a new version of that third party library, they added a new internal dependency to a library you already use and suddenly your executable won't linke anymore. Kind Regards Benjamin Thaut
Sep 19 2013
parent Martin Nowak <code dawg.eu> writes:
On 09/19/2013 01:55 PM, Benjamin Thaut wrote:
 B: You shouldn't mix static with dynamic linking, ODR issues are what
 you get in return.
Well you should not. But it will happen. Espeically if you don't have any control over what others due. E.g. when using third party libraries.
Let me put it differently, if you statically link against a library in your DLL you need to ensure not to leak (export) any symbols from this static library.
Nov 16 2013
prev sibling next sibling parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 07.09.2013 14:47, Benjamin Thaut wrote:
 I propose that we add a command-line-paramter to the compiler (windows
 only) which actually enables dllexport. So all behavior described in the
 DIP will be enabled by default, but to actually make it mark symbols
 with dllexport you have to specifiy a additional command-line-parameter.
 We could call it "-dll" or something.
I don't think the issue is Windows specific, symbol visibility in the GNU toolchain is also bound to the declaration. I remember seeing public (visibility=default) symbols exported from static libraries on OSX.
Sep 07 2013
prev sibling next sibling parent Martin Nowak <code dawg.eu> writes:
On 09/07/2013 02:47 PM, Benjamin Thaut wrote:
 If you compile lib1 into a static library and then copmpile lib2 into a
 DLL which statically links against lib1 both symbols lib1Func and
 lib2Func will be exported from the dll
That might indeed cause some issues, but when you make lib1 part of your DLL by statically linking against it that's what you get. If you just want to use it, you should make lib1 a DLL as well, which also avoids ODR problems. Also could a .DEF file control which symbols are reexported?
Sep 09 2013
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On 09/07/2013 02:47 PM, Benjamin Thaut wrote:
 If you compile lib1 into a static library and then copmpile lib2 into a
 DLL which statically links against lib1 both symbols lib1Func and
 lib2Func will be exported from the dll because lib1Func will get marked
 as dllexport when compiled into a static library and the static library
 transports that information into the linking process of lib2. This
 happens both on win32 and win64. But this is something you really don't
 want. Lets imagine you have a DLL that statically links against phobos.
 Suddenly you will have all phobos exported symbols in your dll. This can
 also lead to double defined symbols in case a user links against your
 dll and against the shared phobos dll. So we have to fix this somehow.
I had another thought about this and tried some examples. If you do transitively link against lib1.lib symbols when using lib2.dll you might get ODR issues if there is another copy of lib1 in your process. This is to be expected because there are two copies of the same library. The safe choice is to make lib1 a DLL too. If you still want to statically link lib1 into lib2.dll you have to be very cautious. An actual problem only occurs when lib1 symbols get used from outside of lib2.dll. Of course a mean to prevent bugs from accidental linking lib1 symbols would be nice. One way of doing this would be to mark all lib1 symbols as PRIVATE in lib2.def. EXPORTS D4lib18lib1FuncFZi PRIVATE Another way would be to write a small tool that turns all exported symbols of a library into non-exported symbols. I'm not sure we should do this in the compiler, because it simply hands over the libs to the linker. We'd have to make a copy of the library first and then do what a tool would do. Another note for the phobos case. You should really ask yourself why does your dll need a private copy of phobos. And if so why can phobos symbols leak out of your dll.
Sep 10 2013
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 11.09.2013 00:39, Martin Nowak wrote:
 On 09/07/2013 02:47 PM, Benjamin Thaut wrote:
 If you compile lib1 into a static library and then copmpile lib2 into a
 DLL which statically links against lib1 both symbols lib1Func and
 lib2Func will be exported from the dll because lib1Func will get marked
 as dllexport when compiled into a static library and the static library
 transports that information into the linking process of lib2. This
 happens both on win32 and win64. But this is something you really don't
 want. Lets imagine you have a DLL that statically links against phobos.
 Suddenly you will have all phobos exported symbols in your dll. This can
 also lead to double defined symbols in case a user links against your
 dll and against the shared phobos dll. So we have to fix this somehow.
I had another thought about this and tried some examples. If you do transitively link against lib1.lib symbols when using lib2.dll you might get ODR issues if there is another copy of lib1 in your process. This is to be expected because there are two copies of the same library. The safe choice is to make lib1 a DLL too. If you still want to statically link lib1 into lib2.dll you have to be very cautious. An actual problem only occurs when lib1 symbols get used from outside of lib2.dll. Of course a mean to prevent bugs from accidental linking lib1 symbols would be nice. One way of doing this would be to mark all lib1 symbols as PRIVATE in lib2.def. EXPORTS D4lib18lib1FuncFZi PRIVATE
That's probably not feasible for a library like phobos. (At least optlink stumbles if the def file gets too large). Also, compressed symbol names cause troubles for OMF.
 Another way would be to write a small tool that turns all exported
 symbols of a library into non-exported symbols.

 I'm not sure we should do this in the compiler, because it simply hands
 over the libs to the linker. We'd have to make a copy of the library
 first and then do what a tool would do.

 Another note for the phobos case. You should really ask yourself why
 does your dll need a private copy of phobos. And if so why can phobos
 symbols leak out of your dll.
AFAIU the discussed scenario is that the phobos library exports a lot of symbols, and this library is used both for static and dynamic linking. An executable statically linked against this library will have all the exports. That still works but is ugly. A DLL to be used as a plugin with a C interface might also want to link statically against phobos to avoid DLL hell. Consider multiple plugins built against different versions of phobos. These would also leak also the symbols.
Sep 10 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 09/11/2013 08:21 AM, Rainer Schuetze wrote:
 AFAIU the discussed scenario is that the phobos library exports a lot of
 symbols, and this library is used both for static and dynamic linking.
 An executable statically linked against this library will have all the
 exports. That still works but is ugly.
Yes and it also has some runtime/memory overhead because the symbol table is much bigger.
 A DLL to be used as a plugin with a C interface might also want to link
 statically against phobos to avoid DLL hell. Consider multiple plugins
 built against different versions of phobos. These would also leak also
 the symbols.
Right, that's the valid use-case. But here the additionally exported symbols are way less of a problem because your loading not linking. And as I wrote before if you link a program with two copies of the same library you're asking for trouble. Anyhow I realized there is even less a use-case for statically linking a lib into a DLL (shared library) while expecting the symbols to be exported. We use that to put druntime into libphobos2.so though. Also one doesn't need exported (visible) symbols for static linking. So how about going into the opposite direction? When the compiler creates a lib it strips all the exports (marks all symbols as hidden) by default. For the very special use-case where one wants to preserve the exports we can add a -libexports flag, only to be used with -lib. Another nice thing is, that dmd writes libs itself and already processes all object files so we could easily integrate the striping in the compiler.
Sep 11 2013
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 12.09.2013 01:16, Martin Nowak wrote:
 On 09/11/2013 08:21 AM, Rainer Schuetze wrote:
 AFAIU the discussed scenario is that the phobos library exports a lot of
 symbols, and this library is used both for static and dynamic linking.
 An executable statically linked against this library will have all the
 exports. That still works but is ugly.
Yes and it also has some runtime/memory overhead because the symbol table is much bigger.
 A DLL to be used as a plugin with a C interface might also want to link
 statically against phobos to avoid DLL hell. Consider multiple plugins
 built against different versions of phobos. These would also leak also
 the symbols.
Right, that's the valid use-case. But here the additionally exported symbols are way less of a problem because your loading not linking. And as I wrote before if you link a program with two copies of the same library you're asking for trouble. Anyhow I realized there is even less a use-case for statically linking a lib into a DLL (shared library) while expecting the symbols to be exported. We use that to put druntime into libphobos2.so though. Also one doesn't need exported (visible) symbols for static linking. So how about going into the opposite direction? When the compiler creates a lib it strips all the exports (marks all symbols as hidden) by default. For the very special use-case where one wants to preserve the exports we can add a -libexports flag, only to be used with -lib. Another nice thing is, that dmd writes libs itself and already processes all object files so we could easily integrate the striping in the compiler.
This sounds interesting. Stripping an existing library isn't even needed because it is normally never created with exports anyway (-lib implies that no exports are created to start with). Stripping object files might be necessary, though, in case they have been built separately with -c.
Sep 11 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 12.09.2013 07:54, schrieb Rainer Schuetze:
 This sounds interesting. Stripping an existing library isn't even needed
 because it is normally never created with exports anyway (-lib implies
 that no exports are created to start with). Stripping object files might
 be necessary, though, in case they have been built separately with -c.
Can't we get around stripping object files by always using -c together with -lib in case the object file is supposed to go into a static library?
Sep 19 2013
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 19.09.2013 13:58, Benjamin Thaut wrote:
 Am 12.09.2013 07:54, schrieb Rainer Schuetze:
 This sounds interesting. Stripping an existing library isn't even needed
 because it is normally never created with exports anyway (-lib implies
 that no exports are created to start with). Stripping object files might
 be necessary, though, in case they have been built separately with -c.
Can't we get around stripping object files by always using -c together with -lib in case the object file is supposed to go into a static library?
That could work, though it has it's own set of side effects, like splitting a module into pseudo modules per function which also ruins running unittests built into the library. Building modules with -c and combining these into a library afterwards avoids this.
Sep 19 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 19.09.2013 20:24, schrieb Rainer Schuetze:
 On 19.09.2013 13:58, Benjamin Thaut wrote:
 Am 12.09.2013 07:54, schrieb Rainer Schuetze:
 This sounds interesting. Stripping an existing library isn't even needed
 because it is normally never created with exports anyway (-lib implies
 that no exports are created to start with). Stripping object files might
 be necessary, though, in case they have been built separately with -c.
Can't we get around stripping object files by always using -c together with -lib in case the object file is supposed to go into a static library?
That could work, though it has it's own set of side effects, like splitting a module into pseudo modules per function which also ruins running unittests built into the library. Building modules with -c and combining these into a library afterwards avoids this.
Ah ok, I didn't know that. How much work would it be to implement object file stripping? I'm not confident I can do this. Kind Regards Benjamin Thaut
Sep 19 2013
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 19.09.2013 21:42, Benjamin Thaut wrote:
 Am 19.09.2013 20:24, schrieb Rainer Schuetze:
 On 19.09.2013 13:58, Benjamin Thaut wrote:
 Am 12.09.2013 07:54, schrieb Rainer Schuetze:
 This sounds interesting. Stripping an existing library isn't even
 needed
 because it is normally never created with exports anyway (-lib implies
 that no exports are created to start with). Stripping object files
 might
 be necessary, though, in case they have been built separately with -c.
Can't we get around stripping object files by always using -c together with -lib in case the object file is supposed to go into a static library?
That could work, though it has it's own set of side effects, like splitting a module into pseudo modules per function which also ruins running unittests built into the library. Building modules with -c and combining these into a library afterwards avoids this.
Ah ok, I didn't know that. How much work would it be to implement object file stripping? I'm not confident I can do this.
I just checked the OMF and COFF docs: it should be possible to wipe out the export records without having to rewrite the object files, so it's not too involved. Don't know about ELF or mach-o, though.
Sep 22 2013
parent Martin Nowak <code dawg.eu> writes:
On 09/23/2013 08:30 AM, Rainer Schuetze wrote:
 I just checked the OMF and COFF docs: it should be possible to wipe out
 the export records without having to rewrite the object files, so it's
 not too involved. Don't know about ELF or mach-o, though.
I tried this and implemented it by rewriting the ELF sym flag for export, so it's was done by a rewrite of the object file. Maybe there are other solutions, but this seemed to be the simplest approach.
Nov 16 2013
prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 27.08.2013 12:12, schrieb Benjamin Thaut:
 The current behaviour of export is not sufficient to create a shared
 runtime on windows. Its alos not working very well in a few other cases.
 For background information please read the links provided in the DIP.
 This DIP tries to solve the problem by imitating the proven preprocessor
 techniques used in C/C++ to create windows dlls.

 Destroy.

 Kind Regards
 Benjamin Thaut
There is yet another problem to be solved. What should happen with tempaltes? Does it make sense to export templates? What happens if a template class marked with export gets instanciated in different DLLs using the same template arguments? Should the TypeInfo objects match? Should templates be exportable in generall? In C++ the only thing I needed so far was exporting static members of templated classes. So the different instances of the template can work on the same data. Kind Regards Benjamin Thaut
Sep 08 2013
next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 08.09.2013 09:33, schrieb Benjamin Thaut:
 Am 27.08.2013 12:12, schrieb Benjamin Thaut:
 The current behaviour of export is not sufficient to create a shared
 runtime on windows. Its alos not working very well in a few other cases.
 For background information please read the links provided in the DIP.
 This DIP tries to solve the problem by imitating the proven preprocessor
 techniques used in C/C++ to create windows dlls.

 Destroy.

 Kind Regards
 Benjamin Thaut
There is yet another problem to be solved. What should happen with tempaltes? Does it make sense to export templates? What happens if a template class marked with export gets instanciated in different DLLs using the same template arguments? Should the TypeInfo objects match? Should templates be exportable in generall? In C++ the only thing I needed so far was exporting static members of templated classes. So the different instances of the template can work on the same data. Kind Regards Benjamin Thaut
Good question. Not sure how Ada, Modula-3, Eiffel, Delphi, MLton solve the issue, even though their generic capabilities are not as powerfull. -- Paulo
Sep 08 2013
prev sibling parent Martin Nowak <code dawg.eu> writes:
On 09/08/2013 09:33 AM, Benjamin Thaut wrote:
 There is yet another problem to be solved. What should happen with
 tempaltes? Does it make sense to export templates?
Yes, we need to export/import symbols from templates. Recently the compiler was changed to skip instantiations when they have already been instantiated in imported modules. There is a problem similar to the lib case though. When you instantiate an exported template you don't want it's symbols exported from your shared library.
 What happens if a template class marked with export gets instanciated in
different DLLs
 using the same template arguments? Should the TypeInfo objects match?
 Should templates be exportable in generall?
Yes TypeInfos should match. For symbols which are allowed to have multiple definitions (like typeinfos from template classes) we can't solely rely on pointer comparison but also need to test typename equality.
Nov 04 2013
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
Can we deal with vtable entries pointing to imported functions on 
Windows? The kind of thing that happens when you inherit across a DLL 
boundary.
Sep 09 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 10.09.2013 05:50, schrieb Martin Nowak:
 Can we deal with vtable entries pointing to imported functions on
 Windows? The kind of thing that happens when you inherit across a DLL
 boundary.
I can't answer that and with dmd's current implementation of export its not possible to try this. I would say we implement the DIP and see on the way if this becomes a issue. -- Kind Regards Benjamin Thaut
Sep 22 2013
parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 23.09.2013 07:22, Benjamin Thaut wrote:
 Am 10.09.2013 05:50, schrieb Martin Nowak:
 Can we deal with vtable entries pointing to imported functions on
 Windows? The kind of thing that happens when you inherit across a DLL
 boundary.
I can't answer that and with dmd's current implementation of export its not possible to try this. I would say we implement the DIP and see on the way if this becomes a issue.
It is the same as initializing a global variable with a pointer into another DLL, i.e. it needs some init code to set the value.
Sep 22 2013