www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP45: fixing the dllimport/dllexport issue

reply Benjamin Thaut <code benjamin-thaut.de> writes:
The current behaviour of export is not sufficient to create a shared 
runtime on windows. Its alos not working very well in a few other cases. 
For background information please read the links provided in the DIP. 
This DIP tries to solve the problem by imitating the proven preprocessor 
techniques used in C/C++ to create windows dlls.

Destroy.

Kind Regards
Benjamin Thaut
Aug 27 2013
next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Well, a link would have been great:

http://wiki.dlang.org/DIP45
Aug 27 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 27 August 2013 at 10:16:24 UTC, Benjamin Thaut wrote:

 Well, a link would have been great:

 http://wiki.dlang.org/DIP45


I don't think that is the way to go.

We need a way to mark a module as being an "header" import for a 
shared lib.

With that solution, you'll ends up duplicating all modules with 
import and export, and putting the export(foobar) all over the 
place in a module.
Aug 27 2013
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 27.08.2013 13:28, schrieb deadalnix:

 On Tuesday, 27 August 2013 at 10:16:24 UTC, Benjamin Thaut wrote:

 Well, a link would have been great:

 http://wiki.dlang.org/DIP45


 I don't think that is the way to go.

 We need a way to mark a module as being an "header" import for a shared
 lib.

 With that solution, you'll ends up duplicating all modules with import
 and export, and putting the export(foobar) all over the place in a module.


Well ideally it should be enough to put one

export(foobar): at the top of the file.

Also I don't see why there is any duplication of modules involved? What 
exactly do you mean?

And how should your header import for a shared lib work, if you link 
against exactly that lib statically?

Kind Regards
Benjamin Thaut
Aug 27 2013
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 8/27/13, Benjamin Thaut <code benjamin-thaut.de> wrote:

 Well, a link would have been great:

 http://wiki.dlang.org/DIP45


Using export(identifier) is not going to be reliable, since chances of
clashes are high.
E.g. if libFoo uses export(Foo) and libBar also uses export(Foo), you
won't be able to distinguish between the two.

Instead I think 'identifier' should be a module name.

However, I have a better idea. Why not only introduce compiler
switches that are based on module names rather than having to annotate
what export does in code? For example:

dmd -m64 -export libB.* -import libA.* -of"libB.dll" dllmain.d libB.d
-L/DLL -L/IMPLIB:"libB.lib" -LlibA.lib

"-export libB.*" means all modules and subpackages of the libB package
should be exported, whereas "-import libA.*" means the opposite for
libA.

This way you don't have to edit any existing code.
Aug 27 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 27.08.2013 15:33, schrieb Andrej Mitrovic:

 On 8/27/13, Benjamin Thaut <code benjamin-thaut.de> wrote:

 Well, a link would have been great:

 http://wiki.dlang.org/DIP45


 Using export(identifier) is not going to be reliable, since chances of
 clashes are high.
 E.g. if libFoo uses export(Foo) and libBar also uses export(Foo), you
 won't be able to distinguish between the two.

 Instead I think 'identifier' should be a module name.

 However, I have a better idea. Why not only introduce compiler
 switches that are based on module names rather than having to annotate
 what export does in code? For example:

 dmd -m64 -export libB.* -import libA.* -of"libB.dll" dllmain.d libB.d
 -L/DLL -L/IMPLIB:"libB.lib" -LlibA.lib

 "-export libB.*" means all modules and subpackages of the libB package
 should be exported, whereas "-import libA.*" means the opposite for
 libA.

 This way you don't have to edit any existing code.


I thought about something along those lines aswell. But wouldn't that 
create a very long command line if you do it for druntime?
Aug 27 2013
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 27.08.2013 15:40, Benjamin Thaut wrote:

 Am 27.08.2013 15:33, schrieb Andrej Mitrovic:

 On 8/27/13, Benjamin Thaut <code benjamin-thaut.de> wrote:

 Well, a link would have been great:

 http://wiki.dlang.org/DIP45


 Using export(identifier) is not going to be reliable, since chances of
 clashes are high.
 E.g. if libFoo uses export(Foo) and libBar also uses export(Foo), you
 won't be able to distinguish between the two.

 Instead I think 'identifier' should be a module name.

 However, I have a better idea. Why not only introduce compiler
 switches that are based on module names rather than having to annotate
 what export does in code? For example:

 dmd -m64 -export libB.* -import libA.* -of"libB.dll" dllmain.d libB.d
 -L/DLL -L/IMPLIB:"libB.lib" -LlibA.lib

 "-export libB.*" means all modules and subpackages of the libB package
 should be exported, whereas "-import libA.*" means the opposite for
 libA.

 This way you don't have to edit any existing code.


 I thought about something along those lines aswell. But wouldn't that
 create a very long command line if you do it for druntime?


I think export (at module granularity) can be handled by building the 
modules with symbols to export using a command line switch "-exportall", 
and all other modules without.

I'd prefer if the compilation against the imported library would be 
agnostic whether it is later linked against a static or dynamic library.

How is this done on linux right now? It does not need "export"/"import" 
to build against a shared phobos library. Is "import" assumed for any 
data access and later removed by some magic in the linker?
Aug 28 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-08-29 08:43, Rainer Schuetze wrote:


 How is this done on linux right now? It does not need "export"/"import"
 to build against a shared phobos library. Is "import" assumed for any
 data access and later removed by some magic in the linker?


"export" is noop on Posix. Every symbol is accessible.

-- 
/Jacob Carlborg
Aug 28 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 29 August 2013 at 06:54:46 UTC, Jacob Carlborg wrote:

 On 2013-08-29 08:43, Rainer Schuetze wrote:


 How is this done on linux right now? It does not need 
 "export"/"import"
 to build against a shared phobos library. Is "import" assumed 
 for any
 data access and later removed by some magic in the linker?


 "export" is noop on Posix. Every symbol is accessible.


C++ is doing its best to move away from it. We should probably 
don't follow that convention.
Aug 29 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 29.08.2013 09:24, schrieb deadalnix:

 On Thursday, 29 August 2013 at 06:54:46 UTC, Jacob Carlborg wrote:

 On 2013-08-29 08:43, Rainer Schuetze wrote:


 How is this done on linux right now? It does not need "export"/"import"
 to build against a shared phobos library. Is "import" assumed for any
 data access and later removed by some magic in the linker?


 "export" is noop on Posix. Every symbol is accessible.


 C++ is doing its best to move away from it. We should probably don't
 follow that convention.


Moving away from what exactly?

-- 
Kind Regards
Benjamin Thaut
Aug 29 2013
parent reply "David Nadlinger" <code klickverbot.at> writes:
On Thursday, 29 August 2013 at 08:36:03 UTC, Benjamin Thaut wrote:

 Am 29.08.2013 09:24, schrieb deadalnix:

 C++ is doing its best to move away from it. We should probably 
 don't
 follow that convention.


 Moving away from what exactly?


Everything being visible by default.
Aug 29 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 29.08.2013 10:46, schrieb David Nadlinger:

 On Thursday, 29 August 2013 at 08:36:03 UTC, Benjamin Thaut wrote:

 Am 29.08.2013 09:24, schrieb deadalnix:

 C++ is doing its best to move away from it. We should probably don't
 follow that convention.


 Moving away from what exactly?


 Everything being visible by default.


So what should be do? So far I like Andrej Mitrovic's solution best. But 
I think we should omit the placement holder so that instead of "-export 
std.*" we use "-export std" which basically means the same thing. 
Otherwise people might geht the idea stuff like this will work "-export 
*.somesubmodule".

Personally I'm ok with any working solution. I wanted to use shared dlls 
since I started using D 3 years ago, and nothing has happend in that 
regard since then.

Kind Regards
Benjamin Thaut
Aug 29 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 29 August 2013 at 09:13:02 UTC, Benjamin Thaut wrote:

 So what should be do? So far I like Andrej Mitrovic's solution 
 best. But I think we should omit the placement holder so that 
 instead of "-export std.*" we use "-export std" which basically 
 means the same thing. Otherwise people might geht the idea 
 stuff like this will work "-export *.somesubmodule".

 Personally I'm ok with any working solution. I wanted to use 
 shared dlls since I started using D 3 years ago, and nothing 
 has happend in that regard since then.

 Kind Regards
 Benjamin Thaut


Here is my proposal.

Everything is NOT exported on unixes system, just like on 
windows. It is consistent accross system and provide 
opportunities for optimizations and faster linkage.

symobol marked as export are understood as ddlexport (or publicly 
visible on unixes)

import declaration marked as export as understood as importing a 
module compiled in a shared object. This time, declaration in the 
imported mopdule marked as export are understood as ddlimport.
Aug 29 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 29.08.2013 11:42, schrieb deadalnix:

 import declaration marked as export as understood as importing a module
 compiled in a shared object. This time, declaration in the imported
 mopdule marked as export are understood as ddlimport.


But what if you import a module that is linked statically? That would 
mean export would be treated as dllimport and it will fail to link 
because the _imp_ symbols are missing when linking statically?
Aug 29 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 08/29/2013 12:03 PM, Benjamin Thaut wrote:

 But what if you import a module that is linked statically? That would
 mean export would be treated as dllimport and it will fail to link
 because the _imp_ symbols are missing when linking statically?


Could we create alias symbols?
Aug 29 2013
next sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 29.08.2013 21:12, schrieb Martin Nowak:

 On 08/29/2013 12:03 PM, Benjamin Thaut wrote:

 But what if you import a module that is linked statically? That would
 mean export would be treated as dllimport and it will fail to link
 because the _imp_ symbols are missing when linking statically?


 Could we create alias symbols?


Yes, but that would add a indirection to all data accesses even if you 
link the library in statically. That would basically remove the benefit 
from linking statically. If we want control over what gets exported and 
what not we will need export anyway, and in my eyes a additional 
compiler switch for windows is not bad enough to justify adding a level 
of indirction to data accesses into static libraries.

Kind Regards
Benjamin Thaut
Aug 29 2013
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On 08/29/2013 09:12 PM, Martin Nowak wrote:

 On 08/29/2013 12:03 PM, Benjamin Thaut wrote:

 But what if you import a module that is linked statically? That would
 mean export would be treated as dllimport and it will fail to link
 because the _imp_ symbols are missing when linking statically?


 Could we create alias symbols?


Indeed this seems to work.
OMF has an ALIAS record (http://www.azillionmonkeys.com/qed/Omfg.pdf)
and COFF has weak externals (http://blog.omega-prime.co.uk/?p=121).

So we could add _imp_* aliases for every exported symbol.
When someone links against the import library they are never used, when 
someone links against a static library they redirect to the actual 
definitions. Would that work?
Aug 29 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 29.08.2013 21:20, schrieb Martin Nowak:

 On 08/29/2013 09:12 PM, Martin Nowak wrote:

 On 08/29/2013 12:03 PM, Benjamin Thaut wrote:

 But what if you import a module that is linked statically? That would
 mean export would be treated as dllimport and it will fail to link
 because the _imp_ symbols are missing when linking statically?


 Could we create alias symbols?


 Indeed this seems to work.
 OMF has an ALIAS record (http://www.azillionmonkeys.com/qed/Omfg.pdf)
 and COFF has weak externals (http://blog.omega-prime.co.uk/?p=121).

 So we could add _imp_* aliases for every exported symbol.
 When someone links against the import library they are never used, when
 someone links against a static library they redirect to the actual
 definitions. Would that work?


Well no its not that simple. You can't just alias it. You still have to 
create a import table. That would mean ALL data accesses to data symbols 
would have another level of indirection. Always. The _imp_ symbols don't 
refer to the data directly, they refer to the location the data is 
stored at. As explained in the article I linked 
(http://blog.omega-prime.co.uk/?p=115) this requires different assembly 
to be generated. To make this work we would always have to generate 
assembly with one level of indirection added for every data symbol 
access, because we can not know if the symbol might be imported from a 
DLL or not. As D tries to achive near C++ performance I would arther 
want to avoid that.
Aug 29 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 08/29/2013 09:28 PM, Benjamin Thaut wrote:

 Well no its not that simple. You can't just alias it. You still have to
 create a import table. That would mean ALL data accesses to data symbols
 would have another level of indirection. Always. The _imp_ symbols don't
 refer to the data directly, they refer to the location the data is
 stored at. As explained in the article I linked
 (http://blog.omega-prime.co.uk/?p=115) this requires different assembly
 to be generated. To make this work we would always have to generate
 assembly with one level of indirection added for every data symbol
 access, because we can not know if the symbol might be imported from a
 DLL or not. As D tries to achive near C++ performance I would arther
 want to avoid that.


So the alias trick would only work for functions.
For data the _imp_* symbol would need to be a pointer to the data?
How about LTO, when statically linking it should be possible to optimize 
away the indirection. Also are there special relocations?

module libA;
export int var;
int* _imp_var = &var; // created by compiler

module foo;
import libA;

void bar()
{
     auto val = var; // creates val = *_imp_var;
}
Aug 29 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 29.08.2013 22:04, schrieb Martin Nowak:

 So the alias trick would only work for functions.


yes


 For data the _imp_* symbol would need to be a pointer to the data?


yes


 How about LTO, when statically linking it should be possible to optimize
 away the indirection.


Rainer Schuetze stated that some linkers are capable of doing this 
optimizations. But I don't know aynthing further about this topic.


 module libA;
 export int var;
 int* _imp_var = &var; // created by compiler

 module foo;
 import libA;

 void bar()
 {
      auto val = var; // creates val = *_imp_var;
 }


Yes that would work. Still there must be a reason why microsoft doesn't 
do stuff like that in their C++ toolchain. Its certanly going to cost 
performance.
Aug 29 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 08/29/2013 10:17 PM, Benjamin Thaut wrote:

 How about LTO, when statically linking it should be possible to optimize
 away the indirection.


 Rainer Schuetze stated that some linkers are capable of doing this
 optimizations. But I don't know aynthing further about this topic.


 module libA;
 export int var;
 int* _imp_var = &var; // created by compiler

 module foo;
 import libA;

 void bar()
 {
      auto val = var; // creates val = *_imp_var;
 }


 Yes that would work. Still there must be a reason why microsoft doesn't
 do stuff like that in their C++ toolchain. Its certanly going to cost
 performance.


I just tested it and it works.

lib.c

int var = 0xdeadbeaf;
int* _imp_var = &var;

main.c

#include <stdio.h>

extern int* _imp_var;
void main()
{
     printf("%d\n", *_imp_var);
}

cl /c /O2 /GL lib.c
cl /O2 /GL main.c lib.obj

get objconv from http://www.agner.org/optimize/

objconv -fasm main.exe

Search for deadbeaf in main.asm to get the symbol number (?_1176).
It directly loads the variable.

mov edx, dword ptr [?_1176]
Aug 29 2013
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 29 August 2013 at 21:40:08 UTC, Martin Nowak wrote:

 On 08/29/2013 10:17 PM, Benjamin Thaut wrote:

 How about LTO, when statically linking it should be possible 
 to optimize
 away the indirection.


 Rainer Schuetze stated that some linkers are capable of doing 
 this
 optimizations. But I don't know aynthing further about this 
 topic.


 module libA;
 export int var;
 int* _imp_var = &var; // created by compiler

 module foo;
 import libA;

 void bar()
 {
     auto val = var; // creates val = *_imp_var;
 }


 Yes that would work. Still there must be a reason why 
 microsoft doesn't
 do stuff like that in their C++ toolchain. Its certanly going 
 to cost
 performance.


 I just tested it and it works.

 lib.c

 int var = 0xdeadbeaf;
 int* _imp_var = &var;

 main.c

 #include <stdio.h>

 extern int* _imp_var;
 void main()
 {
     printf("%d\n", *_imp_var);
 }

 cl /c /O2 /GL lib.c
 cl /O2 /GL main.c lib.obj

 get objconv from http://www.agner.org/optimize/

 objconv -fasm main.exe

 Search for deadbeaf in main.asm to get the symbol number 
 (?_1176).
 It directly loads the variable.

 mov edx, dword ptr [?_1176]


I was doubting my idea, but you conviced me :P
Aug 29 2013
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 29.08.2013 23:40, Martin Nowak wrote:

 On 08/29/2013 10:17 PM, Benjamin Thaut wrote:

 How about LTO, when statically linking it should be possible to optimize
 away the indirection.


 Rainer Schuetze stated that some linkers are capable of doing this
 optimizations. But I don't know aynthing further about this topic.


 module libA;
 export int var;
 int* _imp_var = &var; // created by compiler

 module foo;
 import libA;

 void bar()
 {
      auto val = var; // creates val = *_imp_var;
 }


 Yes that would work. Still there must be a reason why microsoft doesn't
 do stuff like that in their C++ toolchain. Its certanly going to cost
 performance.


 I just tested it and it works.

 lib.c

 int var = 0xdeadbeaf;
 int* _imp_var = &var;

 main.c

 #include <stdio.h>

 extern int* _imp_var;
 void main()
 {
      printf("%d\n", *_imp_var);
 }

 cl /c /O2 /GL lib.c
 cl /O2 /GL main.c lib.obj

 get objconv from http://www.agner.org/optimize/

 objconv -fasm main.exe

 Search for deadbeaf in main.asm to get the symbol number (?_1176).
 It directly loads the variable.

 mov edx, dword ptr [?_1176]


It's a bit easier to see the code by adding debug symbols and using 
dumpbin /disasm main.exe.

Unfortnately, this won't help us a lot, because the intermediate object 
files have some unknown format and in fact are just transferring some 
code representation to the linker that then invokes the compiler and 
optimizer on the full source code. This is very specific to the C/C++ 
toolchain and I don't think we can take advantage of it.
Aug 29 2013
parent Martin Nowak <code dawg.eu> writes:
On 08/30/2013 08:20 AM, Rainer Schuetze wrote:

 It's a bit easier to see the code by adding debug symbols and using
 dumpbin /disasm main.exe.

 Unfortnately, this won't help us a lot, because the intermediate object
 files have some unknown format and in fact are just transferring some
 code representation to the linker that then invokes the compiler and
 optimizer on the full source code. This is very specific to the C/C++
 toolchain and I don't think we can take advantage of it.


This is sadly true for the dmd->COFF->VC link toolchain.
But we'll likely see this from GDC or LDC in the future.
Despite that I don't think this is a huge performance issue at all
we could put all the import data pointers in one section so that they 
are all packed and allow efficient use of the CPU cache.
Aug 30 2013
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 29.08.2013 08:54, Jacob Carlborg wrote:

 On 2013-08-29 08:43, Rainer Schuetze wrote:


 How is this done on linux right now? It does not need "export"/"import"
 to build against a shared phobos library. Is "import" assumed for any
 data access and later removed by some magic in the linker?


 "export" is noop on Posix. Every symbol is accessible.


I meant the import part. How are accesses or data references to data 
inside another shared library implemented?
Aug 29 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 08/29/2013 08:30 PM, Rainer Schuetze wrote:

 I meant the import part. How are accesses or data references to data
 inside another shared library implemented?


References from the data segment use absolute relocations.
References from PIC code use the GOT.
Aug 29 2013
next sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 29.08.2013 21:09, Martin Nowak wrote:

 On 08/29/2013 08:30 PM, Rainer Schuetze wrote:

 I meant the import part. How are accesses or data references to data
 inside another shared library implemented?


 References from the data segment use absolute relocations.
 References from PIC code use the GOT.


So an indirection through the GOT is added to every access to a 
non-static global if you compile with -fPIC?
Aug 29 2013
parent "Wyatt" <wyatt.epp gmail.com> writes:
On Friday, 30 August 2013 at 06:23:02 UTC, Rainer Schuetze wrote:

 On 29.08.2013 21:09, Martin Nowak wrote:

 References from the data segment use absolute relocations.
 References from PIC code use the GOT.


 So an indirection through the GOT is added to every access to a 
 non-static global if you compile with -fPIC?


My understanding was references using the GOT have the same 
number of indirections as in the PLT -- that is, a one-time cost 
for the dynamic linker to dereference and replace an entry with 
the absolute address -- and the major difference is PLT is done 
lazily, while GOT is fixed up in its entirety at runtime.

Having said that, now I wonder if I'm misunderstanding the 
question?

-Wyatt
Aug 30 2013
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 29.08.2013 21:09, Martin Nowak wrote:

 On 08/29/2013 08:30 PM, Rainer Schuetze wrote:

 I meant the import part. How are accesses or data references to data
 inside another shared library implemented?


 References from the data segment use absolute relocations.


Just remembered that we don't have that in Windows, so if we want to 
mimick what C++ does, we'll have to generate initialization-code to be 
run at startup to fill in the proper pointers from the import table.
Aug 30 2013
next sibling parent Martin Nowak <code dawg.eu> writes:
On 08/30/2013 12:50 PM, Rainer Schuetze wrote:

 References from the data segment use absolute relocations.


 Just remembered that we don't have that in Windows, so if we want to
 mimick what C++ does, we'll have to generate initialization-code to be
 run at startup to fill in the proper pointers from the import table.


This is also done at runtime by the loader. The additional startup time 
is one reason why exporting data in ABIs is bad. The other is copy 
relocations for writeable data 
(http://docs.oracle.com/cd/E19082-01/819-0690/chapter4-84604/index.html).
Aug 30 2013
prev sibling parent reply "Kagamin" <spam here.lot> writes:
On Friday, 30 August 2013 at 10:50:40 UTC, Rainer Schuetze wrote:

 Just remembered that we don't have that in Windows


Why? What's the difference between code and data symbols? I 
thought, they're just addresses.
Sep 02 2013
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 03.09.2013 08:17, Kagamin wrote:

 On Friday, 30 August 2013 at 10:50:40 UTC, Rainer Schuetze wrote:

 Just remembered that we don't have that in Windows


 Why? What's the difference between code and data symbols? I thought,
 they're just addresses.


Inside an executable/DLL image, all relocations are relative to the 
image itself, only the import table has references to other DLLs.

To initialize data with a pointer into another DLL, you cannot do that 
with a direct relocation, you have to run some code to copy the value 
from the import table into the data.

Accessing data or functions in another DLL from code already has this 
extra code compiled in.
Sep 03 2013
parent "Kagamin" <spam here.lot> writes:
On Tuesday, 3 September 2013 at 07:02:33 UTC, Rainer Schuetze 
wrote:

 To initialize data with a pointer into another DLL, you cannot 
 do that with a direct relocation, you have to run some code to 
 copy the value from the import table into the data.


Well, yes, static initialization is problematic, but you can have 
statically initialized pointer to an imported function, which is 
the same problem.
Sep 11 2013
prev sibling parent reply "Kagamin" <spam here.lot> writes:
On Thursday, 29 August 2013 at 06:43:39 UTC, Rainer Schuetze 
wrote:

 How is this done on linux right now? It does not need 
 "export"/"import" to build against a shared phobos library. Is 
 "import" assumed for any data access and later removed by some 
 magic in the linker?


I guess, it adds missing functions which jump to the actual 
implementation in so. This is done on windows too in import 
libraries, but export qualifier makes code a little faster 
because you make one indirect call instead of a direct call + 
indirect jump (it can be direct, but for this loader should be 
able to write to the code section to relocate the jump). You also 
only need an import table for such call.
Aug 29 2013
parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 29.08.2013 11:33, Kagamin wrote:

 On Thursday, 29 August 2013 at 06:43:39 UTC, Rainer Schuetze wrote:

 How is this done on linux right now? It does not need
 "export"/"import" to build against a shared phobos library. Is
 "import" assumed for any data access and later removed by some magic
 in the linker?


 I guess, it adds missing functions which jump to the actual
 implementation in so. This is done on windows too in import libraries,
 but export qualifier makes code a little faster because you make one
 indirect call instead of a direct call + indirect jump (it can be
 direct, but for this loader should be able to write to the code section
 to relocate the jump). You also only need an import table for such call.


That's how it works for functions, but what about accesses to data in 
another DLL?

Suppose phobos being built as a shared library, how do you access 
typeid(Object) from another DLL, both from code or inside data, e.g. as 
the base class member of a class info struct?
Aug 29 2013
prev sibling next sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
I found a very good articel on the dllimport/dllexport issue on windows 
explaining both dynamic linking of function symbols and data symbols:

http://blog.omega-prime.co.uk/?p=115
Aug 29 2013
prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
So let me summarize the currently discussed solutions:

1) Use alias symbols
Pros:
  - Better usability (no additional command line parameters needed when 
compiling / linking against DLLs)

Cons:
  - Less efficient code for cross DLL function calls (one additional 
call instruction)
  - Less efficient code for all accesses to global data, no matter if it 
will end up in a DLL or not. That means __gshared variables, global 
shared variables, hidden global data like module info, type info, 
vtables, etc (one additional level of indirection). It might be possible 
to avoid this in most cases using link time optimization. But it is 
unlikely that we can get LTO to work easily.
  - Additional level of indirection might confuse current debuggers

2) Use additional command line
Pros:
  - More efficient code

Cons:
  - Additional command line parameters needed when compiling / linking 
against a DLL. (can be hidden away inside sc.ini for phobos / druntime)
Aug 30 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 30 August 2013 at 08:23:20 UTC, Benjamin Thaut wrote:

 So let me summarize the currently discussed solutions:

 1) Use alias symbols
 Pros:
  - Better usability (no additional command line parameters 
 needed when compiling / linking against DLLs)

 Cons:
  - Less efficient code for cross DLL function calls (one 
 additional call instruction)
  - Less efficient code for all accesses to global data, no 
 matter if it will end up in a DLL or not. That means __gshared 
 variables, global shared variables, hidden global data like 
 module info, type info, vtables, etc (one additional level of 
 indirection). It might be possible to avoid this in most cases 
 using link time optimization. But it is unlikely that we can 
 get LTO to work easily.
  - Additional level of indirection might confuse current 
 debuggers


My understanding is that it can be optimized away. Or can it ?


 2) Use additional command line
 Pros:
  - More efficient code

 Cons:
  - Additional command line parameters needed when compiling / 
 linking against a DLL. (can be hidden away inside sc.ini for 
 phobos / druntime)
Aug 30 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 30.08.2013 11:17, schrieb deadalnix:

 My understanding is that it can be optimized away. Or can it ?


It can in C/C++ because microsoft has really good LTO (link time 
optimization). As Rainer Schuetze explained the format of the object 
files used for LTO is propriatary and unkown to the public. So we can 
not easily use that, and its unlikely we can use it at all. Also for 
32-bit windows it can not be optimized away because optlink does not do LTO.

In theory it could be optimized away, but in pratice I doubt we are 
going to see this kind of optimization in dmd any time soon. And as dmd 
is the only viable option on windows right now I would prefer the better 
performing second solution to this problem
Aug 30 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 30 August 2013 at 09:33:25 UTC, Benjamin Thaut wrote:

 Am 30.08.2013 11:17, schrieb deadalnix:

 My understanding is that it can be optimized away. Or can it ?


 It can in C/C++ because microsoft has really good LTO (link 
 time optimization). As Rainer Schuetze explained the format of 
 the object files used for LTO is propriatary and unkown to the 
 public. So we can not easily use that, and its unlikely we can 
 use it at all. Also for 32-bit windows it can not be optimized 
 away because optlink does not do LTO.


win32 is kind of a dead end anyway. It is still very alive today, 
but we want to aim for the future. And it WILL disappear.

The object format is kind of anoying.


 In theory it could be optimized away, but in pratice I doubt we 
 are going to see this kind of optimization in dmd any time 
 soon. And as dmd is the only viable option on windows right now 
 I would prefer the better performing second solution to this 
 problem


I understand. But sacrificing good design for immediate reward is 
bound to cost way more in the long run if D is successful. And if 
D fails, then it doesn't matter.
Aug 30 2013
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 30.08.2013 11:52, schrieb deadalnix:

 On Friday, 30 August 2013 at 09:33:25 UTC, Benjamin Thaut wrote:

 Am 30.08.2013 11:17, schrieb deadalnix:

 My understanding is that it can be optimized away. Or can it ?


 It can in C/C++ because microsoft has really good LTO (link time
 optimization). As Rainer Schuetze explained the format of the object
 files used for LTO is propriatary and unkown to the public. So we can
 not easily use that, and its unlikely we can use it at all. Also for
 32-bit windows it can not be optimized away because optlink does not
 do LTO.


 win32 is kind of a dead end anyway. It is still very alive today, but we
 want to aim for the future. And it WILL disappear.

 The object format is kind of anoying.


 In theory it could be optimized away, but in pratice I doubt we are
 going to see this kind of optimization in dmd any time soon. And as
 dmd is the only viable option on windows right now I would prefer the
 better performing second solution to this problem


 I understand. But sacrificing good design for immediate reward is bound
 to cost way more in the long run if D is successful. And if D fails,
 then it doesn't matter.


I don't consider good it good design to add lots of indirections just to 
remove a command line parameter to the compiler. Its not like its going 
to make any difference in source code.
Aug 30 2013
prev sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 30.08.2013 10:23, schrieb Benjamin Thaut:

 1) Use alias symbols


I'm convinced now that Martin Nowaks aliasing solution is the better 
solution as there will be less runtime overhead then I originally 
thought. Also the usability will be better. It also leaves room for 
further optimizations by doing some clever pointer patching, in case we 
find a solution that works on x64.

If no one does a veto I'm going to update the DIP with this aliasing 
solution and all other discussed changes. (e.g. making export a attribute)

Kind Regards
Benjamin Thaut


-- 
Kind Regards
Benjamin Thaut
Aug 30 2013
prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
I updated the DIP with all discussed content. Feedback is welcome.

http://wiki.dlang.org/DIP45

Kind Regards
Benjamin Thaut
Sep 01 2013
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 01.09.2013 11:49, Benjamin Thaut wrote:

 I updated the DIP with all discussed content. Feedback is welcome.

 http://wiki.dlang.org/DIP45

 Kind Regards
 Benjamin Thaut


LGTM.

What about the current use case of just exporting some functions for 
being loaded by a C application? It should still work using a def-file 
(which you have to use anyway most of the time due to 
http://d.puremagic.com/issues/show_bug.cgi?id=3956)

Maybe already deep into implementation but how are we going to deal with 
the initialization of data?

module sharedlib;
__gshared int var;

module exe;
import sharedlib;
__gshared int* pvar = &var;

(This kind of relocation also has to be done for stuff like 
TypeInfo_Class). It needs initialization code to fill pvar with the 
correct pointer.

Do we support the same for pointers in TLS?

int* tlsvar = &var;

This might need initialization per thread, or patching the original TLS 
segment.
Sep 01 2013
next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 01.09.2013 17:26, schrieb Rainer Schuetze:

 On 01.09.2013 11:49, Benjamin Thaut wrote:

 I updated the DIP with all discussed content. Feedback is welcome.

 http://wiki.dlang.org/DIP45

 Kind Regards
 Benjamin Thaut


 LGTM.


Lets get this merged? ^^ First someone has to implement it ;-)


 What about the current use case of just exporting some functions for
 being loaded by a C application? It should still work using a def-file
 (which you have to use anyway most of the time due to
 http://d.puremagic.com/issues/show_bug.cgi?id=3956)


Well you can still use a .def file of course. Also no one stops you just 
annotating simple extern(C) functions with the export attribute. I don't 
see any problem there.


 Maybe already deep into implementation but how are we going to deal with
 the initialization of data?

 module sharedlib;
 __gshared int var;

 module exe;
 import sharedlib;
 __gshared int* pvar = &var;

 (This kind of relocation also has to be done for stuff like
 TypeInfo_Class). It needs initialization code to fill pvar with the
 correct pointer.


Yes, you are absolutely correct. I don't know enough about the internals 
of a compiler (dmd in perticular) though to answer this question. Can't 
we use the same mechanism C++ uses to run all its initializers?


 Do we support the same for pointers in TLS?

 int* tlsvar = &var;

 This might need initialization per thread, or patching the original TLS
 segment.


The DIP has a seperate section on TLS variables. Did you miss to read that?

Kind Regards
Benjamin Thaut
Sep 01 2013
next sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 01.09.2013 18:43, Benjamin Thaut wrote:

 Am 01.09.2013 17:26, schrieb Rainer Schuetze:

 On 01.09.2013 11:49, Benjamin Thaut wrote:

 I updated the DIP with all discussed content. Feedback is welcome.

 http://wiki.dlang.org/DIP45

 Kind Regards
 Benjamin Thaut


 LGTM.


 Lets get this merged? ^^ First someone has to implement it ;-)


 What about the current use case of just exporting some functions for
 being loaded by a C application? It should still work using a def-file
 (which you have to use anyway most of the time due to
 http://d.puremagic.com/issues/show_bug.cgi?id=3956)


 Well you can still use a .def file of course. Also no one stops you just
 annotating simple extern(C) functions with the export attribute. I don't
 see any problem there.


It would export the ModuleInfo and related symbols.


 Maybe already deep into implementation but how are we going to deal with
 the initialization of data?

 module sharedlib;
 __gshared int var;

 module exe;
 import sharedlib;
 __gshared int* pvar = &var;

 (This kind of relocation also has to be done for stuff like
 TypeInfo_Class). It needs initialization code to fill pvar with the
 correct pointer.


 Yes, you are absolutely correct. I don't know enough about the internals
 of a compiler (dmd in perticular) though to answer this question. Can't
 we use the same mechanism C++ uses to run all its initializers?


Yes, it would be an option to generate init functions to be run with the 
C initializers. But that might introduce a dependency on initialization 
order. We can write relocation info into another segment and do the 
relocations before runtime init.


 Do we support the same for pointers in TLS?

 int* tlsvar = &var;

 This might need initialization per thread, or patching the original TLS
 segment.


 The DIP has a seperate section on TLS variables. Did you miss to read that?


No. I meant thread global variables that are preinitialized. The 
appropriate relocation doesn't exist, the __acccess_tls_var function has 
to be called in a thread local init function.

Rainer
Sep 01 2013
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 01.09.2013 19:58, schrieb Rainer Schuetze:

 On 01.09.2013 18:43, Benjamin Thaut wrote:

 Am 01.09.2013 17:26, schrieb Rainer Schuetze:

 On 01.09.2013 11:49, Benjamin Thaut wrote:

 I updated the DIP with all discussed content. Feedback is welcome.

 http://wiki.dlang.org/DIP45

 Kind Regards
 Benjamin Thaut


 LGTM.


 Lets get this merged? ^^ First someone has to implement it ;-)


 What about the current use case of just exporting some functions for
 being loaded by a C application? It should still work using a def-file
 (which you have to use anyway most of the time due to
 http://d.puremagic.com/issues/show_bug.cgi?id=3956)


 Well you can still use a .def file of course. Also no one stops you just
 annotating simple extern(C) functions with the export attribute. I don't
 see any problem there.


 It would export the ModuleInfo and related symbols.


Is this a issue? There just would be some unused exports in the dll, 
wouldn't it? If this really is a problem we could add a compiler switch 
to stop implict exporting of symbols.


 Maybe already deep into implementation but how are we going to deal with
 the initialization of data?

 module sharedlib;
 __gshared int var;

 module exe;
 import sharedlib;
 __gshared int* pvar = &var;

 (This kind of relocation also has to be done for stuff like
 TypeInfo_Class). It needs initialization code to fill pvar with the
 correct pointer.


 Yes, you are absolutely correct. I don't know enough about the internals
 of a compiler (dmd in perticular) though to answer this question. Can't
 we use the same mechanism C++ uses to run all its initializers?


 Yes, it would be an option to generate init functions to be run with the
 C initializers. But that might introduce a dependency on initialization
 order. We can write relocation info into another segment and do the
 relocations before runtime init.


I would say whatever works best for the plattform at hand. No need to 
force one way or the other.


 Do we support the same for pointers in TLS?

 int* tlsvar = &var;

 This might need initialization per thread, or patching the original TLS
 segment.


 The DIP has a seperate section on TLS variables. Did you miss to read
 that?


 No. I meant thread global variables that are preinitialized. The
 appropriate relocation doesn't exist, the __acccess_tls_var function has
 to be called in a thread local init function.


Can't we initialize these right before running the shared module 
constructors? To make the implementation less work we could also make 
this a error for now and maybe implement it later. What would you prefer?


 Rainer
Sep 01 2013
prev sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Sunday, 1 September 2013 at 16:43:14 UTC, Benjamin Thaut wrote:

 The DIP has a seperate section on TLS variables. Did you miss 
 to read that?


Can't we just use the normal (g)libc facilities for handling TLS 
in shared objects on Linux? The TLS data from the "DLL" is 
assigned to another tls module, and everything is fine, or am I 
missing something?

David
Sep 01 2013
parent reply Iain Buclaw <ibuclaw ubuntu.com> writes:
On 1 September 2013 19:16, David Nadlinger <code klickverbot.at> wrote:

 On Sunday, 1 September 2013 at 16:43:14 UTC, Benjamin Thaut wrote:

 The DIP has a seperate section on TLS variables. Did you miss to read
 that?



 Can't we just use the normal (g)libc facilities for handling TLS in shared
 objects on Linux? The TLS data from the "DLL" is assigned to another tls
 module, and everything is fine, or am I missing something?

 David


Yes, no need to invent a new cog to go on top of what the base runtime
already offers.

-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';
Sep 01 2013
next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 01.09.2013 21:19, schrieb Iain Buclaw:

 On 1 September 2013 19:16, David Nadlinger <code klickverbot.at> wrote:

 On Sunday, 1 September 2013 at 16:43:14 UTC, Benjamin Thaut wrote:

 The DIP has a seperate section on TLS variables. Did you miss to read
 that?



 Can't we just use the normal (g)libc facilities for handling TLS in shared
 objects on Linux? The TLS data from the "DLL" is assigned to another tls
 module, and everything is fine, or am I missing something?

 David


 Yes, no need to invent a new cog to go on top of what the base runtime
 already offers.


I didn't know linux has a facility for accessing TLS data across shared 
library boundaries. I will put the implementation suggestion into the 
windows specific section then.
Sep 01 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 09/01/2013 09:33 PM, Benjamin Thaut wrote:

 I didn't know linux has a facility for accessing TLS data across shared
 library boundaries. I will put the implementation suggestion into the
 windows specific section then.


Yes, it has a bunch of special relocations and runtime linker support 
for this.
It would be nice if someone could figure out and summarize all the TLS 
index and TIB business on Windows.

http://msdn.microsoft.com/en-us/library/windows/desktop/ms686997(v=vs.85).aspx
http://en.wikipedia.org/wiki/Thread-local_storage#Windows_implementation
http://en.wikipedia.org/wiki/Win32_Thread_Information_Block
Sep 05 2013
next sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 06.09.2013 04:00, schrieb Martin Nowak:

 On 09/01/2013 09:33 PM, Benjamin Thaut wrote:

 I didn't know linux has a facility for accessing TLS data across shared
 library boundaries. I will put the implementation suggestion into the
 windows specific section then.


 Yes, it has a bunch of special relocations and runtime linker support
 for this.
 It would be nice if someone could figure out and summarize all the TLS
 index and TIB business on Windows.

 http://msdn.microsoft.com/en-us/library/windows/desktop/ms686997(v=vs.85).aspx

 http://en.wikipedia.org/wiki/Thread-local_storage#Windows_implementation
 http://en.wikipedia.org/wiki/Win32_Thread_Information_Block


What do we need this information for?

-- 
Kind Regards
Benjamin Thaut
Sep 06 2013
prev sibling parent "David Nadlinger" <code klickverbot.at> writes:
On Friday, 6 September 2013 at 02:00:22 UTC, Martin Nowak wrote:

 Yes, it has a bunch of special relocations and runtime linker 
 support for this.
 It would be nice if someone could figure out and summarize all 
 the TLS index and TIB business on Windows.

 http://msdn.microsoft.com/en-us/library/windows/desktop/ms686997(v=vs.85).aspx
 http://en.wikipedia.org/wiki/Thread-local_storage#Windows_implementation
 http://en.wikipedia.org/wiki/Win32_Thread_Information_Block


Best material I found on Windows TLS so far: 
http://www.nynaeve.net/?s=tls

Haven't really looked into the interaction between multiple DLLs 
at all so far, though.

David
Sep 07 2013
prev sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 01.09.2013 21:19, schrieb Iain Buclaw:

 On 1 September 2013 19:16, David Nadlinger <code klickverbot.at> wrote:

 On Sunday, 1 September 2013 at 16:43:14 UTC, Benjamin Thaut wrote:

 The DIP has a seperate section on TLS variables. Did you miss to read
 that?



 Can't we just use the normal (g)libc facilities for handling TLS in shared
 objects on Linux? The TLS data from the "DLL" is assigned to another tls
 module, and everything is fine, or am I missing something?

 David


 Yes, no need to invent a new cog to go on top of what the base runtime
 already offers.


Basically I thought whatever is inside the Description would go into the 
spec. Everything inside the implementation details is just a suggestion 
for implementing proper dllexport / dllimport on windows plattforms. We 
shouldn't hard wire these implementation suggestions into the spec.
Sep 01 2013
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On 09/01/2013 05:26 PM, Rainer Schuetze wrote:

 Maybe already deep into implementation but how are we going to deal with
 the initialization of data?

 module sharedlib;
 __gshared int var;

 module exe;
 import sharedlib;
 __gshared int* pvar = &var;

 (This kind of relocation also has to be done for stuff like
 TypeInfo_Class). It needs initialization code to fill pvar with the
 correct pointer.


Based on your saying that there are no absolute data relocations for 
this on windows we'd have to perform runtime initialization.
At runtime it would simply copy _imp_var into pvar, right?


 Do we support the same for pointers in TLS?

 int* tlsvar = &var;

 This might need initialization per thread, or patching the original TLS
 segment.


No you can't take the address of a TLS var at compile time.
Sep 05 2013
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 06.09.2013 03:51, schrieb Martin Nowak:

 On 09/01/2013 05:26 PM, Rainer Schuetze wrote:

 Maybe already deep into implementation but how are we going to deal with
 the initialization of data?

 module sharedlib;
 __gshared int var;

 module exe;
 import sharedlib;
 __gshared int* pvar = &var;

 (This kind of relocation also has to be done for stuff like
 TypeInfo_Class). It needs initialization code to fill pvar with the
 correct pointer.


 Based on your saying that there are no absolute data relocations for
 this on windows we'd have to perform runtime initialization.
 At runtime it would simply copy _imp_var into pvar, right?


Yes. As Rainer already mentioned we can either do this by running some 
initialiers before initializing druntime, or we could add additional 
information in a section of the dll and run some generic dll 
initialization code.


 Do we support the same for pointers in TLS?

 int* tlsvar = &var;

 This might need initialization per thread, or patching the original TLS
 segment.


 No you can't take the address of a TLS var at compile time.


Its not taking a address of a TLS variable. It is initializing a TLS 
variable with the address of a __gshared variable.


-- 
Kind Regards
Benjamin Thaut
Sep 06 2013
prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Ok there is a new problem. Consider the following example:

module lib1;

export int lib1Func()
{
   return 5;
}


module lib2;
import lib1;

export int lib2Func()
{
   return lib1Func() + 1;
}


If you compile lib1 into a static library and then copmpile lib2 into a 
DLL which statically links against lib1 both symbols lib1Func and 
lib2Func will be exported from the dll because lib1Func will get marked 
as dllexport when compiled into a static library and the static library 
transports that information into the linking process of lib2. This 
happens both on win32 and win64. But this is something you really don't 
want. Lets imagine you have a DLL that statically links against phobos. 
Suddenly you will have all phobos exported symbols in your dll. This can 
also lead to double defined symbols in case a user links against your 
dll and against the shared phobos dll. So we have to fix this somehow.

I propose that we add a command-line-paramter to the compiler (windows 
only) which actually enables dllexport. So all behavior described in the 
DIP will be enabled by default, but to actually make it mark symbols 
with dllexport you have to specifiy a additional command-line-parameter. 
We could call it "-dll" or something.

In case you want to try it out yourself, here is the repro case: 
http://stuff.benjamin-thaut.de/D/dll2.zip

Kind Regards
Benjamin Thaut
Sep 07 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 7 September 2013 at 12:47:19 UTC, Benjamin Thaut 
wrote:

 I propose that we add a command-line-paramter to the compiler 
 (windows only) which actually enables dllexport. So all 
 behavior described in the DIP will be enabled by default, but 
 to actually make it mark symbols with dllexport you have to 
 specifiy a additional command-line-parameter. We could call it 
 "-dll" or something.


I have to think about it. But I'd like to avoid any special 
behavior for windows language side.

I also kind of feel that we need to export lib1 as well, as you 
may not know what lib2 does with what it gets from lib1 (passing 
objects around for instance). So certainly at least some part of 
lib1 need to be exported.
Sep 07 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 07.09.2013 20:42, schrieb deadalnix:

 On Saturday, 7 September 2013 at 12:47:19 UTC, Benjamin Thaut wrote:

 I propose that we add a command-line-paramter to the compiler (windows
 only) which actually enables dllexport. So all behavior described in
 the DIP will be enabled by default, but to actually make it mark
 symbols with dllexport you have to specifiy a additional
 command-line-parameter. We could call it "-dll" or something.


 I have to think about it. But I'd like to avoid any special behavior for
 windows language side.

 I also kind of feel that we need to export lib1 as well, as you may not
 know what lib2 does with what it gets from lib1 (passing objects around
 for instance). So certainly at least some part of lib1 need to be exported.


I don't agree if that statement. If you have three libs. lib1, lib2 and 
lib3. And lib2 and lib3 link statically against lib1 you are going to 
get linker errors because both lib2 and lib3 contain the symbols of 
lib1. And you don't have any options to avoid that (unless you got into 
the source of lib1 and remove all export attributes, which might not 
even be possible because you don't have the source)
Sep 07 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 7 September 2013 at 21:41:01 UTC, Benjamin Thaut 
wrote:

 I don't agree if that statement. If you have three libs. lib1, 
 lib2 and lib3. And lib2 and lib3 link statically against lib1 
 you are going to get linker errors because both lib2 and lib3 
 contain the symbols of lib1. And you don't have any options to 
 avoid that (unless you got into the source of lib1 and remove 
 all export attributes, which might not even be possible because 
 you don't have the source)


And you should have an error, especially is you pass object from 
one side to the other.

typeid won't match (which will create a mess in the runtime), and 
worse, struct layout may not match (hello memory corruption).

If the problem is that big, we can still have a -noexport flag or 
something, but that doesn't seems really safe to me.
Sep 07 2013
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 08.09.2013 06:19, schrieb deadalnix:

 On Saturday, 7 September 2013 at 21:41:01 UTC, Benjamin Thaut wrote:

 I don't agree if that statement. If you have three libs. lib1, lib2
 and lib3. And lib2 and lib3 link statically against lib1 you are going
 to get linker errors because both lib2 and lib3 contain the symbols of
 lib1. And you don't have any options to avoid that (unless you got
 into the source of lib1 and remove all export attributes, which might
 not even be possible because you don't have the source)


 And you should have an error, especially is you pass object from one
 side to the other.

 typeid won't match (which will create a mess in the runtime), and worse,
 struct layout may not match (hello memory corruption).

 If the problem is that big, we can still have a -noexport flag or
 something, but that doesn't seems really safe to me.


But only assuming that you are passing around objects of that type. I 
see a dll as a module of encapsulation. If the library is only unsed 
internally nothing of its dependencies should get exported. Because if 
they are your users might rely on the fact that they are, and from that 
point on you have to garantuee that all those additional exported 
functions from your dependencies stay in your library, even if you don't 
need the dependency anymore.
Sep 08 2013
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
 I also kind of feel that we need to export lib1 as well, as you may not
 know what lib2 does with what it gets from lib1 (passing objects around
 for instance). So certainly at least some part of lib1 need to be
 exported.




For sure lib2 might forward to some lib1 symbols.
Not sure what might accidentally require linkage.


 I don't agree if that statement. If you have three libs. lib1, lib2 and
 lib3. And lib2 and lib3 link statically against lib1 you are going to
 get linker errors because both lib2 and lib3 contain the symbols of lib1.


A: Why do two DLLs which export the same symbols cause linker errors? 
Because the corresponding import libraries contain the same symbols?
B: You shouldn't mix static with dynamic linking, ODR issues are what 
you get in return.
C: What would happen if you statically link against lib2 and lib3?
Sep 09 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 10.09.2013 05:39, schrieb Martin Nowak:

 I also kind of feel that we need to export lib1 as well, as you may not
 know what lib2 does with what it gets from lib1 (passing objects around
 for instance). So certainly at least some part of lib1 need to be
 exported.




 For sure lib2 might forward to some lib1 symbols.
 Not sure what might accidentally require linkage.


 I don't agree if that statement. If you have three libs. lib1, lib2 and
 lib3. And lib2 and lib3 link statically against lib1 you are going to
 get linker errors because both lib2 and lib3 contain the symbols of lib1.


 A: Why do two DLLs which export the same symbols cause linker errors?


Yes

 Because the corresponding import libraries contain the same symbols?
 B: You shouldn't mix static with dynamic linking, ODR issues are what
 you get in return.


Well you should not. But it will happen. Espeically if you don't have 
any control over what others due. E.g. when using third party libraries.


 C: What would happen if you statically link against lib2 and lib3?


Well you would get a linker error as well. But the reasoing behind this 
is as follows.

You use a third party library that comes as a shared library. That third 
party library uses say libcurl internally. Now you use libcurl too. You 
link against libcurl. But because the third party library you link to 
also uses libcurl you will get linker errors. Also you might not want to 
use the same version of libcurl then the thirdparty library you use. 
This is going to become a problem especially in bigger projects, where 
you are not able to compile the third party libraries yourself because 
you don't have the source code. In such a case it can easily happen that 
you get a new version of that third party library, they added a new 
internal dependency to a library you already use and suddenly your 
executable won't linke anymore.

Kind Regards
Benjamin Thaut
Sep 19 2013
parent Martin Nowak <code dawg.eu> writes:
On 09/19/2013 01:55 PM, Benjamin Thaut wrote:

 B: You shouldn't mix static with dynamic linking, ODR issues are what
 you get in return.


 Well you should not. But it will happen. Espeically if you don't have
 any control over what others due. E.g. when using third party libraries.


Let me put it differently, if you statically link against a library in 
your DLL you need to ensure not to leak (export) any symbols from this 
static library.
Nov 16 2013
prev sibling next sibling parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 07.09.2013 14:47, Benjamin Thaut wrote:

 I propose that we add a command-line-paramter to the compiler (windows
 only) which actually enables dllexport. So all behavior described in the
 DIP will be enabled by default, but to actually make it mark symbols
 with dllexport you have to specifiy a additional command-line-parameter.
 We could call it "-dll" or something.


I don't think the issue is Windows specific, symbol visibility in the 
GNU toolchain is also bound to the declaration. I remember seeing public 
(visibility=default) symbols exported from static libraries on OSX.
Sep 07 2013
prev sibling next sibling parent Martin Nowak <code dawg.eu> writes:
On 09/07/2013 02:47 PM, Benjamin Thaut wrote:

 If you compile lib1 into a static library and then copmpile lib2 into a
 DLL which statically links against lib1 both symbols lib1Func and
 lib2Func will be exported from the dll


That might indeed cause some issues, but when you make lib1 part of your 
DLL by statically linking against it that's what you get.
If you just want to use it, you should make lib1 a DLL as well, which 
also avoids ODR problems.
Also could a .DEF file control which symbols are reexported?
Sep 09 2013
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On 09/07/2013 02:47 PM, Benjamin Thaut wrote:

 If you compile lib1 into a static library and then copmpile lib2 into a
 DLL which statically links against lib1 both symbols lib1Func and
 lib2Func will be exported from the dll because lib1Func will get marked
 as dllexport when compiled into a static library and the static library
 transports that information into the linking process of lib2. This
 happens both on win32 and win64. But this is something you really don't
 want. Lets imagine you have a DLL that statically links against phobos.
 Suddenly you will have all phobos exported symbols in your dll. This can
 also lead to double defined symbols in case a user links against your
 dll and against the shared phobos dll. So we have to fix this somehow.


I had another thought about this and tried some examples.

If you do transitively link against lib1.lib symbols when using lib2.dll 
you might get ODR issues if there is another copy of lib1 in your 
process. This is to be expected because there are two copies of the same 
library. The safe choice is to make lib1 a DLL too.

If you still want to statically link lib1 into lib2.dll you have to be 
very cautious. An actual problem only occurs when lib1 symbols get used 
from outside of lib2.dll.
Of course a mean to prevent bugs from accidental linking lib1 symbols 
would be nice.
One way of doing this would be to mark all lib1 symbols as PRIVATE in 
lib2.def.

EXPORTS
D4lib18lib1FuncFZi PRIVATE

Another way would be to write a small tool that turns all exported 
symbols of a library into non-exported symbols.

I'm not sure we should do this in the compiler, because it simply hands 
over the libs to the linker. We'd have to make a copy of the library 
first and then do what a tool would do.

Another note for the phobos case. You should really ask yourself why 
does your dll need a private copy of phobos. And if so why can phobos 
symbols leak out of your dll.
Sep 10 2013
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 11.09.2013 00:39, Martin Nowak wrote:

 On 09/07/2013 02:47 PM, Benjamin Thaut wrote:

 If you compile lib1 into a static library and then copmpile lib2 into a
 DLL which statically links against lib1 both symbols lib1Func and
 lib2Func will be exported from the dll because lib1Func will get marked
 as dllexport when compiled into a static library and the static library
 transports that information into the linking process of lib2. This
 happens both on win32 and win64. But this is something you really don't
 want. Lets imagine you have a DLL that statically links against phobos.
 Suddenly you will have all phobos exported symbols in your dll. This can
 also lead to double defined symbols in case a user links against your
 dll and against the shared phobos dll. So we have to fix this somehow.


 I had another thought about this and tried some examples.

 If you do transitively link against lib1.lib symbols when using lib2.dll
 you might get ODR issues if there is another copy of lib1 in your
 process. This is to be expected because there are two copies of the same
 library. The safe choice is to make lib1 a DLL too.

 If you still want to statically link lib1 into lib2.dll you have to be
 very cautious. An actual problem only occurs when lib1 symbols get used
 from outside of lib2.dll.
 Of course a mean to prevent bugs from accidental linking lib1 symbols
 would be nice.
 One way of doing this would be to mark all lib1 symbols as PRIVATE in
 lib2.def.

 EXPORTS
 D4lib18lib1FuncFZi PRIVATE


That's probably not feasible for a library like phobos. (At least 
optlink stumbles if the def file gets too large). Also, compressed 
symbol names cause troubles for OMF.


 Another way would be to write a small tool that turns all exported
 symbols of a library into non-exported symbols.

 I'm not sure we should do this in the compiler, because it simply hands
 over the libs to the linker. We'd have to make a copy of the library
 first and then do what a tool would do.

 Another note for the phobos case. You should really ask yourself why
 does your dll need a private copy of phobos. And if so why can phobos
 symbols leak out of your dll.


AFAIU the discussed scenario is that the phobos library exports a lot of 
symbols, and this library is used both for static and dynamic linking. 
An executable statically linked against this library will have all the 
exports. That still works but is ugly.

A DLL to be used as a plugin with a C interface might also want to link 
statically against phobos to avoid DLL hell. Consider multiple plugins 
built against different versions of phobos. These would also leak also 
the symbols.
Sep 10 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 09/11/2013 08:21 AM, Rainer Schuetze wrote:

 AFAIU the discussed scenario is that the phobos library exports a lot of
 symbols, and this library is used both for static and dynamic linking.
 An executable statically linked against this library will have all the
 exports. That still works but is ugly.


Yes and it also has some runtime/memory overhead because the symbol 
table is much bigger.


 A DLL to be used as a plugin with a C interface might also want to link
 statically against phobos to avoid DLL hell. Consider multiple plugins
 built against different versions of phobos. These would also leak also
 the symbols.


Right, that's the valid use-case. But here the additionally exported 
symbols are way less of a problem because your loading not linking.
And as I wrote before if you link a program with two copies of the same 
library you're asking for trouble.

Anyhow I realized there is even less a use-case for statically linking a 
lib into a DLL (shared library) while expecting the symbols to be 
exported. We use that to put druntime into libphobos2.so though.
Also one doesn't need exported (visible) symbols for static linking.

So how about going into the opposite direction?
When the compiler creates a lib it strips all the exports (marks all 
symbols as hidden) by default. For the very special use-case where one 
wants to preserve the exports we can add a -libexports flag, only to be 
used with -lib.
Another nice thing is, that dmd writes libs itself and already processes 
all object files so we could easily integrate the striping in the compiler.
Sep 11 2013
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 12.09.2013 01:16, Martin Nowak wrote:

 On 09/11/2013 08:21 AM, Rainer Schuetze wrote:

 AFAIU the discussed scenario is that the phobos library exports a lot of
 symbols, and this library is used both for static and dynamic linking.
 An executable statically linked against this library will have all the
 exports. That still works but is ugly.


 Yes and it also has some runtime/memory overhead because the symbol
 table is much bigger.


 A DLL to be used as a plugin with a C interface might also want to link
 statically against phobos to avoid DLL hell. Consider multiple plugins
 built against different versions of phobos. These would also leak also
 the symbols.


 Right, that's the valid use-case. But here the additionally exported
 symbols are way less of a problem because your loading not linking.
 And as I wrote before if you link a program with two copies of the same
 library you're asking for trouble.

 Anyhow I realized there is even less a use-case for statically linking a
 lib into a DLL (shared library) while expecting the symbols to be
 exported. We use that to put druntime into libphobos2.so though.
 Also one doesn't need exported (visible) symbols for static linking.

 So how about going into the opposite direction?
 When the compiler creates a lib it strips all the exports (marks all
 symbols as hidden) by default. For the very special use-case where one
 wants to preserve the exports we can add a -libexports flag, only to be
 used with -lib.
 Another nice thing is, that dmd writes libs itself and already processes
 all object files so we could easily integrate the striping in the compiler.


This sounds interesting. Stripping an existing library isn't even needed 
because it is normally never created with exports anyway (-lib implies 
that no exports are created to start with). Stripping object files might 
be necessary, though, in case they have been built separately with -c.
Sep 11 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 12.09.2013 07:54, schrieb Rainer Schuetze:

 This sounds interesting. Stripping an existing library isn't even needed
 because it is normally never created with exports anyway (-lib implies
 that no exports are created to start with). Stripping object files might
 be necessary, though, in case they have been built separately with -c.


Can't we get around stripping object files by always using -c together 
with -lib in case the object file is supposed to go into a static library?
Sep 19 2013
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 19.09.2013 13:58, Benjamin Thaut wrote:

 Am 12.09.2013 07:54, schrieb Rainer Schuetze:

 This sounds interesting. Stripping an existing library isn't even needed
 because it is normally never created with exports anyway (-lib implies
 that no exports are created to start with). Stripping object files might
 be necessary, though, in case they have been built separately with -c.


 Can't we get around stripping object files by always using -c together
 with -lib in case the object file is supposed to go into a static library?


That could work, though it has it's own set of side effects, like 
splitting a module into pseudo modules per function which also ruins 
running unittests built into the library. Building modules with -c and 
combining these into a library afterwards avoids this.
Sep 19 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 19.09.2013 20:24, schrieb Rainer Schuetze:

 On 19.09.2013 13:58, Benjamin Thaut wrote:

 Am 12.09.2013 07:54, schrieb Rainer Schuetze:

 This sounds interesting. Stripping an existing library isn't even needed
 because it is normally never created with exports anyway (-lib implies
 that no exports are created to start with). Stripping object files might
 be necessary, though, in case they have been built separately with -c.


 Can't we get around stripping object files by always using -c together
 with -lib in case the object file is supposed to go into a static
 library?


 That could work, though it has it's own set of side effects, like
 splitting a module into pseudo modules per function which also ruins
 running unittests built into the library. Building modules with -c and
 combining these into a library afterwards avoids this.


Ah ok, I didn't know that. How much work would it be to implement object 
file stripping? I'm not confident I can do this.

Kind Regards
Benjamin Thaut
Sep 19 2013
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 19.09.2013 21:42, Benjamin Thaut wrote:

 Am 19.09.2013 20:24, schrieb Rainer Schuetze:

 On 19.09.2013 13:58, Benjamin Thaut wrote:

 Am 12.09.2013 07:54, schrieb Rainer Schuetze:

 This sounds interesting. Stripping an existing library isn't even
 needed
 because it is normally never created with exports anyway (-lib implies
 that no exports are created to start with). Stripping object files
 might
 be necessary, though, in case they have been built separately with -c.


 Can't we get around stripping object files by always using -c together
 with -lib in case the object file is supposed to go into a static
 library?


 That could work, though it has it's own set of side effects, like
 splitting a module into pseudo modules per function which also ruins
 running unittests built into the library. Building modules with -c and
 combining these into a library afterwards avoids this.


 Ah ok, I didn't know that. How much work would it be to implement object
 file stripping? I'm not confident I can do this.


I just checked the OMF and COFF docs: it should be possible to wipe out 
the export records without having to rewrite the object files, so it's 
not too involved. Don't know about ELF or mach-o, though.
Sep 22 2013
parent Martin Nowak <code dawg.eu> writes:
On 09/23/2013 08:30 AM, Rainer Schuetze wrote:

 I just checked the OMF and COFF docs: it should be possible to wipe out
 the export records without having to rewrite the object files, so it's
 not too involved. Don't know about ELF or mach-o, though.


I tried this and implemented it by rewriting the ELF sym flag for 
export, so it's was done by a rewrite of the object file. Maybe there 
are other solutions, but this seemed to be the simplest approach.
Nov 16 2013
prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 27.08.2013 12:12, schrieb Benjamin Thaut:

 The current behaviour of export is not sufficient to create a shared
 runtime on windows. Its alos not working very well in a few other cases.
 For background information please read the links provided in the DIP.
 This DIP tries to solve the problem by imitating the proven preprocessor
 techniques used in C/C++ to create windows dlls.

 Destroy.

 Kind Regards
 Benjamin Thaut


There is yet another problem to be solved. What should happen with 
tempaltes? Does it make sense to export templates? What happens if a 
template class marked with export gets instanciated in different DLLs 
using the same template arguments? Should the TypeInfo objects match? 
Should templates be exportable in generall?

In C++ the only thing I needed so far was exporting static members of 
templated classes. So the different instances of the template can work 
on the same data.

Kind Regards
Benjamin Thaut
Sep 08 2013
next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 08.09.2013 09:33, schrieb Benjamin Thaut:

 Am 27.08.2013 12:12, schrieb Benjamin Thaut:

 The current behaviour of export is not sufficient to create a shared
 runtime on windows. Its alos not working very well in a few other cases.
 For background information please read the links provided in the DIP.
 This DIP tries to solve the problem by imitating the proven preprocessor
 techniques used in C/C++ to create windows dlls.

 Destroy.

 Kind Regards
 Benjamin Thaut


 There is yet another problem to be solved. What should happen with
 tempaltes? Does it make sense to export templates? What happens if a
 template class marked with export gets instanciated in different DLLs
 using the same template arguments? Should the TypeInfo objects match?
 Should templates be exportable in generall?

 In C++ the only thing I needed so far was exporting static members of
 templated classes. So the different instances of the template can work
 on the same data.

 Kind Regards
 Benjamin Thaut



Good question. Not sure how Ada, Modula-3, Eiffel, Delphi, MLton solve 
the issue, even though their generic capabilities are not as powerfull.

--
Paulo
Sep 08 2013
prev sibling parent Martin Nowak <code dawg.eu> writes:
On 09/08/2013 09:33 AM, Benjamin Thaut wrote:

 There is yet another problem to be solved. What should happen with
 tempaltes? Does it make sense to export templates?


Yes, we need to export/import symbols from templates. Recently the 
compiler was changed to skip instantiations when they have already been 
instantiated in imported modules.
There is a problem similar to the lib case though. When you instantiate 
an exported template you don't want it's symbols exported from your 
shared library.


 What happens if a template class marked with export gets instanciated in
different DLLs
 using the same template arguments? Should the TypeInfo objects match?
 Should templates be exportable in generall?


Yes TypeInfos should match. For symbols which are allowed to have 
multiple definitions (like typeinfos from template classes) we can't 
solely rely on pointer comparison but also need to test typename equality.
Nov 04 2013
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
Can we deal with vtable entries pointing to imported functions on 
Windows? The kind of thing that happens when you inherit across a DLL 
boundary.
Sep 09 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 10.09.2013 05:50, schrieb Martin Nowak:

 Can we deal with vtable entries pointing to imported functions on
 Windows? The kind of thing that happens when you inherit across a DLL
 boundary.


I can't answer that and with dmd's current implementation of export its 
not possible to try this. I would say we implement the DIP and see on 
the way if this becomes a issue.

-- 
Kind Regards
Benjamin Thaut
Sep 22 2013
parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 23.09.2013 07:22, Benjamin Thaut wrote:

 Am 10.09.2013 05:50, schrieb Martin Nowak:

 Can we deal with vtable entries pointing to imported functions on
 Windows? The kind of thing that happens when you inherit across a DLL
 boundary.


 I can't answer that and with dmd's current implementation of export its
 not possible to try this. I would say we implement the DIP and see on
 the way if this becomes a issue.


It is the same as initializing a global variable with a pointer into 
another DLL, i.e. it needs some init code to set the value.
Sep 22 2013