www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - osx shared libraries.

reply bitwise <bitwise.pvt gmail.com> writes:
I'm trying to fix shared libraries for OSX. I've updated sections_osx.d to  
support multiple images, however, the method of notifying druntime when to  
load/unload images needs to be fixed.

Currently, the following method is used to register a callback for when  
images are loaded, which, based on the last conversation on this topic, is  
a no-go. Basically, the problem is that there is no way of unregistering  
the callbacks.
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L76

So the solution is to add __attribute__((constructor/destructor)) calls to  
all D binaries on OSX to initialize the images.

The code looks like this:
http://dpaste.com/0SF46B6

So my question is, what is the best way to get this code into the binary?

I suppose the code could be pre-compiled into an object file and linked by  
dmd during compilation, but adding external dependencies to dmd seems like  
a bad idea.

I would say DMD could generate this code, but where would it go?

I don't think it would be right to add the __constructors__ to a random  
*.o produced by the compiler, because people will expect *.o files to  
behave like *.o files. On the other hand, if the __constructors__ were to  
be added by dmd at link-time, and someone used a linker other than dmd, it  
would not add in the __constructor__ code, and thus generate a defective  
dylib. This is a complicated problem with many things to consider, so I'm  
not exactly sure where to go from here.

Any ideas on this would be much appreciated.

Thanks,
   Bit
Jun 06 2015
next sibling parent reply bitwise <bitwise.pvt gmail.com> writes:
On Sat, 06 Jun 2015 14:52:11 -0400, bitwise <bitwise.pvt gmail.com> wrote:

 Any ideas on this would be much appreciated.

 Thanks,
    Bit


One thought that just occurred to me would be to require that D dynamic  
libraries contain a main entry point, like DllMain on Windows. This seems  
like the only real/reliable option.

The coder would have to explicitly give dmd a module in which to place the  
constructor/destructor code for images, the same way that dmd adds a  
C-main in the file where D-main is found. Enforcing this in the future  
would be a breaking change, but not a dangerous one. It would be a simple  
link-time error which was easy to fix.

Also, requiring all D binaries to have some kind of entry point would  
alleviate the need to ever call Runtime.initialize() explicitly.

   Bit
Jun 06 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2015 12:09 PM, bitwise wrote:

 On Sat, 06 Jun 2015 14:52:11 -0400, bitwise <bitwise.pvt gmail.com> wrote:

 Any ideas on this would be much appreciated.

 Thanks,
    Bit


 One thought that just occurred to me would be to require that D dynamic
 libraries contain a main entry point, like DllMain on Windows. This seems like
 the only real/reliable option.

 The coder would have to explicitly give dmd a module in which to place the
 constructor/destructor code for images, the same way that dmd adds a C-main in
 the file where D-main is found. Enforcing this in the future would be a
breaking
 change, but not a dangerous one. It would be a simple link-time error which was
 easy to fix.

 Also, requiring all D binaries to have some kind of entry point would alleviate
 the need to ever call Runtime.initialize() explicitly.

    Bit


Martin Nowak is the author of most of the OSX DLL support, but he seems to not 
be around lately.

I'd rather have a dynamic library entry point than put a constructor/destructor 
call into every object file.
Jun 06 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Sat, 06 Jun 2015 16:04:33 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:


 On 6/6/2015 12:09 PM, bitwise wrote:

 On Sat, 06 Jun 2015 14:52:11 -0400, bitwise <bitwise.pvt gmail.com>  
 wrote:

 Any ideas on this would be much appreciated.

 Thanks,
    Bit


 One thought that just occurred to me would be to require that D dynamic
 libraries contain a main entry point, like DllMain on Windows. This  
 seems like
 the only real/reliable option.

 The coder would have to explicitly give dmd a module in which to place  
 the
 constructor/destructor code for images, the same way that dmd adds a  
 C-main in
 the file where D-main is found. Enforcing this in the future would be a  
 breaking
 change, but not a dangerous one. It would be a simple link-time error  
 which was
 easy to fix.

 Also, requiring all D binaries to have some kind of entry point would  
 alleviate
 the need to ever call Runtime.initialize() explicitly.

    Bit


 Martin Nowak is the author of most of the OSX DLL support, but he seems  
 to not be around lately.

 I'd rather have a dynamic library entry point than put a  
 constructor/destructor call into every object file.


Right. I was wondering what he was doing there...Just learned what COMDAT  
is ;)

It's starting to look like reworking druntime was the easy part. I've got  
druntime built as a shared library now, and my main project, which thus  
far is an app/graphics/game engine, is working nicely. Of course, all this  
goes out the window with a C-Host app... unless people knew to link the  
shared druntime with their C host app, which doesn't sound intuitive.

I like the idea of a dll entry point though.

I suppose it could look something like this:

enum Reason {
     processAttach,
     processDetach,
     threadAttach,
     threadDetatch
}

bool main(void *handle, Reason reason) {
     return true;
}

For windows, we could simply hijack DllMain().

As for the OSX, we could use an entry point stub like this, with the rest  
of the code in druntime:

// dladdr(&symbol) gets us the mach_image*
void _osx_image_init(void *symbol);
void _osx_image_term(void *symbol);

__attribute__((visibility("hidden"), constructor))
static void _image_init() {
     _osx_image_init((void*)&_image_init);
}
__attribute__((visibility("hidden"), destructor))
static void _image_term(){
     _osx_image_term((void*)&_image_term);
}

I would like to do the above stub in D, but I haven't found a way to  
specify visibility or section of the function yet.

I would rather not do what Martin has done, outputting the entire thing in  
asm, as that would take me a very long time to get right.

Anyways, onward!

Thanks
   Bit
Jun 15 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Mon, 15 Jun 2015 23:26:18 -0400, bitwise <bitwise.pvt gmail.com> wrote:

 I would like to do the above stub in D, but I haven't found a way to  
 specify visibility or section of the function yet.


It actually appears that __attribute__((visibility("hidden")) is redundant  
and making it a constructor already hides it.

So if I can get a D function to go to the "__mod_init_func" section  
instead, I should be set.

   Bit
Jun 15 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Mon, 15 Jun 2015 23:56:14 -0400, bitwise <bitwise.pvt gmail.com> wrote:


 It actually appears that __attribute__((visibility("hidden")) is  
 redundant and making it a constructor already hides it.


derp.. no it's not. Its the fact they were marked static in the C code  
that made them hidden.

If anyone is willing to point me in the right direction here, it would be  
much appreciated ;)
Looking at Martin's github, it doesn't appear that he's back in business  
yet.

Thanks,
   Bit
Jun 16 2015
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-06-16 16:56, bitwise wrote:


 If anyone is willing to point me in the right direction here, it would
 be much appreciated ;)
 Looking at Martin's github, it doesn't appear that he's back in business
 yet.


Why don't you just do what Martin did for Linux, or is that what you 
would like to avoid? The code was a bit more than I first thought it 
would be. You could try doing the same for OS X and see what happens :). 
Just replace the segments and section names with the appropriate OS X 
names and similar stuff.

-- 
/Jacob Carlborg
Jun 16 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Tue, 16 Jun 2015 15:50:32 -0400, Jacob Carlborg <doob me.com> wrote:


 On 2015-06-16 16:56, bitwise wrote:


 If anyone is willing to point me in the right direction here, it would
 be much appreciated ;)
 Looking at Martin's github, it doesn't appear that he's back in business
 yet.


 Why don't you just do what Martin did for Linux, or is that what you  
 would like to avoid? The code was a bit more than I first thought it  
 would be. You could try doing the same for OS X and see what happens :).  
 Just replace the segments and section names with the appropriate OS X  
 names and similar stuff.


Heh... trying ;)

Here, there seems to be a function to put things in ctors/dtors, but it  
looks like it was copy pasted from the Elf code:
https://github.com/D-Programming-Language/dmd/blob/c718790165c3124c61e510d8352b9cb9d8ae0198/src/backend/machobj.c#L1532

I've rewritten the first line as follows, but still working on what I need  
to do to actually put the function in there.
IDXSEC seg = s->Sseg = MachObj::getsegment("__mod_init_func", "__DATA", 2,  
S_MOD_INIT_FUNC_POINTERS);

I think the S_COALESCED flag could be added to the above to allow a  
definition in every obj file like Martin's approach, but I would rather  
have an explicit DllMain and put the init code in that one obj file. It  
seems more intuitive to me.

So if I get Obj::staticctor and Obj::staticdtor working on osx, that's  
half the problem solved... I think.


   Bit
Jun 16 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-06-17 01:19, bitwise wrote:


 Heh... trying ;)

 Here, there seems to be a function to put things in ctors/dtors, but it
 looks like it was copy pasted from the Elf code:
 https://github.com/D-Programming-Language/dmd/blob/c718790165c3124c61e510d8352b9cb9d8ae0198/src/backend/machobj.c#L1532


 I've rewritten the first line as follows, but still working on what I
 need to do to actually put the function in there.
 IDXSEC seg = s->Sseg = MachObj::getsegment("__mod_init_func", "__DATA",
 2, S_MOD_INIT_FUNC_POINTERS);

 I think the S_COALESCED flag could be added to the above to allow a
 definition in every obj file like Martin's approach, but I would rather
 have an explicit DllMain and put the init code in that one obj file. It
 seems more intuitive to me.

 So if I get Obj::staticctor and Obj::staticdtor working on osx, that's
 half the problem solved... I think.


Obj::staticctor does not seem to be used for setting up the runtime. 
This is the code that generates the runtime initialization [1], I 
believe, at least it calls "_d_dso_registry". There's a lot more code 
there than I first hoped.

[1] 
https://github.com/D-Programming-Language/dmd/blob/c718790165c3124c61e510d8352b9cb9d8ae0198/src/backend/elfobj.c#L3183

-- 
/Jacob Carlborg
Jun 16 2015
parent bitwise <bitwise.pvt gmail.com> writes:
On Wed, 17 Jun 2015 02:21:31 -0400, Jacob Carlborg <doob me.com> wrote:


 Obj::staticctor does not seem to be used for setting up the runtime.


I realize that, but I was thinking that I could just generate a DllMain in  
D code[1], and then output it as a special case[2] as is done with regular  
D main:

[1]  
https://github.com/D-Programming-Language/dmd/blob/c718790165c3124c61e510d8352b9cb9d8ae0198/src/mars.c#L228
[2]  
https://github.com/D-Programming-Language/dmd/blob/c718790165c3124c61e510d8352b9cb9d8ae0198/src/mars.c#L1669

For the second step though, I could also call staticctor/staticdtor or  
something.. still workin on it ;)



 There's a lot more code there than I first hoped.


Yeah... Writing out assembly as byte codes like that is a little too much  
for me. I think I can get by with the above solution.

I'm still having trouble finding where functions are exported though.  
Every instance of export_symbol() seems to be check first with isExport(),  
except all symbols are exported by default on OSX.... but from where?
https://github.com/D-Programming-Language/dmd/blob/c718790165c3124c61e510d8352b9cb9d8ae0198/src/toobj.c#L572

It seems like it could be in either out.c or machobj.c, but I still  
haven't found it.

   Bit
Jun 17 2015
prev sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 16 June 2015 at 14:56:35 UTC, bitwise wrote:

 On Mon, 15 Jun 2015 23:56:14 -0400, bitwise 
 <bitwise.pvt gmail.com> wrote:


 It actually appears that __attribute__((visibility("hidden")) 
 is redundant and making it a constructor already hides it.


 derp.. no it's not. Its the fact they were marked static in the 
 C code that made them hidden.

 If anyone is willing to point me in the right direction here, 
 it would be much appreciated ;)
 Looking at Martin's github, it doesn't appear that he's back in 
 business yet.


I can help you with codegen questions if you want to try LDC. The 
implementation I did there for Linux was based on Martin's work 
(but I contributed fixes to a couple of the shared problems and 
made it work with --gc-sections).

You'll have to drop down to the compiler level to influence the 
visibility in either case, though.

  - David
Jun 16 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Tue, 16 Jun 2015 17:45:52 -0400, David Nadlinger <code klickverbot.at>  
wrote:

 I can help you with codegen questions if you want to try LDC.


I actually tried to compile my library with LDC a week or two ago, and got  
ICEd.

I would like to try LDC, so I'll probably give it a go next release. It  
looks like the runtime for LDC would need some work too though. In  
particular, it doesn't seem to support dynamic loading:
https://github.com/ldc-developers/druntime/blob/ldc/src/rt/sections_ldc.d#L379


 You'll have to drop down to the compiler level to influence the  
 visibility in either case, though.


Yeah... I'm still trying to figure out how dmd exports, and decides when  
to export, functions.

In Martin's code, it looks like he's manually outputting assembly as  
bytecodes, but I was hoping for a simpler solution.

Ideally, I would like to copy this approach:
https://github.com/D-Programming-Language/dmd/blob/c718790165c3124c61e510d8352b9cb9d8ae0198/src/mars.c#L228

Then, add some special cases to put the generated function into the right  
section and ensure the symbols are not exported.

Anyways, your Dconf presentation was helpful to explain a few things, so  
thanks for that =D

   Bit
Jun 16 2015
parent reply "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 16 June 2015 at 23:52:26 UTC, bitwise wrote:

 I would like to try LDC, so I'll probably give it a go next 
 release. It looks like the runtime for LDC would need some work 
 too though. In particular, it doesn't seem to support dynamic 
 loading:
 https://github.com/ldc-developers/druntime/blob/ldc/src/rt/sections_ldc.d#L379


Well, this is the old LDC-specific implementation for all 
platforms that have not been converted to the new scheme yet. 
You'd probably want to base your implementation on 
sections_linux, which we use on Linux.

The code that emits the global constructors/destructors is here: 
https://github.com/ldc-developers/ldc/blob/33befca6d72fe267ee8e4179ba670513993e2eb7/gen/module.cpp#L348-L546

  - David
Jun 16 2015
parent bitwise <bitwise.pvt gmail.com> writes:
On Tue, 16 Jun 2015 20:07:03 -0400, David Nadlinger <code klickverbot.at>  
wrote:


 On Tuesday, 16 June 2015 at 23:52:26 UTC, bitwise wrote:

 I would like to try LDC, so I'll probably give it a go next release. It  
 looks like the runtime for LDC would need some work too though. In  
 particular, it doesn't seem to support dynamic loading:
 https://github.com/ldc-developers/druntime/blob/ldc/src/rt/sections_ldc.d#L379


 Well, this is the old LDC-specific implementation for all platforms that  
 have not been converted to the new scheme yet. You'd probably want to  
 base your implementation on sections_linux, which we use on Linux.

 The code that emits the global constructors/destructors is here:  
 https://github.com/ldc-developers/ldc/blob/33befca6d72fe267ee8e4179ba670513993e2eb7/gen/module.cpp#L348-L546

   - David


The llvm codegen is certainly easier on the eyes ;)

When I started porting my library though, I fumbled around with a bunch of  
compilers and build commands trying to get D/C++ interop to work right.  
xcode's default compiler seems to work nicely with dmd, so I went with  
that. Other combinations where giving me strange linker errors.


As far as the runtime code, I understand most of what's going on, but I'm  
still confused about the purpose of ThreadDSO, and what these are for:
  pinLoadedLibraries()
  unpinLoadedLibraries()
  inheritLoadedLibraries()
  cleanupLoadedLibraries()

   Bit
Jun 17 2015
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/15 11:52 AM, bitwise wrote:

 The code looks like this:
 http://dpaste.com/0SF46B6

 So my question is, what is the best way to get this code into the binary?


Where would an approach based on static shared this() and ~this fall 
short? Those would get called when the library is loaded, unless you 
actually need to implement exactly that :o). -- Andrei
Jun 06 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2015 12:47 PM, Andrei Alexandrescu wrote:

 On 6/6/15 11:52 AM, bitwise wrote:

 The code looks like this:
 http://dpaste.com/0SF46B6

 So my question is, what is the best way to get this code into the binary?


 Where would an approach based on static shared this() and ~this fall short?


The druntime must be initialized before those are called.
Jun 06 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-06-06 20:52, bitwise wrote:

 I'm trying to fix shared libraries for OSX. I've updated sections_osx.d
 to support multiple images, however, the method of notifying druntime
 when to load/unload images needs to be fixed.

 Currently, the following method is used to register a callback for when
 images are loaded, which, based on the last conversation on this topic,
 is a no-go. Basically, the problem is that there is no way of
 unregistering the callbacks.
 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L76


 So the solution is to add __attribute__((constructor/destructor)) calls
 to all D binaries on OSX to initialize the images.

 The code looks like this:
 http://dpaste.com/0SF46B6

 So my question is, what is the best way to get this code into the binary?

 I suppose the code could be pre-compiled into an object file and linked
 by dmd during compilation, but adding external dependencies to dmd seems
 like a bad idea.

 I would say DMD could generate this code, but where would it go?


For Linux (and FreeBSD I think) the compiler already generates a 
__attribute__((constructor)) function which calls _d_dso_registry [1]. 
Modify the compiler to do the same thing for OS X.

[1] 
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_elf_shared.d#L321

-- 
/Jacob Carlborg
Jun 06 2015