www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - static ctors in shared libs

reply bitwise <bitwise.pvt gmail.com> writes:
I've been doing some work on shared libraries for OSX, and have 
come across a potential problem, which I'm not sure what to do 
with.

Currently, when a thread is spawned, that thread calls all the 
TLS ctors, then runs the thread entry point function, and finally 
calls the TLS dtors before the thread terminates.

Example, for windows:

https://github.com/D-Programming-Language/druntime/blob/15a227477a344583c4748d95492703901417f4f8/src/core/thread.d#L236

So, the question is, how do dynamic libraries interact here?

Example: A dynamic library contains D modules with TLS ctors. If 
I start a few threads, and then load a dynamic library, shouldn't 
the TLS ctors in the dynamic library be called for each running 
thread? If my assumption is correct, the next question is, how do 
you do this?

I don't think you can hijack each thread and have it run the TLS 
ctors, and you can run them all from the thread loading the 
shared library because of synchronization issues. So what's the 
solution? Should TLS ctors in dynamic libraries simply be 
specified not to run, or could they somehow be run lazily at the 
first TLS access in a dynamic library?

Any thoughts on this would be appreciated.

Thanks,
    Bit
Mar 17 2016
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 17 Mar 2016 20:27:39 +0000
schrieb bitwise <bitwise.pvt gmail.com>:


 I've been doing some work on shared libraries for OSX, and have 
 come across a potential problem, which I'm not sure what to do 
 with.
 
 Currently, when a thread is spawned, that thread calls all the 
 TLS ctors, then runs the thread entry point function, and finally 
 calls the TLS dtors before the thread terminates.
 
 Example, for windows:
 
 https://github.com/D-Programming-Language/druntime/blob/15a227477a344583c4748d95492703901417f4f8/src/core/thread.d#L236
 
 So, the question is, how do dynamic libraries interact here?
 
 Example: A dynamic library contains D modules with TLS ctors. If 
 I start a few threads, and then load a dynamic library, shouldn't 
 the TLS ctors in the dynamic library be called for each running 
 thread? If my assumption is correct, the next question is, how do 
 you do this?
 
 I don't think you can hijack each thread and have it run the TLS 
 ctors, and you can run them all from the thread loading the 
 shared library because of synchronization issues. So what's the 
 solution? Should TLS ctors in dynamic libraries simply be 
 specified not to run, or could they somehow be run lazily at the 
 first TLS access in a dynamic library?
 
 Any thoughts on this would be appreciated.
 
 Thanks,
     Bit
 


It's been some time since I looked at that code, but IIRC TLS ctors for
newly loaded libraries are not run in old threads. There's nothing we
can do about this without help from the C runtime.


As TLS ctors are meant to initialize TLS storage the correct time to
run these is after the per-thread TLS storage allocation. (The C
runtime can actually allocate TLS memory lazily, on the first access to
a TLS variable in a module). The C runtimes of course know when TLS
storage is allocated, but they do not provide hooks we could use.
Mar 17 2016
parent reply Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:
On Thursday, 17 March 2016 at 20:54:57 UTC, Johannes Pfau wrote:

 It's been some time since I looked at that code, but IIRC TLS 
 ctors for newly loaded libraries are not run in old threads. 
 There's nothing we can do about this without help from the C 
 runtime.


Hmm... are all module _destructors_ called when those threads 
exit? That would mean that some destructors may be called without 
the corresponding constructors having run...
Mar 18 2016
parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 18 Mar 2016 14:35:41 +0000
schrieb Marc Sch=C3=BCtz <schuetzm gmx.net>:


 On Thursday, 17 March 2016 at 20:54:57 UTC, Johannes Pfau wrote:

 It's been some time since I looked at that code, but IIRC TLS=20
 ctors for newly loaded libraries are not run in old threads.=20
 There's nothing we can do about this without help from the C=20
 runtime. =20


=20
 Hmm... are all module _destructors_ called when those threads=20
 exit? That would mean that some destructors may be called without=20
 the corresponding constructors having run...


No. Each thread keeps a thread-local list of shared libraries it knows
about. This list is inherited from the parent thread on thread
creation. It also gets updated if a new library is loaded at runtime
(but only for the loading thread!). The runtime only runs constructors
and destructors for libraries in the list. Additionally the GC only
scans TLS memory of libraries in the list.


As an example:
* Create 2 Threads, TA & TB
* TB creates thread TB1
* TB loads library libA
* TB creates new thread TB2
* TA creates new thread TA1

Threads TB and TB2 will run TLS constructors and destructors for libA.
modules. These threads will also scan TLS memory from modules in libA.

Threads TA, TA1 and TB1 will not run TLS constructors or destructors for
libA. Additionally, the GC will not scan TLS memory for libA for these
threads.

So you can only safely use modules from libA in thread TB and TB2.

Relevant links:
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/secti=
ons_elf_shared.d#L198
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/secti=
ons_elf_shared.d#L281

https://github.com/D-Programming-Language/druntime/blob/master/src/core/thr=
ead.d#L385
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/minfo=
.d#L312
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/secti=
ons_elf_shared.d#L49
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/secti=
ons_elf_shared.d#L76
Mar 18 2016
parent bitwise <bitwise.pvt gmail.com> writes:
On Friday, 18 March 2016 at 16:23:00 UTC, Johannes Pfau wrote:

 Am Fri, 18 Mar 2016 14:35:41 +0000
 schrieb Marc Schütz <schuetzm gmx.net>:

 [...]


Thanks for the explanation!

When I first looked at the elf implementation, I was confused by 
what was goign on with 
pinLoadedLibraries()/inheritLoadedLibraries(). With this problem 
in mind though, it all makes sense now.

     Bit
Mar 23 2016
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 17/03/16 21:27, bitwise wrote:

 I've been doing some work on shared libraries for OSX, and have come
 across a potential problem, which I'm not sure what to do with.

 Currently, when a thread is spawned, that thread calls all the TLS
 ctors, then runs the thread entry point function, and finally calls the
 TLS dtors before the thread terminates.

 Example, for windows:

 https://github.com/D-Programming-Language/druntime/blob/15a227477a344583c4748d95492703901417f4f8/src/core/thread.d#L236


 So, the question is, how do dynamic libraries interact here?

 Example: A dynamic library contains D modules with TLS ctors. If I start
 a few threads, and then load a dynamic library, shouldn't the TLS ctors
 in the dynamic library be called for each running thread? If my
 assumption is correct, the next question is, how do you do this?

 I don't think you can hijack each thread and have it run the TLS ctors,
 and you can run them all from the thread loading the shared library
 because of synchronization issues. So what's the solution? Should TLS
 ctors in dynamic libraries simply be specified not to run, or could they
 somehow be run lazily at the first TLS access in a dynamic library?

 Any thoughts on this would be appreciated.


How does it behave on Linux? It already supports dynamic libraries.

-- 
/Jacob Carlborg
Mar 18 2016
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Friday, 18 March 2016 at 18:52:25 UTC, Jacob Carlborg wrote:

 On 17/03/16 21:27, bitwise wrote:

 I've been doing some work on shared libraries for OSX, and 
 have come
 across a potential problem, which I'm not sure what to do with.

 Currently, when a thread is spawned, that thread calls all the 
 TLS
 ctors, then runs the thread entry point function, and finally 
 calls the
 TLS dtors before the thread terminates.

 Example, for windows:

 https://github.com/D-Programming-Language/druntime/blob/15a227477a344583c4748d95492703901417f4f8/src/core/thread.d#L236


 So, the question is, how do dynamic libraries interact here?

 Example: A dynamic library contains D modules with TLS ctors. 
 If I start
 a few threads, and then load a dynamic library, shouldn't the 
 TLS ctors
 in the dynamic library be called for each running thread? If my
 assumption is correct, the next question is, how do you do 
 this?

 I don't think you can hijack each thread and have it run the 
 TLS ctors,
 and you can run them all from the thread loading the shared 
 library
 because of synchronization issues. So what's the solution? 
 Should TLS
 ctors in dynamic libraries simply be specified not to run, or 
 could they
 somehow be run lazily at the first TLS access in a dynamic 
 library?

 Any thoughts on this would be appreciated.


 How does it behave on Linux? It already supports dynamic 
 libraries.


After looking at the elf implementation, Johannes's explanation 
seems to be correct. I don't think it's the best solution though.

I'm wondering if the _exact_ behavior of the current solution was 
intentional. It seems like everything will work fine as long as 
you only use statically linked shared libs, so I'm wondering if 
dynamic linking was fully considered at the time the current 
solution was written. As far as dynamic linking goes, I don't 
like how the current solution works. It can lead to very 
confusing problems, and I think it would be better to do an 
all-or-nothing approach:

-statically linked shared libs always have shared static ctors 
called
-statically linked shared libs always have TLS static ctors 
called for all new threads
-dynamically linked shared libs always have shared static ctors 
called
-dynamically linked shared libs NEVER have TLS static ctors 
called for ANY thread

I can see how one could argue for benefits of the current 
approach, but I don't think it's worth exposing people to that 
kind of confusion, to have partially working TLS static ctors.

Thoughts?

     Bit
Mar 23 2016
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Thursday, 24 March 2016 at 00:56:33 UTC, bitwise wrote:

 Thoughts?

     Bit


Ok..How about "Thought"? One thought will do. I'll take what I 
can get ;)

      Bit
Mar 25 2016
parent reply Joakim <dlang joakim.fea.st> writes:
On Friday, 25 March 2016 at 16:33:26 UTC, bitwise wrote:

 On Thursday, 24 March 2016 at 00:56:33 UTC, bitwise wrote:

 Thoughts?

     Bit


 Ok..How about "Thought"? One thought will do. I'll take what I 
 can get ;)

      Bit


Martin is the guy to talk to, he wrote all that code.  David 
might have an opinion, as he adapted it for ldc.  Most everybody 
else doesn't go into those weeds.
Mar 25 2016
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Saturday, 26 March 2016 at 03:08:13 UTC, Joakim wrote:

 On Friday, 25 March 2016 at 16:33:26 UTC, bitwise wrote:

 On Thursday, 24 March 2016 at 00:56:33 UTC, bitwise wrote:

 Thoughts?

     Bit


 Ok..How about "Thought"? One thought will do. I'll take what I 
 can get ;)

      Bit


 Martin is the guy to talk to, he wrote all that code.  David 
 might have an opinion, as he adapted it for ldc.  Most 
 everybody else doesn't go into those weeds.


I was hoping to have some more people weigh in with their 
experience with plugins, whether or not they're expected to be 
multi-threaded, thread-safe, etc.. I think the current design is 
fragile, and given the limited usage of D shared libraries atm, I 
think it's a good time to come up with something a bit more 
solid/predictable.

Anyways, I guess I'll have to email Martin and David and see what 
they say.

     Bit
Mar 26 2016
parent reply Guillaume Piolat <contact gam3sfrommars.fr> writes:
On Sunday, 27 March 2016 at 03:28:31 UTC, bitwise wrote:

 I was hoping to have some more people weigh in with their 
 experience with plugins, whether or not they're expected to be 
 multi-threaded, thread-safe, etc.. I think the current design 
 is fragile, and given the limited usage of D shared libraries 
 atm, I think it's a good time to come up with something a bit 
 more solid/predictable.


Using OSX shared libraries with both DMD and LDC for plugins, I'm 
trying to avoid all TLS, with LDC the only TLS I have is cached 
Obj-C selectors (pointers) which do not have constructors IIRC.

A real concern is then that Phobos would use TLS internally. 
Apart from that I don't really have an opinion.
Mar 27 2016
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Sunday, 27 March 2016 at 11:38:15 UTC, Guillaume Piolat wrote:

 On Sunday, 27 March 2016 at 03:28:31 UTC, bitwise wrote:

 I was hoping to have some more people weigh in with their 
 experience with plugins, whether or not they're expected to be 
 multi-threaded, thread-safe, etc.. I think the current design 
 is fragile, and given the limited usage of D shared libraries 
 atm, I think it's a good time to come up with something a bit 
 more solid/predictable.


 Using OSX shared libraries with both DMD and LDC for plugins, 
 I'm trying to avoid all TLS, with LDC the only TLS I have is 
 cached Obj-C selectors (pointers) which do not have 
 constructors IIRC.

 A real concern is then that Phobos would use TLS internally. 
 Apart from that I don't really have an opinion.


Ok, thanks!

Phobos should be linked statically even when it's a shared 
library, so it should still have all static ctors called properly.

     Bit
Mar 27 2016
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Sunday, 27 March 2016 at 16:54:53 UTC, bitwise wrote:


 Phobos should be linked statically even when it's a shared 
 library, so it should still have all static ctors called 
 properly.

     Bit


Correction...Phobos would be statically linked to a dynamically 
loaded shared library. This could be a problem.
Mar 27 2016
parent reply Guillaume Piolat <name.lastname gmail.com> writes:
On Sunday, 27 March 2016 at 17:58:01 UTC, bitwise wrote:

 On Sunday, 27 March 2016 at 16:54:53 UTC, bitwise wrote:


 Phobos should be linked statically even when it's a shared 
 library, so it should still have all static ctors called 
 properly.

     Bit


 Correction...Phobos would be statically linked to a dynamically 
 loaded shared library. This could be a problem.


Hasn't seen any practical problem with that, not sure why people 
say it's a bad idea TBH. I like that everything is in a single 
static binary.

The way bigger problem being OSX shared libraries unloading (with 
the leaking workaround), I haven't really followed your work 
recently about it!
Mar 27 2016
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Sunday, 27 March 2016 at 19:31:46 UTC, Guillaume Piolat wrote:

 On Sunday, 27 March 2016 at 17:58:01 UTC, bitwise wrote:

 On Sunday, 27 March 2016 at 16:54:53 UTC, bitwise wrote:


 Phobos should be linked statically even when it's a shared 
 library, so it should still have all static ctors called 
 properly.

     Bit


 Correction...Phobos would be statically linked to a 
 dynamically loaded shared library. This could be a problem.


 Hasn't seen any practical problem with that, not sure why 
 people say it's a bad idea TBH. I like that everything is in a 
 single static binary.


I'm not sure I understand what you're saying here.

What I'm suggesting is that TLS static ctors not be run at all 
for dynamically loaded shared libraries, because at present, they 
are only run in certain circumstances, which can be confusing and 
unreliable. For example, if a program launched a bunch of worker 
threads, and then reloaded a D shared library, the TLS static 
ctors wouldn't be run after the second reload for those worker 
threads. So if the worker threads started accessing the shared 
library, they could encounter uninitialized data.

If TLS static ctors simply weren't run at all for dynamically 
loaded libraries, there would be no confusion. Any D user that 
tried to explore the use of shared libraries, would quickly learn 
that TLS ctors are not called at all for dynamically loaded 
libraries. I suppose then, that an API like 
Runtime.AttachToThread() could be used to explicitly call the TLS 
ctors.


 The way bigger problem being OSX shared libraries unloading 
 (with the leaking workaround), I haven't really followed your 
 work recently about it!


I successfully modified DMD to output ctor/dtors which replace 
the need for the problematic callback, but I have to wait for 
Jacob's work on native TLS to be merged before I can continue.

      Bit
Mar 27 2016
parent Guillaume Piolat <name.lastname gmail.com> writes:
On Sunday, 27 March 2016 at 22:08:57 UTC, bitwise wrote:

 If TLS static ctors simply weren't run at all for dynamically 
 loaded libraries, there would be no confusion. Any D user that 
 tried to explore the use of shared libraries, would quickly 
 learn that TLS ctors are not called at all for dynamically 
 loaded libraries. I suppose then, that an API like 
 Runtime.AttachToThread() could be used to explicitly call the 
 TLS ctors.


That looks more sensible indeed.
Mar 27 2016