www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Ideas for runtime loading of shared libraries.

reply "Martin Nowak" <dawg dawgfoto.de> writes:
I think that I'll defer the support for runtime loading of shared library  
(plugins)
in favor of getting linked shared library support done now.
There are several issues that require more thoughts.

  - Per-thread initialization of modules is somewhat tricky.
    Doing it in Runtime.loadLibrary requires knowledge of shared library  
dependencies
    because different threads might share dependencies but this is not  
provided by libc/libdl.

  - Libraries might not be unloaded as long as GC collected class instances  
still exist because
    finalization fails otherwise.

  - Getting symbols through mangled names is difficult/unstable.

  - D libraries used by a C library should provide proper runtime  
initialization
    even if the C library is used by a D application.

Any ideas or use-cases for plugins are welcome.

martin
Jan 02 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-01-02 20:20, Martin Nowak wrote:

 I think that I'll defer the support for runtime loading of shared
 library (plugins)
 in favor of getting linked shared library support done now.
 There are several issues that require more thoughts.

 - Per-thread initialization of modules is somewhat tricky.
 Doing it in Runtime.loadLibrary requires knowledge of shared library
 dependencies
 because different threads might share dependencies but this is not
 provided by libc/libdl.

 - Libraries might not be unloaded as long as GC collected class
 instances still exist because
 finalization fails otherwise.

 - Getting symbols through mangled names is difficult/unstable.

 - D libraries used by a C library should provide proper runtime
 initialization
 even if the C library is used by a D application.

 Any ideas or use-cases for plugins are welcome.

 martin



- Initializing module infos
- Initializing exception handling tables
- Running module constructors
- Initializing TLS

Then also unload all this when the library is unloaded.

On Mac OS X, can't "_dyld_register_func_for_add_image" be used? Then it 
will work, hopefully, transparently for the user. D libraries used by C 
wouldn't need any different handling. Because they will be linked with 
druntime it can initializing everything with the help of 
"_dyld_register_func_for_add_image".

-- 
/Jacob Carlborg
Jan 02 2012
parent reply "Martin Nowak" <dawg dawgfoto.de> writes:
On Mon, 02 Jan 2012 20:38:50 +0100, Jacob Carlborg <doob me.com> wrote:


 On 2012-01-02 20:20, Martin Nowak wrote:

 I think that I'll defer the support for runtime loading of shared
 library (plugins)
 in favor of getting linked shared library support done now.
 There are several issues that require more thoughts.

 - Per-thread initialization of modules is somewhat tricky.
 Doing it in Runtime.loadLibrary requires knowledge of shared library
 dependencies
 because different threads might share dependencies but this is not
 provided by libc/libdl.

 - Libraries might not be unloaded as long as GC collected class
 instances still exist because
 finalization fails otherwise.

 - Getting symbols through mangled names is difficult/unstable.

 - D libraries used by a C library should provide proper runtime
 initialization
 even if the C library is used by a D application.

 Any ideas or use-cases for plugins are welcome.

 martin



 - Initializing module infos
 - Initializing exception handling tables
 - Running module constructors
 - Initializing TLS

 Then also unload all this when the library is unloaded.


It seems that libraries can't be unloaded deterministically,
because GC finalization still references them.


 On Mac OS X, can't "_dyld_register_func_for_add_image" be used? Then it  
 will work, hopefully, transparently for the user. D libraries used by C  
 wouldn't need any different handling. Because they will be linked with  
 druntime it can initializing everything with the help of  
 "_dyld_register_func_for_add_image".


That was the approach I took and it is partly a dead-end.

I have a mechanism similar to _dyld_register_func_for_add_image
but runtime loaders have no notion of per-thread initialization,
i.e. when two threads load the same library only the first one will
actually cause the image to be loaded.
This implies that the second thread would need to check all
dependencies of the loaded library to do the initialization.
I've written something along this line but it requires to
exploit/rewrite part of the runtime linker.
Using dlmopen on linux would be a terrible inefficient hack
around this issue, it allows to load libraries multiple times.
Jan 02 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-01-02 21:57, Martin Nowak wrote:

 On Mon, 02 Jan 2012 20:38:50 +0100, Jacob Carlborg <doob me.com> wrote:


 On 2012-01-02 20:20, Martin Nowak wrote:

 I think that I'll defer the support for runtime loading of shared
 library (plugins)
 in favor of getting linked shared library support done now.
 There are several issues that require more thoughts.

 - Per-thread initialization of modules is somewhat tricky.
 Doing it in Runtime.loadLibrary requires knowledge of shared library
 dependencies
 because different threads might share dependencies but this is not
 provided by libc/libdl.

 - Libraries might not be unloaded as long as GC collected class
 instances still exist because
 finalization fails otherwise.

 - Getting symbols through mangled names is difficult/unstable.

 - D libraries used by a C library should provide proper runtime
 initialization
 even if the C library is used by a D application.

 Any ideas or use-cases for plugins are welcome.

 martin



 - Initializing module infos
 - Initializing exception handling tables
 - Running module constructors
 - Initializing TLS

 Then also unload all this when the library is unloaded.


 It seems that libraries can't be unloaded deterministically,
 because GC finalization still references them.


 On Mac OS X, can't "_dyld_register_func_for_add_image" be used? Then
 it will work, hopefully, transparently for the user. D libraries used
 by C wouldn't need any different handling. Because they will be linked
 with druntime it can initializing everything with the help of
 "_dyld_register_func_for_add_image".


 That was the approach I took and it is partly a dead-end.

 I have a mechanism similar to _dyld_register_func_for_add_image
 but runtime loaders have no notion of per-thread initialization,
 i.e. when two threads load the same library only the first one will
 actually cause the image to be loaded.
 This implies that the second thread would need to check all
 dependencies of the loaded library to do the initialization.
 I've written something along this line but it requires to
 exploit/rewrite part of the runtime linker.
 Using dlmopen on linux would be a terrible inefficient hack
 around this issue, it allows to load libraries multiple times.


I'm not quite sure I understand. Most of the things that should be done, 
initializing module infos and so on, should only be done once.

-- 
/Jacob Carlborg
Jan 02 2012
next sibling parent reply "Martin Nowak" <dawg dawgfoto.de> writes:
On Tue, 03 Jan 2012 08:20:38 +0100, Jacob Carlborg <doob me.com> wrote:


 On 2012-01-02 21:57, Martin Nowak wrote:

 On Mon, 02 Jan 2012 20:38:50 +0100, Jacob Carlborg <doob me.com> wrote:


 On 2012-01-02 20:20, Martin Nowak wrote:

 I think that I'll defer the support for runtime loading of shared
 library (plugins)
 in favor of getting linked shared library support done now.
 There are several issues that require more thoughts.

 - Per-thread initialization of modules is somewhat tricky.
 Doing it in Runtime.loadLibrary requires knowledge of shared library
 dependencies
 because different threads might share dependencies but this is not
 provided by libc/libdl.

 - Libraries might not be unloaded as long as GC collected class
 instances still exist because
 finalization fails otherwise.

 - Getting symbols through mangled names is difficult/unstable.

 - D libraries used by a C library should provide proper runtime
 initialization
 even if the C library is used by a D application.

 Any ideas or use-cases for plugins are welcome.

 martin



 - Initializing module infos
 - Initializing exception handling tables
 - Running module constructors
 - Initializing TLS

 Then also unload all this when the library is unloaded.


 It seems that libraries can't be unloaded deterministically,
 because GC finalization still references them.


 On Mac OS X, can't "_dyld_register_func_for_add_image" be used? Then
 it will work, hopefully, transparently for the user. D libraries used
 by C wouldn't need any different handling. Because they will be linked
 with druntime it can initializing everything with the help of
 "_dyld_register_func_for_add_image".


 That was the approach I took and it is partly a dead-end.

 I have a mechanism similar to _dyld_register_func_for_add_image
 but runtime loaders have no notion of per-thread initialization,
 i.e. when two threads load the same library only the first one will
 actually cause the image to be loaded.
 This implies that the second thread would need to check all
 dependencies of the loaded library to do the initialization.
 I've written something along this line but it requires to
 exploit/rewrite part of the runtime linker.
 Using dlmopen on linux would be a terrible inefficient hack
 around this issue, it allows to load libraries multiple times.


 I'm not quite sure I understand. Most of the things that should be done,  
 initializing module infos and so on, should only be done once.


Yes most, but not all.
The core issue here is that C++'s __thread doesn't allow dynamic  
initializers,
thus there is no infrastructure to do such things. And really a clean  
approach
would be to extend libc/ld.so.
Jan 03 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-01-03 17:53, Martin Nowak wrote:

 Yes most, but not all.
 The core issue here is that C++'s __thread doesn't allow dynamic
 initializers,
 thus there is no infrastructure to do such things. And really a clean
 approach
 would be to extend libc/ld.so.


First, __thread isn't supported in Mac OS X (if we're talking about 
that). For all the operating systems that do support TLS I'm pretty sure 
that TLS and dynamic libraries work.

This documentation:

http://www.akkadia.org/drepper/tls.pdf

mentions several different TLS modes, some used for dynamic libraries 
and some used for other cases.

-- 
/Jacob Carlborg
Jan 03 2012
next sibling parent reply "Martin Nowak" <dawg dawgfoto.de> writes:
On Tue, 03 Jan 2012 18:44:27 +0100, Jacob Carlborg <doob me.com> wrote:


 On 2012-01-03 17:53, Martin Nowak wrote:

 Yes most, but not all.
 The core issue here is that C++'s __thread doesn't allow dynamic
 initializers,
 thus there is no infrastructure to do such things. And really a clean
 approach
 would be to extend libc/ld.so.


 First, __thread isn't supported in Mac OS X (if we're talking about  
 that). For all the operating systems that do support TLS I'm pretty sure  
 that TLS and dynamic libraries work.

 This documentation:

 http://www.akkadia.org/drepper/tls.pdf

 mentions several different TLS modes, some used for dynamic libraries  
 and some used for other cases.


C++ takes a very simple approach here.
You can have
     __thread int a = 3;
but not
     __thread int a = geta();
     error: 'a' is thread-local and so cannot be dynamically initialized
Jan 03 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-01-03 18:51, Martin Nowak wrote:

 On Tue, 03 Jan 2012 18:44:27 +0100, Jacob Carlborg <doob me.com> wrote:


 On 2012-01-03 17:53, Martin Nowak wrote:

 Yes most, but not all.
 The core issue here is that C++'s __thread doesn't allow dynamic
 initializers,
 thus there is no infrastructure to do such things. And really a clean
 approach
 would be to extend libc/ld.so.


 First, __thread isn't supported in Mac OS X (if we're talking about
 that). For all the operating systems that do support TLS I'm pretty
 sure that TLS and dynamic libraries work.

 This documentation:

 http://www.akkadia.org/drepper/tls.pdf

 mentions several different TLS modes, some used for dynamic libraries
 and some used for other cases.


 C++ takes a very simple approach here.
 You can have
 __thread int a = 3;
 but not
 __thread int a = geta();
 error: 'a' is thread-local and so cannot be dynamically initialized


Oh, you mean dynamic like that. I thought you meant accessing "a" from a 
dynamic library.

-- 
/Jacob Carlborg
Jan 03 2012
prev sibling next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
__thread is supported under Lion via Clang.=20

Sent from my iPhone

On Jan 3, 2012, at 9:44 AM, Jacob Carlborg <doob me.com> wrote:


 On 2012-01-03 17:53, Martin Nowak wrote:

 Yes most, but not all.
 The core issue here is that C++'s __thread doesn't allow dynamic
 initializers,
 thus there is no infrastructure to do such things. And really a clean
 approach
 would be to extend libc/ld.so.


=20
 First, __thread isn't supported in Mac OS X (if we're talking about that).=


 For all the operating systems that do support TLS I'm pretty sure that TLS a=
nd dynamic libraries work.

=20
 This documentation:
=20
 http://www.akkadia.org/drepper/tls.pdf
=20
 mentions several different TLS modes, some used for dynamic libraries and s=


ome used for other cases.

=20
 --=20
 /Jacob Carlborg
Jan 03 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-01-03 22:46, Sean Kelly wrote:

 __thread is supported under Lion via Clang.


Yeah, yeah I know, sorry. I'm keep forgetting that. But we're currently 
not using that and we're still rolling our own implementation.

-- 
/Jacob Carlborg
Jan 03 2012
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
I thought C++0x allowed TLS class instances?

Sent from my iPhone

On Jan 3, 2012, at 9:51 AM, "Martin Nowak" <dawg dawgfoto.de> wrote:


 On Tue, 03 Jan 2012 18:44:27 +0100, Jacob Carlborg <doob me.com> wrote:
=20

 On 2012-01-03 17:53, Martin Nowak wrote:

 Yes most, but not all.
 The core issue here is that C++'s __thread doesn't allow dynamic
 initializers,
 thus there is no infrastructure to do such things. And really a clean
 approach
 would be to extend libc/ld.so.


=20
 First, __thread isn't supported in Mac OS X (if we're talking about that)=




. For all the operating systems that do support TLS I'm pretty sure that TLS=
 and dynamic libraries work.

=20
 This documentation:
=20
 http://www.akkadia.org/drepper/tls.pdf
=20
 mentions several different TLS modes, some used for dynamic libraries and=




 some used for other cases.

=20


=20
 C++ takes a very simple approach here.
 You can have
    __thread int a =3D 3;
 but not
    __thread int a =3D geta();
    error: 'a' is thread-local and so cannot be dynamically initialized
Jan 03 2012
prev sibling next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
The trick seems to be mapping in TLS (on OSX anyway) and running static cror=
e at the right time. Are there other issues as well?

Sent from my iPhone

On Jan 3, 2012, at 8:53 AM, "Martin Nowak" <dawg dawgfoto.de> wrote:


 On Tue, 03 Jan 2012 08:20:38 +0100, Jacob Carlborg <doob me.com> wrote:
=20

 On 2012-01-02 21:57, Martin Nowak wrote:

 On Mon, 02 Jan 2012 20:38:50 +0100, Jacob Carlborg <doob me.com> wrote:
=20

 On 2012-01-02 20:20, Martin Nowak wrote:

 I think that I'll defer the support for runtime loading of shared
 library (plugins)
 in favor of getting linked shared library support done now.
 There are several issues that require more thoughts.
=20
 - Per-thread initialization of modules is somewhat tricky.
 Doing it in Runtime.loadLibrary requires knowledge of shared library
 dependencies
 because different threads might share dependencies but this is not
 provided by libc/libdl.
=20
 - Libraries might not be unloaded as long as GC collected class
 instances still exist because
 finalization fails otherwise.
=20
 - Getting symbols through mangled names is difficult/unstable.
=20
 - D libraries used by a C library should provide proper runtime
 initialization
 even if the C library is used by a D application.
=20
 Any ideas or use-cases for plugins are welcome.
=20
 martin


=20
=20
 - Initializing module infos
 - Initializing exception handling tables
 - Running module constructors
 - Initializing TLS
=20
 Then also unload all this when the library is unloaded.
=20


 It seems that libraries can't be unloaded deterministically,
 because GC finalization still references them.
=20

 On Mac OS X, can't "_dyld_register_func_for_add_image" be used? Then
 it will work, hopefully, transparently for the user. D libraries used
 by C wouldn't need any different handling. Because they will be linked
 with druntime it can initializing everything with the help of
 "_dyld_register_func_for_add_image".
=20


=20
 That was the approach I took and it is partly a dead-end.
=20
 I have a mechanism similar to _dyld_register_func_for_add_image
 but runtime loaders have no notion of per-thread initialization,
 i.e. when two threads load the same library only the first one will
 actually cause the image to be loaded.
 This implies that the second thread would need to check all
 dependencies of the loaded library to do the initialization.
 I've written something along this line but it requires to
 exploit/rewrite part of the runtime linker.
 Using dlmopen on linux would be a terrible inefficient hack
 around this issue, it allows to load libraries multiple times.


=20
 I'm not quite sure I understand. Most of the things that should be done, i=




nitializing module infos and so on, should only be done once.

=20


 Yes most, but not all.
 The core issue here is that C++'s __thread doesn't allow dynamic initializ=


ers,

 thus there is no infrastructure to do such things. And really a clean appr=


oach

 would be to extend libc/ld.so.
Jan 03 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-01-03 18:27, Sean Kelly wrote:

 The trick seems to be mapping in TLS (on OSX anyway) and running static crore
at the right time. Are there other issues as well?


TLS has at least been an issue to me when trying to implement support 
for dynamic libraries on Mac OS X. I have no idea how the 
___tls_get_addr function should be implemented to support dynamic 
libraries, especially not since we're rolling our own implementation and 
there's no documentation to follow.

I don't think there's any problems with the static constructors.

I've posted about my problems before at the DMD internals mailing list:

http://dfeed.kimsufi.thecybershadow.net/discussion/post/F2166CBA-4F77-49F7-9949-9F3666C12840 me.com

-- 
/Jacob Carlborg
Jan 03 2012
parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Tue, 03 Jan 2012 18:56:56 +0100, Jacob Carlborg <doob me.com> wrote:


 On 2012-01-03 18:27, Sean Kelly wrote:

 The trick seems to be mapping in TLS (on OSX anyway) and running static  
 crore at the right time. Are there other issues as well?


 TLS has at least been an issue to me when trying to implement support  
 for dynamic libraries on Mac OS X. I have no idea how the  
 ___tls_get_addr function should be implemented to support dynamic  
 libraries, especially not since we're rolling our own implementation and  
 there's no documentation to follow.

 I don't think there's any problems with the static constructors.

 I've posted about my problems before at the DMD internals mailing list:

 http://dfeed.kimsufi.thecybershadow.net/discussion/post/F2166CBA-4F77-49F7-9949-9F3666C12840 me.com


Without support from the OS (TLS segment register)
and none of the linker (module index as dynamic TLS relocations)
any implementation will be severely flawed/inefficient.

As a workaround you could determine the library that called
__tls_get_addr by the return address, e.g. through  
__builtin_return_address(0),
and a search it among the .text ranges of all loaded libraries.
Jan 03 2012
prev sibling next sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Tue, 03 Jan 2012 18:27:56 +0100, Sean Kelly <sean invisibleduck.org>  
wrote:


 The trick seems to be mapping in TLS (on OSX anyway) and running static  
 crore at the right time. Are there other issues as well?


I was hoping to hook thread local module ctors to TLS initialization
which is already done lazily, but the semantics of 'static this()'
allow to run arbitrary code, so the right time currently is before any
code/data from that library can be accessed by this particular thread.
This necessitates to initialize all library dependencies as well.

Implementing dynamic TLS support for OSX might lead to some useful  
findings.


 Sent from my iPhone

 On Jan 3, 2012, at 8:53 AM, "Martin Nowak" <dawg dawgfoto.de> wrote:


 On Tue, 03 Jan 2012 08:20:38 +0100, Jacob Carlborg <doob me.com> wrote:


 On 2012-01-02 21:57, Martin Nowak wrote:

 On Mon, 02 Jan 2012 20:38:50 +0100, Jacob Carlborg <doob me.com>  
 wrote:


 On 2012-01-02 20:20, Martin Nowak wrote:

 I think that I'll defer the support for runtime loading of shared
 library (plugins)
 in favor of getting linked shared library support done now.
 There are several issues that require more thoughts.

 - Per-thread initialization of modules is somewhat tricky.
 Doing it in Runtime.loadLibrary requires knowledge of shared library
 dependencies
 because different threads might share dependencies but this is not
 provided by libc/libdl.

 - Libraries might not be unloaded as long as GC collected class
 instances still exist because
 finalization fails otherwise.

 - Getting symbols through mangled names is difficult/unstable.

 - D libraries used by a C library should provide proper runtime
 initialization
 even if the C library is used by a D application.

 Any ideas or use-cases for plugins are welcome.

 martin



 - Initializing module infos
 - Initializing exception handling tables
 - Running module constructors
 - Initializing TLS

 Then also unload all this when the library is unloaded.


 It seems that libraries can't be unloaded deterministically,
 because GC finalization still references them.


 On Mac OS X, can't "_dyld_register_func_for_add_image" be used? Then
 it will work, hopefully, transparently for the user. D libraries used
 by C wouldn't need any different handling. Because they will be  
 linked
 with druntime it can initializing everything with the help of
 "_dyld_register_func_for_add_image".


 That was the approach I took and it is partly a dead-end.

 I have a mechanism similar to _dyld_register_func_for_add_image
 but runtime loaders have no notion of per-thread initialization,
 i.e. when two threads load the same library only the first one will
 actually cause the image to be loaded.
 This implies that the second thread would need to check all
 dependencies of the loaded library to do the initialization.
 I've written something along this line but it requires to
 exploit/rewrite part of the runtime linker.
 Using dlmopen on linux would be a terrible inefficient hack
 around this issue, it allows to load libraries multiple times.


 I'm not quite sure I understand. Most of the things that should be  
 done, initializing module infos and so on, should only be done once.


 Yes most, but not all.
 The core issue here is that C++'s __thread doesn't allow dynamic  
 initializers,
 thus there is no infrastructure to do such things. And really a clean  
 approach
 would be to extend libc/ld.so.
Jan 03 2012
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
Shouldn't be terrible then. Have a routine in the lib that returns a referen=
ce to whatever, and have library map it in. Unloading would be tricky though=
, for the reasons you mention. Probably possible though by copying the stuff=
 to be mapped in into GCed memory. Possibly even simply have the GC track th=
at memory in a way similar to how Andeei suggested we handle mmap.=20

Sent from my iPhone

On Jan 3, 2012, at 9:47 AM, "Martin Nowak" <dawg dawgfoto.de> wrote:


 On Tue, 03 Jan 2012 18:27:56 +0100, Sean Kelly <sean invisibleduck.org> wr=


ote:

=20

 The trick seems to be mapping in TLS (on OSX anyway) and running static c=




rore at the right time. Are there other issues as well?

=20


 I was hoping to hook thread local module ctors to TLS initialization
 which is already done lazily, but the semantics of 'static this()'
 allow to run arbitrary code, so the right time currently is before any
 code/data from that library can be accessed by this particular thread.
 This necessitates to initialize all library dependencies as well.
=20
 Implementing dynamic TLS support for OSX might lead to some useful finding=


s.

=20

 Sent from my iPhone
=20
 On Jan 3, 2012, at 8:53 AM, "Martin Nowak" <dawg dawgfoto.de> wrote:
=20

 On Tue, 03 Jan 2012 08:20:38 +0100, Jacob Carlborg <doob me.com> wrote:
=20

 On 2012-01-02 21:57, Martin Nowak wrote:

 On Mon, 02 Jan 2012 20:38:50 +0100, Jacob Carlborg <doob me.com> wrote=










:

=20

 On 2012-01-02 20:20, Martin Nowak wrote:

 I think that I'll defer the support for runtime loading of shared
 library (plugins)
 in favor of getting linked shared library support done now.
 There are several issues that require more thoughts.
=20
 - Per-thread initialization of modules is somewhat tricky.
 Doing it in Runtime.loadLibrary requires knowledge of shared library=















 dependencies
 because different threads might share dependencies but this is not
 provided by libc/libdl.
=20
 - Libraries might not be unloaded as long as GC collected class
 instances still exist because
 finalization fails otherwise.
=20
 - Getting symbols through mangled names is difficult/unstable.
=20
 - D libraries used by a C library should provide proper runtime
 initialization
 even if the C library is used by a D application.
=20
 Any ideas or use-cases for plugins are welcome.
=20
 martin


=20
=20
 - Initializing module infos
 - Initializing exception handling tables
 - Running module constructors
 - Initializing TLS
=20
 Then also unload all this when the library is unloaded.
=20


 It seems that libraries can't be unloaded deterministically,
 because GC finalization still references them.
=20

 On Mac OS X, can't "_dyld_register_func_for_add_image" be used? Then
 it will work, hopefully, transparently for the user. D libraries used=













 by C wouldn't need any different handling. Because they will be linke=












d

 with druntime it can initializing everything with the help of
 "_dyld_register_func_for_add_image".
=20


=20
 That was the approach I took and it is partly a dead-end.
=20
 I have a mechanism similar to _dyld_register_func_for_add_image
 but runtime loaders have no notion of per-thread initialization,
 i.e. when two threads load the same library only the first one will
 actually cause the image to be loaded.
 This implies that the second thread would need to check all
 dependencies of the loaded library to do the initialization.
 I've written something along this line but it requires to
 exploit/rewrite part of the runtime linker.
 Using dlmopen on linux would be a terrible inefficient hack
 around this issue, it allows to load libraries multiple times.


=20
 I'm not quite sure I understand. Most of the things that should be done=








, initializing module infos and so on, should only be done once.

=20


 Yes most, but not all.
 The core issue here is that C++'s __thread doesn't allow dynamic initial=






izers,

 thus there is no infrastructure to do such things. And really a clean ap=






proach

 would be to extend libc/ld.so.
Jan 03 2012
prev sibling next sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Tue, 03 Jan 2012 22:49:57 +0100, Sean Kelly <sean invisibleduck.org>  
wrote:


 Shouldn't be terrible then. Have a routine in the lib that returns a  
 reference to whatever, and have library map it in. Unloading would be  
 tricky though, for the reasons you mention. Probably possible though by  
 copying the stuff to be mapped in into GCed memory.


This is a bad solution it would require to relocate all classinfo
pointers at runtime and even worse move class initializer into a writeable
segment, thus reduce process memory sharing.


 Possibly even simply have the GC track that memory in
 a way similar to how Andeei suggested we handle mmap.


What exactly does he suggest?
But extending the GC seems like a feasible way.
This could be done by a very general interface of the garbage collector.

GC.trackRange(void* p, size_t sz, void function(void* p) finalizer);

OTOH it will be difficult w.r.t. performance.


 Sent from my iPhone

 On Jan 3, 2012, at 9:47 AM, "Martin Nowak" <dawg dawgfoto.de> wrote:


 On Tue, 03 Jan 2012 18:27:56 +0100, Sean Kelly <sean invisibleduck.org>  
 wrote:


 The trick seems to be mapping in TLS (on OSX anyway) and running  
 static crore at the right time. Are there other issues as well?


 I was hoping to hook thread local module ctors to TLS initialization
 which is already done lazily, but the semantics of 'static this()'
 allow to run arbitrary code, so the right time currently is before any
 code/data from that library can be accessed by this particular thread.
 This necessitates to initialize all library dependencies as well.

 Implementing dynamic TLS support for OSX might lead to some useful  
 findings.


 Sent from my iPhone

 On Jan 3, 2012, at 8:53 AM, "Martin Nowak" <dawg dawgfoto.de> wrote:


 On Tue, 03 Jan 2012 08:20:38 +0100, Jacob Carlborg <doob me.com>  
 wrote:


 On 2012-01-02 21:57, Martin Nowak wrote:

 On Mon, 02 Jan 2012 20:38:50 +0100, Jacob Carlborg <doob me.com>  
 wrote:


 On 2012-01-02 20:20, Martin Nowak wrote:

 I think that I'll defer the support for runtime loading of shared
 library (plugins)
 in favor of getting linked shared library support done now.
 There are several issues that require more thoughts.

 - Per-thread initialization of modules is somewhat tricky.
 Doing it in Runtime.loadLibrary requires knowledge of shared  
 library
 dependencies
 because different threads might share dependencies but this is not
 provided by libc/libdl.

 - Libraries might not be unloaded as long as GC collected class
 instances still exist because
 finalization fails otherwise.

 - Getting symbols through mangled names is difficult/unstable.

 - D libraries used by a C library should provide proper runtime
 initialization
 even if the C library is used by a D application.

 Any ideas or use-cases for plugins are welcome.

 martin



 - Initializing module infos
 - Initializing exception handling tables
 - Running module constructors
 - Initializing TLS

 Then also unload all this when the library is unloaded.


 It seems that libraries can't be unloaded deterministically,
 because GC finalization still references them.


 On Mac OS X, can't "_dyld_register_func_for_add_image" be used?  
 Then
 it will work, hopefully, transparently for the user. D libraries  
 used
 by C wouldn't need any different handling. Because they will be  
 linked
 with druntime it can initializing everything with the help of
 "_dyld_register_func_for_add_image".


 That was the approach I took and it is partly a dead-end.

 I have a mechanism similar to _dyld_register_func_for_add_image
 but runtime loaders have no notion of per-thread initialization,
 i.e. when two threads load the same library only the first one will
 actually cause the image to be loaded.
 This implies that the second thread would need to check all
 dependencies of the loaded library to do the initialization.
 I've written something along this line but it requires to
 exploit/rewrite part of the runtime linker.
 Using dlmopen on linux would be a terrible inefficient hack
 around this issue, it allows to load libraries multiple times.


 I'm not quite sure I understand. Most of the things that should be  
 done, initializing module infos and so on, should only be done once.


 Yes most, but not all.
 The core issue here is that C++'s __thread doesn't allow dynamic  
 initializers,
 thus there is no infrastructure to do such things. And really a clean  
 approach
 would be to extend libc/ld.so.
Jan 03 2012
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
That's roughly what he suggested. It's in an old thread, either here or the d=
runtime mailing list. The idea was to have the GC be in charge of releasing e=
ven non-GC memory to ensure that no dangling reference issues exist, IIRC.=20=


Sent from my iPhone

On Jan 3, 2012, at 2:56 PM, "Martin Nowak" <dawg dawgfoto.de> wrote:


 On Tue, 03 Jan 2012 22:49:57 +0100, Sean Kelly <sean invisibleduck.org> wr=


ote:

=20

 Shouldn't be terrible then. Have a routine in the lib that returns a refe=




rence to whatever, and have library map it in. Unloading would be tricky tho=
ugh, for the reasons you mention. Probably possible though by copying the st=
uff to be mapped in into GCed memory.

 This is a bad solution it would require to relocate all classinfo
 pointers at runtime and even worse move class initializer into a writeable=



 segment, thus reduce process memory sharing.
=20

 Possibly even simply have the GC track that memory in
 a way similar to how Andeei suggested we handle mmap.
=20


 What exactly does he suggest?
 But extending the GC seems like a feasible way.
 This could be done by a very general interface of the garbage collector.
=20
 GC.trackRange(void* p, size_t sz, void function(void* p) finalizer);
=20
 OTOH it will be difficult w.r.t. performance.
=20

 Sent from my iPhone
=20
 On Jan 3, 2012, at 9:47 AM, "Martin Nowak" <dawg dawgfoto.de> wrote:
=20

 On Tue, 03 Jan 2012 18:27:56 +0100, Sean Kelly <sean invisibleduck.org> w=






rote:

=20

 The trick seems to be mapping in TLS (on OSX anyway) and running static=








 crore at the right time. Are there other issues as well?

=20


 I was hoping to hook thread local module ctors to TLS initialization
 which is already done lazily, but the semantics of 'static this()'
 allow to run arbitrary code, so the right time currently is before any
 code/data from that library can be accessed by this particular thread.
 This necessitates to initialize all library dependencies as well.
=20
 Implementing dynamic TLS support for OSX might lead to some useful findi=






ngs.

=20

 Sent from my iPhone
=20
 On Jan 3, 2012, at 8:53 AM, "Martin Nowak" <dawg dawgfoto.de> wrote:
=20

 On Tue, 03 Jan 2012 08:20:38 +0100, Jacob Carlborg <doob me.com> wrote=










:

=20

 On 2012-01-02 21:57, Martin Nowak wrote:

 On Mon, 02 Jan 2012 20:38:50 +0100, Jacob Carlborg <doob me.com> wro=














te:

=20

 On 2012-01-02 20:20, Martin Nowak wrote:

 I think that I'll defer the support for runtime loading of shared
 library (plugins)
 in favor of getting linked shared library support done now.
 There are several issues that require more thoughts.
=20
 - Per-thread initialization of modules is somewhat tricky.
 Doing it in Runtime.loadLibrary requires knowledge of shared libra=


















ry

 dependencies
 because different threads might share dependencies but this is not=



















 provided by libc/libdl.
=20
 - Libraries might not be unloaded as long as GC collected class
 instances still exist because
 finalization fails otherwise.
=20
 - Getting symbols through mangled names is difficult/unstable.
=20
 - D libraries used by a C library should provide proper runtime
 initialization
 even if the C library is used by a D application.
=20
 Any ideas or use-cases for plugins are welcome.
=20
 martin


=20
=20
 - Initializing module infos
 - Initializing exception handling tables
 - Running module constructors
 - Initializing TLS
=20
 Then also unload all this when the library is unloaded.
=20


 It seems that libraries can't be unloaded deterministically,
 because GC finalization still references them.
=20

 On Mac OS X, can't "_dyld_register_func_for_add_image" be used? The=
















n

 it will work, hopefully, transparently for the user. D libraries us=
















ed

 by C wouldn't need any different handling. Because they will be lin=
















ked

 with druntime it can initializing everything with the help of
 "_dyld_register_func_for_add_image".
=20


=20
 That was the approach I took and it is partly a dead-end.
=20
 I have a mechanism similar to _dyld_register_func_for_add_image
 but runtime loaders have no notion of per-thread initialization,
 i.e. when two threads load the same library only the first one will
 actually cause the image to be loaded.
 This implies that the second thread would need to check all
 dependencies of the loaded library to do the initialization.
 I've written something along this line but it requires to
 exploit/rewrite part of the runtime linker.
 Using dlmopen on linux would be a terrible inefficient hack
 around this issue, it allows to load libraries multiple times.


=20
 I'm not quite sure I understand. Most of the things that should be do=












ne, initializing module infos and so on, should only be done once.

=20


 Yes most, but not all.
 The core issue here is that C++'s __thread doesn't allow dynamic initi=










alizers,

 thus there is no infrastructure to do such things. And really a clean a=










pproach

 would be to extend libc/ld.so.
Jan 03 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-01-02 21:57, Martin Nowak wrote:

 Then also unload all this when the library is unloaded.


 It seems that libraries can't be unloaded deterministically,
 because GC finalization still references them.


Could you elaborate this, I guess I'm not really familiar with the GC 
finalization.

-- 
/Jacob Carlborg
Jan 02 2012
parent reply "Martin Nowak" <dawg dawgfoto.de> writes:
On Tue, 03 Jan 2012 08:22:47 +0100, Jacob Carlborg <doob me.com> wrote:


 On 2012-01-02 21:57, Martin Nowak wrote:

 Then also unload all this when the library is unloaded.


 It seems that libraries can't be unloaded deterministically,
 because GC finalization still references them.


 Could you elaborate this, I guess I'm not really familiar with the GC  
 finalization.


I stumbled over this issue while doing extended testing.
Consider unloading of a library, this requires you to not
use any data/functions from that library afterward.
If you have instantiated a class from that library the GC
will access it's classinfo when collecting it, but the
classinfo resides in the unloaded library as do finalizers.

Not collection all classes from a shared library before
unloading that library is currently impracticable.

We could allocate classes from a specific allocator associated
with the library, somewhat similar to how TLS is allocated for
each library.
Jan 03 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-01-03 17:49, Martin Nowak wrote:

 On Tue, 03 Jan 2012 08:22:47 +0100, Jacob Carlborg <doob me.com> wrote:


 On 2012-01-02 21:57, Martin Nowak wrote:

 Then also unload all this when the library is unloaded.


 It seems that libraries can't be unloaded deterministically,
 because GC finalization still references them.


 Could you elaborate this, I guess I'm not really familiar with the GC
 finalization.


 I stumbled over this issue while doing extended testing.
 Consider unloading of a library, this requires you to not
 use any data/functions from that library afterward.
 If you have instantiated a class from that library the GC
 will access it's classinfo when collecting it, but the
 classinfo resides in the unloaded library as do finalizers.


Do they? Isn't classinfo part of the module info and the module info 
needs to be collected in one place otherwise ClassInfo.find and similar 
methods won't work correctly?

Or can those functions be implemented to iterate all loaded libraries 
and then look through each module info?


 Not collection all classes from a shared library before
 unloading that library is currently impracticable.


Can't the runtime just run all finalizers for a given library when it's 
unloaded?

-- 
/Jacob Carlborg
Jan 03 2012
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
On 01/02/2012 08:20 PM, Martin Nowak wrote:

 I think that I'll defer the support for runtime loading of shared
 library (plugins)
 in favor of getting linked shared library support done now.
 There are several issues that require more thoughts.

   - Per-thread initialization of modules is somewhat tricky.
     Doing it in Runtime.loadLibrary requires knowledge of shared library
 dependencies
     because different threads might share dependencies but this is not
 provided by libc/libdl.

   - D libraries used by a C library should provide proper runtime
 initialization
     even if the C library is used by a D application.


I now think we should do the initialization directly when loading, even 
though that might cause deadlocks in certain scenarios. It would solve 
all mentioned initialization issues. This will also serialize 
initialization on the loader mutex but I don't expect this to become a 
performance issue.


   - Getting symbols through mangled names is difficult/unstable.


ref T loadSym(T)(string fqn);
void loadSym(T)(string fqn, out T res);

T loadFunc(T)(string fqn);
void loadFunc(T)(string fqn, out T res);

auto lval = loadSym!int("std.foo.bar");
loadSym("std.foo.bar", lval);

auto fptr = loadFunc!(bool function(string) pure)("std.foo.func");
loadFunc("std.foo.func", fptr);


   - Libraries might not be unloaded as long as GC collected class
 instances still exist because
     finalization fails otherwise.


We should extend the GC to allow marking and finalizing of foreign 
memory. This is needed for shared memory too.

// gc.gc
// ...
void trackRange(void* p, size_t sz, void delegate(void*, size_t) dg);
// ...

auto p = malloc(1000);
GC.trackRange(p, 1000, (p, sz) => .free(p));


 Any ideas or use-cases for plugins are welcome.


Still holds.


 martin
Mar 07 2013
parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 07 Mar 2013 12:13:03 +0100
schrieb Martin Nowak <code dawg.eu>:


 On 01/02/2012 08:20 PM, Martin Nowak wrote:
 
 I now think we should do the initialization directly when loading,
 even though that might cause deadlocks in certain scenarios. It would
 solve all mentioned initialization issues. This will also serialize 
 initialization on the loader mutex but I don't expect this to become
 a performance issue.


That's the approach used by other runtime libraries, right? So it
should work in most cases.


 

   - Getting symbols through mangled names is difficult/unstable.




With D methods the problem is that the mangled name contains the module
name, so it'll change for every plugin. This is something a reflection
library has to solve:

 plugin("testFunction") void testFunction(){};

auto lib = SharedLib.load();
reflect.getFunction!(void function())("testFunction", lib);

Where the reflection lib would have to build a name->function pointer
map in the shared library and export a C function "void*
getFunction(string name)" which can be accessed via dlsym. (Some more
work could be done to ensure type safety)
(or we just use extern(C) functions)

For OOP features it might be nice if we can iterate all classes that
implement an Interface

foreach(IWebPage page; lib)

Although the question then is how to construct instances of these
classes. Maybe the factory pattern is better for this.


 

   - Libraries might not be unloaded as long as GC collected class
 instances still exist because
     finalization fails otherwise.


 
 We should extend the GC to allow marking and finalizing of foreign 
 memory. This is needed for shared memory too.


Would it be possible to implement this in a way that shared libraries
can be used without the GC in code which doesn't use the GC (of course
with a loss of convenience & safety?)

For example we could introduce this API:

/**
 * Force unload library. All types contained in this library must have
 * been finalized by now  
 */
 system void forceUnload(SharedLib l);  

/**
 * Internally called by the gc on every scan
 */
 safe private void gcCheckUnload()
{
   foreach(SharedLibrary l; loadedLibs)
     if(shouldCollect(l))
         forceUnload(l);
}

/**
 * Optional additional method to force unloading with GC help:
 * Finalize all types from this SharedLib, then unload. If user
 * code still keeps a reference to a type from this lib it'll crash.
 */
 system void forceGCUnload(SharedLib l)
{
    foreach(void* instance; objects)
    {
        if(inLib(instance), l))
            rt_finalize2(instance);
    }
    forceUnload(l);
}  

As far as I understand the issue, C is necessary if we use the GC but
want to force-unload a library: Even if user code no longer references
types from the shared lib the GC might not have ran the finalizers yet.
So this methods finalizes all objects from the shared lib (even if the
user still keeps a reference) and then unloads the lib.

I think this is useful in a environment with a massive number of
loaded / unloaded plugins. For example webpages could be written as
plugins and the webserver could load/unload those. The webserver has to
make sure that the libraries are really unloaded to free some
resources (AFAIR mono/asp.net implements webpages as such plugins) 


 

 Any ideas or use-cases for plugins are welcome.


 
 Still holds.


Use cases?

* Web pages as plugins
* video/audio filtering framework (avisynth, gstreamer)
* syntax highlighting for unknown languages in text editor
* adding support for a new languages / tools to IDEs 
* Filters in image editing tools
* plugins for webbrowsers (adblocker, dom manipulation)
* Adding new import / export formats to almost any kind of
  editor / viewer
* Adding web site / media information scrapers to media players (xbmc,
  rhythmbox)
Mar 07 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-03-07 14:53, Johannes Pfau wrote:


 For OOP features it might be nice if we can iterate all classes that
 implement an Interface

 foreach(IWebPage page; lib)

 Although the question then is how to construct instances of these
 classes. Maybe the factory pattern is better for this.


Just iterate all classes, like ClassInfo.find does. Add a condition to 
check if a given class implements a given interface. If the correct 
class if found create it with Object.factory. Tango contains a module, 
tango.core.RuntimeTraits, with a function that checks if a class 
implements an interface. I'm pretty sure that Phobos has something similar.

We could create a standard plugin interface in Phobos or druntime, 
something like:

interface Plugin
{
     void initialize ();
     void terminate ();
}

Every plugin should implement this directly or indirectly. "initialize" 
would be called just after the plugin is loaded and "terminate" would be 
called just before unloading.

-- 
/Jacob Carlborg
Mar 07 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/2/2012 11:20 AM, Martin Nowak wrote:

 I think that I'll defer the support for runtime loading of shared library
(plugins)
 in favor of getting linked shared library support done now.
 There are several issues that require more thoughts.

   - Per-thread initialization of modules is somewhat tricky.
     Doing it in Runtime.loadLibrary requires knowledge of shared library
 dependencies
     because different threads might share dependencies but this is not provided
 by libc/libdl.

   - Libraries might not be unloaded as long as GC collected class instances
 still exist because
     finalization fails otherwise.

   - Getting symbols through mangled names is difficult/unstable.

   - D libraries used by a C library should provide proper runtime
initialization
     even if the C library is used by a D application.

 Any ideas or use-cases for plugins are welcome.


My idea for this is straightforward. A D plugin should use a C interface. That 
means, for example, that a D plugin should NOT pass GC allocated references to 
its caller. Only malloc'd data should pass between the plugin and its caller.

The D plugin could still use the shared druntime.so, and could share that usage 
with the D caller of the plugin.
Mar 07 2013
next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, March 07, 2013 13:48:21 Walter Bright wrote:

 My idea for this is straightforward. A D plugin should use a C interface.
 That means, for example, that a D plugin should NOT pass GC allocated
 references to its caller. Only malloc'd data should pass between the plugin
 and its caller.
 
 The D plugin could still use the shared druntime.so, and could share that
 usage with the D caller of the plugin.


That may very well end up being a necessary restriction, but not being able to
pass
GC allocated objects across library boundaries would be a _huge_ downside to
using
shared libraries if that happens. It also would likely be a major source of bugs
as it's pretty much a guarantee that a lot of people would do it unless the
compiler
prevented them in some manner.

- Jonathan M Davis
Mar 07 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 4:44 PM, Jonathan M Davis wrote:

 That may very well end up being a necessary restriction, but not being able to
pass
 GC allocated objects across library boundaries would be a _huge_ downside to
using
 shared libraries if that happens. It also would likely be a major source of
bugs
 as it's pretty much a guarantee that a lot of people would do it unless the
compiler
 prevented them in some manner.


You're probably right. See my other idea on this posted here.
Mar 07 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-03-07 22:48, Walter Bright wrote:


 My idea for this is straightforward. A D plugin should use a C
 interface. That means, for example, that a D plugin should NOT pass GC
 allocated references to its caller. Only malloc'd data should pass
 between the plugin and its caller.


If this restriction is needed I think it's very unfortunate. I had 
really hoped to be able to have an OO interface.

Objective-C can load plugins with an OO interface an Objective-C can be 
used with a GC.

-- 
/Jacob Carlborg
Mar 07 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 11:37 PM, Jacob Carlborg wrote:

 If this restriction is needed I think it's very unfortunate. I had really hoped
 to be able to have an OO interface.

 Objective-C can load plugins with an OO interface an Objective-C can be used
 with a GC.



Read my other posts in this thread!
Mar 08 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/2/2012 11:20 AM, Martin Nowak wrote:

   - Libraries might not be unloaded as long as GC collected class instances
 still exist because
     finalization fails otherwise.


D doesn't guarantee that finalizers will run on GC allocated objects.
Therefore, 
when unloading a dll:

1. run a gc collection
2. for all objects remaining on the heap
        if they have a finalizer and that finalizer points into the dll code
             mark them as not having a finalizer
Mar 07 2013
next sibling parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:khban1$1lm2$1 digitalmars.com...

 On 1/2/2012 11:20 AM, Martin Nowak wrote:

   - Libraries might not be unloaded as long as GC collected class 
 instances
 still exist because
     finalization fails otherwise.


 D doesn't guarantee that finalizers will run on GC allocated objects. 
 Therefore, when unloading a dll:

 1. run a gc collection
 2. for all objects remaining on the heap
        if they have a finalizer and that finalizer points into the dll 
 code
             mark them as not having a finalizer


What if their vtbl points into the dll code?

What about delegates or function pointers that point there?
Mar 07 2013
next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, March 08, 2013 13:18:31 Daniel Murphy wrote:

 What if their vtbl points into the dll code?
 
 What about delegates or function pointers that point there?


Assuming that that sort of thing could be detected, then an exception could be 
thrown when attempting to unload the dll.

- Jonathan M Davis
Mar 07 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 6:18 PM, Daniel Murphy wrote:

 "Walter Bright" <newshound2 digitalmars.com> wrote in message
 news:khban1$1lm2$1 digitalmars.com...

 On 1/2/2012 11:20 AM, Martin Nowak wrote:

    - Libraries might not be unloaded as long as GC collected class
 instances
 still exist because
      finalization fails otherwise.


 D doesn't guarantee that finalizers will run on GC allocated objects.
 Therefore, when unloading a dll:

 1. run a gc collection
 2. for all objects remaining on the heap
         if they have a finalizer and that finalizer points into the dll
 code
              mark them as not having a finalizer


 What if their vtbl points into the dll code?

 What about delegates or function pointers that point there?


These are problems with *any* dynamic dll code. The answer is to tell the user 
"don't do that". The user should NEVER continue to use objects created by that 
dll or delegates/functionpointers/datapointers that refer to it.

The gc problem, however, is not tractable for the user, so it must be dealt
with 
by us, and can be dealt with using the method I described.
Mar 07 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 7:22 PM, Walter Bright wrote:

 On 3/7/2013 6:18 PM, Daniel Murphy wrote:

 What if their vtbl points into the dll code?

 What about delegates or function pointers that point there?


 These are problems with *any* dynamic dll code. The answer is to tell the user
 "don't do that". The user should NEVER continue to use objects created by that
 dll or delegates/functionpointers/datapointers that refer to it.



In essence, dynamically loading/unloading dlls is like calling malloc/free - it 
cannot be made  safe.
Mar 07 2013
parent "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:khbloq$284i$1 digitalmars.com...

 On 3/7/2013 7:22 PM, Walter Bright wrote:

 On 3/7/2013 6:18 PM, Daniel Murphy wrote:

 What if their vtbl points into the dll code?

 What about delegates or function pointers that point there?


 These are problems with *any* dynamic dll code. The answer is to tell the 
 user
 "don't do that". The user should NEVER continue to use objects created by 
 that
 dll or delegates/functionpointers/datapointers that refer to it.



 In essence, dynamically loading/unloading dlls is like calling 
 malloc/free - it cannot be made  safe.


I was thinking about implicit unloading.  If this is actually about explicit 
unloading then it makes sense to just be unsafe.
Mar 07 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 8 March 2013 at 03:22:46 UTC, Walter Bright wrote:

 On 3/7/2013 6:18 PM, Daniel Murphy wrote:

 "Walter Bright" <newshound2 digitalmars.com> wrote in message
 news:khban1$1lm2$1 digitalmars.com...

 On 1/2/2012 11:20 AM, Martin Nowak wrote:

   - Libraries might not be unloaded as long as GC collected 
 class
 instances
 still exist because
     finalization fails otherwise.


 D doesn't guarantee that finalizers will run on GC allocated 
 objects.
 Therefore, when unloading a dll:

 1. run a gc collection
 2. for all objects remaining on the heap
        if they have a finalizer and that finalizer points 
 into the dll
 code
             mark them as not having a finalizer


 What if their vtbl points into the dll code?

 What about delegates or function pointers that point there?


 These are problems with *any* dynamic dll code. The answer is 
 to tell the user "don't do that".


That is completely wrong.
Mar 08 2013
prev sibling parent Martin Nowak <code dawg.eu> writes:
On 03/08/2013 04:22 AM, Walter Bright wrote:

 These are problems with *any* dynamic dll code. The answer is to tell
 the user "don't do that". The user should NEVER continue to use objects
 created by that dll or delegates/functionpointers/datapointers that
 refer to it.

 The gc problem, however, is not tractable for the user, so it must be
 dealt with by us, and can be dealt with using the method I described.


Right that's the point.
You wouldn't expect the following code to work either.

auto h = loadLibrary("lib");
auto f = loadFunc(h, "func");
closeLibrary(h);
f();
Mar 08 2013
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-03-08 01:17, Walter Bright wrote:


 D doesn't guarantee that finalizers will run on GC allocated objects.
 Therefore, when unloading a dll:

 1. run a gc collection
 2. for all objects remaining on the heap
         if they have a finalizer and that finalizer points into the dll
 code
              mark them as not having a finalizer


Could we do something like I suggested in another post:

We could create a standard plugin interface in Phobos or druntime, 
something like:

interface Plugin
{
     void initialize ();
     void terminate ();
}

Every plugin should implement this directly or indirectly. "initialize" 
would be called just after the plugin is loaded and "terminate" would be 
called just before unloading.

The plugin would then have a possibility to cleanup after itself.

-- 
/Jacob Carlborg
Mar 08 2013
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, March 08, 2013 10:09:33 Jacob Carlborg wrote:

 On 2013-03-08 01:17, Walter Bright wrote:

 D doesn't guarantee that finalizers will run on GC allocated objects.
 Therefore, when unloading a dll:
 
 1. run a gc collection
 2. for all objects remaining on the heap
 
         if they have a finalizer and that finalizer points into the dll
 
 code
 
              mark them as not having a finalizer


 
 Could we do something like I suggested in another post:
 
 We could create a standard plugin interface in Phobos or druntime,
 something like:
 
 interface Plugin
 {
      void initialize ();
      void terminate ();
 }
 
 Every plugin should implement this directly or indirectly. "initialize"
 would be called just after the plugin is loaded and "terminate" would be
 called just before unloading.
 
 The plugin would then have a possibility to cleanup after itself.


That's basically what static constructors and destructors do, and it'll be a 
problem if they aren't run when the library is loaded and unloaded.

- Jonathan M Davis
Mar 08 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-03-08 10:34, Jonathan M Davis wrote:


 That's basically what static constructors and destructors do, and it'll be a
 problem if they aren't run when the library is loaded and unloaded.


Right, forgot about those.

-- 
/Jacob Carlborg
Mar 08 2013
prev sibling parent Martin Nowak <code dawg.eu> writes:
On 03/08/2013 01:17 AM, Walter Bright wrote:

 On 1/2/2012 11:20 AM, Martin Nowak wrote:

   - Libraries might not be unloaded as long as GC collected class
 instances
 still exist because
     finalization fails otherwise.


 D doesn't guarantee that finalizers will run on GC allocated objects.
 Therefore, when unloading a dll:

 1. run a gc collection
 2. for all objects remaining on the heap
         if they have a finalizer and that finalizer points into the dll
 code
              mark them as not having a finalizer


Yeah, that's the pragmatic solution. I don't really like the 2. point 
though but that won't be a problem with the current GC.
It also avoids the false pointer problems of the GC.trackRange approach.
Mar 08 2013