www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Dynamic loading,

reply Carl Sturtivant <sturtivant gmail.com> writes:
--f46d0444ec19b9beab04c7aa4474
Content-Type: text/plain; charset=UTF-8

(By dynamic loading I mean using something like the C library function
dlopen to explicitly load a shared object at runtime. I do not mean dynamic
linking in the usual passive sense.)

I just successfully got working a toy example of dynamically loading (from
a D program) a D shared object and then finding and calling functions
defined in the shared object, and not just with C linkage.

The main program, main.d, was compiled and linked somewhat normally, except
for passing the linker via gcc the flags necessary to ensure that the whole
of libphobos2.a is present and that all symbols in the resulting executable
are exposed dynamically.

The shared object source, dload.d was compiled to an object file containing
position independent code by dmd. Then I invoked the linker explicitly and
had it make a shared object without the D runtime system or Phobos. This is
the novel step, and it enables the shared object to resolve its linkage to
the D runtime system and Phobos at the point of being loaded, via callbacks
to the main program. Thus the troubles of D in shared objects are largely
circumvented. There is only one instance of phobos and D-runtime, in the
main program. (Once phobos and druntime are shared objects in the future
somewhere this will work with no code bloat.)

The static initialization code in dload.d is automatically executed when
the shared object libdload.so is loaded by the main program, because the
linker is also passed a flag indicating the static initialization block's
mangled name, dynamically determined from dload.o before linkage to
libdload.so occurs.

Finally, the mangled names of the functions to load are determined by a
call of a function with C linkage in dload.d from main.d that looks up
those names in an associative array initialized in the static
initialization block of dload.d where those mangled names are directly
available, so that full D linkage can be emulated, at least for functions.

One thing: the garbage collector needs to be aware of static and 'global' D
variables in the shared object. Can a technical expert verify that I've
done the right thing to achieve that happy state of affairs in this unusual
context?

So, what's overlooked here? I know that the static initialization code
cannot successfully throw an exception. Yet if a function in the shared
object is called from the main program and throws an exception, all is
well. (Try these.) See my comments in dload.d about this. What is it about
the implementation of exceptions that's problematic here?

All files attached, including a Makefile with the exact options passed to
dmd, gcc and ld.

--f46d0444ec19b9beab04c7aa4474
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

(By dynamic loading I mean using something like the C library function dlop=
en to explicitly load a shared object at runtime. I do not mean dynamic lin=
king in the usual passive sense.)<br><br>I just successfully got working a =
toy example of dynamically loading (from a D program) a D shared object and=
 then finding and calling functions defined in the shared object, and not j=
ust with C linkage. <br>
<br>The main program, main.d, was compiled and linked somewhat normally, ex=
cept for passing the linker via gcc the flags necessary to ensure that the =
whole of libphobos2.a is present and that all symbols in the resulting exec=
utable are exposed dynamically.<br>
<br>The shared object source, dload.d was compiled to an object file contai=
ning position independent code by dmd. Then I invoked the linker explicitly=
 and had it make a shared object without the D runtime system or Phobos. Th=
is is the novel step, and it enables the shared object to resolve its linka=
ge to the D runtime system and Phobos at the point of being loaded, via cal=
lbacks to the main program. Thus the troubles of D in shared objects are la=
rgely circumvented. There is only one instance of phobos and D-runtime, in =
the main program. (Once phobos and druntime are shared objects in the futur=
e somewhere this will work with no code bloat.)<br>
<br>The static initialization code in dload.d is automatically executed whe=
n the shared object libdload.so is loaded by the main program, because the =

;s mangled name, dynamically determined from dload.o before linkage to libd=
load.so occurs.<br>
<br>Finally, the mangled names of the functions to load are determined by a=
 call of a function with C linkage in dload.d from main.d that looks up tho=
se names in an associative array initialized in the static initialization b=
lock of dload.d where those mangled names are directly available, so that f=
ull D linkage can be emulated, at least for functions.<br>



n this unusual context?<br>

ode cannot successfully throw an exception. Yet if a function in the shared=
 object is called from the main program and throws an exception, all is wel=
l. (Try these.) See my comments in dload.d about this. What is it about the=

<br>All files attached, including a Makefile with the exact options passed =
to dmd, gcc and ld.<br>
<br>

--f46d0444ec19b9beab04c7aa4474--
Aug 19 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-08-20 05:40, Carl Sturtivant wrote:

 (By dynamic loading I mean using something like the C library function
 dlopen to explicitly load a shared object at runtime. I do not mean
 dynamic linking in the usual passive sense.)

 I just successfully got working a toy example of dynamically loading
 (from a D program) a D shared object and then finding and calling
 functions defined in the shared object, and not just with C linkage.

 The main program, main.d, was compiled and linked somewhat normally,
 except for passing the linker via gcc the flags necessary to ensure that
 the whole of libphobos2.a is present and that all symbols in the
 resulting executable are exposed dynamically.

 The shared object source, dload.d was compiled to an object file
 containing position independent code by dmd. Then I invoked the linker
 explicitly and had it make a shared object without the D runtime system
 or Phobos. This is the novel step, and it enables the shared object to
 resolve its linkage to the D runtime system and Phobos at the point of
 being loaded, via callbacks to the main program. Thus the troubles of D
 in shared objects are largely circumvented. There is only one instance
 of phobos and D-runtime, in the main program. (Once phobos and druntime
 are shared objects in the future somewhere this will work with no code
 bloat.)

 The static initialization code in dload.d is automatically executed when
 the shared object libdload.so is loaded by the main program, because the
 linker is also passed a flag indicating the static initialization
 block's mangled name, dynamically determined from dload.o before linkage
 to libdload.so occurs.

 Finally, the mangled names of the functions to load are determined by a
 call of a function with C linkage in dload.d from main.d that looks up
 those names in an associative array initialized in the static
 initialization block of dload.d where those mangled names are directly
 available, so that full D linkage can be emulated, at least for functions.

 One thing: the garbage collector needs to be aware of static and
 'global' D variables in the shared object. Can a technical expert verify
 that I've done the right thing to achieve that happy state of affairs in
 this unusual context?

 So, what's overlooked here? I know that the static initialization code
 cannot successfully throw an exception. Yet if a function in the shared
 object is called from the main program and throws an exception, all is
 well. (Try these.) See my comments in dload.d about this. What is it
 about the implementation of exceptions that's problematic here?

 All files attached, including a Makefile with the exact options passed
 to dmd, gcc and ld.


I'm not sure I'm following what you exactly have done here but in 
general this is what needs to be done to make dynamic libraries properly 
work in D :

* Initialize module infos (module constructors and similar)
* Add TLS variables
* Add exception handling tables
* Add GC roots

The above four things need to be extracted from the loaded dynamic 
library and it gets loaded and preferably remove them as well when the 
dynamic library gets unloaded. Currently this is only extracted from the 
running executable. This is platform dependent but usually it's 
extracted using bracketed sections via extern C variables.

-- 
/Jacob Carlborg
Aug 20 2012
next sibling parent reply "Paulo Pinto" <pjmlp progtools.org> writes:
On Monday, 20 August 2012 at 07:26:40 UTC, Jacob Carlborg wrote:

 On 2012-08-20 05:40, Carl Sturtivant wrote:

 (By dynamic loading I mean using something like the C library 
 function
 dlopen to explicitly load a shared object at runtime. I do not 
 mean
 dynamic linking in the usual passive sense.)

 I just successfully got working a toy example of dynamically 
 loading
 (from a D program) a D shared object and then finding and 
 calling
 functions defined in the shared object, and not just with C 
 linkage.

 The main program, main.d, was compiled and linked somewhat 
 normally,
 except for passing the linker via gcc the flags necessary to 
 ensure that
 the whole of libphobos2.a is present and that all symbols in 
 the
 resulting executable are exposed dynamically.

 The shared object source, dload.d was compiled to an object 
 file
 containing position independent code by dmd. Then I invoked 
 the linker
 explicitly and had it make a shared object without the D 
 runtime system
 or Phobos. This is the novel step, and it enables the shared 
 object to
 resolve its linkage to the D runtime system and Phobos at the 
 point of
 being loaded, via callbacks to the main program. Thus the 
 troubles of D
 in shared objects are largely circumvented. There is only one 
 instance
 of phobos and D-runtime, in the main program. (Once phobos and 
 druntime
 are shared objects in the future somewhere this will work with 
 no code
 bloat.)

 The static initialization code in dload.d is automatically 
 executed when
 the shared object libdload.so is loaded by the main program, 
 because the
 linker is also passed a flag indicating the static 
 initialization
 block's mangled name, dynamically determined from dload.o 
 before linkage
 to libdload.so occurs.

 Finally, the mangled names of the functions to load are 
 determined by a
 call of a function with C linkage in dload.d from main.d that 
 looks up
 those names in an associative array initialized in the static
 initialization block of dload.d where those mangled names are 
 directly
 available, so that full D linkage can be emulated, at least 
 for functions.

 One thing: the garbage collector needs to be aware of static 
 and
 'global' D variables in the shared object. Can a technical 
 expert verify
 that I've done the right thing to achieve that happy state of 
 affairs in
 this unusual context?

 So, what's overlooked here? I know that the static 
 initialization code
 cannot successfully throw an exception. Yet if a function in 
 the shared
 object is called from the main program and throws an 
 exception, all is
 well. (Try these.) See my comments in dload.d about this. What 
 is it
 about the implementation of exceptions that's problematic here?

 All files attached, including a Makefile with the exact 
 options passed
 to dmd, gcc and ld.


 I'm not sure I'm following what you exactly have done here but 
 in general this is what needs to be done to make dynamic 
 libraries properly work in D :

 * Initialize module infos (module constructors and similar)
 * Add TLS variables
 * Add exception handling tables
 * Add GC roots

 The above four things need to be extracted from the loaded 
 dynamic library and it gets loaded and preferably remove them 
 as well when the dynamic library gets unloaded. Currently this 
 is only extracted from the running executable. This is platform 
 dependent but usually it's extracted using bracketed sections 
 via extern C variables.


Should this be made automatically by the compiler?

This would be my expectation based on my experience with dynamic 
libraries in Turbo Pascal/Delphi.

--
Paulo
Aug 20 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-08-20 13:38, Paulo Pinto wrote:


 Should this be made automatically by the compiler?

 This would be my expectation based on my experience with dynamic
 libraries in Turbo Pascal/Delphi.


This should be handled automatically by the runtime.

-- 
/Jacob Carlborg
Aug 20 2012
parent reply "Paulo Pinto" <pjmlp progtools.org> writes:
On Monday, 20 August 2012 at 12:15:28 UTC, Jacob Carlborg wrote:

 On 2012-08-20 13:38, Paulo Pinto wrote:


 Should this be made automatically by the compiler?

 This would be my expectation based on my experience with 
 dynamic
 libraries in Turbo Pascal/Delphi.


 This should be handled automatically by the runtime.


Ah ok, from your explanation I understood as something we are 
required to do manually.
Aug 20 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-08-20 16:22, Paulo Pinto wrote:


 Ah ok, from your explanation I understood as something we are required
 to do manually.


I just explained what changes need to be done in order for it to work. 
So we need to modify druntime to do what I listed above.

-- 
/Jacob Carlborg
Aug 20 2012
prev sibling parent reply "Carl Sturtivant" <sturtivant gmail.com> writes:
 I'm not sure I'm following what you exactly have done here but 
 in general this is what needs to be done to make dynamic 
 libraries properly work in D :

 * Initialize module infos (module constructors and similar)
 * Add TLS variables
 * Add exception handling tables
 * Add GC roots

 The above four things need to be extracted from the loaded 
 dynamic library and it gets loaded and preferably remove them 
 as well when the dynamic library gets unloaded. Currently this 
 is only extracted from the running executable. This is platform 
 dependent but usually it's extracted using bracketed sections 
 via extern C variables.


OK, good to know. Any further hints about these, or where I can 
look?

What I've done is use the C dynamic loading library (header 
dlfcn.h) to manually load a shared object written in D, dload.d, 
from a D program (main.d), which then successfully calls 
functions in dload.d that are not defined extern(C).

I am attempting a do-it-yourself dynamic loading in D, where I 
explicitly do all the administration manually to make it work, 
rather than rely upon D to do it automatically. Hence my use of 
the C dynamic loading library, which knows nothing of additional 
work D must do.

Reason for this approach: the newsgroups indicate that dynamic 
loading in D does not work as yet.

What's novel is that I explicitly excluded from the shared object 
anything but code generated directly from dload.d:

dmd -c dload.d -m64 -fPIC
ld dload.o -shared -o libdload.so -m elf_x86_64 -E

And I included the whole of libphobos.a in the build of the main 
program:

dmd -c main.d -m64
gcc main.o -o main -m64 -Wl,-E -ldl -Wl,--whole-archive -lphobos2 
\
-Wl,--no-whole-archive -lcurl -lpthread -lm -lrt

This tactic is in the hope that all parts of D used in the shared 
library will find their linkage in the main program when it's 
dynamically loaded. (And the -E option passed to the linker in 
both cases is to expose all symbols for exactly this purpose.)

Reason for this approach: the newsgroups indicate that D 
runtime/phobos (all currently in libphobos.a it seems) does not 
initialize properly in a shared library, so I ensure that it's 
not present at all, and endeavor to have the shared object 
implicitly use the properly working D runtime/phobos in the main 
executable.

[And besides, at 64 bits all code in shared libraries apparently 
must be position independent, so even if I wanted to link parts 
of libphobos.a into the shared library, I couldn't without 
recompiling libphobos.a with the -fPIC option!]

This bare-bones-in-the-shared-library approach has worked well in 
my toy example. The only thing that apparently doesn't work is if 
an exception is thrown from the static initializer 'static 
this()' in the shared object. If an exception results from a call 
chain initiated by the main program even if thrown from a 
function in the shared object, all is well it seems.

Incidentally, I fibbed slightly about the ld command used to link 
the shared object. It also contains a trailing

   -init=$(shell staticCtor dload.o)

which enables the linker to bind in execution of the static 
initializer 'static this()' so that it runs automatically when 
the shared library is loaded. staticCtor is a script that 
analyzes dload.o to find its name, which is mangled of course.

If anyone can give me any more specific information about what 
else I can make happen manually to complete effective linkage at 
the D level I'd be grateful. I'm not stopping the investigation 
here!
Aug 20 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-08-20 17:16, Carl Sturtivant wrote:

 I'm not sure I'm following what you exactly have done here but in
 general this is what needs to be done to make dynamic libraries
 properly work in D :

 * Initialize module infos (module constructors and similar)
 * Add TLS variables
 * Add exception handling tables
 * Add GC roots

 The above four things need to be extracted from the loaded dynamic
 library and it gets loaded and preferably remove them as well when the
 dynamic library gets unloaded. Currently this is only extracted from
 the running executable. This is platform dependent but usually it's
 extracted using bracketed sections via extern C variables.


 OK, good to know. Any further hints about these, or where I can look?


 From druntime:

* rt.minfo - module infos. The "_moduleinfo_array" variable contains the 
module infos.

* rt.deh2 - exception handling. On Mac OS X the "_deh_eh_array" contains 
the exception handling tables. On the other Posix systems the sections 
bracketed by "_deh_beg" and "_deh_end" contains the tables.

* rt.thread - threading, TLS. On Mac OS X the TLS variables are stored 
in "_tls_data_array". On the other Posix systems they're are again 
stored in bracketed sections: "_tlsstart" and "_tlsend".

* rt.memory - GC roots. I think this module contains code related to 
adding GC roots. rt.memory_osx for Mac OS X.

You can also take a look at work by Martin Nowak:

https://github.com/dawgfoto/druntime/commits/SharedRuntime


 What I've done is use the C dynamic loading library (header dlfcn.h) to
 manually load a shared object written in D, dload.d, from a D program
 (main.d), which then successfully calls functions in dload.d that are
 not defined extern(C).

 I am attempting a do-it-yourself dynamic loading in D, where I
 explicitly do all the administration manually to make it work, rather
 than rely upon D to do it automatically. Hence my use of the C dynamic
 loading library, which knows nothing of additional work D must do.


The functions from dlfcn.h is what D would use as well. Possibly wrapped 
in some D function that does some additional work.

-- 
/Jacob Carlborg
Aug 20 2012
parent reply "Philip Daniels" <foo foo.com> writes:
snip<


If we had dynamic loading, would we be able to do dependency
injection in D?
Aug 22 2012
parent reply "Paulo Pinto" <pjmlp progtools.org> writes:
On Wednesday, 22 August 2012 at 15:51:05 UTC, Philip Daniels 
wrote:

snip<


 If we had dynamic loading, would we be able to do dependency
 injection in D?


Dependency injection does not require dynamic loading per se.

It is all about using interfaces instead of classes, and 
initializing the corresponding instance members.

You just need some piece of code that takes the responsibility of 
locating such interfaces, by registering the classes somehow, or 
by compile time reflection, which gets called on program 
initialization.

You can do this in D today.

--
Paulo
Aug 22 2012
parent reply "Philip Daniels" <foo foo.com> writes:
On Wednesday, 22 August 2012 at 16:57:26 UTC, Paulo Pinto wrote:

 On Wednesday, 22 August 2012 at 15:51:05 UTC, Philip Daniels 
 wrote:

snip<


 If we had dynamic loading, would we be able to do dependency
 injection in D?


 Dependency injection does not require dynamic loading per se.

 It is all about using interfaces instead of classes, and 
 initializing the corresponding instance members.

 You just need some piece of code that takes the responsibility 
 of locating such interfaces, by registering the classes 
 somehow, or by compile time reflection, which gets called on 
 program initialization.

 You can do this in D today.

 --
 Paulo


But wouldn't that require you to link everything together at, 
err, compile time?

What I'm getting at is, would it be possible to port a DI/IoC 
tool such as StructureMap 
(http://docs.structuremap.net/index.html) or Spring to D? This 
can handle tasks such as creating dynamic plug-in architectures. 
For example, given SomeBigApp.exe (not written by me) which looks 
in standard folders for components implementing a particular 
interface, I can just drop my code in that folder and have it 
loaded at runtime. I could even drop it in there after the 
program starts running. I know how to achieve this in the .Net 
world, just wondered if it was possible in D.
Aug 23 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-08-24 00:26, Philip Daniels wrote:


 But wouldn't that require you to link everything together at, err,
 compile time?

 What I'm getting at is, would it be possible to port a DI/IoC tool such
 as StructureMap (http://docs.structuremap.net/index.html) or Spring to
 D? This can handle tasks such as creating dynamic plug-in architectures.
 For example, given SomeBigApp.exe (not written by me) which looks in
 standard folders for components implementing a particular interface, I
 can just drop my code in that folder and have it loaded at runtime. I
 could even drop it in there after the program starts running. I know how
 to achieve this in the .Net world, just wondered if it was possible in D.


Sure, you just need to come up with a couple of conventions. Say for 
example that you have your application in a folder structure like this:

app
  |--plugins
      |---fully.qualified.class.name.so/dll/dylib

The host application inspects the "plugins" folder at runtime grabbing 
all the names of the dynamic libraries. The convention is that in the 
dynamic library there will be a class with the same fully qualified name 
as the name of the dynamic library its located in.

The next convention is that the class must implement an interface, which 
the host application provides. The class also needs to have a default 
constructor (there are ways around that).

Then the host application would do something like this:

interface Plugin
{
     void initialize ();
     // ... other needed methods
}

auto plugin = cast(Plugin) Object.factory("fully.qualified.class.name");
plugin.initialize();

The magic behind this is the "Object.factory" method:

http://dlang.org/phobos/object.html#factory

-- 
/Jacob Carlborg
Aug 24 2012