www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Linux Dynamic Loading of shared libraries

reply "Steve Teale" <steve.teale britseyeview.com> writes:
Martin Nowak's Gihub druntime Page has

module main;
import core.runtime, core.thread;

void main() {
     auto lib = Runtime.loadLibrary("./liba.so");
     auto thr = new Thread({
         auto lib = Runtime.loadLibrary("./liba.so");
         Runtime.unloadLibrary(lib);
     });
     thr.start();
     thr.join();
     Runtime.unloadLibrary(lib);
}

module liba;
import std.stdio;

shared static this() { writeln("shared static this()"); }
shared static ~this() { writeln("shared static ~this()"); }
static this() { writeln("static this()"); }
static ~this() { writeln("static ~this()"); }

Now suppose that my D shared library contains a class, rather 
that just module ctors/dtors, how do I go about creating an 
instance of that class and using its methods?

Steve
Mar 09 2014
next sibling parent reply "Tolga Cakiroglu" <tcak pcak.com> writes:
 Now suppose that my D shared library contains a class, rather 
 that just module ctors/dtors, how do I go about creating an 
 instance of that class and using its methods?

 Steve


For this, you create an "Interface" that matches to the method 
declaration of your class. But notice that instead of defining 
methods, you will define attributes those types' match to that 
class's methods. I did this before and it works. At least with 
Posix "dlsym" function's help.
Mar 09 2014
parent reply "Steve Teale" <steve.teale britseyeview.com> writes:
On Sunday, 9 March 2014 at 14:09:28 UTC, Tolga Cakiroglu wrote:


 For this, you create an "Interface" that matches to the method 
 declaration of your class. But notice that instead of defining 
 methods, you will define attributes those types' match to that 
 class's methods. I did this before and it works. At least with 
 Posix "dlsym" function's help.


OK, so then what goes wrong here:

module exta;

class ExtA
{
    int n;
    this(int _n) { n = _n; }

    int foo() { return ++n; }
}

ExtA getInstance(int n)
{
    return new ExtA(n);
}

compiled with:
dmd exta.d -c -fPIC -shared
dmd exta.o -shared -defaultlib=libphobos2.so -L-rpath=.

module main;
import core.runtime;
import std.stdio;

extern(C) void* dlsym(void*, const char*);
extern(C) void dlclose(void*);

interface ExtA
{
    int foo();
}

void main() {
    void* lib = Runtime.loadLibrary("exta.so");
    if (lib is null)
    {
       writeln("library not loaded");
       return;
    }
    writeln("loaded");

    void* vp = dlsym(lib, 
"_D4exta11getInstanceFiZC4exta4ExtA\0".ptr);
    if (vp is null)
    {
       writeln("symbol not found");
       return;
    }
    writeln("got symbol");
    ExtA function(int) f = cast(ExtA function(int)) vp;
    ExtA x = f(42);
    if (x is null)
    {
       writeln("no class object");
       return;
    }
    int n = x.foo();
    writefln("n = %d", n);
    Runtime.unloadLibrary(lib);
}

compiled with:
dmd -c main.d
dmd main.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.

output:
loaded
got symbol
n = 9
Segmentation fault (core dumped)

The answer should be 43. The segfault happens on the 
Runtime.unloadLibrary(lib); call.

Any ideas?

Steve
Mar 09 2014
parent "Anthony Goins" <neontotem gmail.com> writes:
On Monday, 10 March 2014 at 06:38:35 UTC, Steve Teale wrote:

 On Sunday, 9 March 2014 at 14:09:28 UTC, Tolga Cakiroglu wrote:


 For this, you create an "Interface" that matches to the method 
 declaration of your class. But notice that instead of defining 
 methods, you will define attributes those types' match to that 
 class's methods. I did this before and it works. At least with 
 Posix "dlsym" function's help.


 OK, so then what goes wrong here:

 module exta;

 class ExtA
 {
    int n;
    this(int _n) { n = _n; }

    int foo() { return ++n; }
 }

 ExtA getInstance(int n)
 {
    return new ExtA(n);
 }

 compiled with:
 dmd exta.d -c -fPIC -shared
 dmd exta.o -shared -defaultlib=libphobos2.so -L-rpath=.

 module main;
 import core.runtime;
 import std.stdio;

 extern(C) void* dlsym(void*, const char*);
 extern(C) void dlclose(void*);

 interface ExtA
 {
    int foo();
 }

 void main() {
    void* lib = Runtime.loadLibrary("exta.so");
    if (lib is null)
    {
       writeln("library not loaded");
       return;
    }
    writeln("loaded");

    void* vp = dlsym(lib, 
 "_D4exta11getInstanceFiZC4exta4ExtA\0".ptr);
    if (vp is null)
    {
       writeln("symbol not found");
       return;
    }
    writeln("got symbol");
    ExtA function(int) f = cast(ExtA function(int)) vp;
    ExtA x = f(42);
    if (x is null)
    {
       writeln("no class object");
       return;
    }
    int n = x.foo();
    writefln("n = %d", n);
    Runtime.unloadLibrary(lib);
 }

 compiled with:
 dmd -c main.d
 dmd main.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.

 output:
 loaded
 got symbol
 n = 9
 Segmentation fault (core dumped)

 The answer should be 43. The segfault happens on the 
 Runtime.unloadLibrary(lib); call.

 Any ideas?

 Steve


confusion between main.Exta and exta.ExtA
following worked for me

module exta;

import main;

//===========================================================
class ExtA : ExtA_IF
{
     int n;
     this(int _n) { n = _n; }

     int foo() { return ++n; }
}

ExtA_IF getInstance(int n)
{
     return new ExtA(n);
}
//=============================================================


//==============================================================
module main;
import core.runtime;
import std.stdio;

extern(C) void* dlsym(void*, const char*);
extern(C) void dlclose(void*);

interface ExtA_IF
{
     int foo();
}

void main() {
     void* lib = Runtime.loadLibrary("exta.so");
     if (lib is null)
     {
        writeln("library not loaded");
        return;
     }
     writeln("loaded");

     //use extern (C) to avoid mangling
     void* vp = dlsym(lib,
"_D4exta11getInstanceFiZC4main7ExtA_IF\0".ptr);
     if (vp is null)
     {
        writeln("symbol not found");
        return;
     }
     writeln("got symbol");
     ExtA_IF function(int) f = cast(ExtA_IF function(int)) vp;
     ExtA_IF x = f(42);
     if (x is null)
     {
        writeln("no class object");
        return;
     }
     int n = x.foo();
     writefln("n = %d", n);

     // or free or destroy or whatever it's supposed to be to
     //to avoid the seg fault  (bug?)
     delete x;
     Runtime.unloadLibrary(lib);
}
Mar 10 2014
prev sibling next sibling parent "Anthony Goins" <neontotem gmail.com> writes:
On Sunday, 9 March 2014 at 12:07:22 UTC, Steve Teale wrote:

 Martin Nowak's Gihub druntime Page has

 module main;
 import core.runtime, core.thread;

 void main() {
     auto lib = Runtime.loadLibrary("./liba.so");
     auto thr = new Thread({
         auto lib = Runtime.loadLibrary("./liba.so");
         Runtime.unloadLibrary(lib);
     });
     thr.start();
     thr.join();
     Runtime.unloadLibrary(lib);
 }

 module liba;
 import std.stdio;

 shared static this() { writeln("shared static this()"); }
 shared static ~this() { writeln("shared static ~this()"); }
 static this() { writeln("static this()"); }
 static ~this() { writeln("static ~this()"); }

 Now suppose that my D shared library contains a class, rather 
 that just module ctors/dtors, how do I go about creating an 
 instance of that class and using its methods?

 Steve


If you know the fully qualified name of the class and an interface
auto newClassIF = 
cast(interfacename)object.factory("libmodule.libclass");
Mar 09 2014
prev sibling parent reply "Steve Teale" <steve.teale britseyeview.com> writes:
On Sunday, 9 March 2014 at 12:07:22 UTC, Steve Teale wrote:


 Now suppose that my D shared library contains a class, rather 
 that just module ctors/dtors, how do I go about creating an 
 instance of that class and using its methods?


After wandering down several dead-end paths, and help from other 
contributors, I have finally come up with something that looks 
like the basis of a plugin pattern for Linux DMD using shared 
objects (.so files). This is somewhat long for a forum post. You 
can download this readme and the associated files from 
britseyeview.com/plugin101.tar.bz2

To get started, you need a base class that provides declarations 
for all functions that the plugin will be allowed to use 
externally. Why base class, and not interface? Well I guess 
because interfaces don't provide any information about data. If 
you create a shared library based on an interface, then all the 
shared object methods that reference data in the class that 
implements the interface fail miserably. I'm sure someone will 
explain why - probably some obvious thing I have overlooked.

OK, so my base class is:

module plugin;

class Plugin
{
    int n;
    this(int _n) { n = _n; }

    int foo() { return int.min; }
    void bar() {}
}


The class that implements this base in the shared library is:

module exta;
import plugin;
import std.stdio;
import std.math;

class ExtA: Plugin
{
    double d;
    this(int n) { super(n); d = PI; }

    override int foo() { return ++n; }
    override void bar() { writefln("Done my thing (%f)", d); }
}

Plugin getInstance(int n)
{
    return new ExtA(n);
}

shared static this() {
   writeln("exta.so shared static this");
}

shared static ~this() {
   writeln("exta.so shared static ~this");
}

The module ctor/dtor are included because that has become 
conventional in discussions about dynamic loading. Otherwise, the 
so has the class implementation - ExtA, and a shared method to 
create an instance of same. It includes references to methods in 
Phobos.

The test program is as follows:

module main;
import core.runtime;
import std.stdio;
import plugin;

extern(C) void* dlsym(void*, const char*);

alias Plugin function(int) pfi;

Plugin getPlugin(string name)
{
    void* lib = Runtime.loadLibrary(name~".so");
    if (lib is null)
    {
       writeln("failed to load plugin shared object");
       return null;
    }

    void* vp = dlsym(lib, 
"_D4exta11getInstanceFiZC6plugin6Plugin\0".ptr);
    if (vp is null)
    {
       writeln("plugin creator function not found");
       return null;
    }
    pfi f = cast(pfi) vp;
    Plugin x = f(42);
    if (x is null)
    {
       writeln("creation of plugin failed");
       return null;
    }
    return x;
}

void main()
{
    Plugin x = getPlugin("exta");
    int n = x.foo();
    writefln("n = %d", n);
    x.bar();
}

The long symbol name used in the dlsym() call is of course from 
the .map file generated when the .so file is created

These can be built using the following primitive makefile, whose 
main purpose is to spell out the required compiler flags:

main :
	dmd -c plugin.d
	dmd -c -shared -fPIC exta.d
	dmd exta.o -shared -defaultlib=libphobos2.so -map
	dmd -c main.d
	dmd main.o plugin.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.

This assumes that the plugins will be in the same directory as 
the executable (rpath=.).

Note that there is no call to Runtime.unloadLibrary(). The 
assumption her is that once the plugin has been loaded it will be 
there for the duration of the program. If you want to unload it 
you'll probably have to make sure the plugin object is purged 
from memory first, and I have not discovered how to do that yet 
;=(

Steve
Mar 10 2014
next sibling parent "seany" <seany uni-bonn.de> writes:
Bless you, mate. You described precisely what I was looking for.

Nonetheless, I am still looking for a load and unload command to 
select and reject plugins in runtime.
Jul 29 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Monday, 10 March 2014 at 11:59:20 UTC, Steve Teale wrote:

 Note that there is no call to Runtime.unloadLibrary(). The 
 assumption her is that once the plugin has been loaded it will 
 be there for the duration of the program. If you want to unload 
 it you'll probably have to make sure the plugin object is 
 purged from memory first, and I have not discovered how to do 
 that yet ;=(


A long time ago, Andrei suggested creating a GC interface for 
mapped memory.  I think something similar might be appropriate 
here.  Perhaps the location of instantiated classes could be 
determined by their vtbl pointer?  Then the entire vtbl range 
could be treated as a dynamically allocated struct of sorts, and 
its dtor would queue up a job to unmap the library after 
collection is complete (ie. not immediately, since the order of 
destruction during a collection is undefined).

So... (just thinking out loud) when a library is loaded, you 
basically perform an in-place construction of this LoadedLibrary 
struct on top of the vtbl range.  You'd need an interface similar 
to the "GC tracks memory mapped files" idea to tell the GC to 
track references to this range of memory that lives outside its 
own pool set, and then some kind of post-collection job queue 
that's externally appendable so the LoadedLibrary struct could 
add an unload call when it's collected.  Heck, we could really 
handle all dtors this way, so normal dtors would be inserted at 
the front of the list and special cases like this would be 
inserted at the back.

It doesn't sound tremendously difficult, though we'd need the 
memory-mapped file support API first.  Is there something I'm 
missing that makes this unfeasible?
Jul 29 2014
prev sibling parent "Carl Sturtivant" <sturtivant gmail.com> writes:
Can't retrieve the archive from that URL.
britseyeview.com/plugin101.tar.bz2
Interested, so can you please fix?

On Monday, 10 March 2014 at 11:59:20 UTC, Steve Teale wrote:

 On Sunday, 9 March 2014 at 12:07:22 UTC, Steve Teale wrote:


 Now suppose that my D shared library contains a class, rather 
 that just module ctors/dtors, how do I go about creating an 
 instance of that class and using its methods?


 After wandering down several dead-end paths, and help from 
 other contributors, I have finally come up with something that 
 looks like the basis of a plugin pattern for Linux DMD using 
 shared objects (.so files). This is somewhat long for a forum 
 post. You can download this readme and the associated files 
 from britseyeview.com/plugin101.tar.bz2

 To get started, you need a base class that provides 
 declarations for all functions that the plugin will be allowed 
 to use externally. Why base class, and not interface? Well I 
 guess because interfaces don't provide any information about 
 data. If you create a shared library based on an interface, 
 then all the shared object methods that reference data in the 
 class that implements the interface fail miserably. I'm sure 
 someone will explain why - probably some obvious thing I have 
 overlooked.

 OK, so my base class is:

 module plugin;

 class Plugin
 {
    int n;
    this(int _n) { n = _n; }

    int foo() { return int.min; }
    void bar() {}
 }


 The class that implements this base in the shared library is:

 module exta;
 import plugin;
 import std.stdio;
 import std.math;

 class ExtA: Plugin
 {
    double d;
    this(int n) { super(n); d = PI; }

    override int foo() { return ++n; }
    override void bar() { writefln("Done my thing (%f)", d); }
 }

 Plugin getInstance(int n)
 {
    return new ExtA(n);
 }

 shared static this() {
   writeln("exta.so shared static this");
 }

 shared static ~this() {
   writeln("exta.so shared static ~this");
 }

 The module ctor/dtor are included because that has become 
 conventional in discussions about dynamic loading. Otherwise, 
 the so has the class implementation - ExtA, and a shared method 
 to create an instance of same. It includes references to 
 methods in Phobos.

 The test program is as follows:

 module main;
 import core.runtime;
 import std.stdio;
 import plugin;

 extern(C) void* dlsym(void*, const char*);

 alias Plugin function(int) pfi;

 Plugin getPlugin(string name)
 {
    void* lib = Runtime.loadLibrary(name~".so");
    if (lib is null)
    {
       writeln("failed to load plugin shared object");
       return null;
    }

    void* vp = dlsym(lib, 
 "_D4exta11getInstanceFiZC6plugin6Plugin\0".ptr);
    if (vp is null)
    {
       writeln("plugin creator function not found");
       return null;
    }
    pfi f = cast(pfi) vp;
    Plugin x = f(42);
    if (x is null)
    {
       writeln("creation of plugin failed");
       return null;
    }
    return x;
 }

 void main()
 {
    Plugin x = getPlugin("exta");
    int n = x.foo();
    writefln("n = %d", n);
    x.bar();
 }

 The long symbol name used in the dlsym() call is of course from 
 the .map file generated when the .so file is created

 These can be built using the following primitive makefile, 
 whose main purpose is to spell out the required compiler flags:

 main :
 	dmd -c plugin.d
 	dmd -c -shared -fPIC exta.d
 	dmd exta.o -shared -defaultlib=libphobos2.so -map
 	dmd -c main.d
 	dmd main.o plugin.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.

 This assumes that the plugins will be in the same directory as 
 the executable (rpath=.).

 Note that there is no call to Runtime.unloadLibrary(). The 
 assumption her is that once the plugin has been loaded it will 
 be there for the duration of the program. If you want to unload 
 it you'll probably have to make sure the plugin object is 
 purged from memory first, and I have not discovered how to do 
 that yet ;=(

 Steve
Jul 29 2014