digitalmars.D - Plugins and D programs
- Steve Teale (94/94) Mar 13 2014 One of the primary uses of dynamic loading of libraries might
- Tolga Cakiroglu (9/104) Mar 13 2014 Hi Steve. Well, I had so many experience with this and after many
- Tolga Cakiroglu (1/2) Mar 13 2014 Hmm. I saw that you are using libphobos2.so. I wasn't using this.
- Steve Teale (15/18) Mar 13 2014 Yes, you have to make the program and the plugin both use the
- Steve Teale (91/95) Mar 14 2014 OK, so here's an example using an exemplar base class instead of
- Mathias LANG (23/28) Mar 14 2014 geod@Barsoom:~/Work/BoapNet/Client/Native/SharedTemplate/test$
- Steve Teale (3/11) Mar 14 2014 Ubuntu 12.04 32 bit.
- Steve Teale (7/20) Mar 14 2014 Sure enough, works OK on 12.04 64 bit, so I guess it's a bug.
- Steve Teale (5/7) Mar 14 2014 There's an equivalent in the system already - issue 10719, but
- Sean Kelly (7/14) Mar 13 2014 It might be purely a matter of historic interest at this point,
- Steve Teale (10/17) Mar 14 2014 Sean,
- Jacob Carlborg (6/12) Mar 14 2014 Yes, exactly. It basically allowed him to update code on the fly, when
- Martin Nowak (4/10) Mar 15 2014 But this use object with non-PIC and does full relocation while loading.
- Mathias LANG (6/10) Mar 14 2014 Copied your files, added:
- Steve Teale (2/13) Mar 14 2014 Everything works OK with 64 bit anyway!
- Martin Nowak (23/34) Mar 15 2014 One problem here is that main and the plugin share the definitions in
- Martin Nowak (9/10) Mar 15 2014 Looks like a revenant of
- Martin Nowak (4/16) Mar 15 2014 I prepared a fix here (https://github.com/MartinNowak/dmd/tree/thunkEBX)
- Steve Teale (5/29) Mar 16 2014 Thanks Martin - it's good to have you involved. I'd got as far as
- Steve Teale (3/9) Mar 16 2014 When you've fixed this, you should also be able to mark issue
One of the primary uses of dynamic loading of libraries might
well be to provide plugins. By plugins I mean extensions to an
existing program that can be added to the program at run-time,
and can be written by separate authors who don't necessarily have
access to the source code of the program, but who do understand
the rules provided by the program documentation as to what
capabilities the plugin must have.
The most obvious way to allow a D program to cope with plugins,
is to specify an interface to which the plugin is expected to
conform. Communication between the program and the plugin, after
the latter is loaded would then be restricted to calls provided
by that interface.
One further library method would likely be necessary to allow for
the acquisition of an instance of the plugin, and its attachment
to the program.
So, we could have
module ifd;
interface I
{
string saySomething();
}
The plugin could then be:
module plugin;
import ifd;
import std.stdio;
class Plugin: I
{
this() { writeln("plugin ctor"); }
string saySomething() { return "I am plugin"; }
}
I getInstance()
{
return new Plugin();
}
And our program could be:
module main;
import core.runtime;
import std.stdio;
import ifd;
extern(C) void* dlsym(void*, const char*);
extern(C) void* dlopen(const char*, int);
alias I function() pfi;
I getPlugin(string name)
{
// Take your pick from these two - makes no odds
//void* lib = dlopen("plugin.so\0".ptr, 1);
void* lib = Runtime.loadLibrary(name~".so");
if (lib is null)
{
writeln("failed to load plugin shared object");
return null;
}
void* vp = dlsym(lib, "_D6plugin11getInstanceFZC3ifd1I\0".ptr);
if (vp is null)
{
writeln("plugin creator function not found");
return null;
}
pfi f = cast(pfi) vp;
I x = f();
if (x is null)
{
writeln("creation of plugin failed");
return null;
}
return x;
}
void main()
{
I x = getPlugin("plugin");
writeln(x.saySomething());
}
Unfortunately, the result of running the program is:
steve steve-desktop:~/scratch/piif$ ./main
plugin ctor
Segmentation fault (core dumped)
Which suggests that the library was loaded, the symbol found, and
an instance of plugin created.
The two pieces were built using dmd2.064 with:
main : ifd.d main.d
dmd -c ifd.d
dmd -c main.d
dmd main.o ifd.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.
plugin : plugin.d
dmd -c -shared -fPIC plugin.d
dmd plugin.o -shared -defaultlib=libphobos2.so -map
clean :
rm *.o
rm main
rm plugin.so
Does anyone have any suggestions as to what might be going wrong
here? I have further examples, but I guess I should do them one
at a time.
Steve
Mar 13 2014
Hi Steve. Well, I had so many experience with this and after many
tries, I gave up unfortunately. It continuous giving Segmentation
error.
What I have found is that while the main programme is closing, it
tries to call "rt_finalize" to remove that object from memory,
then it says that method is not defined. I complied programme in
Debug version and used GDB to understand above situation.
(Linux x64, DMD 2.065)
On Thursday, 13 March 2014 at 13:22:55 UTC, Steve Teale wrote:
One of the primary uses of dynamic loading of libraries might
well be to provide plugins. By plugins I mean extensions to an
existing program that can be added to the program at run-time,
and can be written by separate authors who don't necessarily
have access to the source code of the program, but who do
understand the rules provided by the program documentation as
to what capabilities the plugin must have.
The most obvious way to allow a D program to cope with plugins,
is to specify an interface to which the plugin is expected to
conform. Communication between the program and the plugin,
after the latter is loaded would then be restricted to calls
provided by that interface.
One further library method would likely be necessary to allow
for the acquisition of an instance of the plugin, and its
attachment to the program.
So, we could have
module ifd;
interface I
{
string saySomething();
}
The plugin could then be:
module plugin;
import ifd;
import std.stdio;
class Plugin: I
{
this() { writeln("plugin ctor"); }
string saySomething() { return "I am plugin"; }
}
I getInstance()
{
return new Plugin();
}
And our program could be:
module main;
import core.runtime;
import std.stdio;
import ifd;
extern(C) void* dlsym(void*, const char*);
extern(C) void* dlopen(const char*, int);
alias I function() pfi;
I getPlugin(string name)
{
// Take your pick from these two - makes no odds
//void* lib = dlopen("plugin.so\0".ptr, 1);
void* lib = Runtime.loadLibrary(name~".so");
if (lib is null)
{
writeln("failed to load plugin shared object");
return null;
}
void* vp = dlsym(lib,
"_D6plugin11getInstanceFZC3ifd1I\0".ptr);
if (vp is null)
{
writeln("plugin creator function not found");
return null;
}
pfi f = cast(pfi) vp;
I x = f();
if (x is null)
{
writeln("creation of plugin failed");
return null;
}
return x;
}
void main()
{
I x = getPlugin("plugin");
writeln(x.saySomething());
}
Unfortunately, the result of running the program is:
steve steve-desktop:~/scratch/piif$ ./main
plugin ctor
Segmentation fault (core dumped)
Which suggests that the library was loaded, the symbol found,
and an instance of plugin created.
The two pieces were built using dmd2.064 with:
main : ifd.d main.d
dmd -c ifd.d
dmd -c main.d
dmd main.o ifd.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.
plugin : plugin.d
dmd -c -shared -fPIC plugin.d
dmd plugin.o -shared -defaultlib=libphobos2.so -map
clean :
rm *.o
rm main
rm plugin.so
Does anyone have any suggestions as to what might be going
wrong here? I have further examples, but I guess I should do
them one at a time.
Steve
Mar 13 2014
Hmm. I saw that you are using libphobos2.so. I wasn't using this.dmd main.o ifd.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.
Mar 13 2014
On Thursday, 13 March 2014 at 21:20:10 UTC, Tolga Cakiroglu wrote:Yes, you have to make the program and the plugin both use the same runtime, presumably so they are using the same memory allocation system, so you link both with libphobos2.so. It is not just a shut-down artefact, since if I repeat the exercise with an exemplar class instead of an interface, then it will work. But then if I make the exemplar class an instance of an interface, and call one of the methods that is part of the interface it goes back to crashing, so it does seem that this behaviour is linked to interfaces. I will get stuck in with GDB and see if I can find out more, but that's not an agreeable experience, so I thought I would ask first. SteveHmm. I saw that you are using libphobos2.so. I wasn't using this.dmd main.o ifd.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.
Mar 13 2014
On Friday, 14 March 2014 at 06:54:34 UTC, Steve Teale wrote:On Thursday, 13 March 2014 at 21:20:10 UTC, Tolga CakirogluIt is not just a shut-down artefact, since if I repeat the exercise with an exemplar class instead of an interface, then it will work.OK, so here's an example using an exemplar base class instead of an interface. This includes making the exemplar class implement an interface. Here's the exemplar class, and another class that references an instance of it: module bc; import std.stdio; interface I { string saySomething(); } class Bc: I { string saySomething() { return null; }; } class Other { I target; this(I i) { target = i; } void invokeWithCast() { writeln((cast(Bc) target).saySomething()); } void invoke() { writeln(target.saySomething()); } } Then the plugin: module plugin; import bc; import std.stdio; class Plugin: Bc { this() { writeln("plugin ctor"); } override string saySomething() { return "I am plugin"; } } Bc getInstance() { return new Plugin(); } And the program: module main; import core.runtime; import std.stdio; import bc; extern(C) void* dlsym(void*, const char*); alias Bc function() pfi; Bc getPlugin(string name) { void* lib = Runtime.loadLibrary(name~".so"); void* vp = dlsym(lib, "_D6plugin11getInstanceFZC2bc2Bc\0".ptr); pfi f = cast(pfi) vp; Bc x = f(); return x; } void main() { Bc x = getPlugin("plugin"); writeln(x.saySomething()); // OK Other other = new Other(x); other.invokeWithCast(); // OK other.invoke(); // Crash } Built with: main : bc.d main.d dmd -c bc.d dmd -c main.d dmd main.o bc.o -L-L/usr/local/lib -L-ldl -L-lgtkd-2 -defaultlib=libphobos2.so -L-rpath=. plugin : plugin.d dmd -c -shared -fPIC plugin.d dmd plugin.o -shared -defaultlib=libphobos2.so -map clean : rm *.o rm main rm plugin.so Output: steve steve-desktop:~/scratch/pibc$ ./main plugin ctor I am plugin I am plugin Segmentation fault (core dumped) If you comment out the last call via the interface instance with no cast, the program exits cleanly. So the problem does seem to relate to interfaces. So you can do it, but if the base class conforms to some interface, then you have to special-case calls to the interface functions. That rather destroys the utility of classes like Other which provide services to the installed plugin. My gut-feeling question is "is the vtable in the program or the plugin, and does this vary when the plugin implements an interface?" I'm hoping Martin Nowak might join in this discussion. Steve
Mar 14 2014
On Friday, 14 March 2014 at 07:46:25 UTC, Steve Teale wrote:My gut-feeling question is "is the vtable in the program or the plugin, and does this vary when the plugin implements an interface?" I'm hoping Martin Nowak might join in this discussion. Stevegeod Barsoom:~/Work/BoapNet/Client/Native/SharedTemplate/test$ make plugin /home/geod/bin/dmd-2.064.2/linux/bin64/dmd -c -shared -fPIC plugin.d /home/geod/bin/dmd-2.064.2/linux/bin64/dmd plugin.o -shared -defaultlib=libphobos2.so -map geod Barsoom:~/Work/BoapNet/Client/Native/SharedTemplate/test$ make /home/geod/bin/dmd-2.064.2/linux/bin64/dmd -c bc.d /home/geod/bin/dmd-2.064.2/linux/bin64/dmd -c main.d /home/geod/bin/dmd-2.064.2/linux/bin64/dmd main.o bc.o -L-L/usr/local/lib -L-ldl -defaultlib=libphobos2.so -L-rpath=. -L-rpath=/home/geod/bin/dmd-2.064.2/linux/lib64/ geod Barsoom:~/Work/BoapNet/Client/Native/SharedTemplate/test$ ./main plugin ctor I am plugin I am plugin I am plugin Same result with master branch. Long story short: I can't reproduce it on my machine (Debian x86_64). Which platform are you working on ? 32 bits ?
Mar 14 2014
On Friday, 14 March 2014 at 08:15:00 UTC, Mathias LANG wrote:./main plugin ctor I am plugin I am plugin I am plugin Same result with master branch. Long story short: I can't reproduce it on my machine (Debian x86_64). Which platform are you working on ? 32 bits ?Ubuntu 12.04 32 bit. I'll reboot into 64 bit and try there.
Mar 14 2014
On Friday, 14 March 2014 at 08:56:25 UTC, Steve Teale wrote:On Friday, 14 March 2014 at 08:15:00 UTC, Mathias LANG wrote:Sure enough, works OK on 12.04 64 bit, so I guess it's a bug. Probably not core.runtime, because I think the problem still occurs if I use dlopen() directly, so code generation? I'll work up a minimal test case just using an interface, then submit it, yes? Steve./main plugin ctor I am plugin I am plugin I am plugin Same result with master branch. Long story short: I can't reproduce it on my machine (Debian x86_64). Which platform are you working on ? 32 bits ?Ubuntu 12.04 32 bit. I'll reboot into 64 bit and try there.
Mar 14 2014
"Steve Teale" <steve.teale britseyeview.com> On Friday, 14 March 2014 at 09:08:17 UTC, Steve Teale wrote:I'll work up a minimal test case just using an interface, then submit it, yes?There's an equivalent in the system already - issue 10719, but that does not mention the fact that the problem is 32 bit specific. I updated it. Will install 2.065 and test with that.
Mar 14 2014
On Thursday, 13 March 2014 at 13:22:55 UTC, Steve Teale wrote:One of the primary uses of dynamic loading of libraries might well be to provide plugins. By plugins I mean extensions to an existing program that can be added to the program at run-time, and can be written by separate authors who don't necessarily have access to the source code of the program, but who do understand the rules provided by the program documentation as to what capabilities the plugin must have.It might be purely a matter of historic interest at this point, but DDL was basically doing plugins in D ages ago: http://www.dsource.org/projects/ddl I /think/ this was the dynamic loading scheme h3r3tic used in his game engine back in the day. I'm sure someone with a better memory than me could explain it better.
Mar 13 2014
On Thursday, 13 March 2014 at 23:44:23 UTC, Sean Kelly wrote:It might be purely a matter of historic interest at this point, but DDL was basically doing plugins in D ages ago: http://www.dsource.org/projects/ddl I /think/ this was the dynamic loading scheme h3r3tic used in his game engine back in the day. I'm sure someone with a better memory than me could explain it better.Sean, Yes, been there years ago - maybe 2006. I could never get the full edifice working, so I extracted the parts that would enable me to just load a single .o file, and that would work most of the time, but then sometimes it would just crash. Maybe that would work with today's runtime, but the machine that particular code was on got stolen, so I'd have to start from scratch. Steve
Mar 14 2014
On 2014-03-14 00:44, Sean Kelly wrote:It might be purely a matter of historic interest at this point, but DDL was basically doing plugins in D ages ago: http://www.dsource.org/projects/ddlYes, but DDL uses a custom linker.I /think/ this was the dynamic loading scheme h3r3tic used in his game engine back in the day. I'm sure someone with a better memory than me could explain it better.Yes, exactly. It basically allowed him to update code on the fly, when the application was running. He showed this on the Tango conference. -- /Jacob Carlborg
Mar 14 2014
On 03/14/2014 12:44 AM, Sean Kelly wrote:It might be purely a matter of historic interest at this point, but DDL was basically doing plugins in D ages ago: http://www.dsource.org/projects/ddl I /think/ this was the dynamic loading scheme h3r3tic used in his game engine back in the day. I'm sure someone with a better memory than me could explain it better.But this use object with non-PIC and does full relocation while loading. Thus it also requires writable, executable memory which isn't available on many systems due to security issues.
Mar 15 2014
On Thursday, 13 March 2014 at 13:22:55 UTC, Steve Teale wrote:Does anyone have any suggestions as to what might be going wrong here? I have further examples, but I guess I should do them one at a time. SteveCopied your files, added: -L-rpath=/home/geod/bin/dmd-master/linux/lib64/ as dmd is not on the system path for me. It was working. So try to update your compiler and check if that's the reason.
Mar 14 2014
On Friday, 14 March 2014 at 07:22:06 UTC, Mathias LANG wrote:On Thursday, 13 March 2014 at 13:22:55 UTC, Steve Teale wrote:Everything works OK with 64 bit anyway!Does anyone have any suggestions as to what might be going wrong here? I have further examples, but I guess I should do them one at a time. SteveCopied your files, added: -L-rpath=/home/geod/bin/dmd-master/linux/lib64/ as dmd is not on the system path for me. It was working. So try to update your compiler and check if that's the reason.
Mar 14 2014
On 03/13/2014 02:22 PM, Steve Teale wrote:
main : ifd.d main.d
dmd -c ifd.d
dmd -c main.d
dmd main.o ifd.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.
plugin : plugin.d
dmd -c -shared -fPIC plugin.d
dmd plugin.o -shared -defaultlib=libphobos2.so -map
clean :
rm *.o
rm main
rm plugin.so
One problem here is that main and the plugin share the definitions in
ifd.d. Therefor you need to move the interface into it's own shared
library and link both (main and any plugin) against the interface
definition.
main : main.d ifd
dmd -c main.d
dmd main.o -L-ldl -defaultlib=libphobos2.so -L-rpath=. -L./ifd.so
plugin : plugin.d ifd
dmd -c -shared -fPIC plugin.d
dmd plugin.o -shared -defaultlib=libphobos2.so -L./ifd.so
ifd : ifd.d
dmd -c -shared -fPIC ifd.d
dmd ifd.o -shared -defaultlib=libphobos2.so
clean :
rm *.o
rm main ifd.so plugin.so
Note that it was possible to load plugin.so even though it has undefined
symbols because we link with -L--export-symbols (for backtrace symbols).
This linker flag causes the main executable to export it's symbols, so
the plugin will resolve it's undefined symbols by using the ones from
main. I'm still investigating why it crashes when using --export-dynamic
though.
Mar 15 2014
On 03/15/2014 09:50 PM, Martin Nowak wrote:I'm still investigating why it crashes when using --export-dynamic though.Looks like a revenant of https://d.puremagic.com/issues/show_bug.cgi?id=9729. The problem is that nobody sets ebx before calling the interface thunk, therefor resolving the PLT entry fails. _TMP5 LABEL NEAR sub eax, 8 ; 0078 _ 83. E8, 08 jmp _D6plugin6Plugin12saySomethingMFZAya ; 007B _ E9, FFFFFFFC(PLT r)
Mar 15 2014
On 03/15/2014 11:04 PM, Martin Nowak wrote:On 03/15/2014 09:50 PM, Martin Nowak wrote:I prepared a fix here (https://github.com/MartinNowak/dmd/tree/thunkEBX) but it needs more work. Currently it violates the platform ABI because the interface thunk destroys EBX.I'm still investigating why it crashes when using --export-dynamic though.Looks like a revenant of https://d.puremagic.com/issues/show_bug.cgi?id=9729. The problem is that nobody sets ebx before calling the interface thunk, therefor resolving the PLT entry fails. _TMP5 LABEL NEAR sub eax, 8 ; 0078 _ 83. E8, 08 jmp _D6plugin6Plugin12saySomethingMFZAya ; 007B _ E9, FFFFFFFC(PLT r)
Mar 15 2014
On Saturday, 15 March 2014 at 22:56:19 UTC, Martin Nowak wrote:On 03/15/2014 11:04 PM, Martin Nowak wrote:Thanks Martin - it's good to have you involved. I'd got as far as dissasembling it and stepping single instructions in GDB while looking at that, so I could have told you where it crashed, but had no idea what was going onOn 03/15/2014 09:50 PM, Martin Nowak wrote:I prepared a fix here (https://github.com/MartinNowak/dmd/tree/thunkEBX) but it needs more work. Currently it violates the platform ABI because the interface thunk destroys EBX.I'm still investigating why it crashes when using --export-dynamic though.Looks like a revenant of https://d.puremagic.com/issues/show_bug.cgi?id=9729. The problem is that nobody sets ebx before calling the interface thunk, therefor resolving the PLT entry fails. _TMP5 LABEL NEAR sub eax, 8 ; 0078 _ 83. E8, 08 jmp _D6plugin6Plugin12saySomethingMFZAya ; 007B _ E9, FFFFFFFC(PLT r)
Mar 16 2014
On Saturday, 15 March 2014 at 22:56:19 UTC, Martin Nowak wrote:On 03/15/2014 11:04 PM, Martin Nowak wrote:When you've fixed this, you should also be able to mark issue 10719 as fixed ;=)On 03/15/2014 09:50 PM, Martin Nowak wrote:I'm still investigating why it crashes when using --export-dynamic though.
Mar 16 2014