www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Creating and loading D plugins in D app

reply aberba <karabutaworld gmail.com> writes:
Want to create and load plugins written in D into a D app at 
run-time, the kind that can make api calls or extended main app 
with other functionality.

I'm currently interested in it for a vibe.d app. How does these 
stuff work?
Jun 01 2017
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 1 June 2017 at 23:24:13 UTC, aberba wrote:

 Want to create and load plugins written in D into a D app at 
 run-time, the kind that can make api calls or extended main app 
 with other functionality.

 I'm currently interested in it for a vibe.d app. How does these 
 stuff work?


It works using shared libraries.
Jun 01 2017
next sibling parent aberba <karabutaworld gmail.com> writes:
On Friday, 2 June 2017 at 02:05:23 UTC, Stefan Koch wrote:

 On Thursday, 1 June 2017 at 23:24:13 UTC, aberba wrote:

 Want to create and load plugins written in D into a D app at 
 run-time, the kind that can make api calls or extended main 
 app with other functionality.

 I'm currently interested in it for a vibe.d app. How does 
 these stuff work?


 It works using shared libraries.


Oops
Jun 02 2017
prev sibling parent reply aberba <karabutaworld gmail.com> writes:
On Friday, 2 June 2017 at 02:05:23 UTC, Stefan Koch wrote:

 On Thursday, 1 June 2017 at 23:24:13 UTC, aberba wrote:

 Want to create and load plugins written in D into a D app at 
 run-time, the kind that can make api calls or extended main 
 app with other functionality.

 I'm currently interested in it for a vibe.d app. How does 
 these stuff work?


 It works using shared libraries.


I just read it on Wikipedia. An alternative is to use a 
scripting/interpreted language. Working with such approach feels 
unnatural in D (considering pyd, lua-d).


A more sustanable approach would be:

1. Get shared libs to work in D (the best approach for all D code)

1. some kind of embeddable interpreter for a scripting language 
like (a mini js engine) which exposes callable native D APIs at 
runtime

None of which is within my current ability.
Jun 02 2017
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 2 June 2017 at 11:09:05 UTC, aberba wrote:

 1. Get shared libs to work in D (the best approach for all D 
 code)


I have done very little with this myself but other people have so 
it is doable.


 1. some kind of embeddable interpreter for a scripting language 
 like (a mini js engine) which exposes callable native D APIs at 
 runtime


My script.d does this kind of thing
http://dpldocs.info/experimental-docs/arsd.script.html#examples

it is slow though.
Jun 02 2017
next sibling parent reply aberba <karabutaworld gmail.com> writes:
On Friday, 2 June 2017 at 12:19:48 UTC, Adam D. Ruppe wrote:

 On Friday, 2 June 2017 at 11:09:05 UTC, aberba wrote:

 1. Get shared libs to work in D (the best approach for all D 
 code)


 I have done very little with this myself but other people have 
 so it is doable.


 1. some kind of embeddable interpreter for a scripting 
 language like (a mini js engine) which exposes callable native 
 D APIs at runtime


 My script.d does this kind of thing
 http://dpldocs.info/experimental-docs/arsd.script.html#examples

 it is slow though.


Performance doesn't matter now.

Can source of script be reloaded at runtime?

Do I have to wrap external APIs in the "global" object passed as 
argument to "interpreter()"

I'm trying to implement a plugin system for a vibe.d server where 
plugins can be installed to provide additional http routes: 
extending server functionality.
Jun 02 2017
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 2 June 2017 at 13:05:41 UTC, aberba wrote:

 Can source of script be reloaded at runtime?


It is just an ordinary string.


 Do I have to wrap external APIs in the "global" object passed 
 as argument to "interpreter()"


Yes, anything the script calls must be exposed through that. It 
does a decent job automatically wrapping functions though, so 
just assigning them in a list might work.


 I'm trying to implement a plugin system for a vibe.d server 
 where plugins can be installed to provide additional http 
 routes: extending server functionality.


might work.
Jun 02 2017
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Fri, Jun 02, 2017 at 12:19:48PM +0000, Adam D. Ruppe via Digitalmars-d-learn
wrote:

 On Friday, 2 June 2017 at 11:09:05 UTC, aberba wrote:

 1. Get shared libs to work in D (the best approach for all D code)


 
 I have done very little with this myself but other people have so it is
 doable.


[...]

This is not directly related to the OP's question, but recently I wrote
a program that, given a user-specified string, transforms it into D code
using a code template, invokes dmd to compile it into a shared object,
loads the shared object using dlopen(), and looks up the generated
function with dlsym() to obtain a function pointer that can be used for
calling the function. The shared object is unloaded after it's done.

So it's definitely doable, in the sense that I've successfully generated
D code, compiled it into a shared library, loaded it into the running
executable, and can call the code.  Of course, in my case, the code
template is relatively simple so I don't have to worry about things like
module ctors, shared data, non-TLS globals, or GC use.  If you're
planning to support those features, you might need to do a bit more work
after calling dlopen(). At the very least you'd have to use dlsym() to
look up module ctor symbols and run them before calling any other
functions in the shared object, and you may have to initialize druntime
in the shared object too, if it's statically linked.  I'm not sure what
happens if it's dynamically linked, or if it uses its own copy of the
GC.

In any case, this is all possible, it just takes someone to dig into the
details and write the code to make it all work.  Then publish it on
github or dub, and the rest of us can reap the benefits too. ;-)


T

-- 
PNP = Plug 'N' Pray
Jun 02 2017
next sibling parent aberba <karabutaworld gmail.com> writes:
On Friday, 2 June 2017 at 16:36:34 UTC, H. S. Teoh wrote:

 On Fri, Jun 02, 2017 at 12:19:48PM +0000, Adam D. Ruppe via 
 Digitalmars-d-learn wrote:

 [...]


 [...]

 This is not directly related to the OP's question, but recently 
 I wrote a program that, given a user-specified string, 
 transforms it into D code using a code template, invokes dmd to 
 compile it into a shared object, loads the shared object using 
 dlopen(), and looks up the generated function with dlsym() to 
 obtain a function pointer that can be used for calling the 
 function. The shared object is unloaded after it's done.

 [...]


Blog on it?
Jun 03 2017
prev sibling parent reply aberba <karabutaworld gmail.com> writes:
On Friday, 2 June 2017 at 16:36:34 UTC, H. S. Teoh wrote:

 On Fri, Jun 02, 2017 at 12:19:48PM +0000, Adam D. Ruppe via 
 Digitalmars-d-learn wrote:

 On Friday, 2 June 2017 at 11:09:05 UTC, aberba wrote:

 1. Get shared libs to work in D (the best approach for all D 
 code)


 
 I have done very little with this myself but other people have 
 so it is doable.


 [...]

 This is not directly related to the OP's question, but recently 
 I wrote a program that, given a user-specified string, 
 transforms it into D code using a code template, invokes dmd to 
 compile it into a shared object, loads the shared object using 
 dlopen(), and looks up the generated function with dlsym() to 
 obtain a function pointer that can be used for calling the 
 function. The shared object is unloaded after it's done.


Will be of much use to me to see the brief instructions for this. 
I saw the C style on Wikipedia.

Seems the functions loaded needs to be casted from void* to a 
type... before calling. Didn't quite understand that part.


 So it's definitely doable, in the sense that I've successfully 
 generated D code, compiled it into a shared library, loaded it 
 into the running executable, and can call the code.  Of course, 
 in my case, the code template is relatively simple so I don't 
 have to worry about things like module ctors, shared data, 
 non-TLS globals, or GC use.  If you're planning to support 
 those features, you might need to do a bit more work after 
 calling dlopen(). At the very least you'd have to use dlsym() 
 to look up module ctor symbols and run them before calling any 
 other functions in the shared object, and you may have to 
 initialize druntime in the shared object too, if it's 
 statically linked.


That seem like D with its features will be lot of work if I'm 
using loaded code that way.

How about exposing standard interface/apis for loaded object's 
code to query info about the main app at runtime, similar to COM 
(if that's how if even works)


 I'm not sure what happens if it's dynamically linked, or if it 
 uses its own copy of the GC.


Does D work different from C in that regard?


 In any case, this is all possible, it just takes someone to dig 
 into the details and write the code to make it all work.  Then 
 publish it on github or dub, and the rest of us can reap the 
 benefits too. ;-)


 T
Jun 03 2017
parent Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Saturday, 3 June 2017 at 09:41:58 UTC, aberba wrote:

 On Friday, 2 June 2017 at 16:36:34 UTC, H. S. Teoh wrote:

 On Fri, Jun 02, 2017 at 12:19:48PM +0000, Adam D. Ruppe via 
 Digitalmars-d-learn wrote:

 On Friday, 2 June 2017 at 11:09:05 UTC, aberba wrote:

 1. Get shared libs to work in D (the best approach for all 
 D code)


 
 I have done very little with this myself but other people 
 have so it is doable.


 [...]

 This is not directly related to the OP's question, but 
 recently I wrote a program that, given a user-specified 
 string, transforms it into D code using a code template, 
 invokes dmd to compile it into a shared object, loads the 
 shared object using dlopen(), and looks up the generated 
 function with dlsym() to obtain a function pointer that can be 
 used for calling the function. The shared object is unloaded 
 after it's done.


 Will be of much use to me to see the brief instructions for 
 this. I saw the C style on Wikipedia.

 Seems the functions loaded needs to be casted from void* to a 
 type... before calling. Didn't quite understand that part.


Yes, dlsym() returns the address of the object of the shared 
library you requested. The first parameter is the handle to the 
shared object that had been loaded by dlopen(). The second 
parameter is the name of the object one wants the address of.
The address must then be casted to the type of the object. If the 
name was one of a function, one has to cast to a function 
pointer. That's something illegal in strict C (i.e. undefined 
behaviour.) but is something that is required by Posix. It's 
really not very difficult.
Jun 03 2017
prev sibling parent Mike B Johnson <Mikey Ikes.com> writes:
On Friday, 2 June 2017 at 12:19:48 UTC, Adam D. Ruppe wrote:

 On Friday, 2 June 2017 at 11:09:05 UTC, aberba wrote:

 1. Get shared libs to work in D (the best approach for all D 
 code)


 I have done very little with this myself but other people have 
 so it is doable.


 1. some kind of embeddable interpreter for a scripting 
 language like (a mini js engine) which exposes callable native 
 D APIs at runtime


 My script.d does this kind of thing
 http://dpldocs.info/experimental-docs/arsd.script.html#examples

 it is slow though.



You should put a link on the help somewhere so one can get to the 
github page easily to view the source(or have the ability to go 
from the module to the github source easily(small icon next to 
module name in modules list).

This would help peruse the source to see how you came up with 
some of your stuff ;)
Jun 02 2017
prev sibling parent aberba <karabutaworld gmail.com> writes:
On Thursday, 1 June 2017 at 23:24:13 UTC, aberba wrote:

 Want to create and load plugins written in D into a D app at 
 run-time, the kind that can make api calls or extended main app 
 with other functionality.

 I'm currently interested in it for a vibe.d app. How does these 
 stuff work?


Plugin system c++ 
http://blog.nuclex-games.com/tutorials/cxx/plugin-architecture/
Jun 02 2017