www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Emulating DLL

reply Craig <Craig Tonsils.com> writes:
Is it possible to create a D module that has functions in it, and 
then use those functions dynamically at run time emulating DLL 
like functionality?

e.g.,

module MyDL;

import Context;
export void foo(Context)
{
    ...
}

and

module UseDL;
import DL;
import Context;

void main()
{
     DL.Load("MyDL.dl");
     DL.foo(new Context);
     DL.Unload();

     // or
     DL.Loadable("MyDL.dl"); // monitors changes and reloads the 
function safely  when a change is detected. This might require 
some work to make safe.
     DL.foo(new Context);
     DL.Unload();
}


(pseudo)

The idea here is to simply be able to call external code that can 
be hotswapped and such. It requires dealing with relocatable 
code, GC, etc.


It would be portable and non-windows specific and just work.

dmd and ldc of course do not output anything that is reasonable 
as they use the traditional non-portable formats. This leads me 
to think that without modifying them, I'd have to extract the 
code from the DLL/SO and deal with all the problems and then 
generate the .dl file.

The idea is to create simple way to "script" D programs and use 
the D compiler to compile the functionality. It won't be as fast 
as an interpreter but it will have full speed.

I'm just not sure what would be the best way to generate the code 
and do the fixing up.

I'd be willing to hack on the dmd compiler to try and get it to 
generate a more sane file for this process if someone could point 
me to some details.
Mar 18 2019
next sibling parent reply Ethan Watson <gooberman gmail.com> writes:
On Monday, 18 March 2019 at 22:50:57 UTC, Craig wrote:

 Is it possible to create a D module that has functions in it, 
 and then use those functions dynamically at run time emulating 
 DLL like functionality?


On Monday, 18 March 2019 at 22:50:57 UTC, Craig wrote:

 Is it possible to create a D module that has functions in it, 
 and then use those functions dynamically at run time emulating 
 DLL like functionality?


I've talked extensively on this topic at DConf over the last few 
years.

http://dconf.org/2016/talks/watson.html
http://dconf.org/2017/talks/watson.html
http://dconf.org/2018/talks/watson.html

https://github.com/GooberMan/binderoo

It's not a simple thing by any means, there's plenty of little 
implementation details that you'll need to be aware of. But yes, 
it's quite possible.
Mar 18 2019
parent reply Craig <Craig Tonsils.com> writes:
On Monday, 18 March 2019 at 22:59:12 UTC, Ethan Watson wrote:

 On Monday, 18 March 2019 at 22:50:57 UTC, Craig wrote:

 Is it possible to create a D module that has functions in it, 
 and then use those functions dynamically at run time emulating 
 DLL like functionality?


 On Monday, 18 March 2019 at 22:50:57 UTC, Craig wrote:

 Is it possible to create a D module that has functions in it, 
 and then use those functions dynamically at run time emulating 
 DLL like functionality?


 I've talked extensively on this topic at DConf over the last 
 few years.

 http://dconf.org/2016/talks/watson.html
 http://dconf.org/2017/talks/watson.html
 http://dconf.org/2018/talks/watson.html

 https://github.com/GooberMan/binderoo

 It's not a simple thing by any means, there's plenty of little 
 implementation details that you'll need to be aware of. But 
 yes, it's quite possible.


This seems overly complex for what I want and not exactly what I 
need. I'm looking for simply that simply lets me call functions 
that exist in an external dynamically accessible file.

In my host code I will use a function like "foo" but that foo 
will be a function pointer that can point to the external code. I 
don't need a bunch of bells and whistles or complex ways to 
accomplish this.

1. Load up external code,
2. Fixup whatever needs fixing up to make the code work properly
3. Point function pointer to code.
4. Call.

For simple functions such as pure functions step 2 can be skipped 
and hence it is essentially just loading the code in to memory 
and calling it.

I do get that in general it is more complex but other than 
relocatable code, initialization, and the GC, it shouldn't 
require much more work.


For example, with windows I could simply compile to a dll then 
extract the code, or just use LoadLibrary and it effectively does 
all the work(steps 1 & 2 & 3).

LoadLibrary is not portable though and seems excessive to do 
something that should be rather simple unless there is something 
I'm missing that has to be done that is complex. Relocation is 
easy if one has the appropriate locations to fix up which are 
generally included in the binary somewhere. The GC and rt can be 
initialized easily.
Mar 19 2019
next sibling parent reply Ethan <gooberman gmail.com> writes:
On Tuesday, 19 March 2019 at 19:50:15 UTC, Craig wrote:

 For example, with windows I could simply compile to a dll then 
 extract the code, or just use LoadLibrary and it effectively 
 does all the work(steps 1 & 2 & 3).


LoadLibrary performs steps 1 and part of step 2. The DllMain 
function of a library is called separately by Windows for each 
thread in your system, which is where the bulk of step 2 is 
handled. Step 3 isn't handled at all by LoadLibrary, and is 
instead entirely up to you to deal with via the GetProcAddress 
function.

If you want DLLs to operate in that step 1-2-3 manner, then the 
compiler can generate a static library that handles that all for 
you. But, as you might expect, that removes the hot reloading 
capability as that is all handled before WinMain is entered.

If you expect hot reloading to work without effort, you're going 
to have to compile your DLLs with -betterC. The D Runtime is not 
yet in a standalone DLL (unless I've missed an announcement over 
the last couple of months). Each DLL you build will require the D 
runtime to be embedded. Things get really tricky from there.

If you ever use your DLL for more than calling functions with 
basic types, or value aggregates (structs) containing basic 
types, then be aware that without -betterC that there is 
potential for the moduleinfo and typeinfo systems to go a bit 
screwy. Especially once druntime lives in its own DLL, as 
druntime expects those systems to be initialised once at startup. 
POSIX libraries will have the same issues, but I have nod had a 
need to investigate further over the last few months.


 LoadLibrary is not portable though and seems excessive to do 
 something that should be rather simple unless there is 
 something I'm missing that has to be done that is complex.


LoadLibrary is the equivalent to dlopen on POSIX systems, with 
one very important difference: Windows does not provide lazy 
symbol evaluation. As such, extra work is required to handle 
that. The average usecase is covered by the mentioned static lib, 
but any hot reloading is handled in a custom manner by every 
Windows codebase you'll encounter.

The static lib I mention, in fact, redirects every cross-DLL call 
to a jump table. And cross-DLL data sharing gets hairy in 
Windows. Function parameters aren't a problem, but Windows 
explicitly disallows thread local variables to cross the DLL 
boundary for example (even if you expose it for export and 
perform a symbol lookup, it won't work and you'll get garbage 
back).

As I said, it's not a simple thing by any means. My work has 
attempted to hide all that complexity by boiling it all down to 
"include these headers/modules, link this lib, and get on with 
life". If you find the resulting code too complex (and I freely 
admit that I haven't got the integration in to as foolproof a 
manner as I am aiming for) then very definitely read through the 
presentations to get a clear idea of why something that should be 
so simple really isn't.
Mar 20 2019
parent reply Craig <Craig Tonsils.com> writes:
On Wednesday, 20 March 2019 at 14:27:54 UTC, Ethan wrote:

 On Tuesday, 19 March 2019 at 19:50:15 UTC, Craig wrote:

 For example, with windows I could simply compile to a dll then 
 extract the code, or just use LoadLibrary and it effectively 
 does all the work(steps 1 & 2 & 3).


 LoadLibrary performs steps 1 and part of step 2. The DllMain 
 function of a library is called separately by Windows for each 
 thread in your system, which is where the bulk of step 2 is 
 handled. Step 3 isn't handled at all by LoadLibrary, and is 
 instead entirely up to you to deal with via the GetProcAddress 
 function.

 If you want DLLs to operate in that step 1-2-3 manner, then the 
 compiler can generate a static library that handles that all 
 for you. But, as you might expect, that removes the hot 
 reloading capability as that is all handled before WinMain is 
 entered.

 If you expect hot reloading to work without effort, you're 
 going to have to compile your DLLs with -betterC. The D Runtime 
 is not yet in a standalone DLL (unless I've missed an 
 announcement over the last couple of months). Each DLL you 
 build will require the D runtime to be embedded. Things get 
 really tricky from there.

 If you ever use your DLL for more than calling functions with 
 basic types, or value aggregates (structs) containing basic 
 types, then be aware that without -betterC that there is 
 potential for the moduleinfo and typeinfo systems to go a bit 
 screwy. Especially once druntime lives in its own DLL, as 
 druntime expects those systems to be initialised once at 
 startup. POSIX libraries will have the same issues, but I have 
 nod had a need to investigate further over the last few months.


 LoadLibrary is not portable though and seems excessive to do 
 something that should be rather simple unless there is 
 something I'm missing that has to be done that is complex.


 LoadLibrary is the equivalent to dlopen on POSIX systems, with 
 one very important difference: Windows does not provide lazy 
 symbol evaluation. As such, extra work is required to handle 
 that. The average usecase is covered by the mentioned static 
 lib, but any hot reloading is handled in a custom manner by 
 every Windows codebase you'll encounter.

 The static lib I mention, in fact, redirects every cross-DLL 
 call to a jump table. And cross-DLL data sharing gets hairy in 
 Windows. Function parameters aren't a problem, but Windows 
 explicitly disallows thread local variables to cross the DLL 
 boundary for example (even if you expose it for export and 
 perform a symbol lookup, it won't work and you'll get garbage 
 back).

 As I said, it's not a simple thing by any means. My work has 
 attempted to hide all that complexity by boiling it all down to 
 "include these headers/modules, link this lib, and get on with 
 life". If you find the resulting code too complex (and I freely 
 admit that I haven't got the integration in to as foolproof a 
 manner as I am aiming for) then very definitely read through 
 the presentations to get a clear idea of why something that 
 should be so simple really isn't.


You are making this more complicated because you don't understand 
that I don't need a general purpose solution. I simply need to 
call simple functions that do not do much, not full blown 
applications.

For example, Suppose I want to write a fractal display 
application and instead of hard coding the function to display, 
it's external.

e.g.,
import std.complex;
alias C = Complex!double;

C Mandelbrot(C z, C c) { return z*z + c; }


This function does not require much of anything to run. The 
machine code can be copied and pasted at will to any executable 
memory location and ran... it can, in fact, be inlined with a 
little caution(reserve enough space and ignore the stack code). 
No initialization needs to be nor garbage collecting.

For a little more robustness one would want to allow for GC, 
which should not be hard, and context passing so more useful 
functionality can be had.


If D had a Dstring2Machine that compiled the above code to 
machine code then it could be used directly without issue.

Because these functions are deterministic and the host determines 
their usage, there is little issues to worry about.  Hot swapping 
is simply changing a pointer when the function is not being 
called.

As long as certain guidelines are followed one can do this quite 
simply without too much issue and it allows for a scripting like 
solution for D apps. It allows one to inject functionality in to 
an application that runs at full speed and is relatively fast to 
compile.

The idea is not to be able to connect two full blow external apps 
but to connect little pieces of functionality to an app that 
allow for vast flexibility. Since each unit of functionality is 
essentially pure, there are no major issues.

It's this simple:

1. Take source and compile it in to machine code.
2. App extracts machine code and inserts it in to it's memory 
space to be able to execute.

Do you agree that, at least in some cases, this is entirely 
feasible and easy?


I'll give you a hint:

1.

module simple;

void foo() { };

compile directly in to machine code(the most raw form possible 
containing foo).

2.

    App load the machine code, execute, reload and execute if 
desired...


Since foo does nothing it's just a nop. We are just copying 
around a ret. It's impossible for anything to ever go awry here 
excluding buggy code/compiler/etc.


 From here one can expand the model to include something that is 
more useful.

The idea is to expand it enough to make it generally quite useful 
but not so much one has to include the kitchen sink.

I don't need a heavy version of binderoo, I need a light one. 
Something that lets me do basic "scripting" so I can modify the 
application behavior at runtime without recompiling the 
application.

If you think the example is stupid and useless then

int age() { return 43; }

and nothing changes.

of course you will then think that example is stupid because we 
could just put it in a text file and read that rather than go 
through all this trouble.

so then

C calc(C z, C c) { return z*z*z*z*z*z + c*z + c; }

and to get around this without leveraging the D compiler one has 
to make a full blown parser... even though it's still just 
relatively basic code like age. It just includes some 
calculations.

It's no different from foo though. Just basic code.

But we can go further by allowing more information to be passed 
to and from the function. The function can also be impure such as

void foo() { writeln("I'm IMPURE!"); }

which will work without much work(might require fixing up a few 
symbols/pointers).


All it is, is delayed compilation. I'm simply wanting to delay 
the compilation of a piece of code in an app from the main 
compilation. All the other stuff about the OS and TLS are 
irrelevant in this context.

One could sort of do this with the object files and just 
recompile the functions and relink everything. But with a little 
work this step can be avoided since it's not needed and 
slower(but more robust).
Mar 20 2019
parent Ethan Watson <gooberman gmail.com> writes:
On Wednesday, 20 March 2019 at 19:44:52 UTC, Craig wrote:

 Do you agree that, at least in some cases, this is entirely 
 feasible and easy?


I literally answered in the affirmative in my first post.

Note that all that Binderoo does is put function pointers in 
places that looks like real code, and does this automatically for 
DLL recompiles.

Everything you just described in your above post overcomplicates 
it from there. Go wild. But you're not going to get far if you 
choose to expand it past your initial examples (as you've 
suggested you might do) if you don't take the above 
implementation details in to account.

Like I literally said in my first post.
Mar 20 2019
prev sibling parent reply SrMordred <patric.dexheimer gmail.com> writes:
On Tuesday, 19 March 2019 at 19:50:15 UTC, Craig wrote:

Take a look at my lib, its a simple hot-reload external dll lib:

https://github.com/SrMordred/reloaded

I did´nt use it extensively, but its simple enough to be tweaked 
at your own need :)
Mar 20 2019
parent Craig <Craig Tonsils.com> writes:
On Thursday, 21 March 2019 at 02:27:46 UTC, SrMordred wrote:

 On Tuesday, 19 March 2019 at 19:50:15 UTC, Craig wrote:

 Take a look at my lib, its a simple hot-reload external dll lib:

 https://github.com/SrMordred/reloaded

 I did´nt use it extensively, but its simple enough to be 
 tweaked at your own need :)


Thanks, Looks much more like what I need. I'll have to mess 
around with it to see how it works though.
Mar 25 2019
prev sibling parent reply Kagamin <spam here.lot> writes:
There was this old project: http://dsource.org/projects/ddl/
Mar 19 2019
next sibling parent Craig <Craig Tonsils.com> writes:
On Tuesday, 19 March 2019 at 09:18:56 UTC, Kagamin wrote:

 There was this old project: http://dsource.org/projects/ddl/


Thanks, looks pretty similar to what I was thinking. I'll have to 
mess with it a bit but maybe I can tweak any issues to suit my 
agenda.
Mar 19 2019
prev sibling parent Craig <Craig Tonsils.com> writes:
On Tuesday, 19 March 2019 at 09:18:56 UTC, Kagamin wrote:

 There was this old project: http://dsource.org/projects/ddl/


Seems it is too heavily dependent on mago and has a lot of 
depreciated content. Probably not worth trying to update. 
Probably much easier just to hack dmd to export a simpler dll 
format or use dll's directly.
Mar 19 2019