www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Load dynamic libraries with no hand-written bindings!

reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
There is a really cool benefit to having the ImportC feature in 
the language. It enables us to dynamically bind to C libraries 
without having to write any bindings code by hand!

I don't think I'm the first one to figure this out (surely 
someone came up with this stuff before?), but anyway here's the 
trick:

- Step 1: Preprocess your C library header file so you can import 
it in D.
- Step 2: Use the magic of D's compile-time introspection to 
generate function pointers and the associated loader routines.
- Step 3: ???
- Step 4: Profit!

The entire minimal example that works on Windows is in this repo: 
https://github.com/AndrejMitrovic/easybind/blob/master/main.d

Copied here verbatim:

```d
static import portaudio;

import std.meta;
import std.stdio;
import std.traits;
import core.sys.windows.windows;

struct Tuple(_FuncType, string _Name) {
     alias FuncType = _FuncType;
     enum Name = _Name;
}

/* Get the function pointer type of an actual function */
template FuncType(alias symbol) {
     ReturnType!symbol function(Parameters!symbol) func;
     alias FuncType = SetFunctionAttributes!(typeof(func), 
functionLinkage!symbol,
         functionAttributes!(typeof(func)));
}

/* Get a sequence of (Function type, Name) belonging to the 
provided module */
template GetFunctionList(alias Module) {
     alias GetFunctionList = AliasSeq!();
     static foreach (idx, member; __traits(allMembers, Module)) {
         static if (isFunction!(__traits(getMember, Module, 
member))) {
             GetFunctionList = AliasSeq!(GetFunctionList,
                 Tuple!(FuncType!(__traits(getMember, Module, 
member)), member));
         }
     }
}

/* Generate dynamic bindings for all functions in Module and load 
SharedLib */
class Dynamic(alias Module, string SharedLib)
{
     /* Load the shared library */
     static HANDLE dll;
     static this() {
         dll = LoadLibraryA(SharedLib);
         !dll && assert(0);
     }

     /* Declare the function pointers */
     static foreach (Tup; GetFunctionList!Module) {
         mixin("Tup.FuncType " ~ Tup.Name ~ ";");
     }

     /* Load the function pointers */
     this()
     {
         static foreach (Tup; GetFunctionList!Module) {
             *(cast(void**)&__traits(getMember, this, Tup.Name))
                 = cast(void*)GetProcAddress(dll, Tup.Name);
         }
     }
}

void main() {
     // easy!
     auto dynamic = new Dynamic!(portaudio, "portaudio_x64.dll");
     printf("Version info %s\n", dynamic.Pa_GetVersionText());
}
```

And then:

```
$ dub run
Version info PortAudio V19.7.0-devel, revision unknown
```

Pretty neat, huh?
Sep 06 2022
next sibling parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 9/6/22 09:03, Andrej Mitrovic wrote:

 I don't think I'm the first one to figure this out (surely someone came
 up with this stuff before?)
I read from Atila once "Basically, with ImportC every C library can be reflected on, because of D." Yours is a good example. :) Ali
Sep 06 2022
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/6/22 12:03 PM, Andrej Mitrovic wrote:

 Pretty neat, huh?
 
Nice! I wonder if it can be used to generate static bindings as well? As in, just use it to spit out a D module with the proper function/struct definitions? dstep can do this, but maybe easy enough to do with ImportC and a small import file. Still doesn't help with #defines though... -Steve
Sep 06 2022
next sibling parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Tuesday, 6 September 2022 at 16:28:18 UTC, Steven 
Schveighoffer wrote:
 Nice! I wonder if it can be used to generate static bindings as 
 well? As in, just use it to spit out a D module with the proper 
 function/struct definitions?
I'm sure you could set it up, but the point of importc is that there's no benefit to doing that anymore...
Sep 06 2022
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/6/22 12:35 PM, Adam D Ruppe wrote:
 On Tuesday, 6 September 2022 at 16:28:18 UTC, Steven Schveighoffer wrote:
 Nice! I wonder if it can be used to generate static bindings as well? 
 As in, just use it to spit out a D module with the proper 
 function/struct definitions?
I'm sure you could set it up, but the point of importc is that there's no benefit to doing that anymore...
Well, importC doesn't work for everything. Just like dstep doesn't. So generating a static binding allows you to both avoid lots of boilerplate work, but also allows you to get a complete binding for the things that importC cannot do. Also, dstep requires libclang, importC would not. You could have, in an importC shim, a way to expose all the things that aren't exposed (e.g. you have some #defines, just put them in a const array, and then you can introspect them to generate proper enums). I might try this with raylib to see how it works. One thing you won't get is comments though... -Steve
Sep 06 2022
next sibling parent Adam D Ruppe <destructionator gmail.com> writes:
On Tuesday, 6 September 2022 at 16:56:16 UTC, Steven 
Schveighoffer wrote:
 I might try this with raylib to see how it works. One thing you 
 won't get is comments though...
another use for __traits(comment)!
Sep 06 2022
prev sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Tuesday, 6 September 2022 at 16:56:16 UTC, Steven 
Schveighoffer wrote:
 So generating a static binding allows you to both avoid lots of 
 boilerplate work, but also allows you to get a complete binding 
 for the things that importC cannot do.
Ah you mean actually auto-generate bindings so you can save these somewhere so they're more visible (e.g. to IDEs and such)? Yeah that's a neat idea too.
Sep 06 2022
prev sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Tuesday, 6 September 2022 at 16:28:18 UTC, Steven 
Schveighoffer wrote:
 Still doesn't help with #defines though...
I think Walter is working on it: https://forum.dlang.org/post/t5p5ho$1mmc$1 digitalmars.com
Sep 06 2022
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Tuesday, 6 September 2022 at 16:03:03 UTC, Andrej Mitrovic 
wrote:
 ```d
 struct Tuple(_FuncType, string _Name) {
     alias FuncType = _FuncType;
     enum Name = _Name;
 }

 /* Get the function pointer type of an actual function */
 template FuncType(alias symbol) {
     ReturnType!symbol function(Parameters!symbol) func;
     alias FuncType = SetFunctionAttributes!(typeof(func), 
 functionLinkage!symbol,
         functionAttributes!(typeof(func)));
 }

 /* Get a sequence of (Function type, Name) belonging to the 
 provided module */
 template GetFunctionList(alias Module) {
     alias GetFunctionList = AliasSeq!();
     static foreach (idx, member; __traits(allMembers, Module)) {
         static if (isFunction!(__traits(getMember, Module, 
 member))) {
             GetFunctionList = AliasSeq!(GetFunctionList,
                 Tuple!(FuncType!(__traits(getMember, Module, 
 member)), member));
         }
     }
 }
```
Haven't tested, but I think you may be able to simplify this by replacing FuncType!(__traits(getMember, Module, member)) with typeof(&__traits(getMember, Module, member)) ...which would let you eliminate the `FuncType` template entirely.
Sep 06 2022
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Tuesday, 6 September 2022 at 16:50:16 UTC, Paul Backus wrote:
 Haven't tested, but I think you may be able to simplify this by 
 replacing

     FuncType!(__traits(getMember, Module, member))

 with

     typeof(&__traits(getMember, Module, member))

 ...which would let you eliminate the `FuncType` template 
 entirely.
Already tried that, but it doesn't work. Maybe it's just a bug in the backend.
Sep 06 2022
parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 7 September 2022 at 05:29:31 UTC, Andrej Mitrovic 
wrote:
 On Tuesday, 6 September 2022 at 16:50:16 UTC, Paul Backus wrote:
 Haven't tested, but I think you may be able to simplify this 
 by replacing

     FuncType!(__traits(getMember, Module, member))

 with

     typeof(&__traits(getMember, Module, member))

 ...which would let you eliminate the `FuncType` template 
 entirely.
Already tried that, but it doesn't work. Maybe it's just a bug in the backend.
I'd say it's definitely a bug of some kind. This simplified example worked for me: ``` --- lib.c int f(int x) { return x + 1; } --- app.d import lib; typeof(&__traits(getMember, lib, "f")) fptr; pragma(msg, typeof(fptr)); // extern (C) int function(int x) ```
Sep 07 2022
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Wednesday, 7 September 2022 at 13:23:51 UTC, Paul Backus wrote:
 I'd say it's definitely a bug of some kind. This simplified 
 example worked for me:

 ```
 --- lib.c
 int f(int x) { return x + 1; }

 --- app.d
 import lib;

 typeof(&__traits(getMember, lib, "f")) fptr;
 pragma(msg, typeof(fptr)); // extern (C) int function(int x)
 ```
Reduced example: ``` --- lib.c int foo( void ); typedef int Bar(const void *input); --- main.d import lib; import std.meta; import std.traits; template FuncType(alias symbol) { alias FuncType = typeof(&symbol); } template GetFunctionList(alias Module) { alias GetFunctionList = AliasSeq!(); static foreach (idx, member; __traits(allMembers, Module)) { static if (isFunction!(__traits(getMember, Module, member))) { GetFunctionList = AliasSeq!(GetFunctionList, typeof(&__traits(getMember, Module, member))); } } } alias X = GetFunctionList!lib; void main() { } ``` Running: ``` $ dmd lib.c main.d main.d(14): Error: `extern (C) int(const(void)* input)` is a `function` definition and cannot be modified main.d(19): Error: template instance `main.GetFunctionList!(lib)` error instantiating ``` As mentioned in the previous reply it seems that a function typedef trips it up. If there was a way to filter it out, that'd be great.
Sep 07 2022
parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 7 September 2022 at 15:35:29 UTC, Andrej Mitrovic 
wrote:
 As mentioned in the previous reply it seems that a function 
 typedef trips it up. If there was a way to filter it out, 
 that'd be great.
I think the root of the problem here is that `std.traits.isFunction` does not do exactly what you're assuming it does. Specifically, it evaluates to `true` for both function *symbols* and function *types*: ```d import std.traits; void fun() {} static assert(isFunction!fun); // ok static assert(isFunction!(typeof(fun))); // also ok! ``` In order to filter for just function *symbols*, you need to use a more elaborate test: ```d static if ( is(typeof(&__traits(getMember, Module, member)) PtrType) && isFunctionPointer!PtrType ) { GetFunctionList = AliasSeq!(GetFunctionList, PtrType); } ``` This first checks whether you can take the member's address, and then, if you can, checks whether the type of that address is a function pointer. If I use the condition above in your reduced example, it compiles successfully, and the resulting list does not include the `typedef`.
Sep 07 2022
parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Wednesday, 7 September 2022 at 16:05:44 UTC, Paul Backus wrote:
 On Wednesday, 7 September 2022 at 15:35:29 UTC, Andrej Mitrovic 
 wrote:
 As mentioned in the previous reply it seems that a function 
 typedef trips it up. If there was a way to filter it out, 
 that'd be great.
I think the root of the problem here is that `std.traits.isFunction` does not do exactly what you're assuming it does. Specifically, it evaluates to `true` for both function *symbols* and function *types*
Ah yes, that makes sense. I haven't used traits in a long time, shows how rusty I am with them. Thanks for the tips!~
Sep 07 2022
prev sibling next sibling parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
That's pretty interesting, and it showcase the power and 
capabilities of ImportC, nice!

Few notes

- std.meta/std.traits = bloated and slow, i suggest sticking to 
plain __traits and copy/paste the few(2-3) functions that you need

- a whole class and new just to store that? you don't seem to 
make use of inheritance or any OOP features so a simple struct is 
enough imo

If both points are solved, it'll be betterC compatible out of the 
box, a win imo

Also, ``static import`` isn't nice, it leaks all the functions to 
the global scope, a little trick:

```D
import pa = portaudio;

(..)

auto dynamic = Dynamic!(pa, "portaudio_x64.dll")();
```

This should work?
Sep 06 2022
next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Tuesday, 6 September 2022 at 16:53:16 UTC, ryuukk_ wrote:
 - a whole class and new just to store that? you don't seem to 
 make use of inheritance or any OOP features so a simple struct 
 is enough imo
There's no need for a class. It was just an example. You can use static foreach in module scope.
Sep 06 2022
prev sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Tuesday, 6 September 2022 at 16:53:16 UTC, ryuukk_ wrote:
 That's pretty interesting, and it showcase the power and 
 capabilities of ImportC, nice!

 Few notes

 - std.meta/std.traits = bloated and slow, i suggest sticking to 
 plain __traits and copy/paste the few(2-3) functions that you 
 need
Yeah this is the unfortunate thing with traits. If you want to make the code example look neat, you use std.traits. But if you want to make it fast, use __traits or is(typeof( tricks. I wanted the example to be easily understood (and I hacked this in about an hour).
Sep 06 2022
prev sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Tuesday, 6 September 2022 at 16:03:03 UTC, Andrej Mitrovic 
wrote:
 There is a really cool benefit to having the ImportC feature in 
 the language. It enables us to dynamically bind to C libraries 
 without having to write any bindings code by hand!
There are some quirks though. For example the code will try to load this as if it were a function: ```c typedef int PaStreamCallback( const void *input, void *output, unsigned long frameCount, const PaStreamCallbackTimeInfo* timeInfo, PaStreamCallbackFlags statusFlags, void *userData ); ``` But it's actually just a typedef for a callback.
Sep 07 2022