digitalmars.D - Load dynamic libraries with no hand-written bindings!
- Andrej Mitrovic (80/80) Sep 06 2022 There is a really cool benefit to having the ImportC feature in
- =?UTF-8?Q?Ali_=c3=87ehreli?= (5/7) Sep 06 2022 I read from Atila once "Basically, with ImportC every C library can be
- Steven Schveighoffer (8/10) Sep 06 2022 Nice! I wonder if it can be used to generate static bindings as well? As...
- Adam D Ruppe (4/7) Sep 06 2022 I'm sure you could set it up, but the point of importc is that
- Steven Schveighoffer (12/19) Sep 06 2022 Well, importC doesn't work for everything. Just like dstep doesn't.
- Adam D Ruppe (3/5) Sep 06 2022 another use for __traits(comment)!
- Andrej Mitrovic (5/8) Sep 06 2022 Ah you mean actually auto-generate bindings so you can save these
- Andrej Mitrovic (4/5) Sep 06 2022 I think Walter is working on it:
- Paul Backus (8/34) Sep 06 2022 Haven't tested, but I think you may be able to simplify this by
- Andrej Mitrovic (3/10) Sep 06 2022 Already tried that, but it doesn't work. Maybe it's just a bug in
- Paul Backus (12/26) Sep 07 2022 I'd say it's definitely a bug of some kind. This simplified
- Andrej Mitrovic (37/47) Sep 07 2022 Reduced example:
- Paul Backus (28/31) Sep 07 2022 I think the root of the problem here is that
- Andrej Mitrovic (3/12) Sep 07 2022 Ah yes, that makes sense. I haven't used traits in a long time,
- ryuukk_ (18/18) Sep 06 2022 That's pretty interesting, and it showcase the power and
- Andrej Mitrovic (3/6) Sep 06 2022 There's no need for a class. It was just an example. You can use
- Andrej Mitrovic (6/12) Sep 06 2022 Yeah this is the unfortunate thing with traits. If you want to
- Andrej Mitrovic (13/16) Sep 07 2022 There are some quirks though. For example the code will try to
There is a really cool benefit to having the ImportC feature in the language. It enables us to dynamically bind to C libraries without having to write any bindings code by hand! I don't think I'm the first one to figure this out (surely someone came up with this stuff before?), but anyway here's the trick: - Step 1: Preprocess your C library header file so you can import it in D. - Step 2: Use the magic of D's compile-time introspection to generate function pointers and the associated loader routines. - Step 3: ??? - Step 4: Profit! The entire minimal example that works on Windows is in this repo: https://github.com/AndrejMitrovic/easybind/blob/master/main.d Copied here verbatim: ```d static import portaudio; import std.meta; import std.stdio; import std.traits; import core.sys.windows.windows; struct Tuple(_FuncType, string _Name) { alias FuncType = _FuncType; enum Name = _Name; } /* Get the function pointer type of an actual function */ template FuncType(alias symbol) { ReturnType!symbol function(Parameters!symbol) func; alias FuncType = SetFunctionAttributes!(typeof(func), functionLinkage!symbol, functionAttributes!(typeof(func))); } /* Get a sequence of (Function type, Name) belonging to the provided module */ template GetFunctionList(alias Module) { alias GetFunctionList = AliasSeq!(); static foreach (idx, member; __traits(allMembers, Module)) { static if (isFunction!(__traits(getMember, Module, member))) { GetFunctionList = AliasSeq!(GetFunctionList, Tuple!(FuncType!(__traits(getMember, Module, member)), member)); } } } /* Generate dynamic bindings for all functions in Module and load SharedLib */ class Dynamic(alias Module, string SharedLib) { /* Load the shared library */ static HANDLE dll; static this() { dll = LoadLibraryA(SharedLib); !dll && assert(0); } /* Declare the function pointers */ static foreach (Tup; GetFunctionList!Module) { mixin("Tup.FuncType " ~ Tup.Name ~ ";"); } /* Load the function pointers */ this() { static foreach (Tup; GetFunctionList!Module) { *(cast(void**)&__traits(getMember, this, Tup.Name)) = cast(void*)GetProcAddress(dll, Tup.Name); } } } void main() { // easy! auto dynamic = new Dynamic!(portaudio, "portaudio_x64.dll"); printf("Version info %s\n", dynamic.Pa_GetVersionText()); } ``` And then: ``` $ dub run Version info PortAudio V19.7.0-devel, revision unknown ``` Pretty neat, huh?
Sep 06 2022
On 9/6/22 09:03, Andrej Mitrovic wrote:I don't think I'm the first one to figure this out (surely someone came up with this stuff before?)I read from Atila once "Basically, with ImportC every C library can be reflected on, because of D." Yours is a good example. :) Ali
Sep 06 2022
On 9/6/22 12:03 PM, Andrej Mitrovic wrote:Pretty neat, huh?Nice! I wonder if it can be used to generate static bindings as well? As in, just use it to spit out a D module with the proper function/struct definitions? dstep can do this, but maybe easy enough to do with ImportC and a small import file. Still doesn't help with #defines though... -Steve
Sep 06 2022
On Tuesday, 6 September 2022 at 16:28:18 UTC, Steven Schveighoffer wrote:Nice! I wonder if it can be used to generate static bindings as well? As in, just use it to spit out a D module with the proper function/struct definitions?I'm sure you could set it up, but the point of importc is that there's no benefit to doing that anymore...
Sep 06 2022
On 9/6/22 12:35 PM, Adam D Ruppe wrote:On Tuesday, 6 September 2022 at 16:28:18 UTC, Steven Schveighoffer wrote:Well, importC doesn't work for everything. Just like dstep doesn't. So generating a static binding allows you to both avoid lots of boilerplate work, but also allows you to get a complete binding for the things that importC cannot do. Also, dstep requires libclang, importC would not. You could have, in an importC shim, a way to expose all the things that aren't exposed (e.g. you have some #defines, just put them in a const array, and then you can introspect them to generate proper enums). I might try this with raylib to see how it works. One thing you won't get is comments though... -SteveNice! I wonder if it can be used to generate static bindings as well? As in, just use it to spit out a D module with the proper function/struct definitions?I'm sure you could set it up, but the point of importc is that there's no benefit to doing that anymore...
Sep 06 2022
On Tuesday, 6 September 2022 at 16:56:16 UTC, Steven Schveighoffer wrote:I might try this with raylib to see how it works. One thing you won't get is comments though...another use for __traits(comment)!
Sep 06 2022
On Tuesday, 6 September 2022 at 16:56:16 UTC, Steven Schveighoffer wrote:So generating a static binding allows you to both avoid lots of boilerplate work, but also allows you to get a complete binding for the things that importC cannot do.Ah you mean actually auto-generate bindings so you can save these somewhere so they're more visible (e.g. to IDEs and such)? Yeah that's a neat idea too.
Sep 06 2022
On Tuesday, 6 September 2022 at 16:28:18 UTC, Steven Schveighoffer wrote:Still doesn't help with #defines though...I think Walter is working on it: https://forum.dlang.org/post/t5p5ho$1mmc$1 digitalmars.com
Sep 06 2022
On Tuesday, 6 September 2022 at 16:03:03 UTC, Andrej Mitrovic wrote:```d struct Tuple(_FuncType, string _Name) { alias FuncType = _FuncType; enum Name = _Name; } /* Get the function pointer type of an actual function */ template FuncType(alias symbol) { ReturnType!symbol function(Parameters!symbol) func; alias FuncType = SetFunctionAttributes!(typeof(func), functionLinkage!symbol, functionAttributes!(typeof(func))); } /* Get a sequence of (Function type, Name) belonging to the provided module */ template GetFunctionList(alias Module) { alias GetFunctionList = AliasSeq!(); static foreach (idx, member; __traits(allMembers, Module)) { static if (isFunction!(__traits(getMember, Module, member))) { GetFunctionList = AliasSeq!(GetFunctionList, Tuple!(FuncType!(__traits(getMember, Module, member)), member)); } } } ```Haven't tested, but I think you may be able to simplify this by replacing FuncType!(__traits(getMember, Module, member)) with typeof(&__traits(getMember, Module, member)) ...which would let you eliminate the `FuncType` template entirely.
Sep 06 2022
On Tuesday, 6 September 2022 at 16:50:16 UTC, Paul Backus wrote:Haven't tested, but I think you may be able to simplify this by replacing FuncType!(__traits(getMember, Module, member)) with typeof(&__traits(getMember, Module, member)) ...which would let you eliminate the `FuncType` template entirely.Already tried that, but it doesn't work. Maybe it's just a bug in the backend.
Sep 06 2022
On Wednesday, 7 September 2022 at 05:29:31 UTC, Andrej Mitrovic wrote:On Tuesday, 6 September 2022 at 16:50:16 UTC, Paul Backus wrote:I'd say it's definitely a bug of some kind. This simplified example worked for me: ``` --- lib.c int f(int x) { return x + 1; } --- app.d import lib; typeof(&__traits(getMember, lib, "f")) fptr; pragma(msg, typeof(fptr)); // extern (C) int function(int x) ```Haven't tested, but I think you may be able to simplify this by replacing FuncType!(__traits(getMember, Module, member)) with typeof(&__traits(getMember, Module, member)) ...which would let you eliminate the `FuncType` template entirely.Already tried that, but it doesn't work. Maybe it's just a bug in the backend.
Sep 07 2022
On Wednesday, 7 September 2022 at 13:23:51 UTC, Paul Backus wrote:I'd say it's definitely a bug of some kind. This simplified example worked for me: ``` --- lib.c int f(int x) { return x + 1; } --- app.d import lib; typeof(&__traits(getMember, lib, "f")) fptr; pragma(msg, typeof(fptr)); // extern (C) int function(int x) ```Reduced example: ``` --- lib.c int foo( void ); typedef int Bar(const void *input); --- main.d import lib; import std.meta; import std.traits; template FuncType(alias symbol) { alias FuncType = typeof(&symbol); } template GetFunctionList(alias Module) { alias GetFunctionList = AliasSeq!(); static foreach (idx, member; __traits(allMembers, Module)) { static if (isFunction!(__traits(getMember, Module, member))) { GetFunctionList = AliasSeq!(GetFunctionList, typeof(&__traits(getMember, Module, member))); } } } alias X = GetFunctionList!lib; void main() { } ``` Running: ``` $ dmd lib.c main.d main.d(14): Error: `extern (C) int(const(void)* input)` is a `function` definition and cannot be modified main.d(19): Error: template instance `main.GetFunctionList!(lib)` error instantiating ``` As mentioned in the previous reply it seems that a function typedef trips it up. If there was a way to filter it out, that'd be great.
Sep 07 2022
On Wednesday, 7 September 2022 at 15:35:29 UTC, Andrej Mitrovic wrote:As mentioned in the previous reply it seems that a function typedef trips it up. If there was a way to filter it out, that'd be great.I think the root of the problem here is that `std.traits.isFunction` does not do exactly what you're assuming it does. Specifically, it evaluates to `true` for both function *symbols* and function *types*: ```d import std.traits; void fun() {} static assert(isFunction!fun); // ok static assert(isFunction!(typeof(fun))); // also ok! ``` In order to filter for just function *symbols*, you need to use a more elaborate test: ```d static if ( is(typeof(&__traits(getMember, Module, member)) PtrType) && isFunctionPointer!PtrType ) { GetFunctionList = AliasSeq!(GetFunctionList, PtrType); } ``` This first checks whether you can take the member's address, and then, if you can, checks whether the type of that address is a function pointer. If I use the condition above in your reduced example, it compiles successfully, and the resulting list does not include the `typedef`.
Sep 07 2022
On Wednesday, 7 September 2022 at 16:05:44 UTC, Paul Backus wrote:On Wednesday, 7 September 2022 at 15:35:29 UTC, Andrej Mitrovic wrote:Ah yes, that makes sense. I haven't used traits in a long time, shows how rusty I am with them. Thanks for the tips!~As mentioned in the previous reply it seems that a function typedef trips it up. If there was a way to filter it out, that'd be great.I think the root of the problem here is that `std.traits.isFunction` does not do exactly what you're assuming it does. Specifically, it evaluates to `true` for both function *symbols* and function *types*
Sep 07 2022
That's pretty interesting, and it showcase the power and capabilities of ImportC, nice! Few notes - std.meta/std.traits = bloated and slow, i suggest sticking to plain __traits and copy/paste the few(2-3) functions that you need - a whole class and new just to store that? you don't seem to make use of inheritance or any OOP features so a simple struct is enough imo If both points are solved, it'll be betterC compatible out of the box, a win imo Also, ``static import`` isn't nice, it leaks all the functions to the global scope, a little trick: ```D import pa = portaudio; (..) auto dynamic = Dynamic!(pa, "portaudio_x64.dll")(); ``` This should work?
Sep 06 2022
On Tuesday, 6 September 2022 at 16:53:16 UTC, ryuukk_ wrote:- a whole class and new just to store that? you don't seem to make use of inheritance or any OOP features so a simple struct is enough imoThere's no need for a class. It was just an example. You can use static foreach in module scope.
Sep 06 2022
On Tuesday, 6 September 2022 at 16:53:16 UTC, ryuukk_ wrote:That's pretty interesting, and it showcase the power and capabilities of ImportC, nice! Few notes - std.meta/std.traits = bloated and slow, i suggest sticking to plain __traits and copy/paste the few(2-3) functions that you needYeah this is the unfortunate thing with traits. If you want to make the code example look neat, you use std.traits. But if you want to make it fast, use __traits or is(typeof( tricks. I wanted the example to be easily understood (and I hacked this in about an hour).
Sep 06 2022
On Tuesday, 6 September 2022 at 16:03:03 UTC, Andrej Mitrovic wrote:There is a really cool benefit to having the ImportC feature in the language. It enables us to dynamically bind to C libraries without having to write any bindings code by hand!There are some quirks though. For example the code will try to load this as if it were a function: ```c typedef int PaStreamCallback( const void *input, void *output, unsigned long frameCount, const PaStreamCallbackTimeInfo* timeInfo, PaStreamCallbackFlags statusFlags, void *userData ); ``` But it's actually just a typedef for a callback.
Sep 07 2022