www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Pragma mangle and D shared objects

reply Etienne Cimon <etcimon gmail.com> writes:
I haven't been able to find much about pragma mangle. I'd like to do the 
following:

http://forum.dlang.org/thread/hznsrmviciaeirqkjzpy forum.dlang.org#post-zhxnqqubyudteycwudzz:40forum.dlang.org

The part I find ugly is this:

void* vp = dlsym(lib, "_D6plugin11getInstanceFZC2bc2Bc\0".ptr);

I want to write a framework that stores a dynamic library name and 
symbol to execute, and downloads the dynamic library if it's not 
available. This would be in a long-running server/networking 
application, and needs to be simple to use.

The mangling makes it less obvious for the programmer writing a plugin. 
Does mangle make it possible to change this to dlsym(lib, 
"myOwnMangledName"), or would it still have strange symbols?

Also, I've never seen the thunkEBX change merged from here:

http://forum.dlang.org/thread/hznsrmviciaeirqkjzpy forum.dlang.org?page=2#post-lg2lqi:241ga3:241:40digitalmars.com
Oct 25 2014
parent reply "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Sat, Oct 25, 2014 at 09:20:33AM -0400, Etienne Cimon via Digitalmars-d-learn
wrote:

 I haven't been able to find much about pragma mangle. I'd like to do
 the following:
 
 http://forum.dlang.org/thread/hznsrmviciaeirqkjzpy forum.dlang.org#post-zhxnqqubyudteycwudzz:40forum.dlang.org
 
 The part I find ugly is this:
 
 void* vp = dlsym(lib, "_D6plugin11getInstanceFZC2bc2Bc\0".ptr);
 
 I want to write a framework that stores a dynamic library name and
 symbol to execute, and downloads the dynamic library if it's not
 available. This would be in a long-running server/networking
 application, and needs to be simple to use.


Perhaps the .mangleof built-in property might help you here? For
instance:

	---plugin.d---
	class MyClass;
	MyClass getInstance();

	---test.d---
	import std.stdio;
	import plugin;

	void main() {
		writeln(plugin.getInstance.mangleof);
	}

Output:

	_D6plugin11getInstanceFZC6plugin7MyClass

Granted, it's a bit ugly (you have to actually create a module called
'plugin' in order to get the right mangling), but at least it doesn't
require the user to learn how D's mangling scheme works. You just write
a function prototype for the function you're trying to lookup, and call
.mangleof on it.



 The mangling makes it less obvious for the programmer writing a
 plugin. Does mangle make it possible to change this to dlsym(lib,
 "myOwnMangledName"), or would it still have strange symbols?


[...]

What you *could* do, is to use .mangleof and clever regex'ing to make it
possible to do that. For example, something along these lines:

	---dl_support.d---
	alias ReturnType = ... /* whatever type you want */;
	private ReturnType pluginFuncStubName(... /* arguments here */);

	auto loadPluginFunction(string library, string funcName) {
		auto r = regex(`pluginFuncStubName`);
		auto mangledName =
			pluginFuncStubName.mangleof.replace(r, funcName);
		...
		/* mangledName should now be the mangled string you need
		 * to find the symbol */
	}

Basically, use an unambiguous blatantly long name for your function
prototype, and do a search-and-replace to substitute that with the
desired function name.

Note that you still need to have a separate stub per function signature;
so if you want users to be able to load functions of arbitrary
signature, you probably need to make that a template parameter and have
the user pass in the desired function signature. For example:

	auto loadPluginFunction(Signature)(string library, string funcName)
	{
		import std.traits : ReturnType, ParameterTypeTuple;
		import std.regex : regex, replaceFirst;
	
		// Declare a static function with the user-desired
		// signature, with a nicely-substitutable name
		static ReturnType!Signature
			pluginFuncStubName(ParameterTypeTuple!Signature);
		auto r = regex(`pluginFuncStubName`);
		auto symbol = pluginFuncStubName.mangleof.replaceFirst(r, funcName);
	
		// Proof of concept
		import std.stdio;
		writeln(symbol);
	
		// ... Call dlsym to find function here
	
		// Just to make this compile, replace with real function pointer in
		// your code here.
		return null;
	}
	
	void main() {
		auto f1 = loadPluginFunction!(int function(string,int))("mylib", "func1");
		auto f2 = loadPluginFunction!(void function(float))("mylib", "func2");
	}

Output:

	_D4test33__T18loadPluginFunctionTPFAyaiZiZ18loadPluginFunctionFAyaAyaZ18func1FAyaiZi
	_D4test30__T18loadPluginFunctionTPFfZvZ18loadPluginFunctionFAyaAyaZ18func2FfZv

This example isn't complete yet (you need to do something about the
"loadPluginFunction" component in the mangled name), but you should be
able to work out a way of producing the correct mangled name from here.
But at least it demonstrates how you can have a very nice API for your
users -- they just pass in the function prototype of the function they
want, and the library code takes care of deriving the correct mangled
names.

Hope this helps.


T

-- 
Just because you can, doesn't mean you should.
Oct 25 2014
parent reply Etienne Cimon <etcimon gmail.com> writes:
That looks like exactly the solution I need, very clever. It'll take 
some time to wrap my head around it :-P
Oct 25 2014
parent reply Etienne Cimon <etcimon gmail.com> writes:
On 2014-10-25 11:56, Etienne Cimon wrote:

 That looks like exactly the solution I need, very clever. It'll take
 some time to wrap my head around it :-P


Just brainstorming here, but I think every dynamic library should hold a 
utility container (hash map?) that searches for and returns the mangled 
names in itself using regex match. This container would always be in the 
same module/function name in every dynamic library.

The mangling list would load itself using introspection through a 
`shared static this()`. For each module, for each class, insert mangling 
in the hashmap...

This seems ideal because then you basically have libraries that document 
themselves at runtime.

Of course, the return type and arguments would have to be decided in 
advance, but otherwise it allows loading and use of (possibly remote and 
unknown) DLLs, in a very simple way.
Oct 25 2014
parent reply "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Sat, Oct 25, 2014 at 12:15:19PM -0400, Etienne Cimon via Digitalmars-d-learn
wrote:

 On 2014-10-25 11:56, Etienne Cimon wrote:

That looks like exactly the solution I need, very clever. It'll take
some time to wrap my head around it :-P




It's not that complicated, really. It's basically putting together 3
things:

1) .mangleof to extract a mangled symbol from some function declaration
2) regex to replace the function name in the mangled symbol
3) compile-time introspection to build a function declaration out of a
   user-specified signature.



 Just brainstorming here, but I think every dynamic library should hold
 a utility container (hash map?) that searches for and returns the
 mangled names in itself using regex match. This container would always
 be in the same module/function name in every dynamic library.


Actually, the object file (library) itself should already have a list of
exported symbols; you could then use core.demangle to extract the
function signatures from the mangled symbols and construct a hash of all
exported symbols and their types. The only thing is, I don't know of any
cross-platform method of retrieving the list of exported symbols -- the
Posix dlsym() family of functions only allow lookup by explicit symbol
name, no iteration primitives are specified. But the information is
definitely there, since that's how the OS's dynamic linker figures out
how to link dynamic libraries in the first place!


[...]

 Of course, the return type and arguments would have to be decided in
 advance, but otherwise it allows loading and use of (possibly remote
 and unknown) DLLs, in a very simple way.


Well, generally, in order to make use of the functions in the first
place, you'd need some kind of pre-determined return type and parameter
types, otherwise the program couldn't possibly know how to pass
arguments or interpret the return value!

But if you could extract the list of exported symbols from a library,
then you could demangle them to determine their signatures, and thereby
find all symbols matching some given signature.


T

-- 
Curiosity kills the cat. Moral: don't be the cat.
Oct 25 2014
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Saturday, 25 October 2014 at 18:40:23 UTC, H. S. Teoh via 
Digitalmars-d-learn wrote:

 Actually, the object file (library) itself should already have 
 a list of
 exported symbols; you could then use core.demangle to extract 
 the
 function signatures from the mangled symbols and construct a 
 hash of all
 exported symbols and their types. The only thing is, I don't 
 know of any
 cross-platform method of retrieving the list of exported 
 symbols -- the
 Posix dlsym() family of functions only allow lookup by explicit 
 symbol
 name, no iteration primitives are specified. But the 
 information is
 definitely there, since that's how the OS's dynamic linker 
 figures out
 how to link dynamic libraries in the first place!

 T


I wonder what nm uses. AFAIK it works on everything posix.
Oct 25 2014
parent reply "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Sat, Oct 25, 2014 at 08:05:18PM +0000, John Colvin via Digitalmars-d-learn
wrote:

 On Saturday, 25 October 2014 at 18:40:23 UTC, H. S. Teoh via
 Digitalmars-d-learn wrote:

Actually, the object file (library) itself should already have a list
of exported symbols; you could then use core.demangle to extract the
function signatures from the mangled symbols and construct a hash of
all exported symbols and their types. The only thing is, I don't know
of any cross-platform method of retrieving the list of exported
symbols -- the Posix dlsym() family of functions only allow lookup by
explicit symbol name, no iteration primitives are specified. But the
information is definitely there, since that's how the OS's dynamic
linker figures out how to link dynamic libraries in the first place!

T


 
 I wonder what nm uses. AFAIK it works on everything posix.


Not sure what nm uses, but a lot of posix tools for manipulating object
files are based on binutils, which understands the local system's object
file format and deal directly with the binary representation. The
problem is, I don't know of any *standard* system functions that can do
this, so you'd have to rely on OS-specific stuff to make it work, which
is less than ideal.


T

-- 
Indifference will certainly be the downfall of mankind, but who cares? --
Miquel van Smoorenburg
Oct 25 2014
parent reply Etienne Cimon <etcimon gmail.com> writes:
On 2014-10-25 21:26, H. S. Teoh via Digitalmars-d-learn wrote:

 Not sure what nm uses, but a lot of posix tools for manipulating object
 files are based on binutils, which understands the local system's object
 file format and deal directly with the binary representation. The
 problem is, I don't know of any *standard* system functions that can do
 this, so you'd have to rely on OS-specific stuff to make it work, which
 is less than ideal.


Which makes it better to export the mangling into a container at 
compile-time! That way, you can build a standard interface into DLLs so 
that other D application know what they can call =)
Oct 25 2014
parent reply "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Sat, Oct 25, 2014 at 10:54:53PM -0400, Etienne Cimon via Digitalmars-d-learn
wrote:

 On 2014-10-25 21:26, H. S. Teoh via Digitalmars-d-learn wrote:

Not sure what nm uses, but a lot of posix tools for manipulating
object files are based on binutils, which understands the local
system's object file format and deal directly with the binary
representation. The problem is, I don't know of any *standard* system
functions that can do this, so you'd have to rely on OS-specific
stuff to make it work, which is less than ideal.


 
 Which makes it better to export the mangling into a container at
 compile-time! That way, you can build a standard interface into DLLs
 so that other D application know what they can call =)


Hmm. You can probably use __traits(getAllMembers...) to introspect a
library module at compile-time and build a hash based on that, so that
it's completely automated. If you have this available as a mixin, you
could just mixin(exportLibrarySymbols()) in your module to produce the
hash.


T

-- 
"Holy war is an oxymoron." -- Lazarus Long
Oct 25 2014
parent reply Etienne Cimon <etcimon gmail.com> writes:
On 2014-10-25 23:31, H. S. Teoh via Digitalmars-d-learn wrote:

 Hmm. You can probably use __traits(getAllMembers...) to introspect a
 library module at compile-time and build a hash based on that, so that
 it's completely automated. If you have this available as a mixin, you
 could just mixin(exportLibrarySymbols()) in your module to produce the
 hash.


Exactly, or I could also make it export specific functions into the 
hashmap, a little like a router. It seems like a very decent option.
Oct 26 2014
parent Etienne Cimon <etcimon gmail.com> writes:
On 2014-10-26 14:25, Etienne Cimon wrote:

 On 2014-10-25 23:31, H. S. Teoh via Digitalmars-d-learn wrote:

 Hmm. You can probably use __traits(getAllMembers...) to introspect a
 library module at compile-time and build a hash based on that, so that
 it's completely automated. If you have this available as a mixin, you
 could just mixin(exportLibrarySymbols()) in your module to produce the
 hash.


 Exactly, or I could also make it export specific functions into the
 hashmap, a little like a router. It seems like a very decent option.



I found an elegant solution for dealing with dynamic libraries:

https://github.com/bitwise-github/D-Reflection
Nov 06 2014