www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Enhancing DLL/SO behavior with D

reply pragma <pragma_member pathlink.com> writes:
To make an already long story short, I'm looking for a way to dynamically
resolve symbols between dll/so files and their host executable.  This way, one
could unify the TypeInfo tree at runtime (and more) to achive a sort of
runtime-binding similar to what Java does with .class files.*

With portability in mind, I went about researching how to perform a dll detour
(also called a trampoline) in linux using a Shared Object.  

The general idea, is to perform a fixup on the dll after its loaded into memory,
as to circumvent the original call hook for the library.  I know that ELF shared
objects can have external symbols resolved via the ELF loader itself (which
searches for matching symbols provided by in-memory tables); which is kind of
backwards, but may be workable.

With that in mind, it left me with the following questions:

- Is there an ELF, COFF or OMF loader workable under windows out there in F/OSS
land?
- Is there any way to generate an ELF shared object under windows, as an analog
to a dll file (possibly via a coff to elf converter)?
- ELF loading is grand, but isn't there a guide to hacking away at the shared
object after its been loaded into memory (like what we do in windows with dll
detours)?
- Would anyone like more detail or background on the problem and why custom
library loading is important?

(* Java .class files contain a symbol table of what external symbols are needed
to run the class, and what symbols are available to the calling program.  The
exhange of this information, or binding is performed by the classloader at
runtime when the .class file is loaded.  This is in stark contrast to D or C++
where type information is held statically in a dll/so or executable and is not
shared with anything at runtime.)

Thanks in advance,

- EricAnderton at yahoo
Aug 09 2005
next sibling parent reply "Ben Hinkle" <ben.hinkle gmail.com> writes:
"pragma" <pragma_member pathlink.com> wrote in message 
news:ddbn9q$143n$1 digitaldaemon.com...

 To make an already long story short, I'm looking for a way to dynamically
 resolve symbols between dll/so files and their host executable.  This way, 
 one
 could unify the TypeInfo tree at runtime (and more) to achive a sort of
 runtime-binding similar to what Java does with .class files.*


I was playing around with similar goals with dflect (I think... I'm not 
exactly sure what your goals are but I'm guessing). The work-flow was as 
follows: run dflect on the classes/modules you wish to expose (this requires 
some special coding practices like marking ctors as "export") to generate 
MetaInfo code. Compile the MetaInfo classes together with your actual 
classes and make the dll. In the main app load the dll and call the exported 
function getMetaClassCtors() in the dll which returns an AA indexed by class 
name and with value the ctor for the MetaInfo class for the class in 
question. Once you construct the MetaClass you can call it's methods to 
construct instances and get fields and methods etc etc. The AA mapping class 
names to ctors is initialized in the module ctors so the main app needs to 
be able to run the module ctors before getting the AA of MetaInfo.

Dflect also has files for interfacing with C/C++ - like the ability to get a 
global ref to an object so that it doesn't get garbage collected while some 
C code owns the only reference.

Dflect is on the back-burner now, though. I was hoping someone would redo 
std.loader first.


 With portability in mind, I went about researching how to perform a dll 
 detour
 (also called a trampoline) in linux using a Shared Object.

 The general idea, is to perform a fixup on the dll after its loaded into 
 memory,
 as to circumvent the original call hook for the library.  I know that ELF 
 shared
 objects can have external symbols resolved via the ELF loader itself 
 (which
 searches for matching symbols provided by in-memory tables); which is kind 
 of
 backwards, but may be workable.

 With that in mind, it left me with the following questions:

 - Is there an ELF, COFF or OMF loader workable under windows out there in 
 F/OSS
 land?
 - Is there any way to generate an ELF shared object under windows, as an 
 analog
 to a dll file (possibly via a coff to elf converter)?
 - ELF loading is grand, but isn't there a guide to hacking away at the 
 shared
 object after its been loaded into memory (like what we do in windows with 
 dll
 detours)?
 - Would anyone like more detail or background on the problem and why 
 custom
 library loading is important?

 (* Java .class files contain a symbol table of what external symbols are 
 needed
 to run the class, and what symbols are available to the calling program. 
 The
 exhange of this information, or binding is performed by the classloader at
 runtime when the .class file is loaded.  This is in stark contrast to D or 
 C++
 where type information is held statically in a dll/so or executable and is 
 not
 shared with anything at runtime.)


I don't know anything about trampolines so I can't help but it sounds 
interesting. What exactly would the trampoline do? You mention unify 
TypeInfos - is this so that, for example, int's TypeInfo is the same address 
in all the dlls and main executable? Can this stuff be built into 
std.loader?


 Thanks in advance,

 - EricAnderton at yahoo
Aug 09 2005
parent pragma <pragma_member pathlink.com> writes:
In article <ddbrve$1bkm$1 digitaldaemon.com>, Ben Hinkle says...

"pragma" <pragma_member pathlink.com> wrote in message 
news:ddbn9q$143n$1 digitaldaemon.com...

 To make an already long story short, I'm looking for a way to dynamically
 resolve symbols between dll/so files and their host executable.  This way, 
 one
 could unify the TypeInfo tree at runtime (and more) to achive a sort of
 runtime-binding similar to what Java does with .class files.*


I was playing around with similar goals with dflect (I think... I'm not 
exactly sure what your goals are but I'm guessing). The work-flow was as 
follows: run dflect on the classes/modules you wish to expose (this requires 
some special coding practices like marking ctors as "export") to generate 
MetaInfo code. Compile the MetaInfo classes together with your actual 
classes and make the dll. In the main app load the dll and call the exported 
function getMetaClassCtors() in the dll which returns an AA indexed by class 
name and with value the ctor for the MetaInfo class for the class in 
question. Once you construct the MetaClass you can call it's methods to 
construct instances and get fields and methods etc etc. The AA mapping class 
names to ctors is initialized in the module ctors so the main app needs to 
be able to run the module ctors before getting the AA of MetaInfo.

Dflect also has files for interfacing with C/C++ - like the ability to get a 
global ref to an object so that it doesn't get garbage collected while some 
C code owns the only reference.

Dflect is on the back-burner now, though. I was hoping someone would redo 
std.loader first.


Excellent.  That sounds like a worthwhile goal.  I've been very keen on deflect
since you first introduced it, and I'll be eager to see where this goes.



 <snip my discussion about trampolines and ELF loaders>


I don't know anything about trampolines so I can't help but it sounds 
interesting. What exactly would the trampoline do? 


Basically the concept is to load the dll into memory, treat it as data, rewrite
portions of it and then treat it as you normally would.  A trampoline, also
known as a "detour" is explained here:

http://research.microsoft.com/~galenh/Publications/HuntUsenixNt99.pdf


You mention unify 
TypeInfos - is this so that, for example, int's TypeInfo is the same address 
in all the dlls and main executable? Can this stuff be built into 
std.loader?


You hit the nail right on the head.  Although I'd like to see it extended to all
TypeInfo classes, standard and custom.  In a perfect world, yes, it'd be written
directly into std.loader or something similar.

Such a custom loader would take advantage of the concepts outlined by
Detours/Trampolines, so that it may overwrite pointers in the in-memory-dll so
that they point to the 'correct' values.  The catch is: how do we coerce a dll
to create such a table of all in-code references to TypeInfo structures?

Its either that, or somehow rewrite DMD to output code that changes indirect
references to TypeInfo (say in typeid() and cast() calls) to instead call
"getTypeinfo(foo)", which would then use a lookup table.  The table could be
overwritten on load to the correct set of values.

FYI, I'm curious about custom loading COFF or OMF directly, simply to bypass all
the cruft that win32 places on dlls in general.  It be nice since these formats
(by definition) already have all of their external references exposed for
linking;  we'd just be doing it at run-time instead of at compile-time.  As a
side-effect, it would do more than helping the current TypeInfo problem as it
would naturally extend to every kind of symbol.  (AFAIK, this is like emulating
ELF loading under win32) 

- EricAnderton at yahoo
Aug 10 2005
prev sibling parent reply "Carlos Smith" <c_____.s____ sympatico.ca> writes:
"pragma" <pragma_member pathlink.com> wrote in message


 To make an already long story short, I'm looking for a way to dynamically
 resolve symbols between dll/so files and their host executable.


http://edll.sourceforge.net/index.html
Aug 10 2005
parent reply pragma <pragma_member pathlink.com> writes:
In article <ddcpuo$2m16$1 digitaldaemon.com>, Carlos Smith says...

"pragma" <pragma_member pathlink.com> wrote in message


 To make an already long story short, I'm looking for a way to dynamically
 resolve symbols between dll/so files and their host executable.


http://edll.sourceforge.net/index.html


Carlos, you are my hero.  That's one heck of a starting point, even if it
depends on the GCC/MinGW toolchain.

- EricAnderton at yahoo
Aug 10 2005
parent reply "Carlos Smith" <c_____.s____ sympatico.ca> writes:
"pragma" <pragma_member pathlink.com> wrote in message


 Carlos, you are my hero.  That's one heck of a starting point, even if it
 depends on the GCC/MinGW toolchain.


for( i=1.0 ; i > 0 ; i += 1.0 )
    LOL();

The greatest things about Free/OpenSource software, are:

It promote/encourage collaboration instead of competition,

It's "Un pour Tous, Tous pour Un", instead of "We will beat
them all",

It's the realisation of "L'Union fait la Force",

It's a great way to make friends,

And, when Bill Gates will try this, humanity will enter
a new world...
Aug 10 2005
parent pragma <pragma_member pathlink.com> writes:
In article <ddd5h5$453$1 digitaldaemon.com>, Carlos Smith says...

"pragma" <pragma_member pathlink.com> wrote in message


 Carlos, you are my hero.  That's one heck of a starting point, even if it
 depends on the GCC/MinGW toolchain.


for( i=1.0 ; i > 0 ; i += 1.0 )
    LOL();


You're welcome.


The greatest things about Free/OpenSource software, are:

It promote/encourage collaboration instead of competition,

It's "Un pour Tous, Tous pour Un", instead of "We will beat
them all",

It's the realisation of "L'Union fait la Force",

It's a great way to make friends,

And, when Bill Gates will try this, humanity will enter
a new world...


I couldn't have said it better myself.  

In my experience, when I can't just google for answers it means that either it's
right under my nose or its rediculously obscure and possibly offline someplace.


That's why I posted the question here since my research was going nowhere fast.
If I could work with someone, drum up interest in a new project or add to an
existing project in the process: all the better.

Also, I'm long overdue for posting this to the DNG as I've been writing about it
on my DSP project board for a while now. 

Anyway edll will give me a much needed foothold: it uses BFD (object file loader
library for linux) under windows.  I'm sure that it can be ported to DMC/DMD,
but it lacks OMF support.  Otherwise, I'll have to find a way to convert OMF to
COFF or OMF to ELF.

So, is anyone out there willing to donate an OMF loader before I try to write
one? ;)

- EricAnderton at yahoo
Aug 10 2005