www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Dynamic Class Loading Idea

reply "Craig Black" <craigblack2 cox.net> writes:
Dynamic class loading has the requirement that the class library can be 
loaded at run time.  Traditionally, this means that all functions had to be 
bound via function pointers or virtual functions.  This would mean that all 
the local functions would have to be virtual, and static/global functions 
would need to be bound via function pointers somehow.  This is obviously a 
performance issue.

However, self modifying code could be employed to load a class at run time 
without using function pointers and virtual functions.  For each function in 
a dynamically loadable class, we could have an array of pointers that would 
point to the addresses in code where that function is called (call sites). 
Before the class library is loaded, these call sites would simply throw 
exceptions.  When the class is loaded, the code at the call site to throw 
the exception could be overwritten to call the appropriate function.  This 
would eliminate the requirement to use function pointers/virtual functions 
and allow us to do high-performance dynamic class loading.

I don't know if it is possible to modify functions that reside in the code 
segment.  Does the operating system allow this?  If not, we could place 
functions with these dynamic call sites on the heap.

Thoughts?

-Craig
Nov 02 2007
next sibling parent reply "Vladimir Panteleev" <thecybershadow gmail.com> writes:
On Sat, 03 Nov 2007 08:26:42 +0200, Craig Black <craigblack2 cox.net> wrote:


 Dynamic class loading has the requirement that the class library can be
 loaded at run time.  Traditionally, this means that all functions had to be
 bound via function pointers or virtual functions.  This would mean that all
 the local functions would have to be virtual, and static/global functions
 would need to be bound via function pointers somehow.  This is obviously a
 performance issue.

 However, self modifying code could be employed to load a class at run time
 without using function pointers and virtual functions.  For each function in
 a dynamically loadable class, we could have an array of pointers that would
 point to the addresses in code where that function is called (call sites).
 Before the class library is loaded, these call sites would simply throw
 exceptions.  When the class is loaded, the code at the call site to throw
 the exception could be overwritten to call the appropriate function.  This
 would eliminate the requirement to use function pointers/virtual functions
 and allow us to do high-performance dynamic class loading.

 I don't know if it is possible to modify functions that reside in the code
 segment.  Does the operating system allow this?  If not, we could place
 functions with these dynamic call sites on the heap.

 Thoughts?


I like DDL's approach, in which it acts as a run-time linker. It uses
relocation data in OBJ files to find address references and fix them (thus
"relocating" the code for any memory address) when loading the libraries. 

-- 
Best regards,
 Vladimir                          mailto:thecybershadow gmail.com
Nov 03 2007
parent reply "Craig Black" <craigblack2 cox.net> writes:
 I like DDL's approach, in which it acts as a run-time linker. It uses 
 relocation data in OBJ files to find address references and fix them (thus 
 "relocating" the code for any memory address) when loading the libraries.


Does this relocation use function pointers?
Nov 03 2007
next sibling parent reply "Vladimir Panteleev" <thecybershadow gmail.com> writes:
On Sat, 03 Nov 2007 15:40:34 +0200, Craig Black <craigblack2 cox.net> wrote:


 I like DDL's approach, in which it acts as a run-time linker. It uses
 relocation data in OBJ files to find address references and fix them (thus
 "relocating" the code for any memory address) when loading the libraries.


 Does this relocation use function pointers?


"Relocation data" is just a list of addresses which have pointers. I didn't
express myself correctly, relocation data isn't used to resolve cross-module
references. It is only used to "move" that code at a random address. To do
that, the linker/loader adds the address to which the code is relocated to each
DWORD the address of which is specified in the relocation list.

References are resolved by having a list of symbols, and each symbol having an
address of where in this module it's referenced. The linker/loader then simply
places the address of each symbol into the referenced locations.

Since this is done before any code in the module is executed, it should be
possible to implement on all platforms (except platforms where code and data is
strictly separated).

-- 
Best regards,
 Vladimir                          mailto:thecybershadow gmail.com
Nov 03 2007
parent reply "Craig Black" <craigblack2 cox.net> writes:
"Vladimir Panteleev" <thecybershadow gmail.com> wrote in message 
news:op.t0741htdm02fvl cybershadow...

 On Sat, 03 Nov 2007 15:40:34 +0200, Craig Black <craigblack2 cox.net> 
 wrote:


 I like DDL's approach, in which it acts as a run-time linker. It uses
 relocation data in OBJ files to find address references and fix them 
 (thus
 "relocating" the code for any memory address) when loading the 
 libraries.


 Does this relocation use function pointers?


 "Relocation data" is just a list of addresses which have pointers. I 
 didn't express myself correctly, relocation data isn't used to resolve 
 cross-module references. It is only used to "move" that code at a random 
 address. To do that, the linker/loader adds the address to which the code 
 is relocated to each DWORD the address of which is specified in the 
 relocation list.

 References are resolved by having a list of symbols, and each symbol 
 having an address of where in this module it's referenced. The 
 linker/loader then simply places the address of each symbol into the 
 referenced locations.

 Since this is done before any code in the module is executed, it should be 
 possible to implement on all platforms (except platforms where code and 
 data is strictly separated).

 -- 
 Best regards,
 Vladimir                          mailto:thecybershadow gmail.com


Thanks for the explanation.  So then modules can be loaded at run time, and 
this system effectively eliminates function pointers?
Nov 03 2007
parent "Vladimir Panteleev" <thecybershadow gmail.com> writes:
On Sat, 03 Nov 2007 18:28:13 +0200, Craig Black <craigblack2 cox.net> wrote:


 "Vladimir Panteleev" <thecybershadow gmail.com> wrote in message
 news:op.t0741htdm02fvl cybershadow...

 On Sat, 03 Nov 2007 15:40:34 +0200, Craig Black <craigblack2 cox.net>
 wrote:


 I like DDL's approach, in which it acts as a run-time linker. It uses
 relocation data in OBJ files to find address references and fix them
 (thus
 "relocating" the code for any memory address) when loading the
 libraries.


 Does this relocation use function pointers?


 "Relocation data" is just a list of addresses which have pointers. I
 didn't express myself correctly, relocation data isn't used to resolve
 cross-module references. It is only used to "move" that code at a random
 address. To do that, the linker/loader adds the address to which the code
 is relocated to each DWORD the address of which is specified in the
 relocation list.

 References are resolved by having a list of symbols, and each symbol
 having an address of where in this module it's referenced. The
 linker/loader then simply places the address of each symbol into the
 referenced locations.

 Since this is done before any code in the module is executed, it should be
 possible to implement on all platforms (except platforms where code and
 data is strictly separated).

 --
 Best regards,
 Vladimir                          mailto:thecybershadow gmail.com


 Thanks for the explanation.  So then modules can be loaded at run time, and
 this system effectively eliminates function pointers?


Yes, that's the goal and purpose of DDL.

-- 
Best regards,
 Vladimir                          mailto:thecybershadow gmail.com
Nov 03 2007
prev sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Craig Black wrote:

 I like DDL's approach, in which it acts as a run-time linker. It uses
 relocation data in OBJ files to find address references and fix them
 (thus "relocating" the code for any memory address) when loading the
 libraries.


 
 Does this relocation use function pointers?


Here's how I understand it.  DDL is a linker.  The only difference
between DDL and dmd/gcc is that it works at run time instead of compile
time.

What you described is basically what a linker does, hence what DDL does.

	-- Daniel
Nov 03 2007
parent reply "Craig Black" <craigblack2 cox.net> writes:
"Daniel Keep" <daniel.keep.lists gmail.com> wrote in message 
news:fgi777$na4$2 digitalmars.com...

 Craig Black wrote:

 I like DDL's approach, in which it acts as a run-time linker. It uses
 relocation data in OBJ files to find address references and fix them
 (thus "relocating" the code for any memory address) when loading the
 libraries.


 Does this relocation use function pointers?


 Here's how I understand it.  DDL is a linker.  The only difference
 between DDL and dmd/gcc is that it works at run time instead of compile
 time.

 What you described is basically what a linker does, hence what DDL does.

 -- Daniel


That's pretty cool then.  Since we have DDL, can we effectively say that we 
have dynamic class loading?  Or are features missing from DDL that make it 
not a full dynamic class loader?

-Craig
Nov 03 2007
parent reply "Vladimir Panteleev" <thecybershadow gmail.com> writes:
On Sat, 03 Nov 2007 20:47:58 +0200, Craig Black <craigblack2 cox.net> wrote:


 "Daniel Keep" <daniel.keep.lists gmail.com> wrote in message
 news:fgi777$na4$2 digitalmars.com...

 Craig Black wrote:

 I like DDL's approach, in which it acts as a run-time linker. It uses
 relocation data in OBJ files to find address references and fix them
 (thus "relocating" the code for any memory address) when loading the
 libraries.


 Does this relocation use function pointers?


 Here's how I understand it.  DDL is a linker.  The only difference
 between DDL and dmd/gcc is that it works at run time instead of compile
 time.

 What you described is basically what a linker does, hence what DDL does.

 -- Daniel


 That's pretty cool then.  Since we have DDL, can we effectively say that we
 have dynamic class loading?  Or are features missing from DDL that make it
 not a full dynamic class loader?


DDL is, at the moment, defunct when it comes to compatibility with newer
versions of the compiler - and it has been so for a while now. There have been
recent talks of someone getting their hands dirty and bringing it up-to-date.

-- 
Best regards,
 Vladimir                          mailto:thecybershadow gmail.com
Nov 03 2007
parent TomD <t_demmer nospam.web.de> writes:
Vladimir Panteleev Wrote:
[...]

 
 DDL is, at the moment, defunct when it comes to compatibility with newer
versions of the compiler - and it has been so for a while now. There have been
recent talks of someone getting their hands dirty and bringing it up-to-date.
 
 -- 
 Best regards,
  Vladimir                          mailto:thecybershadow gmail.com


Any chance to get this compatible to Phobos? As far as I could see,
the thing relies on (M/T)ango, at least for I/O (which should be quite easily
adjustable). Are there further complications involved?

Ciao
Tom
Nov 04 2007
prev sibling parent reply Bruce Adams <tortoise_74 yeah.who.co.uk> writes:
Daniel Keep Wrote:


 
 
 Craig Black wrote:

 I like DDL's approach, in which it acts as a run-time linker. It uses
 relocation data in OBJ files to find address references and fix them
 (thus "relocating" the code for any memory address) when loading the
 libraries.


 
 Does this relocation use function pointers?


 
 Here's how I understand it.  DDL is a linker.  The only difference
 between DDL and dmd/gcc is that it works at run time instead of compile
 time.
 
 What you described is basically what a linker does, hence what DDL does.
 
 	-- Daniel


Its also what an operating does to load dynamic link libraries or shared
objects.
Nov 03 2007
parent reply "Craig Black" <craigblack2 cox.net> writes:
"Bruce Adams" <tortoise_74 yeah.who.co.uk> wrote in message 
news:fgicuv$11j1$1 digitalmars.com...

 Daniel Keep Wrote:


 Craig Black wrote:

 I like DDL's approach, in which it acts as a run-time linker. It uses
 relocation data in OBJ files to find address references and fix them
 (thus "relocating" the code for any memory address) when loading the
 libraries.


 Does this relocation use function pointers?


 Here's how I understand it.  DDL is a linker.  The only difference
 between DDL and dmd/gcc is that it works at run time instead of compile
 time.

 What you described is basically what a linker does, hence what DDL does.

 -- Daniel


 Its also what an operating does to load dynamic link libraries or shared 
 objects.


Right, but the operating system does not do this when loading dll's or so's 
at run-time.
Nov 03 2007
parent reply Bruce Adams <tortoise_74 yeah.who.co.uk> writes:
Craig Black Wrote:


 
 "Bruce Adams" <tortoise_74 yeah.who.co.uk> wrote in message 
 news:fgicuv$11j1$1 digitalmars.com...

 Daniel Keep Wrote:


 Craig Black wrote:

 I like DDL's approach, in which it acts as a run-time linker. It uses
 relocation data in OBJ files to find address references and fix them
 (thus "relocating" the code for any memory address) when loading the
 libraries.


 Does this relocation use function pointers?


 Here's how I understand it.  DDL is a linker.  The only difference
 between DDL and dmd/gcc is that it works at run time instead of compile
 time.

 What you described is basically what a linker does, hence what DDL does.

 -- Daniel


 Its also what an operating does to load dynamic link libraries or shared 
 objects.


 
 Right, but the operating system does not do this when loading dll's or so's 
 at run-time. 
 


It depends on the operating system but generally yes it does. On some systems
libraries are only loaded when they are first needed. On others they are loaded
whent the program starts. And then there's usually a runtime API of some sort.
LoadLibrary and GetProcAddress on windows. Those only half count because their
too low level.

I don't know much about DDL is it OS specific or portable?
Nov 04 2007
parent "Craig Black" <cblack ara.com> writes:
"Bruce Adams" <tortoise_74 yeah.who.co.uk> wrote in message 
news:fgk1n1$et3$1 digitalmars.com...

 Craig Black Wrote:


 "Bruce Adams" <tortoise_74 yeah.who.co.uk> wrote in message
 news:fgicuv$11j1$1 digitalmars.com...

 Daniel Keep Wrote:


 Craig Black wrote:

 I like DDL's approach, in which it acts as a run-time linker. It 
 uses
 relocation data in OBJ files to find address references and fix 
 them
 (thus "relocating" the code for any memory address) when loading 
 the
 libraries.


 Does this relocation use function pointers?


 Here's how I understand it.  DDL is a linker.  The only difference
 between DDL and dmd/gcc is that it works at run time instead of 
 compile
 time.

 What you described is basically what a linker does, hence what DDL 
 does.

 -- Daniel


 Its also what an operating does to load dynamic link libraries or 
 shared
 objects.


 Right, but the operating system does not do this when loading dll's or 
 so's
 at run-time.


 It depends on the operating system but generally yes it does. On some 
 systems libraries are only loaded when they are first needed. On others 
 they are loaded whent the program starts. And then there's usually a 
 runtime API of some sort. LoadLibrary and GetProcAddress on windows. Those 
 only half count because their too low level.

 I don't know much about DDL is it OS specific or portable?


I think you are talking about delay loaded libraries.  I'm talking about 
libraries that are loaded as plug-ins via LoadLibrary.  In this case linking 
is done with something like GetProcAddress which provides a function 
pointer.
Nov 05 2007