www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - shared libraries in D

reply Iain Buclaw <ibuclaw ubuntu.com> writes:
So I've been prototyping, and have built a fully working shared D2
druntime/phobos library on Linux (will come to caveats in a moment). Just for
sake of visual proof.

[iain natty gdc]$ cat hello.d
import std.stdio;
void main()
{
    writeln("Hello World");
}

[iain natty gdc]$ du -hs a.out
8.0K	a.out
[iain natty gdc]$ ldd a.out
	linux-gate.so.1 =>  (0x00a32000)
	librt.so.1 => /lib/librt.so.1 (0x00c32000)
	libdl.so.2 => /lib/libdl.so.2 (0x00914000)
	libgphobos2 => /usr/local/lib/libgphobos2 (0x0050d000)
	libgdruntime => /usr/local/lib/libgdruntime (0x00a90000)
	libm.so.6 => /lib/libm.so.6 (0x00921000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00e8f000)
	libpthread.so.0 => /lib/libpthread.so.0 (0x007b9000)
	libc.so.6 => /lib/libc.so.6 (0x00110000)
	/lib/ld-linux.so.2 (0x00f07000)



Now, what's the catch?

The curent catch is that *none* of the external static ctors/dtors are called,
because the linked module list (_Dmodule_ref) is generated before the program
enters C main, thus before the shared druntime and phobos libraries have been
loaded into memory (don't know if this is the same for DMD, but I'll assume
that will likely be the case too).

Breakpoint 1, hello.__modinit() () at hello.d:1
(gdb) bt








Each modules own __modinit() is written out as such:

  struct _modref_t {
      _modref_t  * next;
      ModuleInfo m;
  }
  extern (C) _modref_t * _Dmodule_ref;
  private _modref_t our_mod_ref =
    { next: null, m: _ModuleInfo_xxx };
  void ___modinit() { // a static constructor
     our_mod_ref.next = _Dmodule_ref;
     _Dmodule_ref = & our_mod_ref;
  }

These functions are created by the compiler, and inserted into the .ctor list.

Before I progress, posting to ask if anyone has any good implementation ideas
to get this fully functional?

Regards
Iain
Feb 14 2011
parent reply Johannes Pfau <spam example.com> writes:
Iain Buclaw wrote:

So I've been prototyping, and have built a fully working shared D2
druntime/phobos library on Linux (will come to caveats in a moment).
Just for sake of visual proof.


Awesome!


Before I progress, posting to ask if anyone has any good
implementation ideas to get this fully functional?

Regards
Iain


I think dmd/druntime support shared libraries on MacOS. I'm not sure
though whether that also includes using a shared  druntime/phobos.
Probably that code could help (or maybe the MacOS code is too
different, then just ignore this comment ;-))
--=20
Johannes Pfau
Feb 14 2011
parent reply Iain Buclaw <ibuclaw ubuntu.com> writes:
== Quote from Johannes Pfau (spam example.com)'s article

 --Sig_/a6ST_/ke_QlFF5lFi4BGNvw
 Content-Type: text/plain; charset=US-ASCII
 Content-Transfer-Encoding: quoted-printable
 Iain Buclaw wrote:

So I've been prototyping, and have built a fully working shared D2
druntime/phobos library on Linux (will come to caveats in a moment).
Just for sake of visual proof.


 Awesome!

Before I progress, posting to ask if anyone has any good
implementation ideas to get this fully functional?

Regards
Iain


 I think dmd/druntime support shared libraries on MacOS. I'm not sure
 though whether that also includes using a shared  druntime/phobos.
 Probably that code could help (or maybe the MacOS code is too
 different, then just ignore this comment ;-))
 --=20
 Johannes Pfau


The support for OSX uses an entirely different, and questionable implementation.

Regards
Feb 14 2011
next sibling parent reply Iain Buclaw <ibuclaw ubuntu.com> writes:
== Quote from Iain Buclaw (ibuclaw ubuntu.com)'s article

 == Quote from Johannes Pfau (spam example.com)'s article

 --Sig_/a6ST_/ke_QlFF5lFi4BGNvw
 Content-Type: text/plain; charset=US-ASCII
 Content-Transfer-Encoding: quoted-printable
 Iain Buclaw wrote:

So I've been prototyping, and have built a fully working shared D2
druntime/phobos library on Linux (will come to caveats in a moment).
Just for sake of visual proof.


 Awesome!

Before I progress, posting to ask if anyone has any good
implementation ideas to get this fully functional?

Regards
Iain


 I think dmd/druntime support shared libraries on MacOS. I'm not sure
 though whether that also includes using a shared  druntime/phobos.
 Probably that code could help (or maybe the MacOS code is too
 different, then just ignore this comment ;-))
 --=20
 Johannes Pfau


 The support for OSX uses an entirely different, and questionable
implementation.
 Regards


Came across this obscure documentation in the tldp.

"Libraries should export initialization and cleanup routines using the gcc
__attribute__((constructor)) and __attribute__((destructor)) function
attributes."

This is what gdc was doing anyway.


"Constructor routines are executed before dlopen returns (or before main() is
started if the library is loaded at load time). Destructor routines are executed
before dlclose returns (or after exit() or completion of main() if the library
is
loaded at load time)."

This is what should have been happening, but wasn't earlier because:


"Shared libraries must not be compiled with the gcc arguments `-nostartfiles' or
`-nostdlib'. If those arguments are used, the constructor/destructor routines
will
not be executed (unless special measures are taken)."

Whoops! Fixed and it now works. :~)

Will be making shared libraries default in GDC pretty soon now...
Feb 14 2011
next sibling parent reply Christian Kamm <kamm-incasoftware removethis.de> writes:
Iain Buclaw wrote:

 Will be making shared libraries default in GDC pretty soon now...


Did you adjust the GC to check the shared libraries' data sections for 
references? When we looked at this for LDC that turned out to slow down GC 
runs significantly.

I'm pretty sure bearophile benchmarked it at the time.

As far as I remember the main problem was that we were scanning all data 
sections - even of plain C libraries.

Regards,
Christian
Feb 14 2011
parent reply Iain Buclaw <ibuclaw ubuntu.com> writes:
== Quote from Christian Kamm (kamm-incasoftware removethis.de)'s article

 Iain Buclaw wrote:

 Will be making shared libraries default in GDC pretty soon now...


 Did you adjust the GC to check the shared libraries' data sections for
 references? When we looked at this for LDC that turned out to slow down GC
 runs significantly.
 I'm pretty sure bearophile benchmarked it at the time.
 As far as I remember the main problem was that we were scanning all data
 sections - even of plain C libraries.
 Regards,
 Christian


When adding more ranges in the GC to check, slowdowns are inevitable. So far in
my
personal testing, the slowdowns seem pretty negligible in comparison (never more
than 0.300 seconds, though that could be a sign that I'm not pushing it in the
right way).

To prevent the GC from scanning C libraries, I've added an extra check to only
add
ranges that have a D module inside them.

Think:
  ModuleReference *mr;
  for ( mr = _Dmodule_ref; mr; mr = mr.next )
  {
      if ( mr >= start && mr <= end )
      {
          addrange = 1;
          break;
      }
  }

This is viable for GDC, as all platforms share the same implementation. Note,
this
also adds a variable overhead at startup, but I think the unskewed runtime
justifies it to remain.

Regards
Feb 15 2011
parent reply Christian Kamm <kamm-incasoftware removethis.de> writes:
Iain Buclaw wrote:


 == Quote from Christian Kamm (kamm-incasoftware removethis.de)'s article

 Iain Buclaw wrote:

 Will be making shared libraries default in GDC pretty soon now...


 Did you adjust the GC to check the shared libraries' data sections for
 references? When we looked at this for LDC that turned out to slow down
 GC runs significantly.
 I'm pretty sure bearophile benchmarked it at the time.
 As far as I remember the main problem was that we were scanning all data
 sections - even of plain C libraries.
 Regards,
 Christian


 
 When adding more ranges in the GC to check, slowdowns are inevitable. So
 far in my personal testing, the slowdowns seem pretty negligible in
 comparison (never more than 0.300 seconds, though that could be a sign
 that I'm not pushing it in the right way).
 
 To prevent the GC from scanning C libraries, I've added an extra check to
 only add ranges that have a D module inside them.


That sounds good. I was asking because I didn't see any code like this in  
http://bitbucket.org/goshawk/gdc - which repository is this in?

Christian
Feb 15 2011
parent reply Iain Buclaw <ibuclaw ubuntu.com> writes:
== Quote from Christian Kamm (kamm-incasoftware removethis.de)'s article

 Iain Buclaw wrote:

 == Quote from Christian Kamm (kamm-incasoftware removethis.de)'s article

 Iain Buclaw wrote:

 Will be making shared libraries default in GDC pretty soon now...


 Did you adjust the GC to check the shared libraries' data sections for
 references? When we looked at this for LDC that turned out to slow down
 GC runs significantly.
 I'm pretty sure bearophile benchmarked it at the time.
 As far as I remember the main problem was that we were scanning all data
 sections - even of plain C libraries.
 Regards,
 Christian


 When adding more ranges in the GC to check, slowdowns are inevitable. So
 far in my personal testing, the slowdowns seem pretty negligible in
 comparison (never more than 0.300 seconds, though that could be a sign
 that I'm not pushing it in the right way).

 To prevent the GC from scanning C libraries, I've added an extra check to
 only add ranges that have a D module inside them.


 That sounds good. I was asking because I didn't see any code like this in
 http://bitbucket.org/goshawk/gdc - which repository is this in?
 Christian


It's currently on my local workstation, all mashed together with the D2 beta
merge. I have faith that my workstation won't crash and die in the meantime
<touchwood> and will sift through it all (and break it down into sizeable
commits)
once I'm satisfied that there's nothing more I can do.

Regards
Iain
Feb 16 2011
parent Iain Buclaw <ibuclaw ubuntu.com> writes:
== Quote from Iain Buclaw (ibuclaw ubuntu.com)'s article

 == Quote from Christian Kamm (kamm-incasoftware removethis.de)'s article

 Iain Buclaw wrote:

 == Quote from Christian Kamm (kamm-incasoftware removethis.de)'s article

 Iain Buclaw wrote:

 Will be making shared libraries default in GDC pretty soon now...


 Did you adjust the GC to check the shared libraries' data sections for
 references? When we looked at this for LDC that turned out to slow down
 GC runs significantly.
 I'm pretty sure bearophile benchmarked it at the time.
 As far as I remember the main problem was that we were scanning all data
 sections - even of plain C libraries.
 Regards,
 Christian


 When adding more ranges in the GC to check, slowdowns are inevitable. So
 far in my personal testing, the slowdowns seem pretty negligible in
 comparison (never more than 0.300 seconds, though that could be a sign
 that I'm not pushing it in the right way).

 To prevent the GC from scanning C libraries, I've added an extra check to
 only add ranges that have a D module inside them.


 That sounds good. I was asking because I didn't see any code like this in
 http://bitbucket.org/goshawk/gdc - which repository is this in?
 Christian


 It's currently on my local workstation, all mashed together with the D2 beta
 merge. I have faith that my workstation won't crash and die in the meantime
 <touchwood> and will sift through it all (and break it down into sizeable
commits)
 once I'm satisfied that there's nothing more I can do.
 Regards
 Iain


Module/Routine is now up here:
https://bitbucket.org/goshawk/gdc/src/a402fef51a6a/d/druntime/rt/gccmemory.d
Feb 19 2011
prev sibling parent reply Johannes Pfau <spam example.com> writes:
Iain Buclaw wrote:

Came across this obscure documentation in the tldp.

"Libraries should export initialization and cleanup routines using the
gcc __attribute__((constructor)) and __attribute__((destructor))
function attributes."

This is what gdc was doing anyway.


"Constructor routines are executed before dlopen returns (or before
main() is started if the library is loaded at load time). Destructor
routines are executed before dlclose returns (or after exit() or
completion of main() if the library is loaded at load time)."

This is what should have been happening, but wasn't earlier because:


"Shared libraries must not be compiled with the gcc arguments
`-nostartfiles' or `-nostdlib'. If those arguments are used, the
constructor/destructor routines will not be executed (unless special
measures are taken)."

Whoops! Fixed and it now works. :~)

Will be making shared libraries default in GDC pretty soon now...


Great, now I've got 2 questions:

1.) Are shared libraries compiled with gdc abi compitible with dmd?
(if dmd did support shared libraries)

2.) Is it now possible to load a shared d library into a C program? Will
jut loading the library initialize the runtime or is it necessary to
call some runtime setup functions manually? If the C program loads 2 D
libraries, will those share the gc/runtime?

--=20
Johannes Pfau
Feb 15 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-02-15 16:02, Johannes Pfau wrote:

 Iain Buclaw wrote:

 Came across this obscure documentation in the tldp.

 "Libraries should export initialization and cleanup routines using the
 gcc __attribute__((constructor)) and __attribute__((destructor))
 function attributes."

 This is what gdc was doing anyway.


 "Constructor routines are executed before dlopen returns (or before
 main() is started if the library is loaded at load time). Destructor
 routines are executed before dlclose returns (or after exit() or
 completion of main() if the library is loaded at load time)."

 This is what should have been happening, but wasn't earlier because:


 "Shared libraries must not be compiled with the gcc arguments
 `-nostartfiles' or `-nostdlib'. If those arguments are used, the
 constructor/destructor routines will not be executed (unless special
 measures are taken)."

 Whoops! Fixed and it now works. :~)

 Will be making shared libraries default in GDC pretty soon now...


 Great, now I've got 2 questions:

 1.) Are shared libraries compiled with gdc abi compitible with dmd?
 (if dmd did support shared libraries)


I'm not completely sure but the runtimes are different.


 2.) Is it now possible to load a shared d library into a C program? Will
 jut loading the library initialize the runtime or is it necessary to
 call some runtime setup functions manually? If the C program loads 2 D
 libraries, will those share the gc/runtime?


I don't know how Iain has implemented this but it should be easy to do. 
Just add, to the runtime, a C function with __attribute__((constructor)) 
that initializes the runtime.

-- 
/Jacob Carlborg
Feb 15 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-02-14 16:29, Iain Buclaw wrote:

 == Quote from Johannes Pfau (spam example.com)'s article

 --Sig_/a6ST_/ke_QlFF5lFi4BGNvw
 Content-Type: text/plain; charset=US-ASCII
 Content-Transfer-Encoding: quoted-printable
 Iain Buclaw wrote:

 So I've been prototyping, and have built a fully working shared D2
 druntime/phobos library on Linux (will come to caveats in a moment).
 Just for sake of visual proof.


 Awesome!

 Before I progress, posting to ask if anyone has any good
 implementation ideas to get this fully functional?

 Regards
 Iain


 I think dmd/druntime support shared libraries on MacOS. I'm not sure
 though whether that also includes using a shared  druntime/phobos.
 Probably that code could help (or maybe the MacOS code is too
 different, then just ignore this comment ;-))
 --=20
 Johannes Pfau


 The support for OSX uses an entirely different, and questionable
implementation.

 Regards


If you know any other way of implementing it I very open for suggestions.

-- 
/Jacob Carlborg
Feb 14 2011