www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - dynamic library building and loading

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Haven't done any dynamic linking with D and I need to. I'm using dmd 
2.058/Linux at work to build and use dynamic libraries. Here's my attempt:

*** Makefile
all: main lib.so

main: main.d
	dmd main

lib.so: lib.d
	dmd -fPIC -shared lib.d -of./lib.so

*** lib.d
extern(C) int fun(string s)
{
     return 42;
}

*** main.d
import std.stdio;
void main()
{
}

Running make prints:

dmd -fPIC -shared lib.d -of./lib.so
/usr/bin/ld: 
/mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/li
phobos2.a(minfo.o): 
relocation R_X86_64_32 against `a local symbol' can not be used when 
making a shared object; recompile with -fPIC
/mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/rele
se/64/libphobos2.a: 
could not read symbols: Bad value
collect2: ld returned 1 exit status

What steps do I need to take to get off the ground?


Thanks,

Andrei
Sep 26 2012
next sibling parent reply Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
26.09.2012 21:58, Andrei Alexandrescu пишет:

 Haven't done any dynamic linking with D and I need to. I'm using dmd
 2.058/Linux at work to build and use dynamic libraries.


Sorry for the OT (as you are a Linux user), but if you will ever do it 
on Windows, be aware of this nasty trap:
http://d.puremagic.com/issues/show_bug.cgi?id=8130

-- 
Денис В. Шеломовский
Denis V. Shelomovskij
Sep 26 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/12 2:25 PM, Denis Shelomovskij wrote:

 26.09.2012 21:58, Andrei Alexandrescu пишет:

 Haven't done any dynamic linking with D and I need to. I'm using dmd
 2.058/Linux at work to build and use dynamic libraries.


 Sorry for the OT (as you are a Linux user), but if you will ever do it
 on Windows, be aware of this nasty trap:
 http://d.puremagic.com/issues/show_bug.cgi?id=8130


Thanks!

Andrei
Sep 26 2012
prev sibling next sibling parent reply "Michael" <pr m1xa.com> writes:
On Wednesday, 26 September 2012 at 17:57:29 UTC, Andrei
Alexandrescu wrote:

 Haven't done any dynamic linking with D and I need to. I'm 
 using dmd 2.058/Linux at work to build and use dynamic 
 libraries. Here's my attempt:


Maybe it will help you
D:
https://bitbucket.org/AnyCPU/codewithd/raw/de0a2f0ea1af/symload.d

Fortran:
https://bitbucket.org/AnyCPU/codewithd/raw/de0a2f0ea1af/simple.f95

Dmd 2.060 used.

DLL on OpenSUSE 64bit compiled with command:
gfortran -shared -fPIC -O3 -o simple.so simple.f95
Sep 26 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/12 2:59 PM, Michael wrote:

 On Wednesday, 26 September 2012 at 17:57:29 UTC, Andrei
 Alexandrescu wrote:

 Haven't done any dynamic linking with D and I need to. I'm using dmd
 2.058/Linux at work to build and use dynamic libraries. Here's my
 attempt:


 Maybe it will help you
 D:
 https://bitbucket.org/AnyCPU/codewithd/raw/de0a2f0ea1af/symload.d

 Fortran:
 https://bitbucket.org/AnyCPU/codewithd/raw/de0a2f0ea1af/simple.f95

 Dmd 2.060 used.

 DLL on OpenSUSE 64bit compiled with command:
 gfortran -shared -fPIC -O3 -o simple.so simple.f95


Thanks. The loading part is very useful, but I'm still lost when it 
comes to build the shared library itself.

Andrei
Sep 26 2012
parent reply "Michael" <pr m1xa.com> writes:
 Thanks. The loading part is very useful, but I'm still lost 
 when it comes to build the shared library itself.

 Andrei


Program loads dll at runtime using loader which is configured to 
load concrete dll file(s). Like in gtkD 
http://www.dsource.org/projects/gtkd/browser/trunk/src/gtkc/Loader.d
Sep 26 2012
parent reply "nazriel" <spam dzfl.pl> writes:
On Wednesday, 26 September 2012 at 20:10:47 UTC, Michael wrote:

 Thanks. The loading part is very useful, but I'm still lost 
 when it comes to build the shared library itself.

 Andrei


 Program loads dll at runtime using loader which is configured 
 to load concrete dll file(s). Like in gtkD 
 http://www.dsource.org/projects/gtkd/browser/trunk/src/gtkc/Loader.d


Loading Shared lib isn't big issues here.

The bigger one is building Shared library (written in D) and 
running it in host application without issues (EH, shared GC etc).

Andrei, if you find out how to make those things work, please 
share your findings. I'm also in need of using shared libraries.

And yeah, probably Martin Nowak will be the best bet to get 
information from.
Sep 26 2012
next sibling parent "Michael" <pr m1xa.com> writes:
 Loading Shared lib isn't big issues here.

 The bigger one is building Shared library (written in D) and 
 running it in host application without issues (EH, shared GC 
 etc).

 Andrei, if you find out how to make those things work, please 
 share your findings. I'm also in need of using shared libraries.

 And yeah, probably Martin Nowak will be the best bet to get 
 information from.


I understand. As I remember on forum were topic with same 
approach for dll written in D. But I can't find a link.
Sep 26 2012
prev sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Wednesday, 26 September 2012 at 20:15:35 UTC, nazriel wrote:

 On Wednesday, 26 September 2012 at 20:10:47 UTC, Michael wrote:

 Thanks. The loading part is very useful, but I'm still lost 
 when it comes to build the shared library itself.

 Andrei


 Program loads dll at runtime using loader which is configured 
 to load concrete dll file(s). Like in gtkD 
 http://www.dsource.org/projects/gtkd/browser/trunk/src/gtkc/Loader.d


 Loading Shared lib isn't big issues here.

 The bigger one is building Shared library (written in D) and 
 running it in host application without issues (EH, shared GC 
 etc).

 Andrei, if you find out how to make those things work, please 
 share your findings. I'm also in need of using shared libraries.

 And yeah, probably Martin Nowak will be the best bet to get 
 information from.


Native Oberon/BlueBottle OS really a lot in dynamic modules, as 
everything
is dynamically loaded. As far as I know both operating systems 
have a global GC.

Not sure how Spin does it.

Singularity, uses local GC per process and does not support 
dynamic loading.

Bringing this into D's context, I imagine a solution for the GC 
would be to have it shared across all modules. The biggest 
problem I see, is that all shared libraries have to use the same 
D compiler, or a standard GC ABI needs to be defined.

--
Paulo
Sep 26 2012
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-09-26 19:58, Andrei Alexandrescu wrote:

 Haven't done any dynamic linking with D and I need to. I'm using dmd
 2.058/Linux at work to build and use dynamic libraries. Here's my attempt:



 Running make prints:

 dmd -fPIC -shared lib.d -of./lib.so
 /usr/bin/ld:
 /mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/libphobos2.a(minfo.o):
 relocation R_X86_64_32 against `a local symbol' can not be used when
 making a shared object; recompile with -fPIC
 /mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/libphobos2.a:
 could not read symbols: Bad value
 collect2: ld returned 1 exit status

 What steps do I need to take to get off the ground?


You need to properly implement support for dynamic libraries in druntime 
and DMD. I'm not entirely sure if there are some problems left in DMD, 
but druntime is definitely  not ready yet.

In general what needs to be done is to properly handle:

* Module infos
* TLS
* Exception handling tables

Martin Nowak has a branch for this:

https://github.com/dawgfoto/druntime/tree/SharedRuntime

-- 
/Jacob Carlborg
Sep 26 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/12 3:18 PM, Jacob Carlborg wrote:

 Martin Nowak has a branch for this:

 https://github.com/dawgfoto/druntime/tree/SharedRuntime


Thanks, I'll follow up with him.

Andrei
Sep 26 2012
prev sibling next sibling parent reply "Maxim Fomin" <maxim maxim-fomin.ru> writes:
You can build shared libraries on linux by manually compiling 
object files and linking them. On windows last time I tries it 
was not possible.
Sep 26 2012
parent reply Jens Mueller <jens.k.mueller gmx.de> writes:
Maxim Fomin wrote:

 You can build shared libraries on linux by manually compiling object
 files and linking them. On windows last time I tries it was not
 possible.


Can you give detailed steps for doing this on Linux? Because nobody as
far as I know has made this work yet?

Jens
Sep 26 2012
parent reply "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Thursday, 27 September 2012 at 05:52:44 UTC, Jens Mueller
wrote:

 Maxim Fomin wrote:

 You can build shared libraries on linux by manually compiling 
 object
 files and linking them. On windows last time I tries it was not
 possible.


 Can you give detailed steps for doing this on Linux? Because 
 nobody as
 far as I know has made this work yet?

 Jens


Dpaste seems not working, so, sorry for code

----lib.d---
import std.stdio;

static this()
{
	writeln("module ctor");
}

static ~this()
{
	writeln("module dtor");
}

class A
{
	private string text;;
	this(string text)
	{
		writeln("class ctor");
		this.text = text;
	}
	void tell()
	{
		writeln(this.text);
	}
	~this()
	{
		writeln(this.text);
		writeln("dtor");
	}
	static this()
	{
		writeln("static ctor");
	}
	static ~this()
	{
		writeln("static dtor");
	}
}
---------------
-----main.d----
import lib;

void main()
{
	auto a = new A("some text");
	a.tell();
}
---------------

dmd -c -fPIC lib.d
gcc -shared lib.o -o liblib.so
dmd -c main.d
gcc main.o -llib -lphobos2 -lrt -lpthread -L. -Wl,-rpath=.
./a.out
ldd a.out
	linux-vdso.so.1 (0x00007fff703ff000)
	liblib.so => ./liblib.so (0x00007f48158f1000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f48156cd000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f48154b1000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f481510c000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f4815af4000)
Sep 27 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-09-27 10:04, Maxim Fomin wrote:

 On Thursday, 27 September 2012 at 05:52:44 UTC, Jens Mueller
 wrote:

 Maxim Fomin wrote:

 You can build shared libraries on linux by manually compiling object
 files and linking them. On windows last time I tries it was not
 possible.


 Can you give detailed steps for doing this on Linux? Because nobody as
 far as I know has made this work yet?

 Jens


 Dpaste seems not working, so, sorry for code

 ----lib.d---
 import std.stdio;

 static this()
 {
      writeln("module ctor");
 }

 static ~this()
 {
      writeln("module dtor");
 }

 class A
 {
      private string text;;
      this(string text)
      {
          writeln("class ctor");
          this.text = text;
      }
      void tell()
      {
          writeln(this.text);
      }
      ~this()
      {
          writeln(this.text);
          writeln("dtor");
      }
      static this()
      {
          writeln("static ctor");
      }
      static ~this()
      {
          writeln("static dtor");
      }
 }
 ---------------
 -----main.d----
 import lib;

 void main()
 {
      auto a = new A("some text");
      a.tell();
 }
 ---------------

 dmd -c -fPIC lib.d
 gcc -shared lib.o -o liblib.so
 dmd -c main.d
 gcc main.o -llib -lphobos2 -lrt -lpthread -L. -Wl,-rpath=.
 ./a.out
 ldd a.out
      linux-vdso.so.1 (0x00007fff703ff000)
      liblib.so => ./liblib.so (0x00007f48158f1000)
      librt.so.1 => /lib64/librt.so.1 (0x00007f48156cd000)
      libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f48154b1000)
      libc.so.6 => /lib64/libc.so.6 (0x00007f481510c000)
      /lib64/ld-linux-x86-64.so.2 (0x00007f4815af4000)


1. Does this actually run?

2. This is statically linked with druntime and Phobos. What happens when 
you create an executable that links with the D dynamic library?

Last time I tried this (on Mac OS X) I got several symbols missing. This 
was all symbols that are usually pointing to the executable, inserted by 
the compiler. One of them would be "main" and symbols like these:

https://github.com/D-Programming-Language/druntime/blob/master/src/rt/deh2.d#L27

-- 
/Jacob Carlborg
Sep 27 2012
next sibling parent reply "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Thursday, 27 September 2012 at 08:26:08 UTC, Jacob Carlborg 
wrote:

 1. Does this actually run?


If it were non-runnable, I wouldn't posted it.


 2. This is statically linked with druntime and Phobos. What 
 happens when you create an executable that links with the D 
 dynamic library?


Solution depends on a problem. I understood Andrei's post that he 
wanted a .so file or DLL. I told originally that it is possible 
to make shared libraries on linux. Now I see there is some 
misunderstanding. Is the problem in diving D application on 
executables and shared libraries or making druntime+phobos a 
shared library or loading a library at runtime? A was speaking 
about the first.


 Last time I tried this (on Mac OS X) I got several symbols 
 missing. This was all symbols that are usually pointing to the 
 executable, inserted by the compiler. One of them would be 
 "main" and symbols like these:

 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/deh2.d#L27


I have no idea about D support of shared libraries on Mac OS.
Sep 27 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-09-27 10:55, Maxim Fomin wrote:


 If it were non-runnable, I wouldn't posted it.


Ok, I see.


 Solution depends on a problem. I understood Andrei's post that he wanted
 a .so file or DLL. I told originally that it is possible to make shared
 libraries on linux. Now I see there is some misunderstanding. Is the
 problem in diving D application on executables and shared libraries or
 making druntime+phobos a shared library or loading a library at runtime?
 A was speaking about the first.


Obviously he wants to use the shared library, otherwise it would be 
rather useless and pointless.

Last I checked there was a problem with:

* Using a shared library written in D with a D application
* Making druntime and Phobos shared libraries
* Possibly also compiling shared libraries

-- 
/Jacob Carlborg
Sep 27 2012
parent reply "Daniel Kozak" <kozzi11 gmail.com> writes:
On Thursday, 27 September 2012 at 09:16:48 UTC, Jacob Carlborg 
wrote:

 On 2012-09-27 10:55, Maxim Fomin wrote:


 If it were non-runnable, I wouldn't posted it.


 Ok, I see.


 Solution depends on a problem. I understood Andrei's post that 
 he wanted
 a .so file or DLL. I told originally that it is possible to 
 make shared
 libraries on linux. Now I see there is some misunderstanding. 
 Is the
 problem in diving D application on executables and shared 
 libraries or
 making druntime+phobos a shared library or loading a library 
 at runtime?
 A was speaking about the first.


 Obviously he wants to use the shared library, otherwise it 
 would be rather useless and pointless.

 Last I checked there was a problem with:

 * Using a shared library written in D with a D application
 * Making druntime and Phobos shared libraries
 * Possibly also compiling shared libraries


With DMD 2.060 last two points are possible, only first point 
(Using a shared ...) I dont try
Sep 27 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-09-27 12:02, Daniel Kozak wrote:


 With DMD 2.060 last two points are possible, only first point (Using a
 shared ...) I dont try


I really need to try this when I get home.

-- 
/Jacob Carlborg
Sep 27 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-09-27 10:55, Maxim Fomin wrote:

 On Thursday, 27 September 2012 at 08:26:08 UTC, Jacob Carlborg wrote:

 1. Does this actually run?


 If it were non-runnable, I wouldn't posted it.


 2. This is statically linked with druntime and Phobos. What happens
 when you create an executable that links with the D dynamic library?


 Solution depends on a problem. I understood Andrei's post that he wanted
 a .so file or DLL. I told originally that it is possible to make shared
 libraries on linux. Now I see there is some misunderstanding. Is the
 problem in diving D application on executables and shared libraries or
 making druntime+phobos a shared library or loading a library at runtime?
 A was speaking about the first.


Actually, I seriously doubt everything is working as expected. For 
example, what happens when an application loads (via dlopen or similar) 
a D dynamic library:

* Are exception handlers registered
* Are module infos properly registered
* What happens if I call Object.factory, will that find a class in the 
dynamic library
* Are GC sections registered
* What happens when the library is unloaded, will all of the above be 
unregistered and cleaned up

A quick look in rt.minfo in druntime shows the it uses a single array or 
linked list for all module infos. It expect the module infos to be in 
just one place, i.e. in the application. There's no handling when a 
dynamic library is loaded.

Have a look at this:

https://github.com/D-Programming-Language/druntime/blob/master/src/rt/minfo.d#L184

That symbol points to the start of the linked list containing module 
infos. That's only a single linked list, no handling of module infos 
from multiple places.

rt.minfo needs to be changed to use an associative array with the keys 
pointing to images (executables and loaded dynamic libraries) and the 
values mapped to module infos of the given image. It also needs to 
intercept when a library is dynamically loaded, extract the module infos 
and register it with the runtime.

I would think that the same is true for exception handling tables, TLS 
and GC sections.

-- 
/Jacob Carlborg
Sep 27 2012
next sibling parent reply "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Thursday, 27 September 2012 at 17:10:07 UTC, Jacob Carlborg 
wrote:

 On 2012-09-27 10:55, Maxim Fomin wrote:

 On Thursday, 27 September 2012 at 08:26:08 UTC, Jacob Carlborg 
 wrote:

 1. Does this actually run?


 If it were non-runnable, I wouldn't posted it.


 2. This is statically linked with druntime and Phobos. What 
 happens
 when you create an executable that links with the D dynamic 
 library?


 Solution depends on a problem. I understood Andrei's post that 
 he wanted
 a .so file or DLL. I told originally that it is possible to 
 make shared
 libraries on linux. Now I see there is some misunderstanding. 
 Is the
 problem in diving D application on executables and shared 
 libraries or
 making druntime+phobos a shared library or loading a library 
 at runtime?
 A was speaking about the first.


 Actually, I seriously doubt everything is working as expected. 
 For example, what happens when an application loads (via dlopen 
 or similar) a D dynamic library:

 * Are exception handlers registered
 * Are module infos properly registered
 * What happens if I call Object.factory, will that find a class 
 in the dynamic library
 * Are GC sections registered
 * What happens when the library is unloaded, will all of the 
 above be unregistered and cleaned up

 A quick look in rt.minfo in druntime shows the it uses a single 
 array or linked list for all module infos. It expect the module 
 infos to be in just one place, i.e. in the application. There's 
 no handling when a dynamic library is loaded.

 Have a look at this:

 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/minfo.d#L184

 That symbol points to the start of the linked list containing 
 module infos. That's only a single linked list, no handling of 
 module infos from multiple places.

 rt.minfo needs to be changed to use an associative array with 
 the keys pointing to images (executables and loaded dynamic 
 libraries) and the values mapped to module infos of the given 
 image. It also needs to intercept when a library is dynamically 
 loaded, extract the module infos and register it with the 
 runtime.

 I would think that the same is true for exception handling 
 tables, TLS and GC sections.


Posted code doesn't load libraries at runtime, it is just linked 
to shared libraries.
Sep 27 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/27/12 3:37 PM, Maxim Fomin wrote:

 Posted code doesn't load libraries at runtime, it is just linked to
 shared libraries.


Exactly! (I can't believe I'm starting to get the hang of this...) But 
what we ultimately need is true dynamic loading of never-seen modules.

After the initial test I tried to load symbols with dlopen(). Indeed 
this works:

auto p = dlopen("liblib.so", RTLD_LAZY);

And it does work, all fine. To my dismay, as soon as I removed the flag 
"-llib" from the linker command line in the makefile, dlopen() didn't 
want to work anymore.

So I think in order to enable "true" dynamic loading, I'll need to 
generate PIC for druntime and phobos, and then link liblib.so like this:

dmd -fPIC -c lib.d
gcc -shared lib.o -o liblib.so -L/path/to/phobos -lphobos2

Is that correct?

At that point, of course, a variety of issues will need to be resolved 
as people pointed out.


Andrei
Sep 27 2012
next sibling parent reply "Rob T" <rob ucora.com> writes:
For me to get C or C++ to run a D function, I had to do the 
following:

// ====================
// C/C++ library source
// ====================

// sample D function from library
void foo(int i, int j, int k);
// D runtime initialization & shutdown
void init();
void done();

void bar()
{
     foo(6, 7, 8);
}

int main(int argc, char **argv)
{
init();
	bar();
	done();
	return 0;
}


// ================
// D library source
// ================
import std.stdio;
static import core.runtime;

// sample D function for test
extern (C++) //	int foo(int i, int j, int k)
	void foo(int i, int j, int k)
	{
		writefln("i = %s", i);
		writefln("j = %s", j);
		writefln("k = %s", k);
		int t = i + j + k;
		writefln("Total = %s", t);
	}

// Had to initialize and shutdown D system from C/C++.
extern (C++) export void init() { // to be called once after 
loading shared lib
     core.runtime.Runtime.initialize();
}

extern (C++) export void done() { // to be called before 
unloading shared lib
     core.runtime.Runtime.terminate();
}


// I had to include main even though this is a library.
int main()
{
    return 0;
}
Sep 27 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-09-28 01:21, Rob T wrote:

 For me to get C or C++ to run a D function, I had to do the following:

 // ====================
 // C/C++ library source
 // ====================

 // sample D function from library
 void foo(int i, int j, int k);
 // D runtime initialization & shutdown
 void init();
 void done();

 void bar()
 {
      foo(6, 7, 8);
 }

 int main(int argc, char **argv)
 {
 init();
      bar();
      done();
      return 0;
 }


 // ================
 // D library source
 // ================
 import std.stdio;
 static import core.runtime;

 // sample D function for test
 extern (C++) //    int foo(int i, int j, int k)
      void foo(int i, int j, int k)
      {
          writefln("i = %s", i);
          writefln("j = %s", j);
          writefln("k = %s", k);
          int t = i + j + k;
          writefln("Total = %s", t);
      }

 // Had to initialize and shutdown D system from C/C++.
 extern (C++) export void init() { // to be called once after loading
 shared lib
      core.runtime.Runtime.initialize();
 }

 extern (C++) export void done() { // to be called before unloading
 shared lib
      core.runtime.Runtime.terminate();
 }


 // I had to include main even though this is a library.
 int main()
 {
     return 0;
 }


That is not sufficient to have everything work. It might be ok if you 
limit yourself to a C subset of D. On some platforms, calling 
Runtime.initialize, will miss initializing some parts.

The implementation of the runtime initialization contains a lot of code 
duplication and some functions/branches are missing functionality. Example:

https://github.com/D-Programming-Language/druntime/blob/master/src/rt/dmain2.d#L346

The C main function handles the runtime initialization for a regular 
statically linked executable. If you initialize the runtime via 
"rt_init", which is called by "Runtime.initialize", it will miss to 
initialize some parts on some platforms. The C main function should 
really call "rt_init" to remove the code duplication.

Also see this post why everything will not properly work:

http://forum.dlang.org/thread/k3vfm9$1tq$1 digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com


-- 
/Jacob Carlborg
Sep 27 2012
prev sibling parent "Rob T" <rob ucora.com> writes:
On Thursday, 27 September 2012 at 19:57:13 UTC, Andrei 
Alexandrescu wrote:

 So I think in order to enable "true" dynamic loading, I'll need 
 to generate PIC for druntime and phobos, and then link 
 liblib.so like this:

 dmd -fPIC -c lib.d
 gcc -shared lib.o -o liblib.so -L/path/to/phobos -lphobos2

 Is that correct?


Yes, that is correct.

The other thing missing is a phobos2.so for dynamic runtime 
linking.

-rt
Sep 27 2012
prev sibling next sibling parent reply "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Thursday, 27 September 2012 at 17:10:07 UTC, Jacob Carlborg 
wrote:

 Actually, I seriously doubt everything is working as expected.


I tried to check how TLS, EX, etc. (mostly exposed to dll issue) 
are working and here is some kind of test: 
https://github.com/mxfm/sharedtest. Unfortunately scope(exit) 
isn't executed when it is situated in a shared library and which 
calls some throwing function from another shared library. 
Unittests aren't working either. Regarding other parts - they 
seem to work.
Sep 28 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-09-28 20:25, Maxim Fomin wrote:


 I tried to check how TLS, EX, etc. (mostly exposed to dll issue) are
 working and here is some kind of test:
 https://github.com/mxfm/sharedtest. Unfortunately scope(exit) isn't
 executed when it is situated in a shared library and which calls some
 throwing function from another shared library. Unittests aren't working
 either. Regarding other parts - they seem to work.


That's a fairly uninteresting test. You are linking to the dynamic 
library. What's interesting is loading a dynamic library using dlopen, 
or similar. What's the point of using dynamic libraries if you're 
linking with them?

-- 
/Jacob Carlborg
Sep 29 2012
next sibling parent reply "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Saturday, 29 September 2012 at 13:19:01 UTC, Jacob Carlborg 
wrote:

 That's a fairly uninteresting test.


I am not a D developer which means I have no incentive in blindly 
portraying D as a language having shared libraries support when 
in fact it has some issues. I am a D user which has incentive to 
gladly report which part of D does work (or seems to work) and 
which doesn't when making shared libraries.


 You are linking to the dynamic library. What's interesting is 
 loading a dynamic library using dlopen, or similar. What's the 
 point of using dynamic libraries if you're linking with them?


I was not taking about dynamic loading, but about dynamic 
linking. If I understand topic right, the issue is the former, 
not the latter.

BTW, in majority cases dynamic loading gives no advantages over 
dynamic linking (just the opposite: doing unnecessary job which 
can be done by linker and loader). In most cases, when an app is 
being written, it is known which functional from which 
third-party libraries would be used. The only exceptions I can 
name are pluggin support and hacker's binary ELF/PE tools. I can 
count few apps in windows and linux which are used often, which 
dynamically load something, and if they do, they likely load 
pluggins. This was made for cases when app developers by 
definition cannot know beforehand full list of used shared 
libraries. So, the question is just opposite: "What's the point 
of using dynamic loading if you know beforehand which libraries 
you use, which happens almost in all cases?".

But I certainly agree that dynamic loading is essential for some 
applications, if not irreplaceable, and support of it also needed 
to be inspected and improved.
Sep 29 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-09-29 17:40, Maxim Fomin wrote:


 I was not taking about dynamic loading, but about dynamic linking. If I
 understand topic right, the issue is the former, not the latter.


The title of the thread says "... and loading".


 BTW, in majority cases dynamic loading gives no advantages over dynamic
 linking (just the opposite: doing unnecessary job which can be done by
 linker and loader). In most cases, when an app is being written, it is
 known which functional from which third-party libraries would be used.


I agree, in these cases their no point in using dynamic loading.


 The only exceptions I can name are pluggin support and hacker's binary
 ELF/PE tools. I can count few apps in windows and linux which are used
 often, which dynamically load something, and if they do, they likely
 load pluggins. This was made for cases when app developers by definition
 cannot know beforehand full list of used shared libraries.


Exactly, pluggins are one of the main reasons for using dynamic loading. 
An other reason is license issues. There's a D library called Derelict 
which uses dynamic loading to load C libraries (OpenGL, SDL, OpenAL and 
others). One of the many reason for Derelict using dynamic loading is 
due to license issues/complications with these libraries.

http://dsource.org/projects/derelict/


 So, the question is just opposite: "What's the point of using dynamic loading
if
 you know beforehand which libraries you use, which happens almost in all
 cases?".


There's no point in doing that. But what I'm saying is if you know 
beforehand the libraries you will use you can get quite far with static 
libraries.

-- 
/Jacob Carlborg
Sep 29 2012
parent reply "Rob T" <rob ucora.com> writes:
On Saturday, 29 September 2012 at 16:09:14 UTC, Jacob Carlborg 
wrote:

 There's no point in doing that. But what I'm saying is if you 
 know beforehand the libraries you will use you can get quite 
 far with static libraries.


There are plenty of cases where you have to use a dynamically 
loaded lib even if you know before hand what will be loaded. I 
think you understand this if I read your posts correctly.

In my case I have a pre-existing C++ app that is designed to load 
user defined C++ plugins. I wanted to interface D to one of my 
own C++ plugins, but I apparently cannot do it reliably because 
of initialization issues with the GC and possibly some other 
obscure items.

If I can figure out what needs to be done to resolve the problem 
in enough detail, then maybe I can hack the runtime source and 
roll out a solution.

The GC always seems to pop up as a source of problems. For long 
term solution, the GC should be made 100% optional (in practice 
as opposed to in theory), the standard library ought to be made 
to work with or wothout a GC (or simply without), and the GC 
itself should be easily replaceable with alternate versions. I 
think this idea has already been discussed elsewhere, and is on 
the TODO list (I hope!).

--rt
Sep 29 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-09-30 08:41, Rob T wrote:


 There are plenty of cases where you have to use a dynamically loaded lib
 even if you know before hand what will be loaded. I think you understand
 this if I read your posts correctly.

 In my case I have a pre-existing C++ app that is designed to load user
 defined C++ plugins. I wanted to interface D to one of my own C++
 plugins, but I apparently cannot do it reliably because of
 initialization issues with the GC and possibly some other obscure items.


But now we're back at plugins. I think this part of the discussion is 
starting to run in circles and become quite pointless. I think most of 
us know that we need to support all types of libraries. Static and 
dynamic, both for link time and runtime.


 If I can figure out what needs to be done to resolve the problem in
 enough detail, then maybe I can hack the runtime source and roll out a
 solution.


You can have a look at the work done by Martin Nowak:

https://github.com/dawgfoto/druntime/commits/SharedRuntime

He has a couple of other useful branches as well, for example:

https://github.com/dawgfoto/druntime/commits/dmain2Refactoring

On Mac OS X there's also the problem with TLS. There is no native 
support for TLS in Mac OS X prior to Lion (10.7). DMD has rolled its own 
implementation that needs to be adapted to support dynamic libraries.


 The GC always seems to pop up as a source of problems. For long term
 solution, the GC should be made 100% optional (in practice as opposed to
 in theory), the standard library ought to be made to work with or
 wothout a GC (or simply without), and the GC itself should be easily
 replaceable with alternate versions. I think this idea has already been
 discussed elsewhere, and is on the TODO list (I hope!).


You can replace the GC at link time. Here's an example of a stubbed 
implemented of the GC:

http://dsource.org/projects/tango/browser/trunk/tango/core/rt/gc/stub

This is for Tango but the druntime is based on the Tango runtime so I 
would guess most of this would be the same.

-- 
/Jacob Carlborg
Sep 30 2012
parent reply "Rob T" <rob ucora.com> writes:
On Sunday, 30 September 2012 at 10:57:24 UTC, Jacob Carlborg 
wrote:

 On 2012-09-30 08:41, Rob T wrote:

 I think most of us know that we need to support all types of 
 libraries. Static and dynamic, both for link time and runtime.


OK we're all in agreement on this point.


 You can have a look at the work done by Martin Nowak:

 https://github.com/dawgfoto/druntime/commits/SharedRuntime


It seems that an attempt to make the runtime shared is well under 
way. Did anything get into the main dmd branch or has the effort 
been stalled or ...?

I've only recently been learning what's going on with D and why 
it is what it is, so a breif history lesson may be needed once in 
a while.


 You can replace the GC at link time. Here's an example of a 
 stubbed implemented of the GC:
 http://dsource.org/projects/tango/browser/trunk/tango/core/rt/gc/stub


I will look at this too. Thanks for the pointers.

--rt
Sep 30 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-10-01 01:42, Rob T wrote:


 It seems that an attempt to make the runtime shared is well under way.
 Did anything get into the main dmd branch or has the effort been stalled
 or ...?


Seems pretty stalled.


 I will look at this too. Thanks for the pointers.


No problem.

-- 
/Jacob Carlborg
Sep 30 2012
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Sat, 29 Sep 2012 15:19:30 +0200
schrieb Jacob Carlborg <doob me.com>:


 On 2012-09-28 20:25, Maxim Fomin wrote:
 

 I tried to check how TLS, EX, etc. (mostly exposed to dll issue) are
 working and here is some kind of test:
 https://github.com/mxfm/sharedtest. Unfortunately scope(exit) isn't
 executed when it is situated in a shared library and which calls
 some throwing function from another shared library. Unittests
 aren't working either. Regarding other parts - they seem to work.


 
 That's a fairly uninteresting test. You are linking to the dynamic 
 library. What's interesting is loading a dynamic library using
 dlopen, or similar. What's the point of using dynamic libraries if
 you're linking with them?
 


There are some reasons for dynamic libraries linked at compile time,
one is that we have to start somewhere and they are required for
plugins / dynamically loaded libraries as well ;-)

So I started a small test suite for GDC (could be adapted to other
compilers). It currently only tests compile time linking of dynamic
libraries, but adjusting the test to use runtime loading should be
easy. But it's pointless as long as we have no runtime support.
https://github.com/jpf91/dso-test

It should also be enhanced to test multiple shared libraries.

The good news:
* Exception handling is working
* ModuleInfos are working
* unit tests are working
* Static variables, gshared variables, tls variables are working
* Object.factory is working
* Calling functions, passing function pointers, passing classes between
  dso/app is working

The bad news:
* The GC doesn't scan TLS/__gshared/static data in dynamic libraries,
  it only scans the main app.
Oct 01 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-01 12:42, Johannes Pfau wrote:


 There are some reasons for dynamic libraries linked at compile time,
 one is that we have to start somewhere and they are required for
 plugins / dynamically loaded libraries as well ;-)

 So I started a small test suite for GDC (could be adapted to other
 compilers). It currently only tests compile time linking of dynamic
 libraries, but adjusting the test to use runtime loading should be
 easy. But it's pointless as long as we have no runtime support.
 https://github.com/jpf91/dso-test

 It should also be enhanced to test multiple shared libraries.

 The good news:
 * Exception handling is working
 * ModuleInfos are working
 * unit tests are working
 * Static variables, gshared variables, tls variables are working
 * Object.factory is working
 * Calling functions, passing function pointers, passing classes between
    dso/app is working


That's good. Have you tested this with DMD?


 The bad news:
 * The GC doesn't scan TLS/__gshared/static data in dynamic libraries,
    it only scans the main app.


That should be fairly trivial on Mac OS X. But I'm suspecting it won't 
be that easy on Linux.

BTW, is the runtime and phobos statically linked both with the dynamic 
library and the executable?

-- 
/Jacob Carlborg
Oct 01 2012
next sibling parent reply Iain Buclaw <ibuclaw ubuntu.com> writes:
On 1 October 2012 13:34, Jacob Carlborg <doob me.com> wrote:

 On 2012-10-01 12:42, Johannes Pfau wrote:


 There are some reasons for dynamic libraries linked at compile time,
 one is that we have to start somewhere and they are required for
 plugins / dynamically loaded libraries as well ;-)

 So I started a small test suite for GDC (could be adapted to other
 compilers). It currently only tests compile time linking of dynamic
 libraries, but adjusting the test to use runtime loading should be
 easy. But it's pointless as long as we have no runtime support.
 https://github.com/jpf91/dso-test

 It should also be enhanced to test multiple shared libraries.

 The good news:
 * Exception handling is working
 * ModuleInfos are working
 * unit tests are working
 * Static variables, gshared variables, tls variables are working
 * Object.factory is working
 * Calling functions, passing function pointers, passing classes between
    dso/app is working



 That's good. Have you tested this with DMD?



 The bad news:
 * The GC doesn't scan TLS/__gshared/static data in dynamic libraries,
    it only scans the main app.



 That should be fairly trivial on Mac OS X. But I'm suspecting it won't be
 that easy on Linux.

 BTW, is the runtime and phobos statically linked both with the dynamic
 library and the executable?

 --
 /Jacob Carlborg


On Linux, there has already been an runtime implementation written
that scans /proc/self/maps and adds all data sections to the GC that
way.  Whether or not DMD wishes to go down that route is their own
decision.  I am looking into a solution that doesn't have any bearing
on what platform it's running on...


Regards
-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';
Oct 01 2012
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-10-01 14:40, Iain Buclaw wrote:


 On Linux, there has already been an runtime implementation written
 that scans /proc/self/maps and adds all data sections to the GC that
 way.  Whether or not DMD wishes to go down that route is their own
 decision.  I am looking into a solution that doesn't have any bearing
 on what platform it's running on...


Well, /proc isn't available on Mac OS X so I think you have to continue 
looking.

-- 
/Jacob Carlborg
Oct 01 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-10-01 14:40, Iain Buclaw wrote:


 On Linux, there has already been an runtime implementation written
 that scans /proc/self/maps and adds all data sections to the GC that
 way.  Whether or not DMD wishes to go down that route is their own
 decision.  I am looking into a solution that doesn't have any bearing
 on what platform it's running on...


I think this is already working on Mac OS X. It's handled by:

https://github.com/D-Programming-Language/druntime/blob/master/src/rt/memory_osx.d#L82

https://github.com/D-Programming-Language/druntime/blob/master/src/rt/memory_osx.d#L133

-- 
/Jacob Carlborg
Oct 01 2012
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Mon, 01 Oct 2012 14:34:21 +0200
schrieb Jacob Carlborg <doob me.com>:


 On 2012-10-01 12:42, Johannes Pfau wrote:
 

 There are some reasons for dynamic libraries linked at compile time,
 one is that we have to start somewhere and they are required for
 plugins / dynamically loaded libraries as well ;-)

 So I started a small test suite for GDC (could be adapted to other
 compilers). It currently only tests compile time linking of dynamic
 libraries, but adjusting the test to use runtime loading should be
 easy. But it's pointless as long as we have no runtime support.
 https://github.com/jpf91/dso-test

 It should also be enhanced to test multiple shared libraries.

 The good news:
 * Exception handling is working
 * ModuleInfos are working
 * unit tests are working
 * Static variables, gshared variables, tls variables are working
 * Object.factory is working
 * Calling functions, passing function pointers, passing classes
 between dso/app is working


 
 That's good. Have you tested this with DMD?


Not yet.

 

 The bad news:
 * The GC doesn't scan TLS/__gshared/static data in dynamic
 libraries, it only scans the main app.


 
 That should be fairly trivial on Mac OS X. But I'm suspecting it
 won't be that easy on Linux.
 
 BTW, is the runtime and phobos statically linked both with the
 dynamic library and the executable?


I tested two different configurations:

druntime and phobos are shared libraries as well (this is the correct
solution, it's mostly working except for the GC issues)

druntime and phobos static linking: phobos and druntime are statically
linked into the app, libdso.so is not linked against druntime/phobos at
all. (this is a hack to get better test results: With a shared
druntime, you can't call GC.collect twice, because the first call
frees important objects in druntime and then the second call segfaults)
Oct 01 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-10-01 17:03, Johannes Pfau wrote:


 I tested two different configurations:

 druntime and phobos are shared libraries as well (this is the correct
 solution, it's mostly working except for the GC issues)


I agree.


 druntime and phobos static linking: phobos and druntime are statically
 linked into the app, libdso.so is not linked against druntime/phobos at
 all. (this is a hack to get better test results: With a shared
 druntime, you can't call GC.collect twice, because the first call
 frees important objects in druntime and then the second call segfaults)


Ok.

-- 
/Jacob Carlborg
Oct 01 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-01 12:42, Johannes Pfau wrote:


 There are some reasons for dynamic libraries linked at compile time,
 one is that we have to start somewhere and they are required for
 plugins / dynamically loaded libraries as well ;-)

 So I started a small test suite for GDC (could be adapted to other
 compilers). It currently only tests compile time linking of dynamic
 libraries, but adjusting the test to use runtime loading should be
 easy. But it's pointless as long as we have no runtime support.
 https://github.com/jpf91/dso-test


Isn't "dmain2" used when building shared libraries using GDC? That's 
where the implementation of "rt_init" is located.

-- 
/Jacob Carlborg
Oct 01 2012
parent reply Johannes Pfau <nospam example.com> writes:
Am Mon, 01 Oct 2012 15:11:49 +0200
schrieb Jacob Carlborg <doob me.com>:


 On 2012-10-01 12:42, Johannes Pfau wrote:
 

 There are some reasons for dynamic libraries linked at compile time,
 one is that we have to start somewhere and they are required for
 plugins / dynamically loaded libraries as well ;-)

 So I started a small test suite for GDC (could be adapted to other
 compilers). It currently only tests compile time linking of dynamic
 libraries, but adjusting the test to use runtime loading should be
 easy. But it's pointless as long as we have no runtime support.
 https://github.com/jpf91/dso-test


 
 Isn't "dmain2" used when building shared libraries using GDC? That's 
 where the implementation of "rt_init" is located.
 


the problem is that we don't want the C main function in a shared
libgdruntime.so, because you might want to use libgdruntime.so in a
C/C++ app which has it's own main function.

So we currently don't link in dmain2.o into the shared library and it
must be included manually when linking an application.
(But dmain2 also contains some stuff that really should be in
libdruntime.so, so this source file should probably be split up at some
time.)
Oct 01 2012
next sibling parent Iain Buclaw <ibuclaw ubuntu.com> writes:
On 1 October 2012 16:06, Johannes Pfau <nospam example.com> wrote:

 Am Mon, 01 Oct 2012 15:11:49 +0200
 schrieb Jacob Carlborg <doob me.com>:


 On 2012-10-01 12:42, Johannes Pfau wrote:


 There are some reasons for dynamic libraries linked at compile time,
 one is that we have to start somewhere and they are required for
 plugins / dynamically loaded libraries as well ;-)

 So I started a small test suite for GDC (could be adapted to other
 compilers). It currently only tests compile time linking of dynamic
 libraries, but adjusting the test to use runtime loading should be
 easy. But it's pointless as long as we have no runtime support.
 https://github.com/jpf91/dso-test


 Isn't "dmain2" used when building shared libraries using GDC? That's
 where the implementation of "rt_init" is located.


 the problem is that we don't want the C main function in a shared
 libgdruntime.so, because you might want to use libgdruntime.so in a
 C/C++ app which has it's own main function.


That is at least one of the theories behind it anyway. :-)

The more I think about it, the less I think I would want a C++ app to
link against a D shared library though.

With C, there will need to be defined a common interface header -
similar to what used to be mars.h in rt/ but also contain some useful
runtime functions.


Regards
-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';
Oct 01 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-01 17:06, Johannes Pfau wrote:


 the problem is that we don't want the C main function in a shared
 libgdruntime.so, because you might want to use libgdruntime.so in a
 C/C++ app which has it's own main function.

 So we currently don't link in dmain2.o into the shared library and it
 must be included manually when linking an application.
 (But dmain2 also contains some stuff that really should be in
 libdruntime.so, so this source file should probably be split up at some
 time.)


I'm not sure if I follow this correctly or not, but why is this needed 
to be handled manually? If you pass "-shared" to the compiler just skip 
linking dmain2.o, otherwise link with it. Would that work?

-- 
/Jacob Carlborg
Oct 01 2012
parent Johannes Pfau <nospam example.com> writes:
Am Mon, 01 Oct 2012 19:18:46 +0200
schrieb Jacob Carlborg <doob me.com>:


 On 2012-10-01 17:06, Johannes Pfau wrote:
 

 the problem is that we don't want the C main function in a shared
 libgdruntime.so, because you might want to use libgdruntime.so in a
 C/C++ app which has it's own main function.

 So we currently don't link in dmain2.o into the shared library and
 it must be included manually when linking an application.
 (But dmain2 also contains some stuff that really should be in
 libdruntime.so, so this source file should probably be split up at
 some time.)


 
 I'm not sure if I follow this correctly or not, but why is this
 needed to be handled manually? If you pass "-shared" to the compiler
 just skip linking dmain2.o, otherwise link with it. Would that work?
 


Yes something similar would work, it's just not yet implemented.

GDC should detect if we're linking against a shared druntime and then
it should automatically add dmain2.o to the linker command.
Oct 02 2012
prev sibling parent reply "Bottled Gin" <gin bottled.com> writes:
 Actually, I seriously doubt everything is working as expected. 
 For example, what happens when an application loads (via dlopen 
 or similar) a D dynamic library:

 * Are exception handlers registered
 * Are module infos properly registered
 * What happens if I call Object.factory, will that find a class 
 in the dynamic library
 * Are GC sections registered
 * What happens when the library is unloaded, will all of the 
 above be unregistered and cleaned up


Hello D Experts

I am replying to an old thread since I want to know if and how 
the situation has improved over the past few months. In 
particular I want to link and call fairly complex D functions 
from a C++ application.

I am exclusively using Linux. Can somebody please guide me if I 
should expect things to work with current DMD master from github?

Regards
- Puneet
Jun 18 2013
parent reply 1100110 <0b1100110 gmail.com> writes:
On 06/18/2013 10:14 AM, Bottled Gin wrote:

 Actually, I seriously doubt everything is working as expected. For
 example, what happens when an application loads (via dlopen or
 similar) a D dynamic library:

 * Are exception handlers registered
 * Are module infos properly registered
 * What happens if I call Object.factory, will that find a class in the
 dynamic library
 * Are GC sections registered
 * What happens when the library is unloaded, will all of the above be
 unregistered and cleaned up


 Hello D Experts

 I am replying to an old thread since I want to know if and how the
 situation has improved over the past few months. In particular I want to
 link and call fairly complex D functions from a C++ application.

 I am exclusively using Linux. Can somebody please guide me if I should
 expect things to work with current DMD master from github?

 Regards
 - Puneet


There's a DConf talk about shared libraries in D.

Perhaps there's useful information there?
http://www.youtube.com/watch?v=i63VeudjZM4

(I'm watching it now so I don't know how detailed it is.)
Jun 18 2013
parent "Etienne" <etcimon gmail.com> writes:
On Tuesday, 18 June 2013 at 20:23:42 UTC, 1100110 wrote:

 On 06/18/2013 10:14 AM, Bottled Gin wrote:

 Actually, I seriously doubt everything is working as 
 expected. For
 example, what happens when an application loads (via dlopen or
 similar) a D dynamic library:

 * Are exception handlers registered
 * Are module infos properly registered
 * What happens if I call Object.factory, will that find a 
 class in the
 dynamic library
 * Are GC sections registered
 * What happens when the library is unloaded, will all of the 
 above be
 unregistered and cleaned up


 Hello D Experts

 I am replying to an old thread since I want to know if and how 
 the
 situation has improved over the past few months. In particular 
 I want to
 link and call fairly complex D functions from a C++ 
 application.

 I am exclusively using Linux. Can somebody please guide me if 
 I should
 expect things to work with current DMD master from github?

 Regards
 - Puneet


 There's a DConf talk about shared libraries in D.

 Perhaps there's useful information there?
 http://www.youtube.com/watch?v=i63VeudjZM4

 (I'm watching it now so I don't know how detailed it is.)


Hello all,

I'm wondering if this has been resolved to date. I get this error 
on master:

/usr/bin/ld: generated/linux/release/64/libphobos2.so.0.66.o: 
relocation R_X86_64_32 against `.rodata' can not be used when 
making a shared object; recompile with -fPIC
generated/linux/release/64/libphobos2.so.0.66.o: could not read 
symbols: Bad value
collect2: error: ld returned 1 exit status

even though I've built with fPIC:

cc -c  -m64 -fPIC -O3 etc/c/zlib/trees.c 
-ogenerated/linux/release/64/etc/c/zlib/trees.o
cc -c  -m64 -fPIC -O3 etc/c/zlib/uncompr.c 
-ogenerated/linux/release/64/etc/c/zlib/uncompr.o
cc -c  -m64 -fPIC -O3 etc/c/zlib/zutil.c 
-ogenerated/linux/release/64/etc/c/zlib/zutil.o

etc...
Mar 24 2014
prev sibling next sibling parent Jens Mueller <jens.k.mueller gmx.de> writes:
Jacob Carlborg wrote:

 On 2012-09-27 10:04, Maxim Fomin wrote:

On Thursday, 27 September 2012 at 05:52:44 UTC, Jens Mueller
wrote:

Maxim Fomin wrote:

You can build shared libraries on linux by manually compiling object
files and linking them. On windows last time I tries it was not
possible.


Can you give detailed steps for doing this on Linux? Because nobody as
far as I know has made this work yet?

Jens


Dpaste seems not working, so, sorry for code

----lib.d---
import std.stdio;

static this()
{
     writeln("module ctor");
}

static ~this()
{
     writeln("module dtor");
}

class A
{
     private string text;;
     this(string text)
     {
         writeln("class ctor");
         this.text = text;
     }
     void tell()
     {
         writeln(this.text);
     }
     ~this()
     {
         writeln(this.text);
         writeln("dtor");
     }
     static this()
     {
         writeln("static ctor");
     }
     static ~this()
     {
         writeln("static dtor");
     }
}
---------------
-----main.d----
import lib;

void main()
{
     auto a = new A("some text");
     a.tell();
}
---------------

dmd -c -fPIC lib.d
gcc -shared lib.o -o liblib.so
dmd -c main.d
gcc main.o -llib -lphobos2 -lrt -lpthread -L. -Wl,-rpath=.
./a.out
ldd a.out
     linux-vdso.so.1 (0x00007fff703ff000)
     liblib.so => ./liblib.so (0x00007f48158f1000)
     librt.so.1 => /lib64/librt.so.1 (0x00007f48156cd000)
     libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f48154b1000)
     libc.so.6 => /lib64/libc.so.6 (0x00007f481510c000)
     /lib64/ld-linux-x86-64.so.2 (0x00007f4815af4000)


 
 1. Does this actually run?


I just tried.
$ ./a.out
module ctor
static ctor
class ctor
some text
static dtor
module dtor
some text
dtor


 2. This is statically linked with druntime and Phobos. What happens
 when you create an executable that links with the D dynamic library?


a.out is linked dynamically against liblib.so.


 Last time I tried this (on Mac OS X) I got several symbols missing.
 This was all symbols that are usually pointing to the executable,
 inserted by the compiler. One of them would be "main" and symbols
 like these:
 
 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/deh2.d#L27


I'm running Linux. You can test on Mac OS X.
I'm also astonished that this works. This is great.

Jens
Sep 27 2012
prev sibling parent reply "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Thursday, 27 September 2012 at 08:26:08 UTC, Jacob Carlborg
wrote:

 Last time I tried this (on Mac OS X) I got several symbols 
 missing. This was all symbols that are usually pointing to the 
 executable, inserted by the compiler. One of them would be 
 "main" and symbols like these:

 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/deh2.d#L27


"_deh_" problem AFAIK arise when non-D and D code is merged and
can be avoided. For example, assume C/C++ application using D
library. This can be done by writing D main function which
forwards call to C/C++ main function which uses D shared library
code. Again, sorry for code post, dpaste isn't working:
---main.cpp---
extern "C" void foo(void);

#include <stdio.h>

class CPP
{
public:
     CPP() { printf("CPP ctor\n"); }
};

CPP cpp;

extern "C" int c_main_func()
{
     printf("C main function reached\n");
     foo();
     // C/C++ application
     return 0;
}
---dmain.d---
import std.stdio;

extern(C) int c_main_func();

static this()
{
     writeln("main module ctor");
}

static ~this()
{
     writeln("main module dtor");
}

int main()
{
     return c_main_func();
}
---test.d---
import std.stdio;

static this()
{
     writeln("lib module ctor");
}

static ~this()
{
     writeln("lib module dtor");
}

extern(C) void foo()
{
     auto a = new A("data");
     writeln("in foo");

}

class A
{
     string x;
     static this() { writeln("static ctor"); }
     static ~this() { writeln("static dtor"); }
     this(string x) { writeln("ctor"); this.x = x; }
     ~this() { writeln("dtor"); writeln(x); }
}
------





-Wl,-rpath=.

CPP ctor
main module ctor
lib module ctor
static ctor
C main function reached
ctor
in foo
static dtor
lib module dtor
main module dtor
dtor
data

	linux-vdso.so.1 (0x00007fff4cd55000)
	libtest.so => ./libtest.so (0x00007f970520b000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9704fd3000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f9704dcb000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f9704a26000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f970540e000)
Sep 27 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/27/12 6:26 AM, Maxim Fomin wrote:
[snip]

Thanks! I adapted your code as follows and it works with 2.058 on Centos.

*** lib.d
import std.stdio;

extern(C) int fun()
{
     writeln(", world!");
     return 42;
}

*** main.d
import std.stdio;

extern(C) int fun();

void main()
{
     write("Hello");
     fun();
}

*** Makefile

PHOBOS_PATH=/path/to/phobos/

main: main.o liblib.so
	gcc main.o -llib -lphobos2 -lpthread -lrt -L. -L${PHOBOS_PATH} 
-Wl,-rpath=. -o main

liblib.so: lib.d
	dmd -fPIC -c lib.d
	gcc -shared lib.o -o liblib.so

%.o: %.d
	dmd -c $<

clean:
	rm -f main main.o lib.o liblib.so

Building the makefile and running ./main prints the classic message. 
Thanks all for answering, this will get us going.


Andrei
Sep 27 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-09-27 20:25, Andrei Alexandrescu wrote:

 On 9/27/12 6:26 AM, Maxim Fomin wrote:
 [snip]

 Thanks! I adapted your code as follows and it works with 2.058 on Centos.


I seriously doubt that everything is working properly, have a look at my 
reply to Maxim Fomin: 
http://forum.dlang.org/thread/k3vfm9$1tq$1 digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com

-- 
/Jacob Carlborg
Sep 27 2012
prev sibling parent reply "nazriel" <spam dzfl.pl> writes:
On Thursday, 27 September 2012 at 08:03:34 UTC, Maxim Fomin wrote:

 On Thursday, 27 September 2012 at 05:52:44 UTC, Jens Mueller
 wrote:

 Maxim Fomin wrote:

 You can build shared libraries on linux by manually compiling 
 object
 files and linking them. On windows last time I tries it was 
 not
 possible.


 Can you give detailed steps for doing this on Linux? Because 
 nobody as
 far as I know has made this work yet?

 Jens


 Dpaste seems not working, so, sorry for code
[snip]


Hmm, that looks nice, seems that stuff improved from last time I 
checked.

Gonna test this out today. I have project that would gain from 
"Hot swapping" of Shared libraries. Temporarily solution I made 
was just writing plugins in C++.

Btw, sorry for OT.
What exactly doesn't work in Dpaste? It seems to work fine for 
me(TM).
If those are UI glitches, try pressing F5 2-3 times ;D
Sep 27 2012
parent "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Thursday, 27 September 2012 at 13:00:24 UTC, nazriel wrote:

 Btw, sorry for OT.
 What exactly doesn't work in Dpaste? It seems to work fine for 
 me(TM).
 If those are UI glitches, try pressing F5 2-3 times ;D


Dpaste was completely down when I tried to post code. It was in 
the middle of today.
Sep 27 2012
prev sibling parent reply "Daniel Kozak" <kozzi11 gmail.com> writes:
Hi,

I have same issue, but it is possible make shared library,
first of all you have to make shared variant of druntime and 
phobos library, than it should work ok.

Now I am at work, when I come back home I will post some more 
details about this.

Daniel Kozak

On Wednesday, 26 September 2012 at 17:57:29 UTC, Andrei 
Alexandrescu wrote:

 Haven't done any dynamic linking with D and I need to. I'm 
 using dmd 2.058/Linux at work to build and use dynamic 
 libraries. Here's my attempt:

 *** Makefile
 all: main lib.so

 main: main.d
 	dmd main

 lib.so: lib.d
 	dmd -fPIC -shared lib.d -of./lib.so

 *** lib.d
 extern(C) int fun(string s)
 {
     return 42;
 }

 *** main.d
 import std.stdio;
 void main()
 {
 }

 Running make prints:

 dmd -fPIC -shared lib.d -of./lib.so
 /usr/bin/ld: 
 /mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/li
phobos2.a(minfo.o): 
 relocation R_X86_64_32 against `a local symbol' can not be used 
 when making a shared object; recompile with -fPIC
 /mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/rele
se/64/libphobos2.a: 
 could not read symbols: Bad value
 collect2: ld returned 1 exit status

 What steps do I need to take to get off the ground?


 Thanks,

 Andrei
Sep 26 2012
parent reply "Daniel Kozak" <kozzi11 gmail.com> writes:
Now I try it, and it is not required to build shared variant of 
druntime and phobos, only rebuild it with -fPIC



On Thursday, 27 September 2012 at 06:12:38 UTC, Daniel Kozak 
wrote:

 Hi,

 I have same issue, but it is possible make shared library,
 first of all you have to make shared variant of druntime and 
 phobos library, than it should work ok.

 Now I am at work, when I come back home I will post some more 
 details about this.

 Daniel Kozak

 On Wednesday, 26 September 2012 at 17:57:29 UTC, Andrei 
 Alexandrescu wrote:

 Haven't done any dynamic linking with D and I need to. I'm 
 using dmd 2.058/Linux at work to build and use dynamic 
 libraries. Here's my attempt:

 *** Makefile
 all: main lib.so

 main: main.d
 	dmd main

 lib.so: lib.d
 	dmd -fPIC -shared lib.d -of./lib.so

 *** lib.d
 extern(C) int fun(string s)
 {
    return 42;
 }

 *** main.d
 import std.stdio;
 void main()
 {
 }

 Running make prints:

 dmd -fPIC -shared lib.d -of./lib.so
 /usr/bin/ld: 
 /mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/li
phobos2.a(minfo.o): 
 relocation R_X86_64_32 against `a local symbol' can not be 
 used when making a shared object; recompile with -fPIC
 /mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/rele
se/64/libphobos2.a: 
 could not read symbols: Bad value
 collect2: ld returned 1 exit status

 What steps do I need to take to get off the ground?


 Thanks,

 Andrei
Sep 26 2012
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 27 Sep 2012 08:26:36 +0200
schrieb "Daniel Kozak" <kozzi11 gmail.com>:


 Now I try it, and it is not required to build shared variant of 
 druntime and phobos, only rebuild it with -fPIC
 


In the end you'll probably need a shared druntime & phobos: Let's say
your main app doesn't use std.datetime and you statically link
libphobos. Then the linker might strip std.datetime from your
executable. If your shared library now needs std.datetime it won't
work.

The funny part is that on ARM+GDC we can at least build druntime &
phobos as a shared library, but on x86/64 last time I checked it didn't
work because of non-PIC assembly.

https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support
Sep 27 2012
parent reply "Rob T" <rob ucora.com> writes:
On Thursday, 27 September 2012 at 07:54:29 UTC, Johannes Pfau 
wrote:

 Am Thu, 27 Sep 2012 08:26:36 +0200
 schrieb "Daniel Kozak" <kozzi11 gmail.com>:


 Now I try it, and it is not required to build shared variant 
 of druntime and phobos, only rebuild it with -fPIC
 


 In the end you'll probably need a shared druntime & phobos: 
 Let's say
 your main app doesn't use std.datetime and you statically link
 libphobos. Then the linker might strip std.datetime from your
 executable. If your shared library now needs std.datetime it 
 won't
 work.


I'm using C++ main app that dynamically loads shared libs at 
runtime, but I cannot load D shared libs because they cannot be 
linked to the non-fPIC built phobos2/druntime lib. It seems that 
the only way to get things working is to re-build the C++ code so 
that it is linked with non-fPIC phobos2, but this is not 
something I want to be doing.

So if I read this right, I can build phobos/druntime with -fPIC 
from the source code and it will work OK? If so, then why was 
this not done with the latest distrubution package, or is this 
only possible after a certain version number?

--rt
Sep 27 2012
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-09-28 02:04, Rob T wrote:


 I'm using C++ main app that dynamically loads shared libs at runtime,
 but I cannot load D shared libs because they cannot be linked to the
 non-fPIC built phobos2/druntime lib. It seems that the only way to get
 things working is to re-build the C++ code so that it is linked with
 non-fPIC phobos2, but this is not something I want to be doing.

 So if I read this right, I can build phobos/druntime with -fPIC from the
 source code and it will work OK? If so, then why was this not done with
 the latest distrubution package, or is this only possible after a
 certain version number?


No, it will not work ok. Depending on what you do in the D code it might 
work, but everything will not work. Perhaps if you limit yourself to a C 
subset. See one of my other posts:

http://forum.dlang.org/thread/k3vfm9$1tq$1 digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com

-- 
/Jacob Carlborg
Sep 27 2012
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 28 Sep 2012 02:04:11 +0200
schrieb "Rob T" <rob ucora.com>:


 
 So if I read this right, I can build phobos/druntime with -fPIC 
 from the source code and it will work OK? If so, then why was 
 this not done with the latest distrubution package, or is this 
 only possible after a certain version number?
 
 --rt


It shouldn't compile on x86 (32bit) with -fPIC because there's some
incompatible asm code in druntime and phobos. If dmd doesn't warn in
these cases, this code will fail as soon as it's called.

GDC used to warn about these errors, but as we dropped the D inline
assembler that code doesn't affect us anymore and there are no more
warnings.

The fix for this issue is simple: Do not modify the EBX register, or at
least save & restore it.

Old patch, which shows what needs to be done:
https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support#comment-648329

Example asm code which probably doesn't work:
https://github.com/D-Programming-Language/druntime/blob/master/src/core/cpuid.d#L432


I just tried building a shared druntime & phobos with gdc:
Druntime seems to be working, Phobos doesn't link here because
something's wrong with the fstat64 symbol. I haven't done any real
testing and we probably need some additional support in the runtime, but
the compiler part seems to be OK with gdc.
http://gdcproject.org/wiki/SharedRuntime
https://github.com/D-Programming-GDC/gdc/pull/32
Sep 28 2012
next sibling parent Iain Buclaw <ibuclaw ubuntu.com> writes:
On 28 September 2012 12:14, Johannes Pfau <nospam example.com> wrote:

 Am Fri, 28 Sep 2012 02:04:11 +0200
 schrieb "Rob T" <rob ucora.com>:


 So if I read this right, I can build phobos/druntime with -fPIC
 from the source code and it will work OK? If so, then why was
 this not done with the latest distrubution package, or is this
 only possible after a certain version number?

 --rt


 It shouldn't compile on x86 (32bit) with -fPIC because there's some
 incompatible asm code in druntime and phobos. If dmd doesn't warn in
 these cases, this code will fail as soon as it's called.

 GDC used to warn about these errors, but as we dropped the D inline
 assembler that code doesn't affect us anymore and there are no more
 warnings.

 The fix for this issue is simple: Do not modify the EBX register, or at
 least save & restore it.


In GDC D1 phobos, one hack around it was to use db to manually write
out the bytecode instruction (thus bypassing GCC 'ebx' clobbered
checks).  Which was a fiendish workaround to say the least. :~)


-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';
Sep 28 2012
prev sibling parent reply "Rob T" <rob ucora.com> writes:
I am trying out gdc 4.7 branch with -fPIC, but the info I'm 
getting is that even with a successulf PIC build I will still not 
be able to reliably execute D functions directly from C/C++ code.


 Old patch, which shows what needs to be done:
 https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support#comment-648329


Does anyone know why patches like this are taking so [bleeping] 
long to get recognized and implemented?

-rt
Sep 28 2012
parent reply Iain Buclaw <ibuclaw ubuntu.com> writes:
On 28 September 2012 17:43, Rob T <rob ucora.com> wrote:

 I am trying out gdc 4.7 branch with -fPIC, but the info I'm getting is that
 even with a successulf PIC build I will still not be able to reliably
 execute D functions directly from C/C++ code.



 Old patch, which shows what needs to be done:

 https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support#comment-648329



 Does anyone know why patches like this are taking so [bleeping] long to get
 recognized and implemented?

 -rt


Because the [bleeping] runtime does not support running from a shared library.

-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';
Sep 28 2012
parent reply "Rob T" <rob ucora.com> writes:
On Friday, 28 September 2012 at 17:07:16 UTC, Iain Buclaw wrote:

 Because the [bleeping] runtime does not support running from a 
 shared library.


I suppose the answer is very complicated, but why can't the 
runtime execute as a shared lib? Is it a design limitation of the 
runtime model? This sort of problem should have been dealt with 
from the ground up, not now. What can be done to fix it, or is it 
a major redesign + re-write effort?

-rt
Sep 28 2012
next sibling parent reply Iain Buclaw <ibuclaw ubuntu.com> writes:
On 28 September 2012 18:54, Rob T <rob ucora.com> wrote:

 On Friday, 28 September 2012 at 17:07:16 UTC, Iain Buclaw wrote:

 Because the [bleeping] runtime does not support running from a shared
 library.



 I suppose the answer is very complicated, but why can't the runtime execute
 as a shared lib? Is it a design limitation of the runtime model? This sort
 of problem should have been dealt with from the ground up, not now. What can
 be done to fix it, or is it a major redesign + re-write effort?

 -rt


The big implementation issue is that the runtime only passes two data
ranges to the GC to scan. Global data segment and TLS data segment of
the resultant binary/executable.  So any static data stored in shared
libraries aren't scanned / recognised as data we want to keep. So the
use of them may cause sudden unwanted collections.

The way I intend to address it is to have each module handle it's own
gshared/thread data and pass the information to the D runtime during
the module construction stage (.ctor) - there is already something
done this way for _Dmodule_ref - so it may end up being that two new
fields will be tacked onto it; void[] tlsdata, void[] gsharedata;


Regards
-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';
Sep 28 2012
parent "Rob T" <rob ucora.com> writes:
On Friday, 28 September 2012 at 21:26:47 UTC, Iain Buclaw wrote:

 The way I intend to address it is to have each module handle 
 it's own
 gshared/thread data and pass the information to the D runtime 
 during
 the module construction stage (.ctor) - there is already 
 something
 done this way for _Dmodule_ref - so it may end up being that 
 two new
 fields will be tacked onto it; void[] tlsdata, void[] 
 gsharedata;


I re-built libgphobos and libgdruntime with -fPIC and I can now 
successfully create dynamically loaded D libs. I have 
successfully linked a dynamic D lib to a C++ app, executing some 
test code successfully. I have not yet tried to dlopen a dynamic 
D lib from C++, but I will try maybe tomorrow.

My simple dynamic lib test seems to run fine, but I understand 
that there may be problems, such as the GC failing to work 
properly, and perhaps more can/will go wrong as per Jacob's post
http://forum.dlang.org/post/k4219f$uft$1 digitalmars.com

Any idea when/if you will get around to implementing a fix? Wish 
I could help but I've only just started looking at the source 
code, so whatever I try to fix will probably cause more harm than 
good for a while (but it's a start).

--rt
Sep 30 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-09-28 19:54, Rob T wrote:


 I suppose the answer is very complicated, but why can't the runtime
 execute as a shared lib? Is it a design limitation of the runtime model?
 This sort of problem should have been dealt with from the ground up, not
 now. What can be done to fix it, or is it a major redesign + re-write
 effort?


See one of my other posts:

http://forum.dlang.org/thread/k3vfm9$1tq$1 digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com

-- 
/Jacob Carlborg
Sep 29 2012
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/27/12 2:26 AM, Daniel Kozak wrote:

 Now I try it, and it is not required to build shared variant of druntime
 and phobos, only rebuild it with -fPIC


Could you please send a troika composed of one dynlib, one loader using 
it, and a makefile that puts the all together?

Thanks much!


Andrei
Sep 27 2012