www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D shared libraries

reply BB <b.buderman gmail.com> writes:
Tried this on D.gnu but didn't get an answer.  Any feedback here?
------------------------------------------------------------------


 From what I read on newsgroups, this should be possible with gdc on 
linux?  Goal is a plugin type solution.  With the following code, I 
compile intf.d and lib.d together into a shared library, and intf.d and 
main.d into an executable.  Compiles/links fine.  When I run it, I get a 
message like:

Null library handle: libxyz.so.1.0.1: undefined symbol: __data_start

I compiled the .d files with -fPIC and the .so with:
gcc -shared -Wl,-soname,libxyz.so.1 -fPIC.

Is this definitely possible, and if so, what am I doing wrong?  I tried 
gdc 0.24 and the latest off svn with the same results.

TIA.

-------------------------------------------------------------

intf.d:
=========
module intf;
public interface I { public int getId(); }

lib.d:
=========
module lib;
import intf;
class A : I { public int getId() { return 42; } }
extern (C) { I getObj() { return new A(); } }

main.d:
==========
...
void main(char[][] args)
{
     void* hnd = dlopen(toStringz("libxyz.so.1.0.1"), RTLD_NOW);
     if (hnd == null) {
         printf("Null library handle: %s\n", dlerror());
         return 0;
     }
     ...
}
Apr 11 2008
next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
BB wrote:

 Tried this on D.gnu but didn't get an answer.  Any feedback here?


Shared libraries are also supported by the dmd compiler (the -fPIC 
switch is implemented), but not by phobos, simply because nobody has 
gotten around to doing it.
Apr 11 2008
parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
Shared libraries are quite popular.  To be honest, I'm no expert with 
them or I would try my hand (I actually did a bit, but I really don't 
have much experience with linking problems.)

Would you have any suggestions on what one might do to better understand 
the problems Phobos currently has with supporting shared libraries?  If 
I understood the problems, I would be happy to work on a solution.

Also, since it's a bit related - what is your opinion on the concept of ddl?

It (the concept; ddl doesn't work at all for me no matter what I try, 
and seems overcomplicated in ways) seems like a perfect way to bridge 
yet another gap between D and other popular dynamic languages like Ruby, 
Python, Perl, PHP, etc.: ease of importing/using shared code dynamically.

-[Unknown]


Walter Bright wrote:

 BB wrote:

 Tried this on D.gnu but didn't get an answer.  Any feedback here?


 
 Shared libraries are also supported by the dmd compiler (the -fPIC 
 switch is implemented), but not by phobos, simply because nobody has 
 gotten around to doing it.
Apr 12 2008
next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
Assuming you just want to create a shared library of your own, initialization of
the runtime is an issue.  I believe there's a doc page somewhere that describes
how to do this for Phobos, and Tango has rt_init() ad rt_term() routines for
this
purpose.  If the issue is putting just Phobos into a shared lib, changes would
have to be made for how the GC is exposed, and on Windows most things
would need to be exported as well.  This is a pretty big and nasty job (for
Win32 anyway).

== Quote from Unknown W. Brackets (unknown simplemachines.org)'s article

 Shared libraries are quite popular.  To be honest, I'm no expert with
 them or I would try my hand (I actually did a bit, but I really don't
 have much experience with linking problems.)
 Would you have any suggestions on what one might do to better understand
 the problems Phobos currently has with supporting shared libraries?  If
 I understood the problems, I would be happy to work on a solution.
 Also, since it's a bit related - what is your opinion on the concept of ddl?
 It (the concept; ddl doesn't work at all for me no matter what I try,
 and seems overcomplicated in ways) seems like a perfect way to bridge
 yet another gap between D and other popular dynamic languages like Ruby,
 Python, Perl, PHP, etc.: ease of importing/using shared code dynamically.
 -[Unknown]
 Walter Bright wrote:

 BB wrote:

 Tried this on D.gnu but didn't get an answer.  Any feedback here?


 Shared libraries are also supported by the dmd compiler (the -fPIC
 switch is implemented), but not by phobos, simply because nobody has
 gotten around to doing it.
Apr 12 2008
parent "Unknown W. Brackets" <unknown simplemachines.org> writes:
Windows is a huge ordeal.  I'm more interested right now in fixing it on 
Linux (I do not write many Windows programs anyway, and the differences 
between so and dll are many.)

I realize there are runtime issues, but I'm concerned mainly about the 
reason why dmd on Linux is unable to create shared objects, while gdc 
can (although there are many complications.)

-[Unknown]


Sean Kelly wrote:

 Assuming you just want to create a shared library of your own, initialization
of
 the runtime is an issue.  I believe there's a doc page somewhere that describes
 how to do this for Phobos, and Tango has rt_init() ad rt_term() routines for
this
 purpose.  If the issue is putting just Phobos into a shared lib, changes would
 have to be made for how the GC is exposed, and on Windows most things
 would need to be exported as well.  This is a pretty big and nasty job (for
 Win32 anyway).
 
 == Quote from Unknown W. Brackets (unknown simplemachines.org)'s article

 Shared libraries are quite popular.  To be honest, I'm no expert with
 them or I would try my hand (I actually did a bit, but I really don't
 have much experience with linking problems.)
 Would you have any suggestions on what one might do to better understand
 the problems Phobos currently has with supporting shared libraries?  If
 I understood the problems, I would be happy to work on a solution.
 Also, since it's a bit related - what is your opinion on the concept of ddl?
 It (the concept; ddl doesn't work at all for me no matter what I try,
 and seems overcomplicated in ways) seems like a perfect way to bridge
 yet another gap between D and other popular dynamic languages like Ruby,
 Python, Perl, PHP, etc.: ease of importing/using shared code dynamically.
 -[Unknown]
 Walter Bright wrote:

 BB wrote:

 Tried this on D.gnu but didn't get an answer.  Any feedback here?


 Shared libraries are also supported by the dmd compiler (the -fPIC
 switch is implemented), but not by phobos, simply because nobody has
 gotten around to doing it.
Apr 12 2008
prev sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Unknown W. Brackets wrote:

 Shared libraries are quite popular.  To be honest, I'm no expert with 
 them or I would try my hand (I actually did a bit, but I really don't 
 have much experience with linking problems.)
 
 Would you have any suggestions on what one might do to better understand 
 the problems Phobos currently has with supporting shared libraries?  If 
 I understood the problems, I would be happy to work on a solution.



I'd start by examining how DLL support is done for Win32 (i.e. having a 
shared gc and threading system).



 Also, since it's a bit related - what is your opinion on the concept of 
 ddl?
 
 It (the concept; ddl doesn't work at all for me no matter what I try, 
 and seems overcomplicated in ways) seems like a perfect way to bridge 
 yet another gap between D and other popular dynamic languages like Ruby, 
 Python, Perl, PHP, etc.: ease of importing/using shared code dynamically.


I don't know anything about ddl.
Apr 12 2008
parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
Well, I'm more talking about why DMD cannot produce shared libraries 
(using Phobos) on Linux.  It seems to me the problems for Win32 are larger.

As for ddl - essentially, a runtime linker.  Given an obj file, it links 
it against the running program (provided a file listing its references) 
and provides methods to pull its symbols.

This allows libraries to function on Windows similarly to so files 
(reverse linking.)  I believe mingw32 is actually doing something similar.

-[Unknown]


Walter Bright wrote:

 Unknown W. Brackets wrote:

 Shared libraries are quite popular.  To be honest, I'm no expert with 
 them or I would try my hand (I actually did a bit, but I really don't 
 have much experience with linking problems.)

 Would you have any suggestions on what one might do to better 
 understand the problems Phobos currently has with supporting shared 
 libraries?  If I understood the problems, I would be happy to work on 
 a solution.


 
 
 I'd start by examining how DLL support is done for Win32 (i.e. having a 
 shared gc and threading system).
 
 

 Also, since it's a bit related - what is your opinion on the concept 
 of ddl?

 It (the concept; ddl doesn't work at all for me no matter what I try, 
 and seems overcomplicated in ways) seems like a perfect way to bridge 
 yet another gap between D and other popular dynamic languages like 
 Ruby, Python, Perl, PHP, etc.: ease of importing/using shared code 
 dynamically.


 
 I don't know anything about ddl.
Apr 12 2008
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Unknown W. Brackets wrote:

 Well, I'm more talking about why DMD cannot produce shared libraries 
 (using Phobos) on Linux.


DMD can do it. It's just that nobody has fixed Phobos to do it.
Apr 12 2008
parent reply Georg Wrede <georg nospam.org> writes:
Walter Bright wrote:

 Unknown W. Brackets wrote:
 

 Well, I'm more talking about why DMD cannot produce shared libraries 
 (using Phobos) on Linux.


 
 DMD can do it. It's just that nobody has fixed Phobos to do it.


Maybe I'm out of line here, but at this point of this particular thread, 
it would have been prudent to include some remark as to what sort of 
things Phobos might need to become better suited for shared libraries.

Every thread here has more readers than participants, and we must assume 
quite some of them might even consider doing something about the issue, 
given the smallest hint on where to start. Right?
Apr 13 2008
parent "Unknown W. Brackets" <unknown simplemachines.org> writes:
(like me for example, who has decided Walter isn't interested in giving 
him a place to start on making Phobos on Linux capable of generating 
shared objects.)

-[Unknown]


Georg Wrede wrote:

 Walter Bright wrote:

 Unknown W. Brackets wrote:


 Well, I'm more talking about why DMD cannot produce shared libraries 
 (using Phobos) on Linux.


 DMD can do it. It's just that nobody has fixed Phobos to do it.


 
 Maybe I'm out of line here, but at this point of this particular thread, 
 it would have been prudent to include some remark as to what sort of 
 things Phobos might need to become better suited for shared libraries.
 
 Every thread here has more readers than participants, and we must assume 
 quite some of them might even consider doing something about the issue, 
 given the smallest hint on where to start. Right?
Apr 13 2008
prev sibling parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
To compile a shared library, Linux using gdc:

gdmd -op -oflibxyz.so.1.0.1 lib.d -fPIC -q,-rdynamic,-shared
gdmd -op main.d -fPIC -q,-rdynamic -L-ldl

Compiling a shared library on Windows is a bit more complicated 
(although possible with dmd and phobos as they are.)  It involves 
DllMain and def files...

Please note, if you have selinux you may have have to run chcon.

-[Unknown]


BB wrote:

 Tried this on D.gnu but didn't get an answer.  Any feedback here?
 ------------------------------------------------------------------
 
 
  From what I read on newsgroups, this should be possible with gdc on 
 linux?  Goal is a plugin type solution.  With the following code, I 
 compile intf.d and lib.d together into a shared library, and intf.d and 
 main.d into an executable.  Compiles/links fine.  When I run it, I get a 
 message like:
 
 Null library handle: libxyz.so.1.0.1: undefined symbol: __data_start
 
 I compiled the .d files with -fPIC and the .so with:
 gcc -shared -Wl,-soname,libxyz.so.1 -fPIC.
 
 Is this definitely possible, and if so, what am I doing wrong?  I tried 
 gdc 0.24 and the latest off svn with the same results.
 
 TIA.
 
 -------------------------------------------------------------
 
 intf.d:
 =========
 module intf;
 public interface I { public int getId(); }
 
 lib.d:
 =========
 module lib;
 import intf;
 class A : I { public int getId() { return 42; } }
 extern (C) { I getObj() { return new A(); } }
 
 main.d:
 ==========
 ...
 void main(char[][] args)
 {
     void* hnd = dlopen(toStringz("libxyz.so.1.0.1"), RTLD_NOW);
     if (hnd == null) {
         printf("Null library handle: %s\n", dlerror());
         return 0;
     }
     ...
 }
Apr 12 2008
parent reply BB <b.buderman gmail.com> writes:
Thanks very much, that worked.

After being able to call into the .so, I see a segmentation fault at the 
end of main().  Through gdb I see it's crashing in _d_callfinalizer(). 
Locally explicitly deleting the object allocated by the .so also results 
in a sigsegv.  However, calling into the .so to delete the object works 
fine.

What is the reasoning for this - is it that the gc is statically linked 
into both the .so and the executable, so sharing a reference causes the 
local gc to try and clean it up?  If so, what are the guidelines/rules 
to use?

Thanks again.


Unknown W. Brackets wrote:

 To compile a shared library, Linux using gdc:
 
 gdmd -op -oflibxyz.so.1.0.1 lib.d -fPIC -q,-rdynamic,-shared
 gdmd -op main.d -fPIC -q,-rdynamic -L-ldl
 
 Compiling a shared library on Windows is a bit more complicated 
 (although possible with dmd and phobos as they are.) It involves DllMain 
 and def files...
 
 Please note, if you have selinux you may have have to run chcon.
 
 -[Unknown]
 
 
 BB wrote:

 Tried this on D.gnu but didn't get an answer. Any feedback here?
 ------------------------------------------------------------------


 From what I read on newsgroups, this should be possible with gdc on 
 linux? Goal is a plugin type solution. With the following code, I 
 compile intf.d and lib.d together into a shared library, and intf.d 
 and main.d into an executable. Compiles/links fine. When I run it, I 
 get a message like:

 Null library handle: libxyz.so.1.0.1: undefined symbol: __data_start

 I compiled the .d files with -fPIC and the .so with:
 gcc -shared -Wl,-soname,libxyz.so.1 -fPIC.

 Is this definitely possible, and if so, what am I doing wrong? I tried 
 gdc 0.24 and the latest off svn with the same results.

 TIA.

 -------------------------------------------------------------

 intf.d:
 =========
 module intf;
 public interface I { public int getId(); }

 lib.d:
 =========
 module lib;
 import intf;
 class A : I { public int getId() { return 42; } }
 extern (C) { I getObj() { return new A(); } }

 main.d:
 ==========
 ...
 void main(char[][] args)
 {
 void* hnd = dlopen(toStringz("libxyz.so.1.0.1"), RTLD_NOW);
 if (hnd == null) {
 printf("Null library handle: %s\n", dlerror());
 return 0;
 }
 ...
 }
Apr 12 2008
next sibling parent Sean Kelly <sean invisibleduck.org> writes:
Did you initialize the runtime within the shared library?  As for sharing
data between a D shared library and external code, if you want to pass
ownership of GCed data to the external code then the GC will have to
know either that it should just never collect the data or have a means
of scanning external thread stacks.  Tango opted for the second approach,
and you can register external threads via thread_attachThis() and
thread_detachThis() -- both are declared in the Thread module.

== Quote from BB (b.buderman gmail.com)'s article

 Thanks very much, that worked.
 After being able to call into the .so, I see a segmentation fault at the
 end of main().  Through gdb I see it's crashing in _d_callfinalizer().
 Locally explicitly deleting the object allocated by the .so also results
 in a sigsegv.  However, calling into the .so to delete the object works
 fine.
 What is the reasoning for this - is it that the gc is statically linked
 into both the .so and the executable, so sharing a reference causes the
 local gc to try and clean it up?  If so, what are the guidelines/rules
 to use?
 Thanks again.
 Unknown W. Brackets wrote:

 To compile a shared library, Linux using gdc:

 gdmd -op -oflibxyz.so.1.0.1 lib.d -fPIC -q,-rdynamic,-shared
 gdmd -op main.d -fPIC -q,-rdynamic -L-ldl

 Compiling a shared library on Windows is a bit more complicated
 (although possible with dmd and phobos as they are.) It involves DllMain
 and def files...

 Please note, if you have selinux you may have have to run chcon.

 -[Unknown]


 BB wrote:

 Tried this on D.gnu but didn't get an answer. Any feedback here?
 ------------------------------------------------------------------


 From what I read on newsgroups, this should be possible with gdc on
 linux? Goal is a plugin type solution. With the following code, I
 compile intf.d and lib.d together into a shared library, and intf.d
 and main.d into an executable. Compiles/links fine. When I run it, I
 get a message like:

 Null library handle: libxyz.so.1.0.1: undefined symbol: __data_start

 I compiled the .d files with -fPIC and the .so with:
 gcc -shared -Wl,-soname,libxyz.so.1 -fPIC.

 Is this definitely possible, and if so, what am I doing wrong? I tried
 gdc 0.24 and the latest off svn with the same results.

 TIA.

 -------------------------------------------------------------

 intf.d:
 =========
 module intf;
 public interface I { public int getId(); }

 lib.d:
 =========
 module lib;
 import intf;
 class A : I { public int getId() { return 42; } }
 extern (C) { I getObj() { return new A(); } }

 main.d:
 ==========
 ...
 void main(char[][] args)
 {
 void* hnd = dlopen(toStringz("libxyz.so.1.0.1"), RTLD_NOW);
 if (hnd == null) {
 printf("Null library handle: %s\n", dlerror());
 return 0;
 }
 ...
 }
Apr 12 2008
prev sibling parent "Unknown W. Brackets" <unknown simplemachines.org> writes:
I seem to remember they, essentially, share a GC.  I do think they can 
get confused/trip over each other if you're not careful.

Generally, I've just had two functions I call into the so with, which 
init/deinit the library.  To reduce complications and problems like 
you're seeing, I keep the plugins as simple as possible.

IIRC, they share the same gc handle already.  It's possible they both 
may run the gc when they shutdown.  I'd use the functions in std.gc and 
assembler output to debug...

Unfortunately this is one area where D (mostly in its available tools) 
is lacking currently...

-[Unknown]


BB wrote:

 Thanks very much, that worked.
 
 After being able to call into the .so, I see a segmentation fault at the 
 end of main().  Through gdb I see it's crashing in _d_callfinalizer(). 
 Locally explicitly deleting the object allocated by the .so also results 
 in a sigsegv.  However, calling into the .so to delete the object works 
 fine.
 
 What is the reasoning for this - is it that the gc is statically linked 
 into both the .so and the executable, so sharing a reference causes the 
 local gc to try and clean it up?  If so, what are the guidelines/rules 
 to use?
 
 Thanks again.
 
 
 Unknown W. Brackets wrote:

 To compile a shared library, Linux using gdc:

 gdmd -op -oflibxyz.so.1.0.1 lib.d -fPIC -q,-rdynamic,-shared
 gdmd -op main.d -fPIC -q,-rdynamic -L-ldl

 Compiling a shared library on Windows is a bit more complicated 
 (although possible with dmd and phobos as they are.) It involves 
 DllMain and def files...

 Please note, if you have selinux you may have have to run chcon.

 -[Unknown]


 BB wrote:

 Tried this on D.gnu but didn't get an answer. Any feedback here?
 ------------------------------------------------------------------


 From what I read on newsgroups, this should be possible with gdc on 
 linux? Goal is a plugin type solution. With the following code, I 
 compile intf.d and lib.d together into a shared library, and intf.d 
 and main.d into an executable. Compiles/links fine. When I run it, I 
 get a message like:

 Null library handle: libxyz.so.1.0.1: undefined symbol: __data_start

 I compiled the .d files with -fPIC and the .so with:
 gcc -shared -Wl,-soname,libxyz.so.1 -fPIC.

 Is this definitely possible, and if so, what am I doing wrong? I 
 tried gdc 0.24 and the latest off svn with the same results.

 TIA.

 -------------------------------------------------------------

 intf.d:
 =========
 module intf;
 public interface I { public int getId(); }

 lib.d:
 =========
 module lib;
 import intf;
 class A : I { public int getId() { return 42; } }
 extern (C) { I getObj() { return new A(); } }

 main.d:
 ==========
 ...
 void main(char[][] args)
 {
 void* hnd = dlopen(toStringz("libxyz.so.1.0.1"), RTLD_NOW);
 if (hnd == null) {
 printf("Null library handle: %s\n", dlerror());
 return 0;
 }
 ...
 }
Apr 12 2008