www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Problems with shared library (-fPIC) and tango

reply Marius Muja <mariusm cs.ubc.ca> writes:
Hi,

I'm working on a project that needs to be compiled both as a standalone 
executable and (part of it) as a shared library. It's been working like 
that on 32 bit machines without any problems, but now I need to compile 
it on a 64 bit machine (to be able to access more the 3GB of memory).

When I tried to build the project on the 64 bit machine I was unable to 
build the shared library because of the following error:

libgphobos.a(ti_AC.o): relocation R_X86_64_32 against 
`_D17TypeInfo_C6Object6__initZ' can not be used when making a shared 
object; recompile with -fPIC

Since the error refers to an object in the standard library (I'm using 
tango) I tried to build the tango library using -fPIC. Doing that I 
didn't get the above error when I build the project, but it get the 
following assertion when I run it:

tango.core.Exception.AssertException lifetime.d(514): Assertion failure

This makes me think that the tango library doesn't work properly when 
compiled with -fPIC.

Does anyone know a solution to this problem? Is it possible to build a D 
shared library on a 64 bit machine when using tango?


Marius
Jul 04 2008
next sibling parent reply Lars Ivar Igesund <larsivar igesund.net> writes:
Marius Muja wrote:


 Hi,
 
 I'm working on a project that needs to be compiled both as a standalone
 executable and (part of it) as a shared library. It's been working like
 that on 32 bit machines without any problems, but now I need to compile
 it on a 64 bit machine (to be able to access more the 3GB of memory).
 
 When I tried to build the project on the 64 bit machine I was unable to
 build the shared library because of the following error:
 
 libgphobos.a(ti_AC.o): relocation R_X86_64_32 against
 `_D17TypeInfo_C6Object6__initZ' can not be used when making a shared
 object; recompile with -fPIC
 
 Since the error refers to an object in the standard library (I'm using
 tango) I tried to build the tango library using -fPIC. Doing that I
 didn't get the above error when I build the project, but it get the
 following assertion when I run it:
 
 tango.core.Exception.AssertException lifetime.d(514): Assertion failure
 
 This makes me think that the tango library doesn't work properly when
 compiled with -fPIC.
 
 Does anyone know a solution to this problem? Is it possible to build a D
 shared library on a 64 bit machine when using tango?
 
 
 Marius


There is a ticket related to this somewhere. I believe I was told once that
part of the problem is that _all_ of the runtime can't be part of the
dynamic library, but that there needs to be a static portion too with at
least main. The above issue may have more to do with the effect of PIC on
the code though. It would be very good if it could be resolved.

-- 
Lars Ivar Igesund
blog at http://larsivi.net
DSource, #d.tango & #D: larsivi
Dancing the Tango
Jul 05 2008
parent reply e-t172 <e-t172 akegroup.org> writes:
Lars Ivar Igesund a écrit :

 There is a ticket related to this somewhere. I believe I was told once that
 part of the problem is that _all_ of the runtime can't be part of the
 dynamic library, but that there needs to be a static portion too with at
 least main. The above issue may have more to do with the effect of PIC on
 the code though. It would be very good if it could be resolved.


IIRC, there is only one problem when trying to build Tango with -fPIC : 
the garbage collector, which segfaults on collection. Fix the GC and you 
have your Tango shared library.
Jul 05 2008
parent reply Sean Kelly <sean invisibleduck.org> writes:
== Quote from e-t172 (e-t172 akegroup.org)'s article

 Lars Ivar Igesund a écrit :

 There is a ticket related to this somewhere. I believe I was told once that
 part of the problem is that _all_ of the runtime can't be part of the
 dynamic library, but that there needs to be a static portion too with at
 least main. The above issue may have more to do with the effect of PIC on
 the code though. It would be very good if it could be resolved.


 IIRC, there is only one problem when trying to build Tango with -fPIC :
 the garbage collector, which segfaults on collection. Fix the GC and you
 have your Tango shared library.


I believe this is an issue with the static data segments not being represented
in the way that the GC expects in a shared library.  I really need to do some
research to figure out how things are different in this instance.  Though if
someone knows then that would be good too :-)


Sean
Jul 05 2008
next sibling parent reply e-t172 <e-t172 akegroup.org> writes:
Sean Kelly a écrit :

 == Quote from e-t172 (e-t172 akegroup.org)'s article

 Lars Ivar Igesund a écrit :

 There is a ticket related to this somewhere. I believe I was told once that
 part of the problem is that _all_ of the runtime can't be part of the
 dynamic library, but that there needs to be a static portion too with at
 least main. The above issue may have more to do with the effect of PIC on
 the code though. It would be very good if it could be resolved.


 IIRC, there is only one problem when trying to build Tango with -fPIC :
 the garbage collector, which segfaults on collection. Fix the GC and you
 have your Tango shared library.


 
 I believe this is an issue with the static data segments not being represented
 in the way that the GC expects in a shared library.  I really need to do some
 research to figure out how things are different in this instance.  Though if
 someone knows then that would be good too :-)


When I tried to build Tango in a .so (all Tango, not just the runtime), 
there was another (much less important) problem: when the executable 
starts, all the static main() functions of all modules were executed, 
even if I didn't use them. This can be a problem. When I executed my 
"Hello Word", I got messages about cluster initialization...
Jul 06 2008
parent reply Sean Kelly <sean invisibleduck.org> writes:
== Quote from e-t172 (e-t172 akegroup.org)'s article

 Sean Kelly a écrit :

 == Quote from e-t172 (e-t172 akegroup.org)'s article

 Lars Ivar Igesund a écrit :

 There is a ticket related to this somewhere. I believe I was told once that
 part of the problem is that _all_ of the runtime can't be part of the
 dynamic library, but that there needs to be a static portion too with at
 least main. The above issue may have more to do with the effect of PIC on
 the code though. It would be very good if it could be resolved.


 IIRC, there is only one problem when trying to build Tango with -fPIC :
 the garbage collector, which segfaults on collection. Fix the GC and you
 have your Tango shared library.


 I believe this is an issue with the static data segments not being represented
 in the way that the GC expects in a shared library.  I really need to do some
 research to figure out how things are different in this instance.  Though if
 someone knows then that would be good too :-)


 When I tried to build Tango in a .so (all Tango, not just the runtime),
 there was another (much less important) problem: when the executable
 starts, all the static main() functions of all modules were executed,
 even if I didn't use them. This can be a problem. When I executed my
 "Hello Word", I got messages about cluster initialization...


I'm not sure I understand.  The only module with a main() function
should be in the runtime.  Did you mean static this()?  I'll admit
that if so that's still pretty weird.

Sean
Jul 06 2008
parent e-t172 <e-t172 akegroup.org> writes:
Sean Kelly a écrit :

 == Quote from e-t172 (e-t172 akegroup.org)'s article

 Sean Kelly a écrit :

 == Quote from e-t172 (e-t172 akegroup.org)'s article

 Lars Ivar Igesund a écrit :

 There is a ticket related to this somewhere. I believe I was told once that
 part of the problem is that _all_ of the runtime can't be part of the
 dynamic library, but that there needs to be a static portion too with at
 least main. The above issue may have more to do with the effect of PIC on
 the code though. It would be very good if it could be resolved.


 IIRC, there is only one problem when trying to build Tango with -fPIC :
 the garbage collector, which segfaults on collection. Fix the GC and you
 have your Tango shared library.


 I believe this is an issue with the static data segments not being represented
 in the way that the GC expects in a shared library.  I really need to do some
 research to figure out how things are different in this instance.  Though if
 someone knows then that would be good too :-)


 When I tried to build Tango in a .so (all Tango, not just the runtime),
 there was another (much less important) problem: when the executable
 starts, all the static main() functions of all modules were executed,
 even if I didn't use them. This can be a problem. When I executed my
 "Hello Word", I got messages about cluster initialization...


 
 I'm not sure I understand.  The only module with a main() function
 should be in the runtime.  Did you mean static this()?  I'll admit
 that if so that's still pretty weird.


Sorry. I of course meant static this() (module constructors). This was a 
typo.
Jul 06 2008
prev sibling parent reply Thomas Leonard <talex5+d gmail.com> writes:
On Sun, 06 Jul 2008 03:31:44 +0000, Sean Kelly wrote:


 == Quote from e-t172 (e-t172 akegroup.org)'s article

 Lars Ivar Igesund a écrit :

 There is a ticket related to this somewhere. I believe I was told
 once that part of the problem is that _all_ of the runtime can't be
 part of the dynamic library, but that there needs to be a static
 portion too with at least main. The above issue may have more to do
 with the effect of PIC on the code though. It would be very good if
 it could be resolved.


 IIRC, there is only one problem when trying to build Tango with -fPIC :
 the garbage collector, which segfaults on collection. Fix the GC and
 you have your Tango shared library.


 
 I believe this is an issue with the static data segments not being
 represented in the way that the GC expects in a shared library.  I
 really need to do some research to figure out how things are different
 in this instance.  Though if someone knows then that would be good too
 :-)


Could that be why this program crashes?

import std.gc: fullCollect;
import mi = std.moduleinit;

int main(string[] args) {
	//printf("%p\n", mi._moduleinfo_dtors[6]);
	fullCollect();

	return 0;
}

If I uncomment the printf line, it works (I guess because the address 
gets left on the stack). Without calling fullCollect() explicitly, my 
programs work until the GC gets called, then they crash.

I'm using a (rather modified) GDC on Linux, with libgphobos2 as a shared 
library.

How are module variables like _moduleinfo_dtors supposed to get 
registered as roots with the GC?

Thanks,
Oct 23 2008
parent reply Sean Kelly <sean invisibleduck.org> writes:
Thomas Leonard wrote:

 On Sun, 06 Jul 2008 03:31:44 +0000, Sean Kelly wrote:
 

 == Quote from e-t172 (e-t172 akegroup.org)'s article

 Lars Ivar Igesund a écrit :

 There is a ticket related to this somewhere. I believe I was told
 once that part of the problem is that _all_ of the runtime can't be
 part of the dynamic library, but that there needs to be a static
 portion too with at least main. The above issue may have more to do
 with the effect of PIC on the code though. It would be very good if
 it could be resolved.


 IIRC, there is only one problem when trying to build Tango with -fPIC :
 the garbage collector, which segfaults on collection. Fix the GC and
 you have your Tango shared library.


 I believe this is an issue with the static data segments not being
 represented in the way that the GC expects in a shared library.  I
 really need to do some research to figure out how things are different
 in this instance.  Though if someone knows then that would be good too
 :-)


 
 Could that be why this program crashes?
 
 import std.gc: fullCollect;
 import mi = std.moduleinit;
 
 int main(string[] args) {
 	//printf("%p\n", mi._moduleinfo_dtors[6]);
 	fullCollect();
 
 	return 0;
 }
 
 If I uncomment the printf line, it works (I guess because the address 
 gets left on the stack). Without calling fullCollect() explicitly, my 
 programs work until the GC gets called, then they crash.
 
 I'm using a (rather modified) GDC on Linux, with libgphobos2 as a shared 
 library.
 
 How are module variables like _moduleinfo_dtors supposed to get 
 registered as roots with the GC?


The shared data segment(s) of a binary are identified by the runtime at 
startup and the runtime passes them to the GC for scanning.  For 
example, look at initStaticDataPtrs() in this file:

http://dsource.org/projects/tango/browser/trunk/lib/compiler/gdc/memory.d

However, I think the way that the static data segment is identified in a 
shared library is different from in an executable app.  I'm pretty sure 
the Boehm GC handles this properly and have been meaning to look at it 
for clues, but that hasn't made it to the top of my priority list yet.


Sean
Oct 23 2008
parent reply Thomas Leonard <talex5+d gmail.com> writes:
On Thu, 23 Oct 2008 19:59:30 -0700, Sean Kelly wrote:

 Thomas Leonard wrote:

 On Sun, 06 Jul 2008 03:31:44 +0000, Sean Kelly wrote:

 == Quote from e-t172 (e-t172 akegroup.org)'s article

 IIRC, there is only one problem when trying to build Tango with -fPIC
 : the garbage collector, which segfaults on collection. Fix the GC
 and you have your Tango shared library.


 I believe this is an issue with the static data segments not being
 represented in the way that the GC expects in a shared library.  I
 really need to do some research to figure out how things are different
 in this instance.  Though if someone knows then that would be good too
 :-)


 
 Could that be why this program crashes?
 
 import std.gc: fullCollect;
 import mi = std.moduleinit;
 
 int main(string[] args) {
 	//printf("%p\n", mi._moduleinfo_dtors[6]); fullCollect();
 
 	return 0;
 }
 
 If I uncomment the printf line, it works (I guess because the address
 gets left on the stack). Without calling fullCollect() explicitly, my
 programs work until the GC gets called, then they crash.
 
 I'm using a (rather modified) GDC on Linux, with libgphobos2 as a
 shared library.
 
 How are module variables like _moduleinfo_dtors supposed to get
 registered as roots with the GC?


 
 The shared data segment(s) of a binary are identified by the runtime at
 startup and the runtime passes them to the GC for scanning.  For
 example, look at initStaticDataPtrs() in this file:
 
 http://dsource.org/projects/tango/browser/trunk/lib/compiler/gdc/


memory.d

 
 However, I think the way that the static data segment is identified in a
 shared library is different from in an executable app.  I'm pretty sure
 the Boehm GC handles this properly and have been meaning to look at it
 for clues, but that hasn't made it to the top of my priority list yet.


Ah, thanks for the hints. I found some disabled code in gcgcc.d that 
tried to scan /proc/self/maps, and with a bit of tweaking it now works 
for me (tested on x86_64):

http://repo.or.cz/w/delight/core.git?
a=commitdiff;h=83befaa3f72973eb8afd8589d2e1cee4b37ee4fb;hp=03f581cf37409b35521bcfd6f239a14f1fa73221

BTW, would it be possible to get rid of this moduleinfo stuff? It seems 
to cause a lot of problems, because when a program imports a library it 
might not use exactly the same flags that were used when the library was 
compiled (e.g. program is compiled with unit-tests, library wasn't). So 
the program might try to use the moduleinfo, even though it doesn't 
actually exist.

Doesn't the dynamic linker already provide for calling static 
initialisers? I'm not sure.
Oct 24 2008
next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
Thomas Leonard wrote:

 On Thu, 23 Oct 2008 19:59:30 -0700, Sean Kelly wrote:

 Thomas Leonard wrote:

 On Sun, 06 Jul 2008 03:31:44 +0000, Sean Kelly wrote:

 == Quote from e-t172 (e-t172 akegroup.org)'s article

 IIRC, there is only one problem when trying to build Tango with -fPIC
 : the garbage collector, which segfaults on collection. Fix the GC
 and you have your Tango shared library.


 I believe this is an issue with the static data segments not being
 represented in the way that the GC expects in a shared library.  I
 really need to do some research to figure out how things are different
 in this instance.  Though if someone knows then that would be good too
 :-)


 Could that be why this program crashes?

 import std.gc: fullCollect;
 import mi = std.moduleinit;

 int main(string[] args) {
 	//printf("%p\n", mi._moduleinfo_dtors[6]); fullCollect();

 	return 0;
 }

 If I uncomment the printf line, it works (I guess because the address
 gets left on the stack). Without calling fullCollect() explicitly, my
 programs work until the GC gets called, then they crash.

 I'm using a (rather modified) GDC on Linux, with libgphobos2 as a
 shared library.

 How are module variables like _moduleinfo_dtors supposed to get
 registered as roots with the GC?


 The shared data segment(s) of a binary are identified by the runtime at
 startup and the runtime passes them to the GC for scanning.  For
 example, look at initStaticDataPtrs() in this file:

 http://dsource.org/projects/tango/browser/trunk/lib/compiler/gdc/


 memory.d

 However, I think the way that the static data segment is identified in a
 shared library is different from in an executable app.  I'm pretty sure
 the Boehm GC handles this properly and have been meaning to look at it
 for clues, but that hasn't made it to the top of my priority list yet.


 
 Ah, thanks for the hints. I found some disabled code in gcgcc.d that 
 tried to scan /proc/self/maps, and with a bit of tweaking it now works 
 for me (tested on x86_64):
 
 http://repo.or.cz/w/delight/core.git?
 a=commitdiff;h=83befaa3f72973eb8afd8589d2e1cee4b37ee4fb;hp=03f581cf37409b35521bcfd6f239a14f1fa73221


Cool!  I'll see about integrating these changes into druntime.


 BTW, would it be possible to get rid of this moduleinfo stuff? It seems 
 to cause a lot of problems, because when a program imports a library it 
 might not use exactly the same flags that were used when the library was 
 compiled (e.g. program is compiled with unit-tests, library wasn't). So 
 the program might try to use the moduleinfo, even though it doesn't 
 actually exist.


I don't think there's any way around moduleinfo in general, since it's 
the way that static ctors and dtors are called.  However, I'd expect 
everything to work correctly regardless of the flags passed because the 
runtime code simply iterates across a list of ModuleInfo objects and 
calls whatever functions it needs to (after checking if the function 
pointers are non-null).


 Doesn't the dynamic linker already provide for calling static 
 initialisers? I'm not sure.


Possibly the static initializers that C uses, if any, but not the D 
initializers.  However, I've started working on some new runtime 
functions to do dynamic loading and unloading of libraries on druntime, 
and this will do everything needed to properly initialize the library.


Sean
Oct 24 2008
parent =?UTF-8?B?IkrDqXLDtG1lIE0uIEJlcmdlciI=?= <jeberger free.fr> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sean Kelly wrote:

 Thomas Leonard wrote:

 Doesn't the dynamic linker already provide for calling static
 initialisers? I'm not sure.


 
 Possibly the static initializers that C uses, if any, but not the D
 initializers.


	AFAIK, all you need for some code to be run automatically on module
loading/unloading is to put it in the correct section in the object
file or in a properly named function (depending on the linker). For
example, for GNU ld, sections called .init and .fini are executed
upon module loading and unloading respectively. If the D
initializers were put into these, they would be called
automatically. You can look here for more details:
http://www.delorie.com/gnu/docs/gcc/gccint_149.html

		Jerome
<back to lurk mode/>
- --
+------------------------- Jerome M. BERGER ---------------------+
|    mailto:jeberger free.fr      | ICQ:    238062172            |
|    http://jeberger.free.fr/     | Jabber: jeberger jabber.fr   |
+---------------------------------+------------------------------+
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkkC2AMACgkQd0kWM4JG3k88uQCgmW6hYyo93BlaugnUOLzAsisR
hRIAn3AF1nay7Brp347mpZjTbZ4GzWiY
=fbwC
-----END PGP SIGNATURE-----
Oct 25 2008
prev sibling parent e-t172 <e-t172 akegroup.org> writes:
Thomas Leonard wrote:

 On Thu, 23 Oct 2008 19:59:30 -0700, Sean Kelly wrote:

 Thomas Leonard wrote:

 On Sun, 06 Jul 2008 03:31:44 +0000, Sean Kelly wrote:

 == Quote from e-t172 (e-t172 akegroup.org)'s article

 IIRC, there is only one problem when trying to build Tango with -fPIC
 : the garbage collector, which segfaults on collection. Fix the GC
 and you have your Tango shared library.


 I believe this is an issue with the static data segments not being
 represented in the way that the GC expects in a shared library.  I
 really need to do some research to figure out how things are different
 in this instance.  Though if someone knows then that would be good too
 :-)


 Could that be why this program crashes?

 import std.gc: fullCollect;
 import mi = std.moduleinit;

 int main(string[] args) {
 	//printf("%p\n", mi._moduleinfo_dtors[6]); fullCollect();

 	return 0;
 }

 If I uncomment the printf line, it works (I guess because the address
 gets left on the stack). Without calling fullCollect() explicitly, my
 programs work until the GC gets called, then they crash.

 I'm using a (rather modified) GDC on Linux, with libgphobos2 as a
 shared library.

 How are module variables like _moduleinfo_dtors supposed to get
 registered as roots with the GC?


 The shared data segment(s) of a binary are identified by the runtime at
 startup and the runtime passes them to the GC for scanning.  For
 example, look at initStaticDataPtrs() in this file:

 http://dsource.org/projects/tango/browser/trunk/lib/compiler/gdc/


 memory.d

 However, I think the way that the static data segment is identified in a
 shared library is different from in an executable app.  I'm pretty sure
 the Boehm GC handles this properly and have been meaning to look at it
 for clues, but that hasn't made it to the top of my priority list yet.


 
 Ah, thanks for the hints. I found some disabled code in gcgcc.d that 
 tried to scan /proc/self/maps, and with a bit of tweaking it now works 
 for me (tested on x86_64):
 
 http://repo.or.cz/w/delight/core.git?
 a=commitdiff;h=83befaa3f72973eb8afd8589d2e1cee4b37ee4fb;hp=03f581cf37409b35521bcfd6f239a14f1fa73221
 
 BTW, would it be possible to get rid of this moduleinfo stuff? It seems 
 to cause a lot of problems, because when a program imports a library it 
 might not use exactly the same flags that were used when the library was 
 compiled (e.g. program is compiled with unit-tests, library wasn't). So 
 the program might try to use the moduleinfo, even though it doesn't 
 actually exist.
 
 Doesn't the dynamic linker already provide for calling static 
 initialisers? I'm not sure.


Sorry for the ten days delay, I was very busy lately.

When I did some tests compiling Tango as a shared library, I noticed 
that all D static constructors present in the shared library were being 
called, which was kind of annoying at the time (for example, I got 
messages about "tina initialization" for a simple Hello world...).

-- 
Etienne Dechamps / e-t172 - AKE Group

Website: http://www.e-t172.net/
Contact: e-t172 akegroup.org

Phone: +33547414942
Dec 03 2008
prev sibling parent reply e-t172 <e-t172 akegroup.org> writes:
Marius Muja a écrit :

 I'm working on a project that needs to be compiled both as a standalone 
 executable and (part of it) as a shared library. It's been working like 
 that on 32 bit machines without any problems, but now I need to compile 
 it on a 64 bit machine (to be able to access more the 3GB of memory).
 
 When I tried to build the project on the 64 bit machine I was unable to 
 build the shared library because of the following error:
 
 libgphobos.a(ti_AC.o): relocation R_X86_64_32 against 
 `_D17TypeInfo_C6Object6__initZ' can not be used when making a shared 
 object; recompile with -fPIC


You can build a D shared library without including Tango in it (there 
will be undefined references in the shared library, which will be 
resolved when Tango is linked with the main executable).

If you're using GDC, use the -nophoboslib switch when creating your 
shared library to avoid linking to Tango.

Some problems can occur, though. If you happen to modify your shared 
library in such a way that the library makes use of a Tango symbol it 
didn't use before, it won't work unless you relink your executable to 
include the new symbol. That kind of defeats the whole purpose of having 
shared libraries. Of course, one could include all Tango symbols in the 
executable to solve the problem, but then you will end with a huge 
executable because all Tango will be included in it.
Jul 05 2008
parent Marius Muja <mariusm cs.ubc.ca> writes:
e-t172 wrote:


 
 You can build a D shared library without including Tango in it (there 
 will be undefined references in the shared library, which will be 
 resolved when Tango is linked with the main executable).
 
 If you're using GDC, use the -nophoboslib switch when creating your 
 shared library to avoid linking to Tango.



That won't really work for me, my shared library is in fact a matlab MEX 
file, so I'm not creating a main executable in the end, matlab will load 
it when needed.
I'm also building a python binary extension this way, where I have the 
same problems.


 
 Some problems can occur, though. If you happen to modify your shared 
 library in such a way that the library makes use of a Tango symbol it 
 didn't use before, it won't work unless you relink your executable to 
 include the new symbol. That kind of defeats the whole purpose of having 
 shared libraries. Of course, one could include all Tango symbols in the 
 executable to solve the problem, but then you will end with a huge 
 executable because all Tango will be included in it.
Jul 05 2008