www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D hates to be dynamic linked

reply g <g g.g> writes:
it is a real pain trying to make a plugin from d.
I love D2 but I would even abandon it if there is solution with a compiler that
at least supports D1 and has a solution for dynamic linking.
Maybe I'm stupid, but tried dmd and ldc (and tango and phobos). And I don't
know if I want to go GDC.
It is so frustrating that DDL was abandoned, even I grabbed a external branch
an is not so outdated, but probably outdated and w/o documentation.
(btw linux)

is there a not so painful way of making plugins?. Or is there still opportunity
with DDL?

I'm open to both phobos, tango, D1 D2

We NEED a way to make plugins from d. And is a must to use freely the features
of D with out getting dirty (or not so dirty) or worse, limited to some features

/*and for your pleasure some of the pain:
-------------------------------------------------------
g-desktop:~/dynamic/ddl/ddl/Samples$ xfbuild +cldc -L-Wl,-Map
-L-Wl,testHost01.map testHost01.d -I../.. -I/home/g2/ldc/import +oftest01 
.objs/host.o: In function
`_D3ddl11ExportClass44__T11ExportClassTC11testIface0110IHasFooBarZ11ExportClass14__T9newObjectZ9newObjectMFZC11testIface0110IHasFooBar':
host:(.gnu.linkonce.t._D3ddl11ExportClass44__T11ExportClassTC11testIface0110IHasFooBarZ11ExportClass14__T9newObjectZ9newObjectMFZC11testIface01
0IHasFooBar+0x214): undefined reference to `_d_newclass'
.objs/ddl-elf-ELFObjLoader.o:(.rodata+0x28): undefined reference to
`_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader14getLibraryTypeMFZAa'
.objs/ddl-elf-ELFObjLoader.o:(.rodata+0x2c): undefined reference to
`_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader14canLoadLibraryMFS3ddl10FileBuffer10FileBufferZb'
.objs/ddl-elf-ELFObjLoader.o:(.rodata+0x30): undefined reference to
`_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader4loadMFC3ddl14LoaderRegistry14LoaderRegistryS3ddl10FileBuffer10FileBufferZC3ddl14DynamicLibrary14DynamicLibrary'
.objs/ddl-DynamicLibrary.o:(.rodata+0x28): undefined reference to
`_D3ddl14DynamicLibrary14DynamicLibrary9getSymbolMFAaZPS3ddl12ExportSymbol12ExportSymbol'
.objs/ddl-DynamicLibrary.o:(.rodata+0x2c): undefined reference to
`_D3ddl14DynamicLibrary14DynamicLibrary10getModulesMFZAC3ddl13DynamicModule13DynamicModule'
.objs/ddl-DynamicLibrary.o:(.rodata+0x30): undefined reference to
`_D3ddl14DynamicLibrary14DynamicLibrary7getTypeMFZAa'
.objs/ddl-DynamicLibrary.o:(.rodata+0x34): undefined reference to
`_D3ddl14DynamicLibrary14DynamicLibrary13getAttributesMFZHAaAa'
.objs/ddl-DynamicLibrary.o:(.rodata+0x3c): undefined reference to
`_D3ddl14DynamicLibrary14DynamicLibrary18getModuleForSymbolMFAaZC3ddl13DynamicModule13DynamicModule'
.objs/ddl-DynamicLibrary.o:(.rodata+0x40): undefined reference to
`_D3ddl14DynamicLibrary14DynamicLibrary11getResourceMFAaZAh'
.objs/ddl-DynamicModule.o:(.rodata+0x28): undefined reference to
`_D3ddl13DynamicModule13DynamicModule7getNameMFZAa'
.objs/ddl-DynamicModule.o:(.rodata+0x38): undefined reference to
`_D3ddl13DynamicModule13DynamicModule10getSymbolsMFZAS3ddl12ExportSymbol12ExportSymbol'
.objs/ddl-DynamicModule.o:(.rodata+0x3c): undefined reference to
`_D3ddl13DynamicModule13DynamicModule9getSymbolMFAaZPS3ddl12ExportSymbol12ExportSymbol'
.objs/ddl-DynamicModule.o:(.rodata+0x40): undefined reference to
`_D3ddl13DynamicModule13DynamicModule20getSymbolLineNumbersMFZAS3ddl16SymbolLineNumber16SymbolLineNumber'
.objs/ddl-DynamicModule.o:(.rodata+0x44): undefined reference to
`_D3ddl13DynamicModule13DynamicModule13resolveFixupsMFZv'
.objs/ddl-DynamicModule.o:(.rodata+0x48): undefined reference to
`_D3ddl13DynamicModule13DynamicModule10isResolvedMFZb'
.objs/ddl-DynamicLibraryLoader.o:(.rodata+0x28): undefined reference to
`_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader14getLibraryTypeMFZAa'
.objs/ddl-DynamicLibraryLoader.o:(.rodata+0x2c): undefined reference to
`_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader14canLoadLibraryMFS3ddl10FileBuffer10FileBufferZb'
.objs/ddl-DynamicLibraryLoader.o:(.rodata+0x30): undefined reference to
`_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader4loadMFC3ddl14LoaderRegistry14LoaderRegistryS3ddl10FileBuffer10FileBufferZC3ddl14DynamicLibrary14DynamicLibrary'
collect2: ld returned 1 exit status
-------------------------
sorry if that was pointless*/

sorry if *this* is pointless. (I'm a bit frustrateh)
Feb 19 2010
next sibling parent reply "Craig Black" <craigblack2 cox.net> writes:
I am sorry to hear about your problem, and I'm sure there are others who 
share your pain.  If it is any consolation it is helpful to me to hear this, 
since the software that I intend to port to D relies heavily on plug-ins. 
It is disappointing that something as fundamental as plug-ins are such an 
issue in D.  There has been recent talks about adding support for dynamic 
linking, but it doesn't seem to be a high enough priority yet.  Hopefully it 
will get some attention soon.

-Craig

"g" <g g.g> wrote in message news:hln0i0$dm3$1 digitalmars.com...
 it is a real pain trying to make a plugin from d.
 I love D2 but I would even abandon it if there is solution with a compiler 
 that at least supports D1 and has a solution for dynamic linking.
 Maybe I'm stupid, but tried dmd and ldc (and tango and phobos). And I 
 don't know if I want to go GDC.
 It is so frustrating that DDL was abandoned, even I grabbed a external 
 branch an is not so outdated, but probably outdated and w/o documentation.
 (btw linux)

 is there a not so painful way of making plugins?. Or is there still 
 opportunity with DDL?

 I'm open to both phobos, tango, D1 D2

 We NEED a way to make plugins from d. And is a must to use freely the 
 features of D with out getting dirty (or not so dirty) or worse, limited 
 to some features

 /*and for your pleasure some of the pain:
 -------------------------------------------------------
 g-desktop:~/dynamic/ddl/ddl/Samples$ xfbuild 
 +cldc -L-Wl,-Map -L-Wl,testHost01.map 
 testHost01.d -I../.. -I/home/g2/ldc/import +oftest01
 .objs/host.o: In function 
 `_D3ddl11ExportClass44__T11ExportClassTC11testIface0110IHasFooBarZ11ExportClass14__T9newObjectZ9newObjectMFZC11testIface0110IHasFooBar':
 host:(.gnu.linkonce.t._D3ddl11ExportClass44__T11ExportClassTC11testIface0110IHasFooBarZ11ExportClass14__T9newObjectZ9newObjectMFZC11testIface01
0IHasFooBar+0x214): 
 undefined reference to `_d_newclass'
 .objs/ddl-elf-ELFObjLoader.o:(.rodata+0x28): undefined reference to 
 `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader14getLibraryTypeMFZAa'
 .objs/ddl-elf-ELFObjLoader.o:(.rodata+0x2c): undefined reference to 
 `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader14canLoadLibraryMFS3ddl10FileBuffer10FileBufferZb'
 .objs/ddl-elf-ELFObjLoader.o:(.rodata+0x30): undefined reference to 
 `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader4loadMFC3ddl14LoaderRegistry14LoaderRegistryS3ddl10FileBuffer10FileBufferZC3ddl14DynamicLibrary14DynamicLibrary'
 .objs/ddl-DynamicLibrary.o:(.rodata+0x28): undefined reference to 
 `_D3ddl14DynamicLibrary14DynamicLibrary9getSymbolMFAaZPS3ddl12ExportSymbol12ExportSymbol'
 .objs/ddl-DynamicLibrary.o:(.rodata+0x2c): undefined reference to 
 `_D3ddl14DynamicLibrary14DynamicLibrary10getModulesMFZAC3ddl13DynamicModule13DynamicModule'
 .objs/ddl-DynamicLibrary.o:(.rodata+0x30): undefined reference to 
 `_D3ddl14DynamicLibrary14DynamicLibrary7getTypeMFZAa'
 .objs/ddl-DynamicLibrary.o:(.rodata+0x34): undefined reference to 
 `_D3ddl14DynamicLibrary14DynamicLibrary13getAttributesMFZHAaAa'
 .objs/ddl-DynamicLibrary.o:(.rodata+0x3c): undefined reference to 
 `_D3ddl14DynamicLibrary14DynamicLibrary18getModuleForSymbolMFAaZC3ddl13DynamicModule13DynamicModule'
 .objs/ddl-DynamicLibrary.o:(.rodata+0x40): undefined reference to 
 `_D3ddl14DynamicLibrary14DynamicLibrary11getResourceMFAaZAh'
 .objs/ddl-DynamicModule.o:(.rodata+0x28): undefined reference to 
 `_D3ddl13DynamicModule13DynamicModule7getNameMFZAa'
 .objs/ddl-DynamicModule.o:(.rodata+0x38): undefined reference to 
 `_D3ddl13DynamicModule13DynamicModule10getSymbolsMFZAS3ddl12ExportSymbol12ExportSymbol'
 .objs/ddl-DynamicModule.o:(.rodata+0x3c): undefined reference to 
 `_D3ddl13DynamicModule13DynamicModule9getSymbolMFAaZPS3ddl12ExportSymbol12ExportSymbol'
 .objs/ddl-DynamicModule.o:(.rodata+0x40): undefined reference to 
 `_D3ddl13DynamicModule13DynamicModule20getSymbolLineNumbersMFZAS3ddl16SymbolLineNumber16SymbolLineNumber'
 .objs/ddl-DynamicModule.o:(.rodata+0x44): undefined reference to 
 `_D3ddl13DynamicModule13DynamicModule13resolveFixupsMFZv'
 .objs/ddl-DynamicModule.o:(.rodata+0x48): undefined reference to 
 `_D3ddl13DynamicModule13DynamicModule10isResolvedMFZb'
 .objs/ddl-DynamicLibraryLoader.o:(.rodata+0x28): undefined reference to 
 `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader14getLibraryTypeMFZAa'
 .objs/ddl-DynamicLibraryLoader.o:(.rodata+0x2c): undefined reference to 
 `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader14canLoadLibraryMFS3ddl10FileBuffer10FileBufferZb'
 .objs/ddl-DynamicLibraryLoader.o:(.rodata+0x30): undefined reference to 
 `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader4loadMFC3ddl14LoaderRegistry14LoaderRegistryS3ddl10FileBuffer10FileBufferZC3ddl14DynamicLibrary14DynamicLibrary'
 collect2: ld returned 1 exit status
 -------------------------
 sorry if that was pointless*/

 sorry if *this* is pointless. (I'm a bit frustrateh) 
Feb 19 2010
parent Justin Johansson <no spam.com> writes:
Let me preface by saying that I generally try not to be negative and
have a heaps of praise and hope for what the D language and library
designers are attempting to do.  I don't want to spoil the party,
however I think it's fair to recount my own real-world (*) D
experience.

(*) The success or otherwise of the project I'm working on
literally == food or no food to put on my table.  We are not
talking about a hobby; it's a serious pursuit of my livelihood.

Accordingly ...


Commiserations dear friends.

If it is any extra comfort to also share your pain alongside Craig
et. al., let me tell you that I am (was) much in the same boat.

Having spent six months 12x7 working a project in D1, and hoping
that dynamic linking issues (**), might eventually be resolved, I
gave up on D for serious biz.

(**) Shared objects for the Linux platform, though ultimately aiming
for Windows DLLs and a general cross-platform plug-in architecture.

As a result of D's limitations in this regard, I was forced to
back-port my project to C++ some 8 weeks ago.

Now using Eclipse + C Dev Tools (CDT) and GCC as the IDE and
tool-chain respectively, I feel I'm back to really cooking on gas
once again.

"Oh C++, why did I ever leave you?"

Like yourselves, also having previously raised concerns re
dyna-linking on this NG before, most responses from the top to
both my and other users posts on this very topic have generally
been dismissive of the problem.

However, for me, and apart from all the other issues and
frustrations that *you all know about* with D, the dyna-link thing
was the thing that broke the camel's back, becoming the
ultimate show-stopper.

All was not lost though, I learned quite a few things from my 6
month experience with D that have helped to improve my C++ style,
that which was becoming a bit stale after 20 years of it.

To summarize, my feeling is that D is a great experimental language
but you would not want to bet your farm, let alone your life, on it.

Hopefully with the finalization of D2 and publication of TDPL,
things will start to turn around.  One would also hope that by this
time some higher priority will be assigned to taking care of some
of the more menial, yet absolutely fundamental issues affecting
the real-world development and deployment of D applications.

Sincerely, and with best wishes for D,

Justin Johansson


Craig Black wrote:
 I am sorry to hear about your problem, and I'm sure there are others who
 share your pain.  If it is any consolation it is helpful to me to hear
 this, since the software that I intend to port to D relies heavily on
 plug-ins. It is disappointing that something as fundamental as plug-ins
 are such an issue in D.  There has been recent talks about adding
 support for dynamic linking, but it doesn't seem to be a high enough
 priority yet.  Hopefully it will get some attention soon.

 -Craig

 "g" <g g.g> wrote in message news:hln0i0$dm3$1 digitalmars.com...
 it is a real pain trying to make a plugin from d.
 I love D2 but I would even abandon it if there is solution with a
 compiler that at least supports D1 and has a solution for dynamic
 linking.
 Maybe I'm stupid, but tried dmd and ldc (and tango and phobos). And I
 don't know if I want to go GDC.
 It is so frustrating that DDL was abandoned, even I grabbed a external
 branch an is not so outdated, but probably outdated and w/o
 documentation.
 (btw linux)

 is there a not so painful way of making plugins?. Or is there still
 opportunity with DDL?

 I'm open to both phobos, tango, D1 D2

 We NEED a way to make plugins from d. And is a must to use freely the
 features of D with out getting dirty (or not so dirty) or worse,
 limited to some features
Feb 19 2010
prev sibling next sibling parent reply Steve Teale <steve.teale britseyeview.com> writes:
On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:

 it is a real pain trying to make a plugin from d. I love D2 but I would
 even abandon it if there is solution with a compiler that at least
 supports D1 and has a solution for dynamic linking.
There's no way through. I was well on the way to something like Tomcat for D when it dawned on me that which ever way you turn, there's no support for dynamic loading - GDC included (shared libraries don't work). I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there. I just don't understand why this is not at the top of the urgent problems list, unless as I've said before D is just a language that forms a basis for the discussion of cool features that should be in compilers. If that's the case we can all continue to visit the D newsgroups for some intellectual stimulation (in Brit speak that's called intellectual So sad! Steve
Feb 20 2010
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2010-02-20 16.53, Steve Teale wrote:
 On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:

 it is a real pain trying to make a plugin from d. I love D2 but I would
 even abandon it if there is solution with a compiler that at least
 supports D1 and has a solution for dynamic linking.
There's no way through. I was well on the way to something like Tomcat for D when it dawned on me that which ever way you turn, there's no support for dynamic loading - GDC included (shared libraries don't work).
I've made a few dynamic libraries on Mac OS X that work.
 I tried to get some attention for this problem again a couple of weeks
 ago (see the "special treat" thread), and everyone who replied said "yes,
 we really need this", but Walter does not want to go there.

 I just don't understand why this is not at the top of the urgent problems
 list, unless as I've said before D is just a language that forms a basis
 for the discussion of cool features that should be in compilers. If
 that's the case we can all continue to visit the D newsgroups for some
 intellectual stimulation (in Brit speak that's called intellectual


 So sad!

 Steve
Feb 20 2010
prev sibling parent reply Jonathan M Davis <jmdavisProg gmail.com> writes:
Steve Teale wrote:

 On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:
 
 it is a real pain trying to make a plugin from d. I love D2 but I would
 even abandon it if there is solution with a compiler that at least
 supports D1 and has a solution for dynamic linking.
There's no way through. I was well on the way to something like Tomcat for D when it dawned on me that which ever way you turn, there's no support for dynamic loading - GDC included (shared libraries don't work). I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there. I just don't understand why this is not at the top of the urgent problems list, unless as I've said before D is just a language that forms a basis for the discussion of cool features that should be in compilers. If that's the case we can all continue to visit the D newsgroups for some intellectual stimulation (in Brit speak that's called intellectual So sad! Steve
Well, obviously, there are more pressing things from Walter's perspective. For many programs, dynamic libraries are irrelevant, and there are major issues which are relevant to any programming project in D - dynamically linked or otherwise. So, presumably those are more important. I'm sure that dynamically linked libraries will get to the top of the priority queue eventually, but it looks like those who need it will just have to hold tight for now - as irritating as that may be. Being vocal about it might help push it up, though it might not (Walter tends to care about what he cares about whatever others might say). If it's a true showstopper for you, you can always contribute towards getting it working. Most programs can be written just fine without dynamically linked libraries, so I can certainly see why Walter would not consider it a high priority with everything else on his plate, but I do hope that he gets to it eventually. - Jonathan M Davis
Feb 20 2010
parent Eldar Insafutdinov <e.insafutdinov gmail.com> writes:
Jonathan M Davis Wrote:

 Most programs can be written just fine without dynamically linked libraries, 
 so I can certainly see why Walter would not consider it a high priority with 
 everything else on his plate, but I do hope that he gets to it eventually.
 
I disagree here. Yes they can be written, but at some point in time deveopers rewrite it using plugin architecture. From software that I use everyday plugginalble are: file manager(gnome), messenger(pidgin), browser(plugins of different taste, but nevertheless), IDEs(most of them: MSVC, Qt Creator, Eclipse...). I work for the company and all of its software is plugins to bigger programs...
Feb 21 2010
prev sibling next sibling parent reply Eldar Insafutdinov <e.insafutdinov gmail.com> writes:
Steve Teale Wrote:

 On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:
 
 
 I tried to get some attention for this problem again a couple of weeks 
 ago (see the "special treat" thread), and everyone who replied said "yes, 
 we really need this", but Walter does not want to go there.
 
Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Feb 21 2010
parent reply Justin Johansson <no spam.com> writes:
Eldar Insafutdinov wrote:
 Steve Teale Wrote:
 
 On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:


 I tried to get some attention for this problem again a couple of weeks 
 ago (see the "special treat" thread), and everyone who replied said "yes, 
 we really need this", but Walter does not want to go there.
Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.
Feb 21 2010
parent reply Lutger <lutger.blijdestijn gmail.com> writes:
Justin Johansson wrote:

 Eldar Insafutdinov wrote:
 Steve Teale Wrote:
 
 On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:


 I tried to get some attention for this problem again a couple of weeks
 ago (see the "special treat" thread), and everyone who replied said
 "yes, we really need this", but Walter does not want to go there.
Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.
At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?
Feb 21 2010
next sibling parent reply Justin Johansson <no spam.com> writes:
Lutger wrote:
 Justin Johansson wrote:
 
 Eldar Insafutdinov wrote:
 Steve Teale Wrote:

 On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:


 I tried to get some attention for this problem again a couple of weeks
 ago (see the "special treat" thread), and everyone who replied said
 "yes, we really need this", but Walter does not want to go there.
Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.
At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?
Specifically my roadblock was the inability to create shared objects (.so files) on Linux platform (analogous to DLLs in Windows). This was required for the plug-in architecture that I was targeting.
Feb 21 2010
parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Another important question: does the application still blow up if an
exception gets thrown across the DLL boundary?
Feb 24 2010
parent Sean Kelly <sean invisibleduck.org> writes:
Daniel Keep Wrote:

 
 Another important question: does the application still blow up if an
 exception gets thrown across the DLL boundary?
If you called Runtime.loadLibrary() to load the DLL then it shouldn't.
Feb 24 2010
prev sibling next sibling parent Steve Teale <steve.teale britseyeview.com> writes:
On Sun, 21 Feb 2010 14:06:44 +0100, Lutger wrote:

 Justin Johansson wrote:
 
 Eldar Insafutdinov wrote:
 Steve Teale Wrote:
 
 On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:


 I tried to get some attention for this problem again a couple of
 weeks ago (see the "special treat" thread), and everyone who replied
 said "yes, we really need this", but Walter does not want to go
 there.
Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.
At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?
I have not got to the bottom of this yet - been doing building work to get a house finished that was deemed to have higher priority. Just to keep things basic, I did quite a bit of investigation on the creation of Linux shared libraries. I could not do it cleanly with either DMD or GDC. There's an account of what I tried on the blog section of http:// www.britseyeview.com/GDC-newbie.html. I think a major stumbling block is that Phobos is not a shared library, so the dynamic linking process can't just pull in stuff it needs when loading your shared library plugin. In addition though I think there may be a problem with the runtime. I had difficulties with a symbol called __data_start (or similar). I could see this in the host program, but it was not available to the dynamically loaded object. Also it is not 100% clear whether DMD can actually generate position independent code. I had previously tried to simplify DDL to make it at least be able to load a single object file, and in Windows, I had limited success, but it seemed to me that throwing a lot of effort into the ancient OMF format was not the way to go. I have not had time yet to attempt a similar thing with ELF. So there's more than just dynamic loading, the DMD Windows back end is a bit of a blast from the past anyway. I'm sorry this is rather vague, but I'm not just bleating without having tried some things. Steve
Feb 21 2010
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Sun, 21 Feb 2010 08:06:44 -0500, Lutger <lutger.blijdestijn gmail.com>  
wrote:
 Justin Johansson wrote:

 Eldar Insafutdinov wrote:
 Steve Teale Wrote:

 On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:


 I tried to get some attention for this problem again a couple of weeks
 ago (see the "special treat" thread), and everyone who replied said
 "yes, we really need this", but Walter does not want to go there.
Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.
At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?
I know one issue in D1 is the GC. That's been fixed in D2 with druntime.
Feb 21 2010
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
Hi,

as I'm also hitting some problems with DLLs, here are some issues that I 
am now aware of (sorry, can't tell for linux shared objects, but I guess 
the situation is similar):

1. For D2, there is a major blocker with DLLs loaded after intialization 
on XP because of no TLS support from the OS. There is a simple 
workaround for single-threaded application (just setting FS:2c to a 
pointer to _tlsstart), but I'm considering a full emulation of the TLS 
initialization.

2. No multi-threading support: I've added a function to std.thread that 
allows adding a thread to the Thread.allThreads array, so it can be 
called from DLLMain/THREAD_ATTACH. That way the GC can suspend these, 
and allocations from different threads don't trample onto each other. Is 
there anything else needed for full multi-threading support?

My patch is currently D1/Win32 only, but should not be too hard to 
extend to other platforms. Attaching to already existing threads is a 
bit harder, though, and it might not always be desired, because these 
threads might never call into D-code.

3. To try things out, I am playing around with the mydll-example, and it 
shows some quirks that you need to get around: the implementation-file 
for the DLL needs to use a file name different from the module name, 
because you need another file specifying the exports. Unfortunately this 
cannot be the di-file created from the source, because it contains two 
much information that will cause references to symbols not actually 
exported.

Maybe some command line switch is needed to just write the exports into 
the di file without any implementations. Even better: import the 
implementation file, but don't create unnecessary references. (I think 
it's the module-initialization that's been called because of the import 
- maybe some modifier to the import could remove it).

4. The documentation on the website states, that you should use

	DATA            PRELOAD SINGLE

in the def-file. That probably is still there as a historical note to 
win 3.1, it will cause different processes to trample on each others 
data segments. This has caused a few hours of debugging, so please 
remove it, so others don't fall into the same trap. The mydll-example 
does not use the statement.

5. To share gc-collected objects between different DLLs, a common 
phobos-DLL seems necessary. Extracting the GC into a separate DLL and 
using the proxy-mechanism to attach any other client-DLL to it seems 
feasable, but are there other things that need to be shared between 
different phobos-instances? What about exception-handling?

I'd say, if there is a way to put all public symbols into a def-file, 
then compile phobos into a DLL and create the import library and use 
this instead of a static library. The multi-threading-issue for DLLs 
needs to be solved before, though.

I'm currently attacking 1. and 2., but only on windows at the moment. 
I'll add patches and reports into bugzilla later...

Rainer

Lutger wrote:
 Justin Johansson wrote:
 
 Eldar Insafutdinov wrote:
 Steve Teale Wrote:

 On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:


 I tried to get some attention for this problem again a couple of weeks
 ago (see the "special treat" thread), and everyone who replied said
 "yes, we really need this", but Walter does not want to go there.
Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.
At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?
Feb 21 2010
next sibling parent Justin Johansson <no spam.com> writes:
Rainer Schuetze wrote:
 Hi,

 as I'm also hitting some problems with DLLs, here are some issues 
that I am now aware of (sorry, can't tell for linux shared objects, but I guess the situation is similar):
 1. For D2, there is a major blocker with DLLs loaded after 
intialization on XP because of no TLS support from the OS. There is a simple workaround for single-threaded application (just setting FS:2c to a pointer to _tlsstart), but I'm considering a full emulation of the TLS initialization.
 2. No multi-threading support: I've added a function to std.thread 
that allows adding a thread to the Thread.allThreads array, so it can be called from DLLMain/THREAD_ATTACH. That way the GC can suspend these, and allocations from different threads don't trample onto each other. Is there anything else needed for full multi-threading support?
 My patch is currently D1/Win32 only, but should not be too hard to 
extend to other platforms. Attaching to already existing threads is a bit harder, though, and it might not always be desired, because these threads might never call into D-code.
 3. To try things out, I am playing around with the mydll-example, and 
it shows some quirks that you need to get around: the implementation-file for the DLL needs to use a file name different from the module name, because you need another file specifying the exports. Unfortunately this cannot be the di-file created from the source, because it contains two much information that will cause references to symbols not actually exported.
 Maybe some command line switch is needed to just write the exports 
into the di file without any implementations. Even better: import the implementation file, but don't create unnecessary references. (I think it's the module-initialization that's been called because of the import - maybe some modifier to the import could remove it).
 4. The documentation on the website states, that you should use

     DATA            PRELOAD SINGLE

 in the def-file. That probably is still there as a historical note to 
win 3.1, it will cause different processes to trample on each others data segments. This has caused a few hours of debugging, so please remove it, so others don't fall into the same trap. The mydll-example does not use the statement.
 5. To share gc-collected objects between different DLLs, a common 
phobos-DLL seems necessary. Extracting the GC into a separate DLL and using the proxy-mechanism to attach any other client-DLL to it seems feasable, but are there other things that need to be shared between different phobos-instances? What about exception-handling?





 I'd say, if there is a way to put all public symbols into a def-file, 
then compile phobos into a DLL and create the import library and use this instead of a static library. The multi-threading-issue for DLLs needs to be solved before, though. Hi Rainer, Congrats. Sounds like you are giving the problem a lot of thought. Just regarding the public symbols though, I hazard a guess that you might run into a name-mangling problem here. Don't want to put a damper on your effort though; just a forewarning that this might be an issue. Perhaps someone else who knows about exporting DLL symbols might be gracious enough to chime in. Cheers Justin
Feb 22 2010
prev sibling next sibling parent reply Justin Johansson <no spam.com> writes:
Rainer Schuetze wrote:

 5. To share gc-collected objects between different DLLs, a common 
 phobos-DLL seems necessary. Extracting the GC into a separate DLL and 
 using the proxy-mechanism to attach any other client-DLL to it seems 
 feasable, but are there other things that need to be shared between 
 different phobos-instances? What about exception-handling?
 
Sounds like you have the right idea; that sounds similar to the way Microsoft does it with MFC DLL's. If I recall correctly, when MFC applications are DLL based, you link with a common C runtime DLL as well. This way all memory allocations and frees are handled by the common C runtime DLL (i.e. single point of responsibility). Presumably a similar regime for D and the GC would be necessary.
Feb 22 2010
parent reply Roald Ribe <rr.nospam nospam.teikom.no> writes:
Justin Johansson wrote:
 Rainer Schuetze wrote:
 
 5. To share gc-collected objects between different DLLs, a common 
 phobos-DLL seems necessary. Extracting the GC into a separate DLL and 
 using the proxy-mechanism to attach any other client-DLL to it seems 
 feasable, but are there other things that need to be shared between 
 different phobos-instances? What about exception-handling?
Sounds like you have the right idea; that sounds similar to the way Microsoft does it with MFC DLL's. If I recall correctly, when MFC applications are DLL based, you link with a common C runtime DLL as well. This way all memory allocations and frees are handled by the common C runtime DLL (i.e. single point of responsibility). Presumably a similar regime for D and the GC would be necessary.
Not only MFC. Plain C and C++ programs as well. This is why most MS- Windows based compiled languages comes with two versions of the runtime lib, one for static link and one for dynamic link (DLL). If not, there will always be problems with DLL's in that language sharing file handles, memory, sockets, threads and so on. The flip side of the coin is "DLL hell" (use Google). It is less of a problem in newer versions of MS-Windows than it used to be, but necessary to consider when supporting older platforms. The Linux/BSD/unix side I know less about, but since they have a common location for shared libs and support multiple versions of the same lib (AFAIK) the problems are less there. Roald
Feb 22 2010
parent Bane <branimir.milosavljevic gmail.com> writes:
Just to get it righ: this means GC will be in dl tool so people that complain
about exe size and gc bloat will be happy about it?

Roald Ribe Wrote:

 Justin Johansson wrote:
 Rainer Schuetze wrote:
 
 5. To share gc-collected objects between different DLLs, a common 
 phobos-DLL seems necessary. Extracting the GC into a separate DLL and 
 using the proxy-mechanism to attach any other client-DLL to it seems 
 feasable, but are there other things that need to be shared between 
 different phobos-instances? What about exception-handling?
Sounds like you have the right idea; that sounds similar to the way Microsoft does it with MFC DLL's. If I recall correctly, when MFC applications are DLL based, you link with a common C runtime DLL as well. This way all memory allocations and frees are handled by the common C runtime DLL (i.e. single point of responsibility). Presumably a similar regime for D and the GC would be necessary.
Not only MFC. Plain C and C++ programs as well. This is why most MS- Windows based compiled languages comes with two versions of the runtime lib, one for static link and one for dynamic link (DLL). If not, there will always be problems with DLL's in that language sharing file handles, memory, sockets, threads and so on. The flip side of the coin is "DLL hell" (use Google). It is less of a problem in newer versions of MS-Windows than it used to be, but necessary to consider when supporting older platforms. The Linux/BSD/unix side I know less about, but since they have a common location for shared libs and support multiple versions of the same lib (AFAIK) the problems are less there. Roald
Feb 23 2010
prev sibling next sibling parent Lutger <lutger.blijdestijn gmail.com> writes:
Thank you and everybody else for the explanation. Perhaps a shared phobos 
lib distributed with dmd will become a reality after all, not too far in the 
future.  
Feb 22 2010
prev sibling next sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 22 Feb 2010 02:46:31 -0500, Rainer Schuetze <r.sagitario gmx.de>  
wrote:
 5. To share gc-collected objects between different DLLs, a common  
 phobos-DLL seems necessary. Extracting the GC into a separate DLL and  
 using the proxy-mechanism to attach any other client-DLL to it seems  
 feasable, but are there other things that need to be shared between  
 different phobos-instances? What about exception-handling?
druntime (D2) already supports sharing the GC between DLLs.
Feb 23 2010
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
I've seen that, too, but have not tried it yet. As Roald has also 
pointed out, there are other things you might want to share between dlls 
(threads, file handles, etc.).

Robert Jacques wrote:
 On Mon, 22 Feb 2010 02:46:31 -0500, Rainer Schuetze <r.sagitario gmx.de> 
 wrote:
 5. To share gc-collected objects between different DLLs, a common 
 phobos-DLL seems necessary. Extracting the GC into a separate DLL and 
 using the proxy-mechanism to attach any other client-DLL to it seems 
 feasable, but are there other things that need to be shared between 
 different phobos-instances? What about exception-handling?
druntime (D2) already supports sharing the GC between DLLs.
Feb 23 2010
parent "Robert Jacques" <sandford jhu.edu> writes:
On Wed, 24 Feb 2010 02:38:58 -0500, Rainer Schuetze <r.sagitario gmx.de>  
wrote:

 I've seen that, too, but have not tried it yet. As Roald has also  
 pointed out, there are other things you might want to share between dlls  
 (threads, file handles, etc.).
Given the GC has to know about all threads, I assume those are shared as well. As for file handles, those are just data variables, which can be easily shared between DLLs. The bug problem, as I understand it, with DLLs is global data requires manual re-routing, etc. to work.
Feb 24 2010
prev sibling parent reply Don <nospam nospam.com> writes:
Rainer Schuetze wrote:
 Hi,
 
 as I'm also hitting some problems with DLLs, here are some issues that I 
 am now aware of (sorry, can't tell for linux shared objects, but I guess 
 the situation is similar):
 
 1. For D2, there is a major blocker with DLLs loaded after intialization 
 on XP because of no TLS support from the OS. There is a simple 
 workaround for single-threaded application (just setting FS:2c to a 
 pointer to _tlsstart), but I'm considering a full emulation of the TLS 
 initialization.
That's incredibly difficult. Have you read this? http://www.nynaeve.net/?p=189 It'd be great if you can get it to work. But I fear that for Win32, DMD is going to have to switch to explicit TLS.
Feb 24 2010
next sibling parent Walter Bright <newshound1 digitalmars.com> writes:
Don wrote:
 Rainer Schuetze wrote:
 Hi,

 as I'm also hitting some problems with DLLs, here are some issues that 
 I am now aware of (sorry, can't tell for linux shared objects, but I 
 guess the situation is similar):

 1. For D2, there is a major blocker with DLLs loaded after 
 intialization on XP because of no TLS support from the OS. There is a 
 simple workaround for single-threaded application (just setting FS:2c 
 to a pointer to _tlsstart), but I'm considering a full emulation of 
 the TLS initialization.
That's incredibly difficult. Have you read this? http://www.nynaeve.net/?p=189 It'd be great if you can get it to work. But I fear that for Win32, DMD is going to have to switch to explicit TLS.
I talked to the guy who wrote that. He wrote some workaround code to make TLS work on older Windows versions, and I asked him if we could license it. He was reluctant, saying it was experimental only. So yeah, rolling our own may be the only viable option.
Feb 24 2010
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
I knew that blog, but I expected to find the necessary data somewhere in 
the "undocumented" (but still available) structures.

But I had to dig a little deeper into ntdll myself and came up with a 
patch that is part of the multithread-support for DLLs:

http://d.puremagic.com/issues/show_bug.cgi?id=3885

It emulates the behaviour at process startup time, and modifies the data 
structures so that windows handles the DLL as if it was loaded with the 
process. This has the nice benefit that you don't need to handle the 
creation of new threads.

The downsides are:
- DLL-unload is disabled, but I think unloading a DLL that uses TLS is 
problematic anyway)
- It will leak some pointer-arrays in favor of not causing crashes due 
to other threads using TLS at same time as reallocating these arrays.

As XP is not a fast moving target anymore, I hope that ntdll will not 
change a lot, so it will not need adjustment for every windows-update. 
I've checked SP2 and SP3, which look very much the same with respect to 
TLS implementation.

Rainer

Don wrote:
 Rainer Schuetze wrote:
 Hi,

 as I'm also hitting some problems with DLLs, here are some issues that 
 I am now aware of (sorry, can't tell for linux shared objects, but I 
 guess the situation is similar):

 1. For D2, there is a major blocker with DLLs loaded after 
 intialization on XP because of no TLS support from the OS. There is a 
 simple workaround for single-threaded application (just setting FS:2c 
 to a pointer to _tlsstart), but I'm considering a full emulation of 
 the TLS initialization.
That's incredibly difficult. Have you read this? http://www.nynaeve.net/?p=189 It'd be great if you can get it to work. But I fear that for Win32, DMD is going to have to switch to explicit TLS.
Mar 06 2010
parent reply Don <nospam nospam.com> writes:
Rainer Schuetze wrote:
 I knew that blog, but I expected to find the necessary data somewhere in 
 the "undocumented" (but still available) structures.
 
 But I had to dig a little deeper into ntdll myself and came up with a 
 patch that is part of the multithread-support for DLLs:
 
 http://d.puremagic.com/issues/show_bug.cgi?id=3885
 
 It emulates the behaviour at process startup time, and modifies the data 
 structures so that windows handles the DLL as if it was loaded with the 
 process. This has the nice benefit that you don't need to handle the 
 creation of new threads.
 
 The downsides are:
 - DLL-unload is disabled, but I think unloading a DLL that uses TLS is 
 problematic anyway)
 - It will leak some pointer-arrays in favor of not causing crashes due 
 to other threads using TLS at same time as reallocating these arrays.
 
 As XP is not a fast moving target anymore, I hope that ntdll will not 
 change a lot, so it will not need adjustment for every windows-update. 
 I've checked SP2 and SP3, which look very much the same with respect to 
 TLS implementation.
 
 Rainer
You're totally awesome. This is a game-changer.
Mar 09 2010
parent Rainer Schuetze <r.sagitario gmx.de> writes:
Don wrote:
 Rainer Schuetze wrote:
 I knew that blog, but I expected to find the necessary data somewhere 
 in the "undocumented" (but still available) structures.

 But I had to dig a little deeper into ntdll myself and came up with a 
 patch that is part of the multithread-support for DLLs:

 http://d.puremagic.com/issues/show_bug.cgi?id=3885

 It emulates the behaviour at process startup time, and modifies the 
 data structures so that windows handles the DLL as if it was loaded 
 with the process. This has the nice benefit that you don't need to 
 handle the creation of new threads.

 The downsides are:
 - DLL-unload is disabled, but I think unloading a DLL that uses TLS is 
 problematic anyway)
 - It will leak some pointer-arrays in favor of not causing crashes due 
 to other threads using TLS at same time as reallocating these arrays.

 As XP is not a fast moving target anymore, I hope that ntdll will not 
 change a lot, so it will not need adjustment for every windows-update. 
 I've checked SP2 and SP3, which look very much the same with respect 
 to TLS implementation.

 Rainer
You're totally awesome. This is a game-changer.
Thanks :-) The patch might still need a little polishing before finding it's way into druntime, though...
Mar 10 2010
prev sibling next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
g wrote:
 (btw linux)
The first step is to make phobos into a shared .so library. Is anyone willing to try it and sort out just what the issues are?
Feb 21 2010
next sibling parent reply Igor Lesik <curoles yahoo.com> writes:
The first step is to make phobos into a shared .so library. Is anyone willing
to try it and sort out just what the issues are?
I may give it a try, if nobody else wants to jump on it.
Feb 21 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Igor Lesik wrote:
 The first step is to make phobos into a shared .so library. Is
 anyone willing to try it and sort out just what the issues are?
I may give it a try, if nobody else wants to jump on it.
Thanks! dmd has the first step, it supports -fPIC.
Feb 21 2010
parent reply Steve Teale <steve.teale britseyeview.com> writes:
On Sun, 21 Feb 2010 16:05:47 -0800, Walter Bright wrote:

 Igor Lesik wrote:
 The first step is to make phobos into a shared .so library. Is anyone
 willing to try it and sort out just what the issues are?
I may give it a try, if nobody else wants to jump on it.
Thanks! dmd has the first step, it supports -fPIC.
So it's real? There's still something wrong though - I had a big bash at shared library about a week ago - no joy. Great Walter - thanks. Igor - please get in touch, I may be able to help. Steve
Feb 24 2010
parent Igor Lesik <curoles yahoo.com> writes:
 Igor Lesik wrote:
 The first step is to make phobos into a shared .so library. Is anyone
 willing to try it and sort out just what the issues are?
I may give it a try, if nobody else wants to jump on it.
Thanks! dmd has the first step, it supports -fPIC.
So it's real? There's still something wrong though - I had a big bash at shared library about a week ago - no joy. Great Walter - thanks. Igor - please get in touch, I may be able to help. Steve
Sure. Most probably I will need help. Thanks.
Feb 24 2010
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2010-02-21 19.50, Walter Bright wrote:
 g wrote:
 (btw linux)
The first step is to make phobos into a shared .so library. Is anyone willing to try it and sort out just what the issues are?
I've been trying to do this from time to time on Mac OS X, I've also tried with tango. The issues I've encountered are undefined symbols: Undefined symbols: "__deh_beg", referenced from: __deh_beg$non_lazy_ptr in deh2.o "__deh_end", referenced from: __deh_end$non_lazy_ptr in deh2.o "__Dmain", referenced from: _main in dmain2.o _main in dmain2.o "__minfo_beg", referenced from: __minfo_beg$non_lazy_ptr in moduleinit.o "__minfo_end", referenced from: __minfo_end$non_lazy_ptr in moduleinit.o GDC doesn't have this problem. This is how GDC solves some of the above undefined symbols: __Dmain: GDC doesn't have the C and D main functions in dmain2.d, instead it has another file with the main functions that should not be linked if you build it as a dynamic/shared library. I tried to do this with tango and dmd but then when I built a executable I got undefined symbol on the C main function. __minfo_beg and __minfo_end: Neither GDC or LDC have these. I don't remember how they handle this but it can probably be solved using the function "getsegbyname" or any of the similar functions that exist. For this I would need more precise information in which segment and section they are located. __deh_beg and __deh_end: Both GDC and LDC uses a different exception handling implementation than dmd and don't have these. I have no idea what to do about these. /Jacob Carlborg
Feb 22 2010
prev sibling parent reply Bane <branimir.milosavljevic gmail.com> writes:
A nasty typo, should be "...GC will be in dll too so people..."

 Just to get it righ: this means GC will be in dl tool so people that complain
about exe size and gc bloat will be happy about it?
Feb 23 2010
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
Yes, GC is the main issue, as you want to pass gc-collected objects 
across dll boundaries. I'd say the size of the executables is just a 
secondary issue, though it would also benefit.

Bane wrote:
 A nasty typo, should be "...GC will be in dll too so people..."
 
 Just to get it righ: this means GC will be in dl tool so people that complain
about exe size and gc bloat will be happy about it?
Feb 23 2010
parent Walter Bright <newshound1 digitalmars.com> writes:
Rainer Schuetze wrote:
 Yes, GC is the main issue, as you want to pass gc-collected objects 
 across dll boundaries. I'd say the size of the executables is just a 
 secondary issue, though it would also benefit.
The GC itself doesn't have to be in a dll for this to work. The library is already set up to hand off the GC. What it doesn't do at the moment is hand off the thread management.
Feb 24 2010