digitalmars.D - D hates to be dynamic linked
- g (35/35) Feb 19 2010 it is a real pain trying to make a plugin from d.
- Craig Black (9/72) Feb 19 2010 I am sorry to hear about your problem, and I'm sure there are others who...
- Justin Johansson (44/73) Feb 19 2010 Let me preface by saying that I generally try not to be negative and
- Steve Teale (15/18) Feb 20 2010 There's no way through. I was well on the way to something like Tomcat
- Jacob Carlborg (2/20) Feb 20 2010
- Jonathan M Davis (15/39) Feb 20 2010 Well, obviously, there are more pressing things from Walter's perspectiv...
- Eldar Insafutdinov (3/7) Feb 21 2010 I disagree here. Yes they can be written, but at some point in time deve...
- Eldar Insafutdinov (2/9) Feb 21 2010 Is it really true? That's a big shame. Being able to define and use plug...
- Justin Johansson (5/16) Feb 21 2010 Yes it is really true.
- Lutger (4/25) Feb 21 2010 At the risk of asking a dumb question, what is the major roadblock? Is i...
- Justin Johansson (4/29) Feb 21 2010 Specifically my roadblock was the inability to create shared objects
- Daniel Keep (2/2) Feb 24 2010 Another important question: does the application still blow up if an
- Sean Kelly (2/5) Feb 24 2010 If you called Runtime.loadLibrary() to load the DLL then it shouldn't.
- Steve Teale (24/53) Feb 21 2010 I have not got to the bottom of this yet - been doing building work to
- Robert Jacques (3/32) Feb 21 2010 I know one issue in D1 is the GC. That's been fixed in D2 with druntime.
- Rainer Schuetze (50/75) Feb 21 2010 Hi,
- Justin Johansson (44/56) Feb 22 2010 that I am now aware of (sorry, can't tell for linux shared objects, but
- Justin Johansson (7/13) Feb 22 2010 Sounds like you have the right idea; that sounds similar to the
- Roald Ribe (13/28) Feb 22 2010 Not only MFC. Plain C and C++ programs as well. This is why most MS-
- Bane (2/35) Feb 23 2010
- Lutger (3/3) Feb 22 2010 Thank you and everybody else for the explanation. Perhaps a shared phobo...
- Robert Jacques (3/8) Feb 23 2010 druntime (D2) already supports sharing the GC between DLLs.
- Rainer Schuetze (4/13) Feb 23 2010 I've seen that, too, but have not tried it yet. As Roald has also
- Robert Jacques (6/9) Feb 24 2010 Given the GC has to know about all threads, I assume those are shared as...
- Don (5/16) Feb 24 2010 That's incredibly difficult. Have you read this?
- Walter Bright (5/24) Feb 24 2010 I talked to the guy who wrote that. He wrote some workaround code to
- Rainer Schuetze (20/39) Mar 06 2010 I knew that blog, but I expected to find the necessary data somewhere in...
- Don (2/27) Mar 09 2010 You're totally awesome. This is a game-changer.
- Rainer Schuetze (3/31) Mar 10 2010 Thanks :-) The patch might still need a little polishing before finding
- Walter Bright (3/4) Feb 21 2010 The first step is to make phobos into a shared .so library. Is anyone
- Igor Lesik (2/3) Feb 21 2010 I may give it a try, if nobody else wants to jump on it.
- Walter Bright (2/6) Feb 21 2010 Thanks! dmd has the first step, it supports -fPIC.
- Steve Teale (6/13) Feb 24 2010 So it's real? There's still something wrong though - I had a big bash at...
- Igor Lesik (2/14) Feb 24 2010 Sure. Most probably I will need help. Thanks.
- Jacob Carlborg (32/36) Feb 22 2010 I've been trying to do this from time to time on Mac OS X, I've also
- Bane (1/2) Feb 23 2010
- Rainer Schuetze (4/8) Feb 23 2010 Yes, GC is the main issue, as you want to pass gc-collected objects
- Walter Bright (4/7) Feb 24 2010 The GC itself doesn't have to be in a dll for this to work. The library
it is a real pain trying to make a plugin from d. I love D2 but I would even abandon it if there is solution with a compiler that at least supports D1 and has a solution for dynamic linking. Maybe I'm stupid, but tried dmd and ldc (and tango and phobos). And I don't know if I want to go GDC. It is so frustrating that DDL was abandoned, even I grabbed a external branch an is not so outdated, but probably outdated and w/o documentation. (btw linux) is there a not so painful way of making plugins?. Or is there still opportunity with DDL? I'm open to both phobos, tango, D1 D2 We NEED a way to make plugins from d. And is a must to use freely the features of D with out getting dirty (or not so dirty) or worse, limited to some features /*and for your pleasure some of the pain: ------------------------------------------------------- g-desktop:~/dynamic/ddl/ddl/Samples$ xfbuild +cldc -L-Wl,-Map -L-Wl,testHost01.map testHost01.d -I../.. -I/home/g2/ldc/import +oftest01 .objs/host.o: In function `_D3ddl11ExportClass44__T11ExportClassTC11testIface0110IHasFooBarZ11ExportClass14__T9newObjectZ9newObjectMFZC11testIface0110IHasFooBar': host:(.gnu.linkonce.t._D3ddl11ExportClass44__T11ExportClassTC11testIface0110IHasFooBarZ11ExportClass14__T9newObjectZ9newObjectMFZC11testIface01 0IHasFooBar+0x214): undefined reference to `_d_newclass' .objs/ddl-elf-ELFObjLoader.o:(.rodata+0x28): undefined reference to `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader14getLibraryTypeMFZAa' .objs/ddl-elf-ELFObjLoader.o:(.rodata+0x2c): undefined reference to `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader14canLoadLibraryMFS3ddl10FileBuffer10FileBufferZb' .objs/ddl-elf-ELFObjLoader.o:(.rodata+0x30): undefined reference to `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader4loadMFC3ddl14LoaderRegistry14LoaderRegistryS3ddl10FileBuffer10FileBufferZC3ddl14DynamicLibrary14DynamicLibrary' .objs/ddl-DynamicLibrary.o:(.rodata+0x28): undefined reference to `_D3ddl14DynamicLibrary14DynamicLibrary9getSymbolMFAaZPS3ddl12ExportSymbol12ExportSymbol' .objs/ddl-DynamicLibrary.o:(.rodata+0x2c): undefined reference to `_D3ddl14DynamicLibrary14DynamicLibrary10getModulesMFZAC3ddl13DynamicModule13DynamicModule' .objs/ddl-DynamicLibrary.o:(.rodata+0x30): undefined reference to `_D3ddl14DynamicLibrary14DynamicLibrary7getTypeMFZAa' .objs/ddl-DynamicLibrary.o:(.rodata+0x34): undefined reference to `_D3ddl14DynamicLibrary14DynamicLibrary13getAttributesMFZHAaAa' .objs/ddl-DynamicLibrary.o:(.rodata+0x3c): undefined reference to `_D3ddl14DynamicLibrary14DynamicLibrary18getModuleForSymbolMFAaZC3ddl13DynamicModule13DynamicModule' .objs/ddl-DynamicLibrary.o:(.rodata+0x40): undefined reference to `_D3ddl14DynamicLibrary14DynamicLibrary11getResourceMFAaZAh' .objs/ddl-DynamicModule.o:(.rodata+0x28): undefined reference to `_D3ddl13DynamicModule13DynamicModule7getNameMFZAa' .objs/ddl-DynamicModule.o:(.rodata+0x38): undefined reference to `_D3ddl13DynamicModule13DynamicModule10getSymbolsMFZAS3ddl12ExportSymbol12ExportSymbol' .objs/ddl-DynamicModule.o:(.rodata+0x3c): undefined reference to `_D3ddl13DynamicModule13DynamicModule9getSymbolMFAaZPS3ddl12ExportSymbol12ExportSymbol' .objs/ddl-DynamicModule.o:(.rodata+0x40): undefined reference to `_D3ddl13DynamicModule13DynamicModule20getSymbolLineNumbersMFZAS3ddl16SymbolLineNumber16SymbolLineNumber' .objs/ddl-DynamicModule.o:(.rodata+0x44): undefined reference to `_D3ddl13DynamicModule13DynamicModule13resolveFixupsMFZv' .objs/ddl-DynamicModule.o:(.rodata+0x48): undefined reference to `_D3ddl13DynamicModule13DynamicModule10isResolvedMFZb' .objs/ddl-DynamicLibraryLoader.o:(.rodata+0x28): undefined reference to `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader14getLibraryTypeMFZAa' .objs/ddl-DynamicLibraryLoader.o:(.rodata+0x2c): undefined reference to `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader14canLoadLibraryMFS3ddl10FileBuffer10FileBufferZb' .objs/ddl-DynamicLibraryLoader.o:(.rodata+0x30): undefined reference to `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader4loadMFC3ddl14LoaderRegistry14LoaderRegistryS3ddl10FileBuffer10FileBufferZC3ddl14DynamicLibrary14DynamicLibrary' collect2: ld returned 1 exit status ------------------------- sorry if that was pointless*/ sorry if *this* is pointless. (I'm a bit frustrateh)
Feb 19 2010
I am sorry to hear about your problem, and I'm sure there are others who share your pain. If it is any consolation it is helpful to me to hear this, since the software that I intend to port to D relies heavily on plug-ins. It is disappointing that something as fundamental as plug-ins are such an issue in D. There has been recent talks about adding support for dynamic linking, but it doesn't seem to be a high enough priority yet. Hopefully it will get some attention soon. -Craig "g" <g g.g> wrote in message news:hln0i0$dm3$1 digitalmars.com...it is a real pain trying to make a plugin from d. I love D2 but I would even abandon it if there is solution with a compiler that at least supports D1 and has a solution for dynamic linking. Maybe I'm stupid, but tried dmd and ldc (and tango and phobos). And I don't know if I want to go GDC. It is so frustrating that DDL was abandoned, even I grabbed a external branch an is not so outdated, but probably outdated and w/o documentation. (btw linux) is there a not so painful way of making plugins?. Or is there still opportunity with DDL? I'm open to both phobos, tango, D1 D2 We NEED a way to make plugins from d. And is a must to use freely the features of D with out getting dirty (or not so dirty) or worse, limited to some features /*and for your pleasure some of the pain: ------------------------------------------------------- g-desktop:~/dynamic/ddl/ddl/Samples$ xfbuild +cldc -L-Wl,-Map -L-Wl,testHost01.map testHost01.d -I../.. -I/home/g2/ldc/import +oftest01 .objs/host.o: In function `_D3ddl11ExportClass44__T11ExportClassTC11testIface0110IHasFooBarZ11ExportClass14__T9newObjectZ9newObjectMFZC11testIface0110IHasFooBar': host:(.gnu.linkonce.t._D3ddl11ExportClass44__T11ExportClassTC11testIface0110IHasFooBarZ11ExportClass14__T9newObjectZ9newObjectMFZC11testIface01 0IHasFooBar+0x214): undefined reference to `_d_newclass' .objs/ddl-elf-ELFObjLoader.o:(.rodata+0x28): undefined reference to `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader14getLibraryTypeMFZAa' .objs/ddl-elf-ELFObjLoader.o:(.rodata+0x2c): undefined reference to `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader14canLoadLibraryMFS3ddl10FileBuffer10FileBufferZb' .objs/ddl-elf-ELFObjLoader.o:(.rodata+0x30): undefined reference to `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader4loadMFC3ddl14LoaderRegistry14LoaderRegistryS3ddl10FileBuffer10FileBufferZC3ddl14DynamicLibrary14DynamicLibrary' .objs/ddl-DynamicLibrary.o:(.rodata+0x28): undefined reference to `_D3ddl14DynamicLibrary14DynamicLibrary9getSymbolMFAaZPS3ddl12ExportSymbol12ExportSymbol' .objs/ddl-DynamicLibrary.o:(.rodata+0x2c): undefined reference to `_D3ddl14DynamicLibrary14DynamicLibrary10getModulesMFZAC3ddl13DynamicModule13DynamicModule' .objs/ddl-DynamicLibrary.o:(.rodata+0x30): undefined reference to `_D3ddl14DynamicLibrary14DynamicLibrary7getTypeMFZAa' .objs/ddl-DynamicLibrary.o:(.rodata+0x34): undefined reference to `_D3ddl14DynamicLibrary14DynamicLibrary13getAttributesMFZHAaAa' .objs/ddl-DynamicLibrary.o:(.rodata+0x3c): undefined reference to `_D3ddl14DynamicLibrary14DynamicLibrary18getModuleForSymbolMFAaZC3ddl13DynamicModule13DynamicModule' .objs/ddl-DynamicLibrary.o:(.rodata+0x40): undefined reference to `_D3ddl14DynamicLibrary14DynamicLibrary11getResourceMFAaZAh' .objs/ddl-DynamicModule.o:(.rodata+0x28): undefined reference to `_D3ddl13DynamicModule13DynamicModule7getNameMFZAa' .objs/ddl-DynamicModule.o:(.rodata+0x38): undefined reference to `_D3ddl13DynamicModule13DynamicModule10getSymbolsMFZAS3ddl12ExportSymbol12ExportSymbol' .objs/ddl-DynamicModule.o:(.rodata+0x3c): undefined reference to `_D3ddl13DynamicModule13DynamicModule9getSymbolMFAaZPS3ddl12ExportSymbol12ExportSymbol' .objs/ddl-DynamicModule.o:(.rodata+0x40): undefined reference to `_D3ddl13DynamicModule13DynamicModule20getSymbolLineNumbersMFZAS3ddl16SymbolLineNumber16SymbolLineNumber' .objs/ddl-DynamicModule.o:(.rodata+0x44): undefined reference to `_D3ddl13DynamicModule13DynamicModule13resolveFixupsMFZv' .objs/ddl-DynamicModule.o:(.rodata+0x48): undefined reference to `_D3ddl13DynamicModule13DynamicModule10isResolvedMFZb' .objs/ddl-DynamicLibraryLoader.o:(.rodata+0x28): undefined reference to `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader14getLibraryTypeMFZAa' .objs/ddl-DynamicLibraryLoader.o:(.rodata+0x2c): undefined reference to `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader14canLoadLibraryMFS3ddl10FileBuffer10FileBufferZb' .objs/ddl-DynamicLibraryLoader.o:(.rodata+0x30): undefined reference to `_D3ddl20DynamicLibraryLoader20DynamicLibraryLoader4loadMFC3ddl14LoaderRegistry14LoaderRegistryS3ddl10FileBuffer10FileBufferZC3ddl14DynamicLibrary14DynamicLibrary' collect2: ld returned 1 exit status ------------------------- sorry if that was pointless*/ sorry if *this* is pointless. (I'm a bit frustrateh)
Feb 19 2010
Let me preface by saying that I generally try not to be negative and have a heaps of praise and hope for what the D language and library designers are attempting to do. I don't want to spoil the party, however I think it's fair to recount my own real-world (*) D experience. (*) The success or otherwise of the project I'm working on literally == food or no food to put on my table. We are not talking about a hobby; it's a serious pursuit of my livelihood. Accordingly ... Commiserations dear friends. If it is any extra comfort to also share your pain alongside Craig et. al., let me tell you that I am (was) much in the same boat. Having spent six months 12x7 working a project in D1, and hoping that dynamic linking issues (**), might eventually be resolved, I gave up on D for serious biz. (**) Shared objects for the Linux platform, though ultimately aiming for Windows DLLs and a general cross-platform plug-in architecture. As a result of D's limitations in this regard, I was forced to back-port my project to C++ some 8 weeks ago. Now using Eclipse + C Dev Tools (CDT) and GCC as the IDE and tool-chain respectively, I feel I'm back to really cooking on gas once again. "Oh C++, why did I ever leave you?" Like yourselves, also having previously raised concerns re dyna-linking on this NG before, most responses from the top to both my and other users posts on this very topic have generally been dismissive of the problem. However, for me, and apart from all the other issues and frustrations that *you all know about* with D, the dyna-link thing was the thing that broke the camel's back, becoming the ultimate show-stopper. All was not lost though, I learned quite a few things from my 6 month experience with D that have helped to improve my C++ style, that which was becoming a bit stale after 20 years of it. To summarize, my feeling is that D is a great experimental language but you would not want to bet your farm, let alone your life, on it. Hopefully with the finalization of D2 and publication of TDPL, things will start to turn around. One would also hope that by this time some higher priority will be assigned to taking care of some of the more menial, yet absolutely fundamental issues affecting the real-world development and deployment of D applications. Sincerely, and with best wishes for D, Justin Johansson Craig Black wrote:I am sorry to hear about your problem, and I'm sure there are others who share your pain. If it is any consolation it is helpful to me to hear this, since the software that I intend to port to D relies heavily on plug-ins. It is disappointing that something as fundamental as plug-ins are such an issue in D. There has been recent talks about adding support for dynamic linking, but it doesn't seem to be a high enough priority yet. Hopefully it will get some attention soon. -Craig "g" <g g.g> wrote in message news:hln0i0$dm3$1 digitalmars.com...it is a real pain trying to make a plugin from d. I love D2 but I would even abandon it if there is solution with a compiler that at least supports D1 and has a solution for dynamic linking. Maybe I'm stupid, but tried dmd and ldc (and tango and phobos). And I don't know if I want to go GDC. It is so frustrating that DDL was abandoned, even I grabbed a external branch an is not so outdated, but probably outdated and w/o documentation. (btw linux) is there a not so painful way of making plugins?. Or is there still opportunity with DDL? I'm open to both phobos, tango, D1 D2 We NEED a way to make plugins from d. And is a must to use freely the features of D with out getting dirty (or not so dirty) or worse, limited to some features
Feb 19 2010
On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:it is a real pain trying to make a plugin from d. I love D2 but I would even abandon it if there is solution with a compiler that at least supports D1 and has a solution for dynamic linking.There's no way through. I was well on the way to something like Tomcat for D when it dawned on me that which ever way you turn, there's no support for dynamic loading - GDC included (shared libraries don't work). I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there. I just don't understand why this is not at the top of the urgent problems list, unless as I've said before D is just a language that forms a basis for the discussion of cool features that should be in compilers. If that's the case we can all continue to visit the D newsgroups for some intellectual stimulation (in Brit speak that's called intellectual So sad! Steve
Feb 20 2010
On 2010-02-20 16.53, Steve Teale wrote:On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:I've made a few dynamic libraries on Mac OS X that work.it is a real pain trying to make a plugin from d. I love D2 but I would even abandon it if there is solution with a compiler that at least supports D1 and has a solution for dynamic linking.There's no way through. I was well on the way to something like Tomcat for D when it dawned on me that which ever way you turn, there's no support for dynamic loading - GDC included (shared libraries don't work).I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there. I just don't understand why this is not at the top of the urgent problems list, unless as I've said before D is just a language that forms a basis for the discussion of cool features that should be in compilers. If that's the case we can all continue to visit the D newsgroups for some intellectual stimulation (in Brit speak that's called intellectual So sad! Steve
Feb 20 2010
Steve Teale wrote:On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:Well, obviously, there are more pressing things from Walter's perspective. For many programs, dynamic libraries are irrelevant, and there are major issues which are relevant to any programming project in D - dynamically linked or otherwise. So, presumably those are more important. I'm sure that dynamically linked libraries will get to the top of the priority queue eventually, but it looks like those who need it will just have to hold tight for now - as irritating as that may be. Being vocal about it might help push it up, though it might not (Walter tends to care about what he cares about whatever others might say). If it's a true showstopper for you, you can always contribute towards getting it working. Most programs can be written just fine without dynamically linked libraries, so I can certainly see why Walter would not consider it a high priority with everything else on his plate, but I do hope that he gets to it eventually. - Jonathan M Davisit is a real pain trying to make a plugin from d. I love D2 but I would even abandon it if there is solution with a compiler that at least supports D1 and has a solution for dynamic linking.There's no way through. I was well on the way to something like Tomcat for D when it dawned on me that which ever way you turn, there's no support for dynamic loading - GDC included (shared libraries don't work). I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there. I just don't understand why this is not at the top of the urgent problems list, unless as I've said before D is just a language that forms a basis for the discussion of cool features that should be in compilers. If that's the case we can all continue to visit the D newsgroups for some intellectual stimulation (in Brit speak that's called intellectual So sad! Steve
Feb 20 2010
Jonathan M Davis Wrote:Most programs can be written just fine without dynamically linked libraries, so I can certainly see why Walter would not consider it a high priority with everything else on his plate, but I do hope that he gets to it eventually.I disagree here. Yes they can be written, but at some point in time deveopers rewrite it using plugin architecture. From software that I use everyday plugginalble are: file manager(gnome), messenger(pidgin), browser(plugins of different taste, but nevertheless), IDEs(most of them: MSVC, Qt Creator, Eclipse...). I work for the company and all of its software is plugins to bigger programs...
Feb 21 2010
Steve Teale Wrote:On Fri, 19 Feb 2010 16:41:20 -0500, g wrote: I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there.Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Feb 21 2010
Eldar Insafutdinov wrote:Steve Teale Wrote:Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.On Fri, 19 Feb 2010 16:41:20 -0500, g wrote: I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there.Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Feb 21 2010
Justin Johansson wrote:Eldar Insafutdinov wrote:At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?Steve Teale Wrote:Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.On Fri, 19 Feb 2010 16:41:20 -0500, g wrote: I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there.Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Feb 21 2010
Lutger wrote:Justin Johansson wrote:Specifically my roadblock was the inability to create shared objects (.so files) on Linux platform (analogous to DLLs in Windows). This was required for the plug-in architecture that I was targeting.Eldar Insafutdinov wrote:At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?Steve Teale Wrote:Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.On Fri, 19 Feb 2010 16:41:20 -0500, g wrote: I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there.Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Feb 21 2010
Another important question: does the application still blow up if an exception gets thrown across the DLL boundary?
Feb 24 2010
Daniel Keep Wrote:Another important question: does the application still blow up if an exception gets thrown across the DLL boundary?If you called Runtime.loadLibrary() to load the DLL then it shouldn't.
Feb 24 2010
On Sun, 21 Feb 2010 14:06:44 +0100, Lutger wrote:Justin Johansson wrote:I have not got to the bottom of this yet - been doing building work to get a house finished that was deemed to have higher priority. Just to keep things basic, I did quite a bit of investigation on the creation of Linux shared libraries. I could not do it cleanly with either DMD or GDC. There's an account of what I tried on the blog section of http:// www.britseyeview.com/GDC-newbie.html. I think a major stumbling block is that Phobos is not a shared library, so the dynamic linking process can't just pull in stuff it needs when loading your shared library plugin. In addition though I think there may be a problem with the runtime. I had difficulties with a symbol called __data_start (or similar). I could see this in the host program, but it was not available to the dynamically loaded object. Also it is not 100% clear whether DMD can actually generate position independent code. I had previously tried to simplify DDL to make it at least be able to load a single object file, and in Windows, I had limited success, but it seemed to me that throwing a lot of effort into the ancient OMF format was not the way to go. I have not had time yet to attempt a similar thing with ELF. So there's more than just dynamic loading, the DMD Windows back end is a bit of a blast from the past anyway. I'm sorry this is rather vague, but I'm not just bleating without having tried some things. SteveEldar Insafutdinov wrote:At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?Steve Teale Wrote:Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.On Fri, 19 Feb 2010 16:41:20 -0500, g wrote: I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there.Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Feb 21 2010
On Sun, 21 Feb 2010 08:06:44 -0500, Lutger <lutger.blijdestijn gmail.com> wrote:Justin Johansson wrote:I know one issue in D1 is the GC. That's been fixed in D2 with druntime.Eldar Insafutdinov wrote:At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?Steve Teale Wrote:Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.On Fri, 19 Feb 2010 16:41:20 -0500, g wrote: I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there.Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Feb 21 2010
Hi, as I'm also hitting some problems with DLLs, here are some issues that I am now aware of (sorry, can't tell for linux shared objects, but I guess the situation is similar): 1. For D2, there is a major blocker with DLLs loaded after intialization on XP because of no TLS support from the OS. There is a simple workaround for single-threaded application (just setting FS:2c to a pointer to _tlsstart), but I'm considering a full emulation of the TLS initialization. 2. No multi-threading support: I've added a function to std.thread that allows adding a thread to the Thread.allThreads array, so it can be called from DLLMain/THREAD_ATTACH. That way the GC can suspend these, and allocations from different threads don't trample onto each other. Is there anything else needed for full multi-threading support? My patch is currently D1/Win32 only, but should not be too hard to extend to other platforms. Attaching to already existing threads is a bit harder, though, and it might not always be desired, because these threads might never call into D-code. 3. To try things out, I am playing around with the mydll-example, and it shows some quirks that you need to get around: the implementation-file for the DLL needs to use a file name different from the module name, because you need another file specifying the exports. Unfortunately this cannot be the di-file created from the source, because it contains two much information that will cause references to symbols not actually exported. Maybe some command line switch is needed to just write the exports into the di file without any implementations. Even better: import the implementation file, but don't create unnecessary references. (I think it's the module-initialization that's been called because of the import - maybe some modifier to the import could remove it). 4. The documentation on the website states, that you should use DATA PRELOAD SINGLE in the def-file. That probably is still there as a historical note to win 3.1, it will cause different processes to trample on each others data segments. This has caused a few hours of debugging, so please remove it, so others don't fall into the same trap. The mydll-example does not use the statement. 5. To share gc-collected objects between different DLLs, a common phobos-DLL seems necessary. Extracting the GC into a separate DLL and using the proxy-mechanism to attach any other client-DLL to it seems feasable, but are there other things that need to be shared between different phobos-instances? What about exception-handling? I'd say, if there is a way to put all public symbols into a def-file, then compile phobos into a DLL and create the import library and use this instead of a static library. The multi-threading-issue for DLLs needs to be solved before, though. I'm currently attacking 1. and 2., but only on windows at the moment. I'll add patches and reports into bugzilla later... Rainer Lutger wrote:Justin Johansson wrote:Eldar Insafutdinov wrote:At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?Steve Teale Wrote:Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.On Fri, 19 Feb 2010 16:41:20 -0500, g wrote: I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there.Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Feb 21 2010
Rainer Schuetze wrote:Hi, as I'm also hitting some problems with DLLs, here are some issuesthat I am now aware of (sorry, can't tell for linux shared objects, but I guess the situation is similar):1. For D2, there is a major blocker with DLLs loaded afterintialization on XP because of no TLS support from the OS. There is a simple workaround for single-threaded application (just setting FS:2c to a pointer to _tlsstart), but I'm considering a full emulation of the TLS initialization.2. No multi-threading support: I've added a function to std.threadthat allows adding a thread to the Thread.allThreads array, so it can be called from DLLMain/THREAD_ATTACH. That way the GC can suspend these, and allocations from different threads don't trample onto each other. Is there anything else needed for full multi-threading support?My patch is currently D1/Win32 only, but should not be too hard toextend to other platforms. Attaching to already existing threads is a bit harder, though, and it might not always be desired, because these threads might never call into D-code.3. To try things out, I am playing around with the mydll-example, andit shows some quirks that you need to get around: the implementation-file for the DLL needs to use a file name different from the module name, because you need another file specifying the exports. Unfortunately this cannot be the di-file created from the source, because it contains two much information that will cause references to symbols not actually exported.Maybe some command line switch is needed to just write the exportsinto the di file without any implementations. Even better: import the implementation file, but don't create unnecessary references. (I think it's the module-initialization that's been called because of the import - maybe some modifier to the import could remove it).4. The documentation on the website states, that you should use DATA PRELOAD SINGLE in the def-file. That probably is still there as a historical note towin 3.1, it will cause different processes to trample on each others data segments. This has caused a few hours of debugging, so please remove it, so others don't fall into the same trap. The mydll-example does not use the statement.5. To share gc-collected objects between different DLLs, a commonphobos-DLL seems necessary. Extracting the GC into a separate DLL and using the proxy-mechanism to attach any other client-DLL to it seems feasable, but are there other things that need to be shared between different phobos-instances? What about exception-handling?I'd say, if there is a way to put all public symbols into a def-file,then compile phobos into a DLL and create the import library and use this instead of a static library. The multi-threading-issue for DLLs needs to be solved before, though. Hi Rainer, Congrats. Sounds like you are giving the problem a lot of thought. Just regarding the public symbols though, I hazard a guess that you might run into a name-mangling problem here. Don't want to put a damper on your effort though; just a forewarning that this might be an issue. Perhaps someone else who knows about exporting DLL symbols might be gracious enough to chime in. Cheers Justin
Feb 22 2010
Rainer Schuetze wrote:5. To share gc-collected objects between different DLLs, a common phobos-DLL seems necessary. Extracting the GC into a separate DLL and using the proxy-mechanism to attach any other client-DLL to it seems feasable, but are there other things that need to be shared between different phobos-instances? What about exception-handling?Sounds like you have the right idea; that sounds similar to the way Microsoft does it with MFC DLL's. If I recall correctly, when MFC applications are DLL based, you link with a common C runtime DLL as well. This way all memory allocations and frees are handled by the common C runtime DLL (i.e. single point of responsibility). Presumably a similar regime for D and the GC would be necessary.
Feb 22 2010
Justin Johansson wrote:Rainer Schuetze wrote:Not only MFC. Plain C and C++ programs as well. This is why most MS- Windows based compiled languages comes with two versions of the runtime lib, one for static link and one for dynamic link (DLL). If not, there will always be problems with DLL's in that language sharing file handles, memory, sockets, threads and so on. The flip side of the coin is "DLL hell" (use Google). It is less of a problem in newer versions of MS-Windows than it used to be, but necessary to consider when supporting older platforms. The Linux/BSD/unix side I know less about, but since they have a common location for shared libs and support multiple versions of the same lib (AFAIK) the problems are less there. Roald5. To share gc-collected objects between different DLLs, a common phobos-DLL seems necessary. Extracting the GC into a separate DLL and using the proxy-mechanism to attach any other client-DLL to it seems feasable, but are there other things that need to be shared between different phobos-instances? What about exception-handling?Sounds like you have the right idea; that sounds similar to the way Microsoft does it with MFC DLL's. If I recall correctly, when MFC applications are DLL based, you link with a common C runtime DLL as well. This way all memory allocations and frees are handled by the common C runtime DLL (i.e. single point of responsibility). Presumably a similar regime for D and the GC would be necessary.
Feb 22 2010
Just to get it righ: this means GC will be in dl tool so people that complain about exe size and gc bloat will be happy about it? Roald Ribe Wrote:Justin Johansson wrote:Rainer Schuetze wrote:Not only MFC. Plain C and C++ programs as well. This is why most MS- Windows based compiled languages comes with two versions of the runtime lib, one for static link and one for dynamic link (DLL). If not, there will always be problems with DLL's in that language sharing file handles, memory, sockets, threads and so on. The flip side of the coin is "DLL hell" (use Google). It is less of a problem in newer versions of MS-Windows than it used to be, but necessary to consider when supporting older platforms. The Linux/BSD/unix side I know less about, but since they have a common location for shared libs and support multiple versions of the same lib (AFAIK) the problems are less there. Roald5. To share gc-collected objects between different DLLs, a common phobos-DLL seems necessary. Extracting the GC into a separate DLL and using the proxy-mechanism to attach any other client-DLL to it seems feasable, but are there other things that need to be shared between different phobos-instances? What about exception-handling?Sounds like you have the right idea; that sounds similar to the way Microsoft does it with MFC DLL's. If I recall correctly, when MFC applications are DLL based, you link with a common C runtime DLL as well. This way all memory allocations and frees are handled by the common C runtime DLL (i.e. single point of responsibility). Presumably a similar regime for D and the GC would be necessary.
Feb 23 2010
Thank you and everybody else for the explanation. Perhaps a shared phobos lib distributed with dmd will become a reality after all, not too far in the future.
Feb 22 2010
On Mon, 22 Feb 2010 02:46:31 -0500, Rainer Schuetze <r.sagitario gmx.de> wrote:5. To share gc-collected objects between different DLLs, a common phobos-DLL seems necessary. Extracting the GC into a separate DLL and using the proxy-mechanism to attach any other client-DLL to it seems feasable, but are there other things that need to be shared between different phobos-instances? What about exception-handling?druntime (D2) already supports sharing the GC between DLLs.
Feb 23 2010
I've seen that, too, but have not tried it yet. As Roald has also pointed out, there are other things you might want to share between dlls (threads, file handles, etc.). Robert Jacques wrote:On Mon, 22 Feb 2010 02:46:31 -0500, Rainer Schuetze <r.sagitario gmx.de> wrote:5. To share gc-collected objects between different DLLs, a common phobos-DLL seems necessary. Extracting the GC into a separate DLL and using the proxy-mechanism to attach any other client-DLL to it seems feasable, but are there other things that need to be shared between different phobos-instances? What about exception-handling?druntime (D2) already supports sharing the GC between DLLs.
Feb 23 2010
On Wed, 24 Feb 2010 02:38:58 -0500, Rainer Schuetze <r.sagitario gmx.de> wrote:I've seen that, too, but have not tried it yet. As Roald has also pointed out, there are other things you might want to share between dlls (threads, file handles, etc.).Given the GC has to know about all threads, I assume those are shared as well. As for file handles, those are just data variables, which can be easily shared between DLLs. The bug problem, as I understand it, with DLLs is global data requires manual re-routing, etc. to work.
Feb 24 2010
Rainer Schuetze wrote:Hi, as I'm also hitting some problems with DLLs, here are some issues that I am now aware of (sorry, can't tell for linux shared objects, but I guess the situation is similar): 1. For D2, there is a major blocker with DLLs loaded after intialization on XP because of no TLS support from the OS. There is a simple workaround for single-threaded application (just setting FS:2c to a pointer to _tlsstart), but I'm considering a full emulation of the TLS initialization.That's incredibly difficult. Have you read this? http://www.nynaeve.net/?p=189 It'd be great if you can get it to work. But I fear that for Win32, DMD is going to have to switch to explicit TLS.
Feb 24 2010
Don wrote:Rainer Schuetze wrote:I talked to the guy who wrote that. He wrote some workaround code to make TLS work on older Windows versions, and I asked him if we could license it. He was reluctant, saying it was experimental only. So yeah, rolling our own may be the only viable option.Hi, as I'm also hitting some problems with DLLs, here are some issues that I am now aware of (sorry, can't tell for linux shared objects, but I guess the situation is similar): 1. For D2, there is a major blocker with DLLs loaded after intialization on XP because of no TLS support from the OS. There is a simple workaround for single-threaded application (just setting FS:2c to a pointer to _tlsstart), but I'm considering a full emulation of the TLS initialization.That's incredibly difficult. Have you read this? http://www.nynaeve.net/?p=189 It'd be great if you can get it to work. But I fear that for Win32, DMD is going to have to switch to explicit TLS.
Feb 24 2010
I knew that blog, but I expected to find the necessary data somewhere in the "undocumented" (but still available) structures. But I had to dig a little deeper into ntdll myself and came up with a patch that is part of the multithread-support for DLLs: http://d.puremagic.com/issues/show_bug.cgi?id=3885 It emulates the behaviour at process startup time, and modifies the data structures so that windows handles the DLL as if it was loaded with the process. This has the nice benefit that you don't need to handle the creation of new threads. The downsides are: - DLL-unload is disabled, but I think unloading a DLL that uses TLS is problematic anyway) - It will leak some pointer-arrays in favor of not causing crashes due to other threads using TLS at same time as reallocating these arrays. As XP is not a fast moving target anymore, I hope that ntdll will not change a lot, so it will not need adjustment for every windows-update. I've checked SP2 and SP3, which look very much the same with respect to TLS implementation. Rainer Don wrote:Rainer Schuetze wrote:Hi, as I'm also hitting some problems with DLLs, here are some issues that I am now aware of (sorry, can't tell for linux shared objects, but I guess the situation is similar): 1. For D2, there is a major blocker with DLLs loaded after intialization on XP because of no TLS support from the OS. There is a simple workaround for single-threaded application (just setting FS:2c to a pointer to _tlsstart), but I'm considering a full emulation of the TLS initialization.That's incredibly difficult. Have you read this? http://www.nynaeve.net/?p=189 It'd be great if you can get it to work. But I fear that for Win32, DMD is going to have to switch to explicit TLS.
Mar 06 2010
Rainer Schuetze wrote:I knew that blog, but I expected to find the necessary data somewhere in the "undocumented" (but still available) structures. But I had to dig a little deeper into ntdll myself and came up with a patch that is part of the multithread-support for DLLs: http://d.puremagic.com/issues/show_bug.cgi?id=3885 It emulates the behaviour at process startup time, and modifies the data structures so that windows handles the DLL as if it was loaded with the process. This has the nice benefit that you don't need to handle the creation of new threads. The downsides are: - DLL-unload is disabled, but I think unloading a DLL that uses TLS is problematic anyway) - It will leak some pointer-arrays in favor of not causing crashes due to other threads using TLS at same time as reallocating these arrays. As XP is not a fast moving target anymore, I hope that ntdll will not change a lot, so it will not need adjustment for every windows-update. I've checked SP2 and SP3, which look very much the same with respect to TLS implementation. RainerYou're totally awesome. This is a game-changer.
Mar 09 2010
Don wrote:Rainer Schuetze wrote:Thanks :-) The patch might still need a little polishing before finding it's way into druntime, though...I knew that blog, but I expected to find the necessary data somewhere in the "undocumented" (but still available) structures. But I had to dig a little deeper into ntdll myself and came up with a patch that is part of the multithread-support for DLLs: http://d.puremagic.com/issues/show_bug.cgi?id=3885 It emulates the behaviour at process startup time, and modifies the data structures so that windows handles the DLL as if it was loaded with the process. This has the nice benefit that you don't need to handle the creation of new threads. The downsides are: - DLL-unload is disabled, but I think unloading a DLL that uses TLS is problematic anyway) - It will leak some pointer-arrays in favor of not causing crashes due to other threads using TLS at same time as reallocating these arrays. As XP is not a fast moving target anymore, I hope that ntdll will not change a lot, so it will not need adjustment for every windows-update. I've checked SP2 and SP3, which look very much the same with respect to TLS implementation. RainerYou're totally awesome. This is a game-changer.
Mar 10 2010
g wrote:(btw linux)The first step is to make phobos into a shared .so library. Is anyone willing to try it and sort out just what the issues are?
Feb 21 2010
The first step is to make phobos into a shared .so library. Is anyone willing to try it and sort out just what the issues are?I may give it a try, if nobody else wants to jump on it.
Feb 21 2010
Igor Lesik wrote:Thanks! dmd has the first step, it supports -fPIC.The first step is to make phobos into a shared .so library. Is anyone willing to try it and sort out just what the issues are?I may give it a try, if nobody else wants to jump on it.
Feb 21 2010
On Sun, 21 Feb 2010 16:05:47 -0800, Walter Bright wrote:Igor Lesik wrote:So it's real? There's still something wrong though - I had a big bash at shared library about a week ago - no joy. Great Walter - thanks. Igor - please get in touch, I may be able to help. SteveThanks! dmd has the first step, it supports -fPIC.The first step is to make phobos into a shared .so library. Is anyone willing to try it and sort out just what the issues are?I may give it a try, if nobody else wants to jump on it.
Feb 24 2010
Sure. Most probably I will need help. Thanks.Igor Lesik wrote:So it's real? There's still something wrong though - I had a big bash at shared library about a week ago - no joy. Great Walter - thanks. Igor - please get in touch, I may be able to help. SteveThanks! dmd has the first step, it supports -fPIC.The first step is to make phobos into a shared .so library. Is anyone willing to try it and sort out just what the issues are?I may give it a try, if nobody else wants to jump on it.
Feb 24 2010
On 2010-02-21 19.50, Walter Bright wrote:g wrote:I've been trying to do this from time to time on Mac OS X, I've also tried with tango. The issues I've encountered are undefined symbols: Undefined symbols: "__deh_beg", referenced from: __deh_beg$non_lazy_ptr in deh2.o "__deh_end", referenced from: __deh_end$non_lazy_ptr in deh2.o "__Dmain", referenced from: _main in dmain2.o _main in dmain2.o "__minfo_beg", referenced from: __minfo_beg$non_lazy_ptr in moduleinit.o "__minfo_end", referenced from: __minfo_end$non_lazy_ptr in moduleinit.o GDC doesn't have this problem. This is how GDC solves some of the above undefined symbols: __Dmain: GDC doesn't have the C and D main functions in dmain2.d, instead it has another file with the main functions that should not be linked if you build it as a dynamic/shared library. I tried to do this with tango and dmd but then when I built a executable I got undefined symbol on the C main function. __minfo_beg and __minfo_end: Neither GDC or LDC have these. I don't remember how they handle this but it can probably be solved using the function "getsegbyname" or any of the similar functions that exist. For this I would need more precise information in which segment and section they are located. __deh_beg and __deh_end: Both GDC and LDC uses a different exception handling implementation than dmd and don't have these. I have no idea what to do about these. /Jacob Carlborg(btw linux)The first step is to make phobos into a shared .so library. Is anyone willing to try it and sort out just what the issues are?
Feb 22 2010
A nasty typo, should be "...GC will be in dll too so people..."Just to get it righ: this means GC will be in dl tool so people that complain about exe size and gc bloat will be happy about it?
Feb 23 2010
Yes, GC is the main issue, as you want to pass gc-collected objects across dll boundaries. I'd say the size of the executables is just a secondary issue, though it would also benefit. Bane wrote:A nasty typo, should be "...GC will be in dll too so people..."Just to get it righ: this means GC will be in dl tool so people that complain about exe size and gc bloat will be happy about it?
Feb 23 2010
Rainer Schuetze wrote:Yes, GC is the main issue, as you want to pass gc-collected objects across dll boundaries. I'd say the size of the executables is just a secondary issue, though it would also benefit.The GC itself doesn't have to be in a dll for this to work. The library is already set up to hand off the GC. What it doesn't do at the moment is hand off the thread management.
Feb 24 2010