www.digitalmars.com         C & C++   DMDScript  

D.gnu - This week's experiment build gdc with tls disabled.

reply "Iain Buclaw" <ibuclaw ubuntu.com> writes:
We really need to understand this problem first before can start 
merging Martin's shared library work into gdc.

Running testsuite:

=== gdc Summary ===




FAIL: runnable/testaa.d execution test


Running unittests:

./libdruntime/unittest
Success!

./src/unittest
Floating point exception (core dumped)


This has me surprised.
Dec 16 2013
next sibling parent "Iain Buclaw" <ibuclaw ubuntu.com> writes:
On Monday, 16 December 2013 at 15:46:59 UTC, Iain Buclaw wrote:

 We really need to understand this problem first before can 
 start merging Martin's shared library work into gdc.

 Running testsuite:

 === gdc Summary ===




 FAIL: runnable/testaa.d execution test


OF course, the problem that I am running into here is in the same 
spirit as:

http://d.puremagic.com/issues/show_bug.cgi?id=11139
Dec 16 2013
prev sibling parent reply "Iain Buclaw" <ibuclaw ubuntu.com> writes:
Gthreads and Emulated TLS are now in GDC!!!

https://github.com/D-Programming-GDC/GDC/commit/62554bfe1b35ee4f586634a76c65d83ebfa871ef

The bug noted above still exists, however what I am going to do 
is update and hook emulated TLS into the GC.
Dec 17 2013
next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 17 December 2013 at 17:48:40 UTC, Iain Buclaw wrote:

 Gthreads and Emulated TLS are now in GDC!!!

 https://github.com/D-Programming-GDC/GDC/commit/62554bfe1b35ee4f586634a76c65d83ebfa871ef

 The bug noted above still exists, however what I am going to do 
 is update and hook emulated TLS into the GC.


Warning, n00b question ahead:

What are the wider implications for this?

Will this fix the problems with tls for embedded targets, windows 
xp and os x < 10.7?
Dec 17 2013
next sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On 17 December 2013 18:35, John Colvin <john.loughran.colvin gmail.com> wrote:

 On Tuesday, 17 December 2013 at 17:48:40 UTC, Iain Buclaw wrote:

 Gthreads and Emulated TLS are now in GDC!!!


 https://github.com/D-Programming-GDC/GDC/commit/62554bfe1b35ee4f586634a76c65d83ebfa871ef

 The bug noted above still exists, however what I am going to do is update
 and hook emulated TLS into the GC.



 Warning, n00b question ahead:

 What are the wider implications for this?

 Will this fix the problems with tls for embedded targets, windows xp and os
 x < 10.7?


Yes.
Dec 17 2013
prev sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 17 December 2013 19:08, Iain Buclaw <ibuclaw gdcproject.org> wrote:

 On 17 December 2013 18:35, John Colvin <john.loughran.colvin gmail.com> wrote:

 On Tuesday, 17 December 2013 at 17:48:40 UTC, Iain Buclaw wrote:

 Gthreads and Emulated TLS are now in GDC!!!


 https://github.com/D-Programming-GDC/GDC/commit/62554bfe1b35ee4f586634a76c65d83ebfa871ef

 The bug noted above still exists, however what I am going to do is update
 and hook emulated TLS into the GC.



 Warning, n00b question ahead:

 What are the wider implications for this?

 Will this fix the problems with tls for embedded targets, windows xp and os
 x < 10.7?


 Yes.


And will also make them primed and ready for shared library support too.

Native TLS support on linux will have to wait a while before I get
that working... (but you could always build with --disable-tls)
Dec 17 2013
parent reply "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 17 December 2013 at 19:09:31 UTC, Iain Buclaw wrote:

 Native TLS support on linux will have to wait a while before I 
 get
 that working... (but you could always build with --disable-tls)


Wait, so GDC doesn't (entirely) support native TLS on Linux?! Or 
are you just talking about shared libraries?

David
Dec 17 2013
parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 17 December 2013 19:34, David Nadlinger <code klickverbot.at> wrote:

 On Tuesday, 17 December 2013 at 19:09:31 UTC, Iain Buclaw wrote:

 Native TLS support on linux will have to wait a while before I get
 that working... (but you could always build with --disable-tls)



 Wait, so GDC doesn't (entirely) support native TLS on Linux?! Or are you
 just talking about shared libraries?


Shared libraries (sorry, it made sense to me when I wrote it)... The
hidden subtext that you didn't understand being, I'm holding back on
Martin's patches for now.
Dec 17 2013
parent reply "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 17 December 2013 at 20:07:41 UTC, Iain Buclaw wrote:

 The hidden subtext that you didn't understand being, I'm 
 holding back on Martin's patches for now.


That subtext isn't exactly hidden, given the first sentence in 
your first post. ;)

Do you have a plan yet regarding how to implement module 
discovery for shared libraries? Would be a good idea to 
coordinate efforts and find a solution that works for all the 
backends, as Martin has been suggesting as well.

David
Dec 17 2013
next sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On 17 December 2013 20:12, David Nadlinger <code klickverbot.at> wrote:

 On Tuesday, 17 December 2013 at 20:07:41 UTC, Iain Buclaw wrote:

 The hidden subtext that you didn't understand being, I'm holding back on
 Martin's patches for now.



 That subtext isn't exactly hidden, given the first sentence in your first
 post. ;)

 Do you have a plan yet regarding how to implement module discovery for
 shared libraries? Would be a good idea to coordinate efforts and find a
 solution that works for all the backends, as Martin has been suggesting as
 well.


Not yet, but that will be my focus after getting shared libs with
emulated TLS working.
Dec 17 2013
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 17 Dec 2013 21:12:41 +0100
schrieb "David Nadlinger" <code klickverbot.at>:


 On Tuesday, 17 December 2013 at 20:07:41 UTC, Iain Buclaw wrote:

 The hidden subtext that you didn't understand being, I'm 
 holding back on Martin's patches for now.


 
 That subtext isn't exactly hidden, given the first sentence in 
 your first post. ;)
 
 Do you have a plan yet regarding how to implement module 
 discovery for shared libraries? Would be a good idea to 
 coordinate efforts and find a solution that works for all the 
 backends, as Martin has been suggesting as well.
 
 David


I hope I'm not talking bullshit here as I'm not 100% sure what's meant
with 'module discovery'.

But if the question is how to get all ModuleInfos in a library, we
should probably not forget the way C handles it's 'module
constructors', aka the '.ctors' / '.dtors' section.

They use linker scripts to make sure these sections are always
correctly bracketed by _start and _end symbols and we could do the same
for a '.moduleinfo' section and for the standard tls sections.

Here's the relevant part of a linker script:
------------------------
  .ctors          :
  {
    /* gcc uses crtbegin.o to find the start of
       the constructors, so we make sure it is
       first.  Because this is a wildcard, it
       doesn't matter if the user does not
       actually link against crtbegin.o; the
       linker won't look for a file to match a
       wildcard.  The wildcard also means that it
       doesn't matter which directory crtbegin.o
       is in.  */
    KEEP (*crtbegin.o(.ctors))
    KEEP (*crtbegin?.o(.ctors))
    /* We don't want to include the .ctor section from
       the crtend.o file until after the sorted ctors.
       The .ctor section from the crtend file contains the
       end of ctors marker and it must be last */
    KEEP (*(EXCLUDE_FILE (*crtend.o *crtend?.o ) .ctors))
    KEEP (*(SORT(.ctors.*)))
    KEEP (*(.ctors))
  }
------------------------

We can then access the bracket-symbols from a module-constructor (or a
c-like .ctor constructor) in the library? So we'd have to add
drtbegin.o / drtend.o files to every d library or executable.

Ideally we'd get the default linker scripts changed, but that may take
some time if the binutils guys even agree. It's also possible to
1) give a linker script to ld to replace the default script (not so
   good for us)
2) give ld a script to extend the default script (could work
   for .moduleinfo section, I'm not sure about TLS)

One big problem is of course portability. But it seems such a solution
might be more portable than a solution relying on runtime
linkers (especially considering gcc already uses this internally for c++
support)
Dec 18 2013
next sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 18 December 2013 15:39, Johannes Pfau <nospam example.com> wrote:

 Am Tue, 17 Dec 2013 21:12:41 +0100
 schrieb "David Nadlinger" <code klickverbot.at>:


 On Tuesday, 17 December 2013 at 20:07:41 UTC, Iain Buclaw wrote:

 The hidden subtext that you didn't understand being, I'm
 holding back on Martin's patches for now.


 That subtext isn't exactly hidden, given the first sentence in
 your first post. ;)

 Do you have a plan yet regarding how to implement module
 discovery for shared libraries? Would be a good idea to
 coordinate efforts and find a solution that works for all the
 backends, as Martin has been suggesting as well.

 David


 I hope I'm not talking bullshit here as I'm not 100% sure what's meant
 with 'module discovery'.


I'm not so sure about 'module discovery' either.  At least, in
emulated TLS, it has a completely different concept - each thread has
a dynamically allocated range (effectively, a void**[] on the heap
that gets destroyed upon thread termination) which is shared amongst
all modules / loaded libraries in D for the duration of the thread.
So when it comes to calling getTLSRange() - what is effectively
happening is:

  void**[] tlsarray = gthread_getspecific(emutls_key);
  return cast(void[]) tlsarray[0 .. $];

What I'm hoping is that whatever Martin has done, it is compatible
with this way of doing things...
Dec 18 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-12-18 18:36, Iain Buclaw wrote:


 I'm not so sure about 'module discovery' either.  At least, in
 emulated TLS, it has a completely different concept - each thread has
 a dynamically allocated range (effectively, a void**[] on the heap
 that gets destroyed upon thread termination) which is shared amongst
 all modules / loaded libraries in D for the duration of the thread.
 So when it comes to calling getTLSRange() - what is effectively
 happening is:

    void**[] tlsarray = gthread_getspecific(emutls_key);
    return cast(void[]) tlsarray[0 .. $];

 What I'm hoping is that whatever Martin has done, it is compatible
 with this way of doing things...


After the changes Margin has done DMD now emits .ctors and .dtors 
sections to to every executable and shared library. These will call 
"_d_dso_registry", passing in, among other things, the start and end of 
the module infos.

To get the TLS data the runtime iterates the sections of the 
executable/shared library and collections these.

I don't know if that will be compatible with how it's done in GDC.

This is only for Linux, FreeBSD and Mac OS X behaves as it did before. 
FreeBSD uses bracked sections and Mac OS X doing something similar as 
Linux does now, the dynamic linker has API's for this.

-- 
/Jacob Carlborg
Dec 18 2013
prev sibling parent Johannes Pfau <nospam example.com> writes:
Am Wed, 18 Dec 2013 17:36:53 +0000
schrieb Iain Buclaw <ibuclaw gdcproject.org>:


 On 18 December 2013 15:39, Johannes Pfau <nospam example.com> wrote:

 Am Tue, 17 Dec 2013 21:12:41 +0100
 schrieb "David Nadlinger" <code klickverbot.at>:


 On Tuesday, 17 December 2013 at 20:07:41 UTC, Iain Buclaw wrote:

 The hidden subtext that you didn't understand being, I'm
 holding back on Martin's patches for now.


 That subtext isn't exactly hidden, given the first sentence in
 your first post. ;)

 Do you have a plan yet regarding how to implement module
 discovery for shared libraries? Would be a good idea to
 coordinate efforts and find a solution that works for all the
 backends, as Martin has been suggesting as well.

 David


 I hope I'm not talking bullshit here as I'm not 100% sure what's
 meant with 'module discovery'.


 
 I'm not so sure about 'module discovery' either.  At least, in
 emulated TLS, it has a completely different concept - each thread has
 a dynamically allocated range (effectively, a void**[] on the heap
 that gets destroyed upon thread termination) which is shared amongst
 all modules / loaded libraries in D for the duration of the thread.
 So when it comes to calling getTLSRange() - what is effectively
 happening is:
 
   void**[] tlsarray = gthread_getspecific(emutls_key);
   return cast(void[]) tlsarray[0 .. $];
 
 What I'm hoping is that whatever Martin has done, it is compatible
 with this way of doing things...


I guess you can add the scan code here:
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/tlsgc.d#L62
Dec 19 2013
prev sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Wednesday, 18 December 2013 at 15:39:30 UTC, Johannes Pfau 
wrote:

 I hope I'm not talking bullshit here as I'm not 100% sure 
 what's meant
 with 'module discovery'.

 But if the question is how to get all ModuleInfos in a library, 
 […]


Yes, this was the idea.


 We can then access the bracket-symbols from a 
 module-constructor (or a
 c-like .ctor constructor) in the library? So we'd have to add
 drtbegin.o / drtend.o files to every d library or executable.


That is pretty much how it is done in DMD and LDC as well. The 
only difference is that instead of modifying the linker scripts 
to accommodate this, we emit the ModuleInfo references to custom 
sections. The GNU toolchain (and we are in highly system-specific 
territory here anyway) never changes the order of custom 
sections, which you can also verify using 
__attribute__((section("...))) in GCC. Thus, if you emit your 
relevant symbols into three sections like this,

a.o
---
_minfo_beg: moduleInfoBeginTag
_minfo: moduleInfoForA
_minfo_end: moduleInfoEndTag

b.o
---
_minfo_beg: moduleInfoBeginTag
_minfo: moduleInfoForB
_minfo_end: moduleInfoEndTag

the linked result will be

_minfo_beg: moduleInfoBeginTag
_minfo: moduleInfoForA moduleInfoForB
_minfo_end: moduleInfoEndTag

and you can just use [moduleInfoBeginTag, moduleInfoEndTag) to 
get a list of all available ModuleInfos.


Of course, once shared libraries come into play, you have to 
consider quite a few subtleties regarding symbol visibility and 
so on. Also, due to a bug in LLVM, LDC can't emit weak symbols 
with custom section names, so we end up with multiple copies of 
the begin/end symbols and the C .ctor/.dtor table entries.

But in general, this works quite well, and is certainly much more 
portable than requiring changes to the linker script.

David
Dec 19 2013
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 19 Dec 2013 18:59:42 +0100
schrieb "David Nadlinger" <code klickverbot.at>:


 Yes, this was the idea.
 

 We can then access the bracket-symbols from a 
 module-constructor (or a
 c-like .ctor constructor) in the library? So we'd have to add
 drtbegin.o / drtend.o files to every d library or executable.


 
 That is pretty much how it is done in DMD and LDC as well. The 
 only difference is that instead of modifying the linker scripts 
 to accommodate this, we emit the ModuleInfo references to custom 
 sections. The GNU toolchain (and we are in highly system-specific 
 territory here anyway) never changes the order of custom 
 sections, which you can also verify using 
 __attribute__((section("...))) in GCC. Thus, if you emit your 
 relevant symbols into three sections like this,
 
 a.o
 ---
 _minfo_beg: moduleInfoBeginTag
 _minfo: moduleInfoForA
 _minfo_end: moduleInfoEndTag
 
 b.o
 ---
 _minfo_beg: moduleInfoBeginTag
 _minfo: moduleInfoForB
 _minfo_end: moduleInfoEndTag
 
 the linked result will be
 
 _minfo_beg: moduleInfoBeginTag
 _minfo: moduleInfoForA moduleInfoForB
 _minfo_end: moduleInfoEndTag
 
 and you can just use [moduleInfoBeginTag, moduleInfoEndTag) to 
 get a list of all available ModuleInfos.
 
 
 Of course, once shared libraries come into play, you have to 
 consider quite a few subtleties regarding symbol visibility and 
 so on.


Yes, I got that part. Begin/End symbols should be private to the library
and accessed from a constructor function in that library which then
passes the addresses of these symbols to the _d_dso_registry function.

(And for TLS if we had these symbols tey would also be 'private' to
different threads, so we'd have to register them at thread startup for
every dso but AFAICS the currect DMD code basically does that already)

However, I'd prefer to pass an additional object file to ld which
contains this constructor instead of placing this code into every
object file, but that's probably just a personal preference.


 Also, due to a bug in LLVM, LDC can't emit weak symbols 
 with custom section names, so we end up with multiple copies of 
 the begin/end symbols and the C .ctor/.dtor table entries.
 
 But in general, this works quite well, and is certainly much more 
 portable than requiring changes to the linker script.
 
 David


Sounds like that could work. But as the module section is a custom
section anyway we wouldn't have to replace/modify the default linker
script - we can pass a custom script to ld which just handles the
".minfo" section. That should be just as portable as relying on the
"don't reorder sections" behavior: Works everywhere where GNU Binutils
LD/GOLD are used. (Emitting 3 sections is a clever trick, but it feels
like a hack imho. I'm also not sure if we can control the order in
which sections are emitted in GCC)

Even if we don't want to go that route I'll try to write a proof of
concept this weekend as I really want to know now whether this will
work.


It seems like getting the TLS section is the more interesting part. We
can't emit sections around the TLS section so IIRC the current dmd
implementation therefore relies on the runtime linker and libc specific
interfaces?

I think asking the binutils maintainers to add __tdata_begin,
__tdata_end, __tbss_begin and __tbss_end markers to the tdata and tbss
sections would be a nice long-term solution, or is there some issue
with that?

If the maintainers don't want to add these extra symbols for every
library we might ask for special treatment of dso_start.o/dso_end.o
files, similar to what they do for crtbegin.o/crtend.o for C++.


I'm wondering though if ld actually reorders symbols in a section by
default? I know there are some --sort-section options, but does it sort
by default? IIRC we used to use TLS bracketing in GDC and Iain said
there were some issues. But if we placed the begin/end markers into
extra object files and supplied them as first/last file to the linker,
wouldn't that work for now?
Dec 19 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-19 20:51, Johannes Pfau wrote:


 Sounds like that could work. But as the module section is a custom
 section anyway we wouldn't have to replace/modify the default linker
 script - we can pass a custom script to ld which just handles the
 ".minfo" section. That should be just as portable as relying on the
 "don't reorder sections" behavior: Works everywhere where GNU Binutils
 LD/GOLD are used. (Emitting 3 sections is a clever trick, but it feels
 like a hack imho. I'm also not sure if we can control the order in
 which sections are emitted in GCC)


The trick with brackting sections doesn't work on Mac OS X. DMD had some 
problems with that. It basically broke on every new major release of Mac 
OS X, the linker changed all the time. If I recall correctly it either 
removed the empty sections or reordered the sections.


 It seems like getting the TLS section is the more interesting part. We
 can't emit sections around the TLS section so IIRC the current dmd
 implementation therefore relies on the runtime linker and libc specific
 interfaces?


Yes, using dl_iterate_phdr: http://linux.die.net/man/3/dl_iterate_phdr.


 I think asking the binutils maintainers to add __tdata_begin,
 __tdata_end, __tbss_begin and __tbss_end markers to the tdata and tbss
 sections would be a nice long-term solution, or is there some issue
 with that?


How does that work? Do people need to update their linkers for that to work?

-- 
/Jacob Carlborg
Dec 19 2013
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 20 Dec 2013 08:40:28 +0100
schrieb Jacob Carlborg <doob me.com>:


 I think asking the binutils maintainers to add __tdata_begin,
 __tdata_end, __tbss_begin and __tbss_end markers to the tdata and
 tbss sections would be a nice long-term solution, or is there some
 issue with that?


 
 How does that work? Do people need to update their linkers for that
 to work?
 


Likely yes, that's why I said 'longterm' solution. You need to edit the
default linker script. Then there are 2 options: We can force certain
object files to be at the start or the end of the section. So we could
introduce dso_start.o and dso_end.o files containing the start/end
sections and those would always be placed as the first/last objects in
a section. (This is how the C++/C runtime calls the constructors in the
.ctors section: crtbegin.o contains a __CTOR_LIST__ symbol)

It's also possible to directly instruct the linker to insert a symbol
at the start of the section and at the end of the section. This way you
don't need special object files but the symbols will appear always,
even for C/C++ programs.

In theory it's always possible to use your own linker script without
updating the compiler, but that's not a practical solution. Implicit
linker-script which only extend the default script can be used for
custom sections, but AFAIK this doesn't work for standard sections.
Dec 20 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-12-20 10:13, Johannes Pfau wrote:


 Likely yes, that's why I said 'longterm' solution. You need to edit the
 default linker script. Then there are 2 options: We can force certain
 object files to be at the start or the end of the section. So we could
 introduce dso_start.o and dso_end.o files containing the start/end
 sections and those would always be placed as the first/last objects in
 a section. (This is how the C++/C runtime calls the constructors in the
 .ctors section: crtbegin.o contains a __CTOR_LIST__ symbol)

 It's also possible to directly instruct the linker to insert a symbol
 at the start of the section and at the end of the section. This way you
 don't need special object files but the symbols will appear always,
 even for C/C++ programs.

 In theory it's always possible to use your own linker script without
 updating the compiler, but that's not a practical solution. Implicit
 linker-script which only extend the default script can be used for
 custom sections, but AFAIK this doesn't work for standard sections.


Ok, I see.

-- 
/Jacob Carlborg
Dec 20 2013
prev sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 20 December 2013 07:40, Jacob Carlborg <doob me.com> wrote:

 On 2013-12-19 20:51, Johannes Pfau wrote:


 Sounds like that could work. But as the module section is a custom
 section anyway we wouldn't have to replace/modify the default linker
 script - we can pass a custom script to ld which just handles the
 ".minfo" section. That should be just as portable as relying on the
 "don't reorder sections" behavior: Works everywhere where GNU Binutils
 LD/GOLD are used. (Emitting 3 sections is a clever trick, but it feels
 like a hack imho. I'm also not sure if we can control the order in
 which sections are emitted in GCC)



 The trick with brackting sections doesn't work on Mac OS X. DMD had some
 problems with that. It basically broke on every new major release of Mac OS
 X, the linker changed all the time. If I recall correctly it either removed
 the empty sections or reordered the sections.


If using GDC on OSX, then you be going through the emutls module I've
got set-up.

As for module discovery, we already generate this in the GDC backend,
which makes it then the job of runtime to pick-up, sort and run the
ctors on all modules:

 attribute("constructor")
void __modinit()
{
  extern (C) __gshared ModuleReference* _Dmodule_ref;
  static private ModuleReference __mymod = ModuleReference(null, minfo);

  __mymod.next = _Dmodule_ref;
  _Dmodule_ref = &__mymod;
}



Unless I'm missing something in how module discovery is supposed to
work, this could be instead amended to:

 attribute("constructor")
void __modinit()
{
  extern (C) void __register_module(ModuleInfo *);

  __register_module(&minfo);
}

So that any lazily loaded modules would be recorded to an existing
list and trigger a run-once ctor run.

Regards
Iain
Dec 20 2013
parent reply "David Nadlinger" <code klickverbot.at> writes:
On Friday, 20 December 2013 at 10:54:56 UTC, Iain Buclaw wrote:

 As for module discovery, we already generate this in the GDC 
 backend,
 which makes it then the job of runtime to pick-up, sort and run 
 the
 ctors on all modules:

  attribute("constructor")
 void __modinit()
 {
   extern (C) __gshared ModuleReference* _Dmodule_ref;
   static private ModuleReference __mymod = 
 ModuleReference(null, minfo);

   __mymod.next = _Dmodule_ref;
   _Dmodule_ref = &__mymod;
 }



 Unless I'm missing something in how module discovery is 
 supposed to
 work, this could be instead amended to:

  attribute("constructor")
 void __modinit()
 {
   extern (C) void __register_module(ModuleInfo *);

   __register_module(&minfo);
 }

 So that any lazily loaded modules would be recorded to an 
 existing
 list and trigger a run-once ctor run.


The _Dmodule_ref approach is virtually the same as we have been 
doing in LDC up to now. The problem with regard to runtime 
loading of shared libraries is that you need to run certain code 
(i.e. the current _d_dso_registry in DMD's/LDC's druntime) when 
you know that you have collected all the modules. You can't do 
this from a .ctor, because if you also use them for constructing 
the module list, you'd need to call the registry function in the 
last .ctor to run. And at least if you don't know the order in 
which they are written to the executable, you can't know that.

It might be possible to work around this requirement by rewriting 
the druntime internals to perform all the collision checking, 
etc. one-by-one, but I didn't investigate this in more detail.

David
Dec 20 2013
parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 20 December 2013 13:06, David Nadlinger <code klickverbot.at> wrote:

 On Friday, 20 December 2013 at 10:54:56 UTC, Iain Buclaw wrote:

 As for module discovery, we already generate this in the GDC backend,
 which makes it then the job of runtime to pick-up, sort and run the
 ctors on all modules:

  attribute("constructor")
 void __modinit()
 {
   extern (C) __gshared ModuleReference* _Dmodule_ref;
   static private ModuleReference __mymod = ModuleReference(null, minfo);

   __mymod.next = _Dmodule_ref;
   _Dmodule_ref = &__mymod;
 }



 Unless I'm missing something in how module discovery is supposed to
 work, this could be instead amended to:

  attribute("constructor")
 void __modinit()
 {
   extern (C) void __register_module(ModuleInfo *);

   __register_module(&minfo);
 }

 So that any lazily loaded modules would be recorded to an existing
 list and trigger a run-once ctor run.



 The _Dmodule_ref approach is virtually the same as we have been doing in LDC
 up to now. The problem with regard to runtime loading of shared libraries is
 that you need to run certain code (i.e. the current _d_dso_registry in
 DMD's/LDC's druntime) when you know that you have collected all the modules.
 You can't do this from a .ctor, because if you also use them for
 constructing the module list, you'd need to call the registry function in
 the last .ctor to run. And at least if you don't know the order in which
 they are written to the executable, you can't know that.

 It might be possible to work around this requirement by rewriting the
 druntime internals to perform all the collision checking, etc. one-by-one,
 but I didn't investigate this in more detail.

 David


It should be possible if it's druntime handling all module loading (if
you circumvent the module load handlers, don't expect it to work
properly).

eg:

loadModule(mod);  // .ctors are ran and modules self register
themselves to 'mod'
mod.sortCtors();
mod.runSharedCtors();
// etc...
Dec 20 2013
parent "David Nadlinger" <code klickverbot.at> writes:
On Friday, 20 December 2013 at 14:55:46 UTC, Iain Buclaw wrote:

 It should be possible if it's druntime handling all module 
 loading (if
 you circumvent the module load handlers, don't expect it to work
 properly).

 eg:

 loadModule(mod);  // .ctors are ran and modules self register
 themselves to 'mod'
 mod.sortCtors();
 mod.runSharedCtors();
 // etc...


If you just load modules using druntime (or another helper 
function we control), then you can of course do the 
initialization like this.

You'd lose the benefit of seamlessly being able to load D 
libraries from e.g. C/C++ plugin hosts, though.

David
Dec 20 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-19 18:59, David Nadlinger wrote:


 That is pretty much how it is done in DMD and LDC as well. The only
 difference is that instead of modifying the linker scripts to
 accommodate this, we emit the ModuleInfo references to custom sections.
 The GNU toolchain (and we are in highly system-specific territory here
 anyway) never changes the order of custom sections, which you can also
 verify using __attribute__((section("...))) in GCC. Thus, if you emit
 your relevant symbols into three sections like this,


Does that really work reliably? As far as I remember, the trick with 
brackting sections doesn't work on Mac OS X. DMD had some problems with 
that. It basically broke on every new major release of Mac OS X, the 
linker changed all the time. If I recall correctly it either removed the 
empty sections or reordered the sections.

But the dynamic linker on Mac OS X has an API to access these sections 
easily anyway.

-- 
/Jacob Carlborg
Dec 19 2013
parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 20 Dec 2013 08:43:12 +0100
schrieb Jacob Carlborg <doob me.com>:


 On 2013-12-19 18:59, David Nadlinger wrote:
 

 That is pretty much how it is done in DMD and LDC as well. The only
 difference is that instead of modifying the linker scripts to
 accommodate this, we emit the ModuleInfo references to custom
 sections. The GNU toolchain (and we are in highly system-specific
 territory here anyway) never changes the order of custom sections,
 which you can also verify using __attribute__((section("...))) in
 GCC. Thus, if you emit your relevant symbols into three sections
 like this,


 
 Does that really work reliably? As far as I remember, the trick with 
 brackting sections doesn't work on Mac OS X. DMD had some problems
 with that. It basically broke on every new major release of Mac OS X,
 the linker changed all the time. If I recall correctly it either
 removed the empty sections or reordered the sections.
 
 But the dynamic linker on Mac OS X has an API to access these
 sections easily anyway.
 


Seems like the OSX linker is not based on the GNU linker and doesn't
have linker scripts either. There's some "sectorder" argument which
might be useful though.

But relying on the runtime on OSX isn't bad as there's only one runtime
anyway. On linux we have different libcs and non-standard interfaces
which isn't optimal.
Dec 20 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-12-20 10:27, Johannes Pfau wrote:


 Seems like the OSX linker is not based on the GNU linker and doesn't
 have linker scripts either. There's some "sectorder" argument which
 might be useful though.


It's probably BSD.


 But relying on the runtime on OSX isn't bad as there's only one runtime
 anyway. On linux we have different libcs and non-standard interfaces
 which isn't optimal.


Ok, I see.

-- 
/Jacob Carlborg
Dec 20 2013
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 17 Dec 2013 18:48:38 +0100
schrieb "Iain Buclaw" <ibuclaw ubuntu.com>:


 Gthreads and Emulated TLS are now in GDC!!!
 
 https://github.com/D-Programming-GDC/GDC/commit/62554bfe1b35ee4f586634a76c65d83ebfa871ef
 
 The bug noted above still exists, however what I am going to do 
 is update and hook emulated TLS into the GC.


A question about the emutls support:
I guess it's not compatible with C/C++ emutls right now? So if we have:
test.c:
__thread int a;

and test.d:
extern(C) int a;

This wouldn't work right now?

I think it could be possible to treat thread local variables in
extern(C) code specially and hook it up with the standard GCC emutls.
This way we could make it work, however we likely can't make the GC
scan extern(C) thread local variables then.
Dec 19 2013
parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 19 December 2013 11:47, Johannes Pfau <nospam example.com> wrote:

 Am Tue, 17 Dec 2013 18:48:38 +0100
 schrieb "Iain Buclaw" <ibuclaw ubuntu.com>:


 Gthreads and Emulated TLS are now in GDC!!!

 https://github.com/D-Programming-GDC/GDC/commit/62554bfe1b35ee4f586634a76c65d83ebfa871ef

 The bug noted above still exists, however what I am going to do
 is update and hook emulated TLS into the GC.


 A question about the emutls support:
 I guess it's not compatible with C/C++ emutls right now? So if we have:
 test.c:
 __thread int a;

 and test.d:
 extern(C) int a;

 This wouldn't work right now?

 I think it could be possible to treat thread local variables in
 extern(C) code specially and hook it up with the standard GCC emutls.
 This way we could make it work, however we likely can't make the GC
 scan extern(C) thread local variables then.


It's compatible with libgcc emutls.
Dec 19 2013
parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 19 Dec 2013 15:39:28 +0000
schrieb Iain Buclaw <ibuclaw gdcproject.org>:


 On 19 December 2013 11:47, Johannes Pfau <nospam example.com> wrote:

 Am Tue, 17 Dec 2013 18:48:38 +0100
 schrieb "Iain Buclaw" <ibuclaw ubuntu.com>:


 
 It's compatible with libgcc emutls.


Of course! I totally missed the extern(C) in emutls.d. So emutls.d just
overwrites the default emutls implementation in libgcc?

I guess this works fine for D apps as long as libgcc is a static
library.

What about a shared libgcc? Wouldn't the compiler then pickup the
emutls function in libgcc instead of libgdruntme?

The most difficult case is probably a C app loading a
shared D library?
Dec 19 2013
parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On 19 December 2013 16:34, Johannes Pfau <nospam example.com> wrote:

 Am Thu, 19 Dec 2013 15:39:28 +0000
 schrieb Iain Buclaw <ibuclaw gdcproject.org>:


 On 19 December 2013 11:47, Johannes Pfau <nospam example.com> wrote:

 Am Tue, 17 Dec 2013 18:48:38 +0100
 schrieb "Iain Buclaw" <ibuclaw ubuntu.com>:


 It's compatible with libgcc emutls.


 Of course! I totally missed the extern(C) in emutls.d. So emutls.d just
 overwrites the default emutls implementation in libgcc?

 I guess this works fine for D apps as long as libgcc is a static
 library.

 What about a shared libgcc? Wouldn't the compiler then pickup the
 emutls function in libgcc instead of libgdruntme?


It picks up the libgdruntime function in both cases.



 The most difficult case is probably a C app loading a
 shared D library?


That I will worry about when we get to that point.
Dec 19 2013