www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - static init cycle detection problem

reply =?UTF-8?B?IsOYaXZpbmQi?= <oivind.loe gmail.com> writes:
I am struggeling to get around the cycle detection kicking in 
when I have static init in modules that depend on eachother.

I have seen some threads on 'fixes' for this, e.g. adding a 
 standalone property to the module or similar. Has there been any 
progress on this?

If not would it be possible in e.g. main() to get a list of all 
compiled-in modules, and then iterate over them and call an init 
function where it exists? As long as there is a way to list the 
name of the modules at compile-time, this should be pretty easy..?

-Øivind
Sep 19 2012
next sibling parent reply "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On Wed, 19 Sep 2012 22:25:46 +0200, =C3=98ivind <oivind.loe gmail.com> w=
rote:


 I am struggeling to get around the cycle detection kicking in when I  =



 have static init in modules that depend on eachother.

 I have seen some threads on 'fixes' for this, e.g. adding a  standalon=


e  =


 property to the module or similar. Has there been any progress on this=


?

 If not would it be possible in e.g. main() to get a list of all  =



 compiled-in modules, and then iterate over them and call an init  =



 function where it exists? As long as there is a way to list the name o=


f  =


 the modules at compile-time, this should be pretty easy..?


There's no way to get that list at compile-time, because object files ma=
y
be added at link-time. However, D has a ModuleInfo object, which contain=
s
information on all modules in the program:

import std.stdio;
void main( ) {
     foreach( m; ModuleInfo ) {
         writeln( m.name );
     }
}

For details on how this object works, have a look-see at
src/druntime/src/object_.d in your DMD installation folder.

I'm not sure what you're asking for is possible even given this object,
but it's probably the closest you'll (easily) get.

-- =

Simen
Sep 19 2012
parent reply =?UTF-8?B?IsOYaXZpbmQi?= <oivind.loe gmail.com> writes:
On Wednesday, 19 September 2012 at 20:56:17 UTC, Simen Kjaeraas 
wrote:

 On Wed, 19 Sep 2012 22:25:46 +0200, Øivind 
 <oivind.loe gmail.com> wrote:


 I am struggeling to get around the cycle detection kicking in 
 when I have static init in modules that depend on eachother.

 I have seen some threads on 'fixes' for this, e.g. adding a 
  standalone property to the module or similar. Has there been 
 any progress on this?

 If not would it be possible in e.g. main() to get a list of 
 all compiled-in modules, and then iterate over them and call 
 an init function where it exists? As long as there is a way to 
 list the name of the modules at compile-time, this should be 
 pretty easy..?


 There's no way to get that list at compile-time, because object 
 files may
 be added at link-time. However, D has a ModuleInfo object, 
 which contains
 information on all modules in the program:

 import std.stdio;
 void main( ) {
     foreach( m; ModuleInfo ) {
         writeln( m.name );
     }
 }

 For details on how this object works, have a look-see at
 src/druntime/src/object_.d in your DMD installation folder.

 I'm not sure what you're asking for is possible even given this 
 object,
 but it's probably the closest you'll (easily) get.


If I define a function:

void getMembers() {

}

in one of my modules, all modules including this get the 
xgetMembers property of the ModuleInfo struct set. It seems like 
I should be able to do something with this, but i then get the 
following assertion failure:

./dboss(_d_assertm+0x26) [0x8aa1da]
./dboss() [0x7c8555]
./dboss(boss.core.proc.ProcBase!(boss.core.boss.Boss, 
"boss").ProcBase boss.core.proc.ProcBase!(boss.core.boss.Boss, 
"boss").ProcBase.__ctor()+0x31) [0x797805]
./dboss(boss.core.boss.Boss 
boss.core.boss.Boss.__ctor(immutable(char)[][])+0x21) [0x791099]
./dboss(_Dmain+0x33) [0x7488fb]
./dboss(extern (C) int rt.dmain2.main(int, char**).void 
runMain()+0x1c) [0x8aaa70]
./dboss(extern (C) int rt.dmain2.main(int, char**).void 
tryExec(scope void delegate())+0x2a) [0x8aa3ea]
./dboss(extern (C) int rt.dmain2.main(int, char**).void 
runAll()+0x3b) [0x8aaab7]
./dboss(extern (C) int rt.dmain2.main(int, char**).void 
tryExec(scope void delegate())+0x2a) [0x8aa3ea]
./dboss(main+0xd1) [0x8aa375]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) 
[0x7f1d0edcc76d]

Removing the function from the module makes the assertion failure 
go away.. What is the cause of this? And what is the getMembers 
function used for?


Another way of approaching this would be if I could feed a list 
of modules into DMD during compile time. In C++, i would be able 
to do this by passing a define to g++ on the command line when 
invoking it. Is it possible to do something similar with DMD? 
E.g. create a list of modules before invoking DMD, pass it to 
DMD, and use this at compile time to import and call my init 
function?
Sep 19 2012
parent =?UTF-8?B?IsOYaXZpbmQi?= <oivind.loe gmail.com> writes:
 Another way of approaching this would be if I could feed a list 
 of modules into DMD during compile time. In C++, i would be 
 able to do this by passing a define to g++ on the command line 
 when invoking it. Is it possible to do something similar with 
 DMD? E.g. create a list of modules before invoking DMD, pass it 
 to DMD, and use this at compile time to import and call my init 
 function?


I can of course just generate a separate init.d file containing 
all of this info and then use that for static init of my modules. 
Think that should work :)
Sep 19 2012
prev sibling next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Wednesday, September 19, 2012 22:25:46 Øivind wrote:

 I am struggeling to get around the cycle detection kicking in
 when I have static init in modules that depend on eachother.
 
 I have seen some threads on 'fixes' for this, e.g. adding a
  standalone property to the module or similar. Has there been any
 progress on this?


The solution is to do something like what std.stdio does. Instead of having a 
static constructor, it has an extern(C) function which does the same, and then 
it has std/stdiobase.d which has a static constructor which calls that 
extern(C) function in std.stdio. As long as you don't actually have any inter-
dependencies, you can make it work.

Walter has refused suggestions which involve telling the compiler that there 
isn't actually dependency (via an attribute or whatnot) on the grounds that 
you're throwing away the compiler's checks, and it's too easy to screw up. In 
general, you can refactor things so that you don't have the circular 
dependency anymore, and if you can't, and the static constructors aren't 
actually forming a circular dependency, then you can pull the same trick that 
std.stdio does.


 If not would it be possible in e.g. main() to get a list of all
 compiled-in modules, and then iterate over them and call an init
 function where it exists? As long as there is a way to list the
 name of the modules at compile-time, this should be pretty easy..?


It will never be possible to determine the full list of modules at compile 
time due to C's linking model. For instance, you could link together modules 
which were built years apart. There's no way that they could know about each 
other. And since you could be using .di files, and the circular dependencies 
could actually be really indirect, the compiler can't possibly have all of the 
information that it needs to determine circular dependencies. A linker could 
do it but not as long as we use standard, C linkers. And that's not going to 
change.

So, the situation sucks, but there _are_ workarounds. The main one of course 
is to simply limit how much your modules import each other so that the few 
places that you need static constructors don't import each other even 
indirectly, but regardless, it can be worked around as long there isn't a true
circular dependency.

- Jonathan M Davis
Sep 19 2012
parent reply =?UTF-8?B?IsOYaXZpbmQi?= <oivind.loe gmail.com> writes:
On Thursday, 20 September 2012 at 00:23:33 UTC, Jonathan M Davis 
wrote:

 On Wednesday, September 19, 2012 22:25:46 Øivind wrote:

 I am struggeling to get around the cycle detection kicking in
 when I have static init in modules that depend on eachother.
 
 I have seen some threads on 'fixes' for this, e.g. adding a
  standalone property to the module or similar. Has there been 
 any
 progress on this?


 The solution is to do something like what std.stdio does. 
 Instead of having a
 static constructor, it has an extern(C) function which does the 
 same, and then
 it has std/stdiobase.d which has a static constructor which 
 calls that
 extern(C) function in std.stdio. As long as you don't actually 
 have any inter-
 dependencies, you can make it work.

 Walter has refused suggestions which involve telling the 
 compiler that there
 isn't actually dependency (via an attribute or whatnot) on the 
 grounds that
 you're throwing away the compiler's checks, and it's too easy 
 to screw up. In
 general, you can refactor things so that you don't have the 
 circular
 dependency anymore, and if you can't, and the static 
 constructors aren't
 actually forming a circular dependency, then you can pull the 
 same trick that
 std.stdio does.


 If not would it be possible in e.g. main() to get a list of all
 compiled-in modules, and then iterate over them and call an 
 init
 function where it exists? As long as there is a way to list the
 name of the modules at compile-time, this should be pretty 
 easy..?


 It will never be possible to determine the full list of modules 
 at compile
 time due to C's linking model. For instance, you could link 
 together modules
 which were built years apart. There's no way that they could 
 know about each
 other. And since you could be using .di files, and the circular 
 dependencies
 could actually be really indirect, the compiler can't possibly 
 have all of the
 information that it needs to determine circular dependencies. A 
 linker could
 do it but not as long as we use standard, C linkers. And that's 
 not going to
 change.

 So, the situation sucks, but there _are_ workarounds. The main 
 one of course
 is to simply limit how much your modules import each other so 
 that the few
 places that you need static constructors don't import each 
 other even
 indirectly, but regardless, it can be worked around as long 
 there isn't a true
 circular dependency.

 - Jonathan M Davis


Thanks for the explination. The trick you talk about has worked 
for me before, but now I really need static init of modules that 
depend on eachother. Only solution I can think of is to list all 
modules in a script and generate d code from this to call init 
functions.. Then pass it to dmd along with the other code.

Sad that we can't get this to work like it really should. D is a 
very flexible language in many ways, and it seems strange that it 
should not allow the stuff I want to do here..

-Øivind
Sep 20 2012
next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, September 20, 2012 20:49:32 Øivind wrote:

 Thanks for the explination. The trick you talk about has worked
 for me before, but now I really need static init of modules that
 depend on eachother. Only solution I can think of is to list all
 modules in a script and generate d code from this to call init
 functions.. Then pass it to dmd along with the other code.
 
 Sad that we can't get this to work like it really should. D is a
 very flexible language in many ways, and it seems strange that it
 should not allow the stuff I want to do here..


If you have a true circular dependency, then you're just plain screwed and 
need to redesign your code. That's fundamental and _can't_ be fixed.

Off the top of my head, the only reason that you wouldn't be able to use the 
trick that std.stdio uses but not have a true circular dependency is if your 
initializing const or immutable variables in a static constructor. But then 
the solution is to refactor your code so that you don't have the circular 
dependency, even if it's just moving those variables into a separate module, 
which is generally quite feasible, even if it's not necessarily quite what you 
wanted.

- Jonathan M Davis
Sep 20 2012
prev sibling parent Brad Roberts <braddr puremagic.com> writes:
On Thu, 20 Sep 2012, Jonathan M Davis wrote:


 On Thursday, September 20, 2012 20:49:32 ?ivind wrote:

 Thanks for the explination. The trick you talk about has worked
 for me before, but now I really need static init of modules that
 depend on eachother. Only solution I can think of is to list all
 modules in a script and generate d code from this to call init
 functions.. Then pass it to dmd along with the other code.
 
 Sad that we can't get this to work like it really should. D is a
 very flexible language in many ways, and it seems strange that it
 should not allow the stuff I want to do here..


 
 If you have a true circular dependency, then you're just plain screwed and 
 need to redesign your code. That's fundamental and _can't_ be fixed.
 
 Off the top of my head, the only reason that you wouldn't be able to use the 
 trick that std.stdio uses but not have a true circular dependency is if your 
 initializing const or immutable variables in a static constructor. But then 
 the solution is to refactor your code so that you don't have the circular 
 dependency, even if it's just moving those variables into a separate module, 
 which is generally quite feasible, even if it's not necessarily quite what you 
 wanted.
 
 - Jonathan M Davis
 


I'm not picking on Jonathan's response, just needed an entry point to the 
thread...

Static initializers are overused in D, particularly phobos and druntime, 
imho.  They're a particular problem when it comes to use in library code.  
Mixing static and dynamic linkage along with initialization == things 
break in wierd ways.  I'd _much_ rather see them moved into explicit api 
calls or removed entirely where possible.

Every static initializtion is unavoidable (by users of the 
library) overhead.

My 2 cents,
Brad
Sep 20 2012
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 19 Sep 2012 16:25:46 -0400, =C3=98ivind <oivind.loe gmail.com> w=
rote:


 I am struggeling to get around the cycle detection kicking in when I  =



 have static init in modules that depend on eachother.

 I have seen some threads on 'fixes' for this, e.g. adding a  standalon=


e  =


 property to the module or similar. Has there been any progress on this=


?

No.  If two initializations truly don't depend on each other, you can  =

initialize in their own module.

The issue is that you have to move *all* the initializations into their =
 =

own module.  The compiler doesn't keep track of complex dependencies, it=
's  =

all on a module-level.

I redesigned the static cycle detection code a while back (it was broken=
  =

(allowed certain cycles) and much less informative), and I can say it wa=
s  =

a very difficult problem to solve.


 If not would it be possible in e.g. main() to get a list of all  =



 compiled-in modules, and then iterate over them and call an init  =



 function where it exists? As long as there is a way to list the name o=


f  =


 the modules at compile-time, this should be pretty easy..?


No, you have no idea at compile-time what other modules will be linked i=
n.

You need to do the cycle detection at link-time or run-time, and as long=
  =

as we use standard C linkers, we cannot add that feature.

-Steve
Sep 24 2012