www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - Module init ordering bug - analysis and code example

reply pragma <pragma_member pathlink.com> writes:
Walter,
in learning how D handles module constructors, I came across the same bug that
others have stumbled across.  I did a little sleuthing to discover what was
causing this problem. ;)

My analysis (see below) showed that modules without constructors (static
this(){}) cannot be prioritized in _moduleCtor(), nor will their imports be
prioritized, since without such a constructor a module isnt't added to the
_moduleinfo_array.  Is this correct?

If so (and if you don't mind), I'd like to propose that dmd be changed to add
all modules (not just those with ctors) to the _moduleinfo_array.  This would
not only fix these particular init problems, but would also be very useful for
reflection purposes.

Many, many thanks,
Eric



[Example]
The expected init order was: mod4, mod3, mod2, mod1 as that is the order of
import and the order of dependency by function call.  Instead the modules are
run in a different order: mod2, mod1, mod4.  Inserting a module constructor into
mod3 forces the correct init order.

//// mod1.d ////
import std.stdio;
import std.moduleinit;
import mod2;

static this(){
writefln("mod1");
}

void main(){
writefln("\n--MODULE INFO--\n");

foreach(ModuleInfo info; _moduleinfo_array){
writefln("%s", info.name);
foreach(ModuleInfo imported; info.importedModules){
writefln("\timport %s",imported.name);
}
}
}


//// mod2.d ////
import std.stdio;
import mod3;

static this(){
writefln("mod2");
mod3.foo();
}


//// mod3.d ////
import std.stdio;
import mod4;

/* Uncomment to force correct ordering
static this(){
writefln("mod3");
}
*/

static void foo(){
writefln("\tmod3.foo()");
mod4.bar();
}


//// mod4.d ////
import std.stdio;

static this(){
writefln("mod4");
}

static void bar(){
writefln("\tmod4.bar()");
}


//// Output ////
mod2
mod3.foo()
mod4.bar()
mod1
mod4

--MODULE INFO--

mod1
import mod2
mod2
mod4
format
utf
string
adi
import string
gcbits
import string
thread

- Pragma
[[ EricAnderton at (static this{}) yahoo dot com ]]
Sep 02 2004
next sibling parent reply pragma <pragma_member pathlink.com> writes:
Update:
The problem gets worse with 'private import'.  Module ctor dependencies seem to
only be added if the 'private' declaration is dropped, regardless of the
presence of a module ctor.  

An revised version of my example (with private imports used) is attached.  The
output shows that mod1 no longer shows a dependency upon mod2.

mod1
mod2
mod3.foo()
mod4.bar()
mod4

--MODULE INFO--

mod1
mod2
mod4
format
utf
string
adi
import string
gcbits
import string
thread

-Pragma

In article <ch7s0p$mi8$1 digitaldaemon.com>, pragma says...

Walter,
in learning how D handles module constructors, I came across the same bug that
others have stumbled across.  I did a little sleuthing to discover what was
causing this problem. ;)

My analysis (see below) showed that modules without constructors (static
this(){}) cannot be prioritized in _moduleCtor(), nor will their imports be
prioritized, since without such a constructor a module isnt't added to the
_moduleinfo_array.  Is this correct?

If so (and if you don't mind), I'd like to propose that dmd be changed to add
all modules (not just those with ctors) to the _moduleinfo_array.  This would
not only fix these particular init problems, but would also be very useful for
reflection purposes.

Many, many thanks,
Eric



[Example]
The expected init order was: mod4, mod3, mod2, mod1 as that is the order of
import and the order of dependency by function call.  Instead the modules are
run in a different order: mod2, mod1, mod4.  Inserting a module constructor into
mod3 forces the correct init order.

//// mod1.d ////
import std.stdio;
import std.moduleinit;
import mod2;

static this(){
writefln("mod1");
}

void main(){
writefln("\n--MODULE INFO--\n");

foreach(ModuleInfo info; _moduleinfo_array){
writefln("%s", info.name);
foreach(ModuleInfo imported; info.importedModules){
writefln("\timport %s",imported.name);
}
}
}


//// mod2.d ////
import std.stdio;
import mod3;

static this(){
writefln("mod2");
mod3.foo();
}


//// mod3.d ////
import std.stdio;
import mod4;

/* Uncomment to force correct ordering
static this(){
writefln("mod3");
}
*/

static void foo(){
writefln("\tmod3.foo()");
mod4.bar();
}


//// mod4.d ////
import std.stdio;

static this(){
writefln("mod4");
}

static void bar(){
writefln("\tmod4.bar()");
}


//// Output ////
mod2
mod3.foo()
mod4.bar()
mod1
mod4

--MODULE INFO--

mod1
import mod2
mod2
mod4
format
utf
string
adi
import string
gcbits
import string
thread

- Pragma
[[ EricAnderton at (static this{}) yahoo dot com ]]



begin 0644 mod.zip


M)/B2QU%G\2)(`;:=L02.9.8WFOROY%O/C4*-]*M[;>%?X\B1(*R`3NC2Z3V.

M<HA1Z-H`^B.'8PA6FZ.P5MR^'2<NF0UDID6K>J]_3L,82N8O,EQ?A!M%I/?0
MO7, /^:/L)Y02P,$%````` `I8XB,=$M9)]>````> ````8```!M;V0R+F3C
MY>+ET <"A=S\%".]%`40FY>KH"BS++$D52$SMR"_J$2AN"1%#X
MC(&BO%S%)8DEF<D*)1F9Q1J:U;Q<Y469):EI.7D:2B"3E32!BD!J]=+R\S5`


M:,TBM$X: =+"HE`TN'K$ES^KUL[U:PV.<,V>4XI901EF%A_!LTCT"BPA"N6'
M-44G)0_ZI-(/;VLV(8WS*_==6]`-%?:Q9L3&/+(K4Z \_LG?=-^\&WNO;7'W


M,I,52C(RBS4TJWFYRHLR2U+3<O(TE$!F*&D"%=4BJ2O+STQ12$HL0E,;4P*V

MR4U1T,U/R\U/T4NM2%4`TH8 P A$&(,($P!02P$"%``4````"`"DCB(QX>SC
MF\4```!O`0``! `````````!`"``MH$`````;6]D,2YD4$L!`A0`%````` `

M`!0````(`*6.(C$(X/=7DP```-H````&``````````$`(`"V 6L!``!M;V0S
M+F102P$"%``4````"`"ECB(QAM&#EV,```")````! `````````!`"``MH$B
M` ``;6]D-"YD4$L!`A0`%````` `MXXB,><=*U0;````( ```` `````````
H`0` `/^!J0(``&UA:V4N8F%T4$L%! `````%``4`! $``.H"````````
`
end
Sep 02 2004
next sibling parent "antiAlias" <fu bar.com> writes:
Yep; I just posted you a reply on dsource.

The issue is with the importedModules[], which is supposed to house a list
of imported modules (filtered to include only those which require static
initialization) ~ per-module.

Unfortunately, if you add /any/ kind of attribute to an import statement, it
is excluded from importedModules[]. Entire chains of dependency are thus
lost, and static constructors are never executed. Anything project that (for
example) makes use "private import" is thus inviting doom.


"pragma" <pragma_member pathlink.com> wrote in message
news:ch84ft$r1b$1 digitaldaemon.com...
Update:
The problem gets worse with 'private import'.  Module ctor dependencies seem
to
only be added if the 'private' declaration is dropped, regardless of the
presence of a module ctor.

An revised version of my example (with private imports used) is attached.
The
output shows that mod1 no longer shows a dependency upon mod2.

mod1
mod2
mod3.foo()
mod4.bar()
mod4

--MODULE INFO--

mod1
mod2
mod4
format
utf
string
adi
import string
gcbits
import string
thread

-Pragma

In article <ch7s0p$mi8$1 digitaldaemon.com>, pragma says...

Walter,
in learning how D handles module constructors, I came across the same bug


that

others have stumbled across.  I did a little sleuthing to discover what was
causing this problem. ;)

My analysis (see below) showed that modules without constructors (static
this(){}) cannot be prioritized in _moduleCtor(), nor will their imports be
prioritized, since without such a constructor a module isnt't added to the
_moduleinfo_array.  Is this correct?

If so (and if you don't mind), I'd like to propose that dmd be changed to


add

all modules (not just those with ctors) to the _moduleinfo_array.  This


would

not only fix these particular init problems, but would also be very useful


for

reflection purposes.

Many, many thanks,
Eric



[Example]
The expected init order was: mod4, mod3, mod2, mod1 as that is the order of



import and the order of dependency by function call.  Instead the modules


are

run in a different order: mod2, mod1, mod4.  Inserting a module constructor


into

mod3 forces the correct init order.

//// mod1.d ////
import std.stdio;
import std.moduleinit;
import mod2;

static this(){
writefln("mod1");
}

void main(){
writefln("\n--MODULE INFO--\n");

foreach(ModuleInfo info; _moduleinfo_array){
writefln("%s", info.name);
foreach(ModuleInfo imported; info.importedModules){
writefln("\timport %s",imported.name);
}
}
}


//// mod2.d ////
import std.stdio;
import mod3;

static this(){
writefln("mod2");
mod3.foo();
}


//// mod3.d ////
import std.stdio;
import mod4;

/* Uncomment to force correct ordering
static this(){
writefln("mod3");
}
*/

static void foo(){
writefln("\tmod3.foo()");
mod4.bar();
}


//// mod4.d ////
import std.stdio;

static this(){
writefln("mod4");
}

static void bar(){
writefln("\tmod4.bar()");
}


//// Output ////
mod2
mod3.foo()
mod4.bar()
mod1
mod4

--MODULE INFO--

mod1
import mod2
mod2
mod4
format
utf
string
adi
import string
gcbits
import string
thread

- Pragma
[[ EricAnderton at (static this{}) yahoo dot com ]]
Sep 02 2004
prev sibling parent "Walter" <newshound digitalmars.com> writes:
"pragma" <pragma_member pathlink.com> wrote in message
news:ch84ft$r1b$1 digitaldaemon.com...

 Update:
 The problem gets worse with 'private import'.  Module ctor dependencies


seem to

 only be added if the 'private' declaration is dropped, regardless of the
 presence of a module ctor.


I'm glad you were able to track this down. Now I can fix it!
Sep 02 2004
prev sibling parent reply "Walter" <newshound digitalmars.com> writes:
D's only requirement for ordering module constructors between modules is
that the imported module constructors must happen first. They are not
ordered by the order of the import statements - or at least such order is an
artifact of the implementation.

You should therefore be able to control the order of module construction by
making sure that each module depending on another module constructor imports
that module.

"pragma" <pragma_member pathlink.com> wrote in message
news:ch7s0p$mi8$1 digitaldaemon.com...

 Walter,
 in learning how D handles module constructors, I came across the same bug


that

 others have stumbled across.  I did a little sleuthing to discover what


was

 causing this problem. ;)

 My analysis (see below) showed that modules without constructors (static
 this(){}) cannot be prioritized in _moduleCtor(), nor will their imports


be

 prioritized, since without such a constructor a module isnt't added to the
 _moduleinfo_array.  Is this correct?

 If so (and if you don't mind), I'd like to propose that dmd be changed to


add

 all modules (not just those with ctors) to the _moduleinfo_array.  This


would

 not only fix these particular init problems, but would also be very useful


for

 reflection purposes.

 Many, many thanks,
 Eric



 [Example]
 The expected init order was: mod4, mod3, mod2, mod1 as that is the order


of

 import and the order of dependency by function call.  Instead the modules


are

 run in a different order: mod2, mod1, mod4.  Inserting a module


constructor into

 mod3 forces the correct init order.

 //// mod1.d ////
 import std.stdio;
 import std.moduleinit;
 import mod2;

 static this(){
 writefln("mod1");
 }

 void main(){
 writefln("\n--MODULE INFO--\n");

 foreach(ModuleInfo info; _moduleinfo_array){
 writefln("%s", info.name);
 foreach(ModuleInfo imported; info.importedModules){
 writefln("\timport %s",imported.name);
 }
 }
 }


 //// mod2.d ////
 import std.stdio;
 import mod3;

 static this(){
 writefln("mod2");
 mod3.foo();
 }


 //// mod3.d ////
 import std.stdio;
 import mod4;

 /* Uncomment to force correct ordering
 static this(){
 writefln("mod3");
 }
 */

 static void foo(){
 writefln("\tmod3.foo()");
 mod4.bar();
 }


 //// mod4.d ////
 import std.stdio;

 static this(){
 writefln("mod4");
 }

 static void bar(){
 writefln("\tmod4.bar()");
 }


 //// Output ////
 mod2
 mod3.foo()
 mod4.bar()
 mod1
 mod4

 --MODULE INFO--

 mod1
 import mod2
 mod2
 mod4
 format
 utf
 string
 adi
 import string
 gcbits
 import string
 thread

 - Pragma
 [[ EricAnderton at (static this{}) yahoo dot com ]]
Sep 02 2004
next sibling parent Ben Hinkle <bhinkle4 juno.com> writes:
So are you saying the supplied example isn't a bug? How about something like

mod1.m:
module mod1;
import mod2;
static this() { mod2.run(); }

mod2.m:
module mod2;
import mod3;
void run() { mod3.run(); }

mod3.m:
module mod3;
void* never_null;
static this() { never_null = new int; }
void run(){ printf("is never_null null? %p eek!\n",never_null); }

I would think this problem makes module initialization very easy to get
around. How about a default (trivial) module constructor gets generated for
a module if any of its imported modules have a constructor.

Walter wrote:


 D's only requirement for ordering module constructors between modules is
 that the imported module constructors must happen first. They are not
 ordered by the order of the import statements - or at least such order is
 an artifact of the implementation.
 
 You should therefore be able to control the order of module construction
 by making sure that each module depending on another module constructor
 imports that module.
 
 "pragma" <pragma_member pathlink.com> wrote in message
 news:ch7s0p$mi8$1 digitaldaemon.com...

 Walter,
 in learning how D handles module constructors, I came across the same bug


 that

 others have stumbled across.  I did a little sleuthing to discover what


 was

 causing this problem. ;)

 My analysis (see below) showed that modules without constructors (static
 this(){}) cannot be prioritized in _moduleCtor(), nor will their imports


 be

 prioritized, since without such a constructor a module isnt't added to
 the
 _moduleinfo_array.  Is this correct?

 If so (and if you don't mind), I'd like to propose that dmd be changed to


 add

 all modules (not just those with ctors) to the _moduleinfo_array.  This


 would

 not only fix these particular init problems, but would also be very
 useful


 for

 reflection purposes.

 Many, many thanks,
 Eric



 [Example]
 The expected init order was: mod4, mod3, mod2, mod1 as that is the order


 of

 import and the order of dependency by function call.  Instead the modules


 are

 run in a different order: mod2, mod1, mod4.  Inserting a module


 constructor into

 mod3 forces the correct init order.

 //// mod1.d ////
 import std.stdio;
 import std.moduleinit;
 import mod2;

 static this(){
 writefln("mod1");
 }

 void main(){
 writefln("\n--MODULE INFO--\n");

 foreach(ModuleInfo info; _moduleinfo_array){
 writefln("%s", info.name);
 foreach(ModuleInfo imported; info.importedModules){
 writefln("\timport %s",imported.name);
 }
 }
 }


 //// mod2.d ////
 import std.stdio;
 import mod3;

 static this(){
 writefln("mod2");
 mod3.foo();
 }


 //// mod3.d ////
 import std.stdio;
 import mod4;

 /* Uncomment to force correct ordering
 static this(){
 writefln("mod3");
 }
 */

 static void foo(){
 writefln("\tmod3.foo()");
 mod4.bar();
 }


 //// mod4.d ////
 import std.stdio;

 static this(){
 writefln("mod4");
 }

 static void bar(){
 writefln("\tmod4.bar()");
 }


 //// Output ////
 mod2
 mod3.foo()
 mod4.bar()
 mod1
 mod4

 --MODULE INFO--

 mod1
 import mod2
 mod2
 mod4
 format
 utf
 string
 adi
 import string
 gcbits
 import string
 thread

 - Pragma
 [[ EricAnderton at (static this{}) yahoo dot com ]]
Sep 02 2004
prev sibling parent reply pragma <pragma_member pathlink.com> writes:
In article <ch8c1i$tdd$1 digitaldaemon.com>, Walter says...

D's only requirement for ordering module constructors between modules is
that the imported module constructors must happen first. They are not
ordered by the order of the import statements - or at least such order is an
artifact of the implementation.

You should therefore be able to control the order of module construction by
making sure that each module depending on another module constructor imports
that module.


Gotcha.  Thank you for confirming this, as that's pretty much where all my
hacking lead me.  (Actually it was pretty much in plain D for everyone in
moduleinit.d, but that's another story) :)

However, the order of operations really isn't so crucial since it visits all the
dependencies recursively, and flags everything on the way out to keep from
running things more than once. (its a good compact design!) We also now know
what conditions dmd pays attention to (at least in 0.101) when adding nodes to
this constructor list.

As to the problem at hand: I feel that having dmd add all modules to the list
(rather than just those with ctors) will be a huge improvement to the language.

Thanks Walter!

- Pragma
[[ Eric Anderton at (_moduleCtor) yahoo dot com ]]
Sep 02 2004
parent reply "Walter" <newshound digitalmars.com> writes:
"pragma" <pragma_member pathlink.com> wrote in message
news:ch8fa0$ujm$1 digitaldaemon.com...

 As to the problem at hand: I feel that having dmd add all modules to the


list

 (rather than just those with ctors) will be a huge improvement to the


language.

The problem with that is that then two modules could not import each other.
Sep 06 2004
parent reply pragma <pragma_member pathlink.com> writes:
In article <chjl73$259k$1 digitaldaemon.com>, Walter says...

"pragma" <pragma_member pathlink.com> wrote in message
news:ch8fa0$ujm$1 digitaldaemon.com...

 As to the problem at hand: I feel that having dmd add all modules to the


list

 (rather than just those with ctors) will be a huge improvement to the


language.

The problem with that is that then two modules could not import each other.


Would this be only if both modules have static ctors, just one, or none at all?

I would think that the latter two cases would never be a problem given the way
moduleinit.d is already coded: it's already assuming that there are modules in
the list w/o ctors as it is checking for that condition (lines 111 and 121).
Just put modules on the list with null ctors/dtors if none are declared and it
should work fine (IMO).

And you already know the first case is already being handled quite nicely. :)

Or have I missed the point completely and there is some other case that's
causing a problem?

- Pragma
[[Eric Anderton at yahoo dot com]]
Sep 08 2004
parent reply "Walter" <newshound digitalmars.com> writes:
"pragma" <pragma_member pathlink.com> wrote in message
news:chn51k$r3a$1 digitaldaemon.com...

 In article <chjl73$259k$1 digitaldaemon.com>, Walter says...

"pragma" <pragma_member pathlink.com> wrote in message
news:ch8fa0$ujm$1 digitaldaemon.com...

 As to the problem at hand: I feel that having dmd add all modules to






the

list

 (rather than just those with ctors) will be a huge improvement to the


language.

The problem with that is that then two modules could not import each




other.

 Would this be only if both modules have static ctors, just one, or none at


all?

 I would think that the latter two cases would never be a problem given the


way

 moduleinit.d is already coded: it's already assuming that there are


modules in

 the list w/o ctors as it is checking for that condition (lines 111 and


121).

 Just put modules on the list with null ctors/dtors if none are declared


and it

 should work fine (IMO).

 And you already know the first case is already being handled quite nicely.


:)

 Or have I missed the point completely and there is some other case that's
 causing a problem?


The problem is if A imports B, and B imports A, whose initialization code
gets run first? The imports imply a dependency, so we have "A requires that
B's initializers run first" coupled with "B requires that A's initializers
run first." There's no resolution. The only one I can think of is that a
module with no initializers doesn't have a requirement that the imports get
initialized first. And I think that is always correct - that if there *is*
such a dependency in the code, then there's something else wrong.
Sep 09 2004
parent reply pragma <pragma_member pathlink.com> writes:
In article <chr7l5$v3p$1 digitaldaemon.com>, Walter says...

 Or have I missed the point completely and there is some other case that's
 causing a problem?


The problem is if A imports B, and B imports A, whose initialization code
gets run first? The imports imply a dependency, so we have "A requires that
B's initializers run first" coupled with "B requires that A's initializers
run first." There's no resolution. 


I see your point: the problem /is/ deeper than just running static ctors in
order.  It also gets worse when you throw additional modules into such an
"import cycle".


The only one I can think of is that a
module with no initializers doesn't have a requirement that the imports get
initialized first. And I think that is always correct - that if there *is*
such a dependency in the code, then there's something else wrong.


You're talking about the relaxed init rules already in phobos, correct?
_moduleCtor2() treats modules with and without ctors differently, in the manner
you prescribe. 

I think there may be another way.

Perhaps mixing the two cases would be best: if the imported module doesn't have
a ctor then don't worry if its flagged as 'MIctorstart', but throw an error if
it has a ctor and is flagged as such.  That way you'd be able to detect init
cycles regarless of the intermediate imported module types (modules with or w/o
ctors).  In addition, modules w/o ctors can cycle all they want.

This is (slightly) different from the current behavior which is dependenent upon
the current module type.  Instead, it relies on the /imported/ module type.

If I'm correct this would be the new behavior from this, and previously
mentioned, changes:
- 'import' and 'private import' now behave the same (new)
- all modules are included in the init sequence  (new) 
- imported modules w/o ctors can participate in cyclic imports. (somewhat new)
- modules w/static ctors cannot participate in cyclic imports. (same)
- (implied) cycles of mixed module types are not allowed (new)

- Pragma
[[ eric anderton at yahoo dot com ]]
Sep 10 2004
parent reply "Walter" <newshound digitalmars.com> writes:
"pragma" <pragma_member pathlink.com> wrote in message
news:chsfc1$1j7d$1 digitaldaemon.com...

 If I'm correct this would be the new behavior from this, and previously
 mentioned, changes:
 - 'import' and 'private import' now behave the same (new)
 - all modules are included in the init sequence  (new)
 - imported modules w/o ctors can participate in cyclic imports. (somewhat


new)

 - modules w/static ctors cannot participate in cyclic imports. (same)
 - (implied) cycles of mixed module types are not allowed (new)


Ok, but *why* should a module that has no static ctors have a dependency on
other module static ctors?
Sep 10 2004
parent pragma <pragma_member pathlink.com> writes:
In article <chtp3c$26mj$1 digitaldaemon.com>, Walter says...

Ok, but *why* should a module that has no static ctors have a dependency on
other module static ctors?


Easy.  A module w/o any static ctors may have functions that are callable from
*other* module ctors.  It may have no initalization requirements (and rightly so
since it doesn't have a static ctor), but it may depend on the behavior of
additional modules that in turn rely on their own static ctors to run first.

Here's a more concrete example. :)

// api.d (just an interface, so there's no module ctor here)
private import logger;
private import ilogger;

public static ILogger getDefaultLogger(){ return logger.defaultLogger; }


// ilogger.d
interface ILogger{ /*methods omitted*/}


// logger.d
private import ILogger;

class Logger: ILogger{ /*methods omitted*/ }
private Logger defaultLogger;
static this(){ defaultLogger = new Logger();


// client.d (not in the library at all)
import api.d
import ilogger.d

/** this is where it would break: getDefaultLogger() might return null **/
static this(){ api.getDefaultLogger().log("started"); }


The above example yields unpredictable results because one cannot guarantee that
logger.d will be initalized before client.d.  (In fact, one cannot gaurantee
that it *won't* initalize before client.d either)  If api.d, a module w/o static
init code, were included in the init dependency chain, then this would run
predictably and as expected.

- Pragma
[[ Eric Anderton at yahoo dot com ]]
Sep 11 2004