www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Fixing cyclic import static construction problems

reply Walter Bright <newshound2 digitalmars.com> writes:
For discussion:

Cyclical Imports

Problem:

---- a.d ----
     module a;
     import b;
     static this () { ... }
---- b.d ----
     module b;
     import a;
     static this() { ... }
-------------

Static constructors for a module are only run after static constructors
for all its imports are run. Circular imports, such as the above, are
detected at run time and the program is aborted.

This can in general be solved by moving the static constructor(s) into
a third module, c.d, which does not import a or b. But, people find this
to be unnatural.

Proposed Solution:

Add a pragma,

     pragma(cyclic_imports);

This can appear anywhere in a module, and applies globally to that module.
It means that static constructors from imports that are not part of the 
cycle
are run first, and that the static constructor for this module may be 
run before
the static constructors of other modules that are part of the cycle.

If any static constructors in such a module with the pragma have
the  safe attribute, that is a compile time error.
Nov 28 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/28/12 9:34 PM, Walter Bright wrote:

 For discussion:


[snip]

I'd say we better finish const, immutable, and shared first.

Andrei
Nov 28 2012
next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei 
Alexandrescu wrote:

 I'd say we better finish const, immutable, and shared first.


If we put off every easy fix until all the hard fixes are done, 
it means we have longer wait times on everything....
Nov 28 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/28/12 10:34 PM, Adam D. Ruppe wrote:

 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei Alexandrescu wrote:

 I'd say we better finish const, immutable, and shared first.


 If we put off every easy fix until all the hard fixes are done, it means
 we have longer wait times on everything....


On the other hand if we work on every easy and unimportant fix we miss 
big on the important stuff.

http://c2.com/cgi/wiki?FourQuadrants


Andrei
Nov 29 2012
prev sibling next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:


 I'd say we better finish const, immutable, and shared first.


There are few things left to implement for purity (they are 
listed in Bugzilla), but what's left to do for const and 
immutable?

Bye,
bearophile
Nov 28 2012
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
 what's left to do for const and immutable?


I guess the answer is too much long, so please ignore the 
question.

Bye,
bearophile
Nov 28 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/29/12 12:07 AM, bearophile wrote:

 what's left to do for const and immutable?


 I guess the answer is too much long, so please ignore the question.


Ars longa vita brevis est.

Andrei
Nov 29 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/28/12 11:26 PM, bearophile wrote:

 Andrei Alexandrescu:


 I'd say we better finish const, immutable, and shared first.


 There are few things left to implement for purity (they are listed in
 Bugzilla), but what's left to do for const and immutable?


Construction flow and copy conversion.

Andrei
Nov 29 2012
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 11/29/2012 02:50 PM, Andrei Alexandrescu wrote:

 On 11/28/12 11:26 PM, bearophile wrote:

 There are few things left to implement for purity (they are listed in
 Bugzilla), but what's left to do for const and immutable?


 Construction flow and copy conversion.


... meaning e.g. being able to effectively .dup or .idup any class, struct,
etc.?
Nov 30 2012
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, November 28, 2012 22:19:54 Andrei Alexandrescu wrote:

 On 11/28/12 9:34 PM, Walter Bright wrote:

 For discussion:


 [snip]
 
 I'd say we better finish const, immutable, and shared first.


Both problems need to be addressed, and this one is probably easier. It also 
has a huge impact on std.benchmark, so I would have thought that you'd be more 
in favor of it.

However, we do have a tendancy to bring up problems like this, discuss them 
for a while, and then let them be more or less forgotten for a while. It keeps 
happening with stuff like const ref / auto ref, shared, const and Object, const 
postblit constructors,  trusted blocks, etc. And we need to actually get them 
sorted out.

- Jonathan M Davis
Nov 28 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/29/12 12:25 AM, Jonathan M Davis wrote:

 On Wednesday, November 28, 2012 22:19:54 Andrei Alexandrescu wrote:

 On 11/28/12 9:34 PM, Walter Bright wrote:

 For discussion:


 [snip]

 I'd say we better finish const, immutable, and shared first.


 Both problems need to be addressed, and this one is probably easier. It also
 has a huge impact on std.benchmark, so I would have thought that you'd be more
 in favor of it.


My perception is that we need to tackle the major issues at this point: 
process and qualifiers (including shared).


 However, we do have a tendancy to bring up problems like this, discuss them
 for a while, and then let them be more or less forgotten for a while. It keeps
 happening with stuff like const ref / auto ref, shared, const and Object, const
 postblit constructors,  trusted blocks, etc. And we need to actually get them
 sorted out.


Enhancement requests are the right place for those. Now that we have 
forum.dlang.org we can link to discussions, too.


Andrei
Nov 29 2012
prev sibling parent reply "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei 
Alexandrescu wrote:

 On 11/28/12 9:34 PM, Walter Bright wrote:

 For discussion:


 [snip]

 I'd say we better finish const, immutable, and shared first.

 Andrei


+1

Fully agree.

Cyclic imports are a minor nuisance that can be easily solvable 
with better code architecture.

Turbo Pascal/Delphi is the only language that I know fully allows 
cyclic dependencies between modules. So this is not that 
important for most people.

--
Paulo
Nov 29 2012
next sibling parent reply "Max Samukha" <maxsamukha gmail.com> writes:
On Thursday, 29 November 2012 at 11:39:20 UTC, Paulo Pinto wrote:

 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei 
 Alexandrescu wrote:

 On 11/28/12 9:34 PM, Walter Bright wrote:

 For discussion:


 [snip]

 I'd say we better finish const, immutable, and shared first.

 Andrei


 +1

 Fully agree.

 Cyclic imports are a minor nuisance that can be easily solvable 
 with better code architecture.


Show me please how to solve that problem easily with acceptable 
results, would you?


 Turbo Pascal/Delphi is the only language that I know fully 
 allows cyclic dependencies between modules. So this is not that 
 important for most people.

 --
 Paulo
Nov 29 2012
parent reply "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 29 November 2012 at 12:04:28 UTC, Max Samukha wrote:

 On Thursday, 29 November 2012 at 11:39:20 UTC, Paulo Pinto 
 wrote:

 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei 
 Alexandrescu wrote:

 On 11/28/12 9:34 PM, Walter Bright wrote:

 For discussion:


 [snip]

 I'd say we better finish const, immutable, and shared first.

 Andrei


 +1

 Fully agree.

 Cyclic imports are a minor nuisance that can be easily 
 solvable with better code architecture.


 Show me please how to solve that problem easily with acceptable 
 results, would you?


You just need to have a better architecture.

In 20 years of software development experience I never found a 
case were this wasn't possible.

Maybe you care to provide an example?

--
Paulo
Nov 29 2012
next sibling parent reply "Max Samukha" <maxsamukha gmail.com> writes:
On Thursday, 29 November 2012 at 15:18:11 UTC, Paulo Pinto wrote:

 On Thursday, 29 November 2012 at 12:04:28 UTC, Max Samukha 
 wrote:

 On Thursday, 29 November 2012 at 11:39:20 UTC, Paulo Pinto 
 wrote:

 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei 
 Alexandrescu wrote:

 On 11/28/12 9:34 PM, Walter Bright wrote:

 For discussion:


 [snip]

 I'd say we better finish const, immutable, and shared first.

 Andrei


 +1

 Fully agree.

 Cyclic imports are a minor nuisance that can be easily 
 solvable with better code architecture.


 Show me please how to solve that problem easily with 
 acceptable results, would you?


 You just need to have a better architecture.

 In 20 years of software development experience I never found a 
 case were this wasn't possible.


That's an argument from authority, sorry.


 Maybe you care to provide an example?


The general problem is constructing global data structures based 
on data introspected at compile-time.

My specific problem is extending scarce runtime type information 
provided by the language with something usable for runtime 
reflection. With lots of detail omitted:

module reflect;

Meta[string] metas;
mixin template Reflect(alias object) {
     static this()
     {
         auto m = meta!(object);
         metas[m.fullName] ~= m;
     }
}


module a;
import reflect;

struct S
{
}
mixin Reflect!S;

The meta-object for S is automatically made available at runtime 
through the global metas array. Note that we do not want to force 
the user to register the meta-object manually because then it 
would not be a "better architecture".

The important (Andrei somehow thinks it is not) requirement is 
there must not be circular dependency issues for the users of the 
"reflect" module.
Nov 29 2012
next sibling parent reply Manfred Nowak <svv1999 hotmail.com> writes:
Max Samukha wrote:


 there must not be circular dependency issues


The model shown does not force circular dependencies, because 
every receiver module only imports one module: `module reflect'. 
And I can not see, that this module imports any other module.

Therefore `module relect' and its `import' statements are not 
responsible for any circular dependency issue.

Where am I wrong?

-manfred
Nov 29 2012
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 11/29/2012 09:30 PM, Manfred Nowak wrote:

 Max Samukha wrote:


 there must not be circular dependency issues


 The model shown does not force circular dependencies, because
 every receiver module only imports one module: `module reflect'.
 And I can not see, that this module imports any other module.

 Therefore `module relect' and its `import' statements are not
 responsible for any circular dependency issue.

 Where am I wrong?

 -manfred


A precondition for (pseudo-) circular dependency issues of this kind are 
static constructors. The static constructors are what module reflect 
imposes in its users in order to operate.
Nov 29 2012
parent reply Manfred Nowak <svv1999 hotmail.com> writes:
Timon Gehr wrote:


 what module reflect imposes


Thank you.

It seems to be true then, that no issue would be generated if 
the compiler assumes, that Walters pragma is existent in every 
module.

-manfred
Nov 29 2012
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, November 30, 2012 03:10:57 Manfred Nowak wrote:

 Timon Gehr wrote:

 what module reflect imposes


 
 Thank you.
 
 It seems to be true then, that no issue would be generated if
 the compiler assumes, that Walters pragma is existent in every
 module.


So, you're suggesting that the runtime should just not worry about circular 
dependencies between modules at all? What if you have something like

module a;
import b;

immutable Foo foo;
shared static this()
{
 foo = new immutable(Foo)(bar);
}

module b;
import a;

immutable Bar bar;
shared static this()
{
 bar = new immutable(Bar)(foo);
}

That's a true circular dependency. It's a big problem if that's allowed. D 
rightly disallows it. The problem is that in the vast majority of cases, there 
is no such circular dependency, so the circular dependency detection disallows 
many valid use cases.

Or do you mean something else by the suggestion that the existence of Walter's 
pragma can be assumed? I don't know what else you could mean other than
suggesting that assuming that it's present in all modules which is the same
as throwing away circular dependency detection entirely.

- Jonathan M Davis
Nov 29 2012
parent Manfred Nowak <svv1999 hotmail.com> writes:
Jonathan M Davis wrote:


 Or do you mean something else by the suggestion that the
 existence of Walter's pragma can be assumed?


In

http://forum.dlang.org/thread/k96hj2$2lus$1 digitalmars.com?
page=4#post-k98o7j:24qb6:241:40digitalmars.com

Walter pointed this out for a clique of imports. All but one 
members of a clique are to have that pragma anyway---and Walter 
claims, that there is no error introduced, when the remaining 
member has that pragma too.

-manfred
Dec 02 2012
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 29 November 2012 at 16:51:29 UTC, Max Samukha wrote:

 On Thursday, 29 November 2012 at 15:18:11 UTC, Paulo Pinto 
 wrote:

 On Thursday, 29 November 2012 at 12:04:28 UTC, Max Samukha 
 wrote:

 On Thursday, 29 November 2012 at 11:39:20 UTC, Paulo Pinto 
 wrote:

 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei 
 Alexandrescu wrote:

 On 11/28/12 9:34 PM, Walter Bright wrote:

 For discussion:


 [snip]

 I'd say we better finish const, immutable, and shared first.

 Andrei


 +1

 Fully agree.

 Cyclic imports are a minor nuisance that can be easily 
 solvable with better code architecture.


 Show me please how to solve that problem easily with 
 acceptable results, would you?


 You just need to have a better architecture.

 In 20 years of software development experience I never found a 
 case were this wasn't possible.


 That's an argument from authority, sorry.


 Maybe you care to provide an example?


 The general problem is constructing global data structures 
 based on data introspected at compile-time.

 My specific problem is extending scarce runtime type 
 information provided by the language with something usable for 
 runtime reflection. With lots of detail omitted:

 module reflect;

 Meta[string] metas;
 mixin template Reflect(alias object) {
     static this()
     {
         auto m = meta!(object);
         metas[m.fullName] ~= m;
     }
 }


 module a;
 import reflect;

 struct S
 {
 }
 mixin Reflect!S;

 The meta-object for S is automatically made available at 
 runtime through the global metas array. Note that we do not 
 want to force the user to register the meta-object manually 
 because then it would not be a "better architecture".


For me too much use of meta capabilities is not a better 
architecture,
as quite often it leads to write only code.

If you are alreading writing code on the client side for 
initialization, like your mixin definition, the client already 
needs to make the reflect module of its existence, so why not 
call an initialization function that avoids the static 
constructors issues altogether?


--
Paulo
Nov 29 2012
prev sibling parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 11/29/2012 5:51 PM, Max Samukha wrote:

 On Thursday, 29 November 2012 at 15:18:11 UTC, Paulo Pinto wrote:

 Maybe you care to provide an example?


 The general problem is constructing global data structures based on data
 introspected at compile-time.

 My specific problem is extending scarce runtime type information
 provided by the language with something usable for runtime reflection.
 With lots of detail omitted:

 module reflect;

 Meta[string] metas;
 mixin template Reflect(alias object) {
      static this()
      {
          auto m = meta!(object);
          metas[m.fullName] ~= m;
      }
 }


 module a;
 import reflect;

 struct S
 {
 }
 mixin Reflect!S;

 The meta-object for S is automatically made available at runtime through
 the global metas array. Note that we do not want to force the user to
 register the meta-object manually because then it would not be a "better
 architecture".

 The important (Andrei somehow thinks it is not) requirement is there
 must not be circular dependency issues for the users of the "reflect"
 module.


How about running your own set of "constructors" searching the module 
info array, searching for specific classes in the module that are added 
by a mixin:

----------------------------------
module register;

RegisterBase[string] registry;

void doRegister(string name, RegisterBase r) { registry[name] = r; }

class RegisterBase
{
	abstract void _register();
}

template Register(string name)
{
	enum string Register = "
	class Register : RegisterBase
	{
		override void _register() { doRegister(\"" ~ name ~ "\", this); }
	}
	";
}

void registerAll()
{
	foreach(m; ModuleInfo)
	{
		TypeInfo_Class[] clss = m.localClasses();
		foreach(c; clss)
		{
			if(c.base is RegisterBase.classinfo)
			{
				if(auto reg = cast(RegisterBase) c.create())
				{
					reg._register();
				}
			}
		}
	}
}


-----------------------
module a;
import register;

mixin(Register!"a");

-----------------------
module main;
import std.stdio;
import register;

void main()
{
	registerAll();
	foreach(a, o; registry)
		writeln(a, " ", o);
}

This might also work for the benchmark module.
Nov 30 2012
prev sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, November 29, 2012 16:18:10 Paulo Pinto wrote:

 On Thursday, 29 November 2012 at 12:04:28 UTC, Max Samukha wrote:

 On Thursday, 29 November 2012 at 11:39:20 UTC, Paulo Pinto
 
 wrote:

 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei
 
 Alexandrescu wrote:

 On 11/28/12 9:34 PM, Walter Bright wrote:

 For discussion:


 [snip]
 
 I'd say we better finish const, immutable, and shared first.
 
 Andrei


 
 +1
 
 Fully agree.
 
 Cyclic imports are a minor nuisance that can be easily
 solvable with better code architecture.


 
 Show me please how to solve that problem easily with acceptable
 results, would you?


 
 You just need to have a better architecture.
 
 In 20 years of software development experience I never found a
 case were this wasn't possible.
 
 Maybe you care to provide an example?


Considering that you need static constructors to initialize any static 
variables at runtime (and in the case of const and immutable variables, you 
_can't_ work around that by doing the initialization later - it _must_ be in 
the static constructor), and all it takes for the compiler to declare a 
circular import is to have two modules import each other - even indirectly - 
it becomes extremely easy to run into this problem. And if you're dealing with 
immutable static variables which are initialized at runtime, you can't fix it 
unless you can contort your modules so that they don't depend on each other, 
and with a lot of modules, that can become extremely difficult. We had to strip 
out all static constructors from std.datetime because of this, and in order to 
fix it, we had to do some nasty voodoo with casting in order to lazily 
initialize some variables which are supposed to be immutable. _None_ of that 
sort of thing should be necessary. As it stands, it's basically bad practice 
to use static constructors, because it's so easy to end up with circular 
dependencies, and they can involve modules which don't seem even vaguely 
related because of other stuff that they import and are often a royal pain to 
debug and fix, if you even can without seriously revamping your design. And for 
a library like Phobos which needs to avoid breaking backwards compatibility, 
redesigning things to avoid such circular dependencies isn't necessarily even 
possible, because those redesigns would break backwards compatibliity.

We _need_ a solution for this. Whether now is the best time tackle it is 
another matter, but the current situation with static constructors is 
ridiculous. There are features which require them, but you have to avoid them 
or you're going to run into circular dependency issues very easily.

This is a case of some great design decisions in D have leading to a very bad 
design, and it needs to be fixed. Fortunately, it shouldn't be all that hard to 
fix with an attribute of some kind. But prior to now, Walter wouldn't even 
consider it. I do think though that the idea of using only one pragma instead 
of an attribute/pragma per static constructor is a bad idea because of the 
silent breakage that it would cause, as has been pointed out in this thread. 
So, his proposed solution needs some tweaking.

- Jonathan M Davis
Nov 29 2012
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, November 29, 2012 12:39:19 Paulo Pinto wrote:

 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei
 
 Alexandrescu wrote:

 On 11/28/12 9:34 PM, Walter Bright wrote:

 For discussion:


 [snip]
 
 I'd say we better finish const, immutable, and shared first.
 
 Andrei


 
 +1
 
 Fully agree.
 
 Cyclic imports are a minor nuisance that can be easily solvable
 with better code architecture.
 
 Turbo Pascal/Delphi is the only language that I know fully allows
 cyclic dependencies between modules. So this is not that
 important for most people.


Basic features in the language require static constructors (e.g. static 
variables frequently do), and some things just can't be done without them. 
Andrei's std.benchmark proposal actually doesn't work, because in order to do 
what it does, in needs to mixin in a static constructor and static destructor 
(to set up the benchmarking and record it at the end, I believe). That 
completely falls apart due to cyclical imports. It doesn't actually cause any 
true circular dependencies, but it's treated as such anyway. A _lot_ of us 
have run into this problem and a number of us consider it to be a huge design 
problem in the language. Personally, I'd put it towards the top of design 
mistakes at this point, and I'm very glad to see Walter actually finally being 
willing to do something about this. In the past when I've brought up similar 
solutions, he's been completely opposed to them.

Yes, we have other major issues that need to be resolved, and those may very 
well be of a higher priority, but this is still a very import issue, and it's 
the sort of issue that can probably be implemented with minimal effort. The 
main thing is getting Walter to agree to it and how it will be done. Once a 
decision is made, it wouldn't surprise me if someone like Kenji were able to 
get it done very quickly.

- Jonathan M Davis
Nov 29 2012
next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 29 November 2012 at 12:17:49 UTC, Jonathan M Davis 
wrote:

 On Thursday, November 29, 2012 12:39:19 Paulo Pinto wrote:

 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei
 
 Alexandrescu wrote:

 On 11/28/12 9:34 PM, Walter Bright wrote:

 For discussion:


 [snip]
 
 I'd say we better finish const, immutable, and shared first.
 
 Andrei


 
 +1
 
 Fully agree.
 
 Cyclic imports are a minor nuisance that can be easily solvable
 with better code architecture.
 
 Turbo Pascal/Delphi is the only language that I know fully 
 allows
 cyclic dependencies between modules. So this is not that
 important for most people.


 Basic features in the language require static constructors 
 (e.g. static
 variables frequently do), and some things just can't be done 
 without them.
 Andrei's std.benchmark proposal actually doesn't work, because 
 in order to do
 what it does, in needs to mixin in a static constructor and 
 static destructor
 (to set up the benchmarking and record it at the end, I 
 believe). That
 completely falls apart due to cyclical imports. It doesn't 
 actually cause any
 true circular dependencies, but it's treated as such anyway. A 
 _lot_ of us
 have run into this problem and a number of us consider it to be 
 a huge design
 problem in the language. Personally, I'd put it towards the top 
 of design
 mistakes at this point, and I'm very glad to see Walter 
 actually finally being
 willing to do something about this. In the past when I've 
 brought up similar
 solutions, he's been completely opposed to them.

 Yes, we have other major issues that need to be resolved, and 
 those may very
 well be of a higher priority, but this is still a very import 
 issue, and it's
 the sort of issue that can probably be implemented with minimal 
 effort. The
 main thing is getting Walter to agree to it and how it will be 
 done. Once a
 decision is made, it wouldn't surprise me if someone like Kenji 
 were able to
 get it done very quickly.

 - Jonathan M Davis


Maybe I should keep my mouth shut, because actually I don't 
really use D besides some toy experiments, as there is no place 
for D in the type of work I currently do, either on the job or 
privately.

I like however to point people to D as a possible C++ successor 
as way to do some language publicity, and think that these type 
of discussions might scare D newbies away in what concerns 
language stability, hence my post.

--
Paulo
Nov 29 2012
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
11/29/2012 4:17 PM, Jonathan M Davis пишет:

 On Thursday, November 29, 2012 12:39:19 Paulo Pinto wrote:

 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei

 Alexandrescu wrote:

 On 11/28/12 9:34 PM, Walter Bright wrote:

 For discussion:


 [snip]

 I'd say we better finish const, immutable, and shared first.

 Andrei


 +1

 Fully agree.

 Cyclic imports are a minor nuisance that can be easily solvable
 with better code architecture.

 Turbo Pascal/Delphi is the only language that I know fully allows
 cyclic dependencies between modules. So this is not that
 important for most people.


 Basic features in the language require static constructors (e.g. static
 variables frequently do), and some things just can't be done without them.
 Andrei's std.benchmark proposal actually doesn't work, because in order to do
 what it does, in needs to mixin in a static constructor and static destructor
 (to set up the benchmarking and record it at the end, I believe). That
 completely falls apart due to cyclical imports.


I do suspect it could be healed by inverting the control flow (or rather 
bringing it back).

Basically there is this pattern:

module a;
...
mixin ScheduleForBenchmark;

Same for module b, c, d...

And then compiling all of this with -version benchmark should run the 
benchmarks.

It already has another problem: there has to be D main stuffed in there 
somehow. It can't be in std.benchmark as it then will have to be 
included in phobos(.lib|.a) and it can't be magically turned on/off with 
-version switch either unless std.benchmark is passed directly to the 
compiler.

The other solution would be: make a separate my_benchmark.d module that 
contains:
import a,b,c,d...;

void main(){
    runBenchmarks!(a,b,c,d...)();
}

And that's it.
In any case I usually like having some separation between different sets 
of modules to benchmark so that I can start them in a 'clean room' scenario.


 It doesn't actually cause any
 true circular dependencies, but it's treated as such anyway. A _lot_ of us
 have run into this problem and a number of us consider it to be a huge design
 problem in the language. Personally, I'd put it towards the top of design
 mistakes at this point, and I'm very glad to see Walter actually finally being
 willing to do something about this. In the past when I've brought up similar
 solutions, he's been completely opposed to them.

 Yes, we have other major issues that need to be resolved, and those may very
 well be of a higher priority, but this is still a very import issue, and it's
 the sort of issue that can probably be implemented with minimal effort. The
 main thing is getting Walter to agree to it and how it will be done. Once a
 decision is made, it wouldn't surprise me if someone like Kenji were able to
 get it done very quickly.

 - Jonathan M Davis


-- 
Dmitry Olshansky
Nov 29 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/29/12 10:54 AM, Dmitry Olshansky wrote:

 Basically there is this pattern:

 module a;
 ...
 mixin ScheduleForBenchmark;

 Same for module b, c, d...


Correct.


 And then compiling all of this with -version benchmark should run the
 benchmarks.

 It already has another problem: there has to be D main stuffed in there
 somehow. It can't be in std.benchmark as it then will have to be
 included in phobos(.lib|.a) and it can't be magically turned on/off with
 -version switch either unless std.benchmark is passed directly to the
 compiler.

 The other solution would be: make a separate my_benchmark.d module that
 contains:
 import a,b,c,d...;

 void main(){
 runBenchmarks!(a,b,c,d...)();
 }


That's workable. I'm hoping, however, to make benchmarks as easy and as 
trivial to define as unittests.


Andrei
Nov 29 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-11-29 16:58, Andrei Alexandrescu wrote:


 That's workable. I'm hoping, however, to make benchmarks as easy and as
 trivial to define as unittests.


Unit tests have the same problem. One need to create a module containing 
a main function and importing the modules one wants to benchmark/test.

One idea would be to add more support for this in the compiler.

Another idea, that might better and is easier to implement, is tool that:

1. Takes a couple of modules on the command line
2. Create a main module which imports the modules on the command line
3. Compile all the modules
4. Run the benchmarks or tests

-- 
/Jacob Carlborg
Nov 29 2012
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 29 November 2012 at 12:17:49 UTC, Jonathan M Davis 
wrote:

 Basic features in the language require static constructors 
 (e.g. static
 variables frequently do), and some things just can't be done 
 without them.
 Andrei's std.benchmark proposal actually doesn't work, because 
 in order to do
 what it does, in needs to mixin in a static constructor and 
 static destructor
 (to set up the benchmarking and record it at the end, I 
 believe). That
 completely falls apart due to cyclical imports. It doesn't 
 actually cause any
 true circular dependencies, but it's treated as such anyway. A 
 _lot_ of us
 have run into this problem and a number of us consider it to be 
 a huge design
 problem in the language. Personally, I'd put it towards the top 
 of design
 mistakes at this point, and I'm very glad to see Walter 
 actually finally being
 willing to do something about this. In the past when I've 
 brought up similar
 solutions, he's been completely opposed to them.

 Yes, we have other major issues that need to be resolved, and 
 those may very
 well be of a higher priority, but this is still a very import 
 issue, and it's
 the sort of issue that can probably be implemented with minimal 
 effort. The
 main thing is getting Walter to agree to it and how it will be 
 done. Once a
 decision is made, it wouldn't surprise me if someone like Kenji 
 were able to
 get it done very quickly.

 - Jonathan M Davis


That is understood, but Let me explain how I see things.

We are here adding yet a new feature, however small. Nothing 
stabilize when adding new feature all the time.

The annoyance exist, is real, but is not THAT bad, and 'm pretty 
sure the proposed solution is likely to backfire. Firstly because 
a module can have several constructors, and more can be added by 
mixins. Ensuring that constructor A will not create dependancy 
cycle error may silently break other constructor of the module.

Considering the problem static constructor solve, I'd be happy to 
see some think out of the box. The way things work here may be 
broken. Many people consider that D runtime should evolve in a 
way that promote fibers and map them over system threads (this 
have many benefit : the program adapt autmatically to the 
provided hardware, yields can be introduced directly into the 
runtime or some libraries to hide IO latencies, and many other 
things), and static constructor/variables really get into the way 
(you would have to run all static constructors each time you 
start a new fiber or reset one to reuse it).

I really wish we can focus on figuring out the uses cases we have 
for static constructor and start the reflection from theses sues 
case and not from the feature itself.
Nov 29 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-11-29 19:17, deadalnix wrote:


 That is understood, but Let me explain how I see things.

 We are here adding yet a new feature, however small. Nothing stabilize
 when adding new feature all the time.

 The annoyance exist, is real, but is not THAT bad, and 'm pretty sure
 the proposed solution is likely to backfire. Firstly because a module
 can have several constructors, and more can be added by mixins. Ensuring
 that constructor A will not create dependancy cycle error may silently
 break other constructor of the module.

 Considering the problem static constructor solve, I'd be happy to see
 some think out of the box. The way things work here may be broken. Many
 people consider that D runtime should evolve in a way that promote
 fibers and map them over system threads (this have many benefit : the
 program adapt autmatically to the provided hardware, yields can be
 introduced directly into the runtime or some libraries to hide IO
 latencies, and many other things), and static constructor/variables
 really get into the way (you would have to run all static constructors
 each time you start a new fiber or reset one to reuse it).

 I really wish we can focus on figuring out the uses cases we have for
 static constructor and start the reflection from theses sues case and
 not from the feature itself.




-- 
/Jacob Carlborg
Nov 29 2012
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, November 29, 2012 21:08:58 Jacob Carlborg wrote:




They just let you blow your foot off. All static variables can be directly 
initialized at runtime, so it's easy to use variables before they're actually 
initialized. I don't know how they decide what order to run static 
constructors in, but AFAIK, it never worries about circular dependencies. 
We're only running into this problem beacuse we're trying to provide higher 
safety and better guarantees with regards to when and how variables are 
initialized.

- Jonathan M Davis
Nov 29 2012
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 29 November 2012 at 21:43:30 UTC, Jonathan M Davis 
wrote:

 On Thursday, November 29, 2012 21:08:58 Jacob Carlborg wrote:


 similar.


 They just let you blow your foot off. All static variables can 
 be directly
 initialized at runtime, so it's easy to use variables before 
 they're actually
 initialized. I don't know how they decide what order to run 
 static
 constructors in, but AFAIK, it never worries about circular 
 dependencies.
 We're only running into this problem beacuse we're trying to 
 provide higher
 safety and better guarantees with regards to when and how 
 variables are
 initialized.

 - Jonathan M Davis


Java have static block to pre initialize stuff before it is used. 
But I'm not sure how they react in case of cyclic dependancies.
Nov 29 2012
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 11/29/2012 10:45 PM, deadalnix wrote:

 On Thursday, 29 November 2012 at 21:43:30 UTC, Jonathan M Davis wrote:

 On Thursday, November 29, 2012 21:08:58 Jacob Carlborg wrote:




 They just let you blow your foot off. All static variables can be
 directly
 initialized at runtime, so it's easy to use variables before they're
 actually
 initialized. I don't know how they decide what order to run static
 constructors in, but AFAIK, it never worries about circular dependencies.
 We're only running into this problem beacuse we're trying to provide
 higher
 safety and better guarantees with regards to when and how variables are
 initialized.

 - Jonathan M Davis


 Java have static block to pre initialize stuff before it is used. But
 I'm not sure how they react in case of cyclic dependancies.




arbitrary side-effecting code to run. Circular dependencies are not 
considered at all. If two static constructors depend on each other's 
statically initialized parts, at least one of them will see 
uninitialized data.
Nov 29 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-11-29 22:43, Jonathan M Davis wrote:


 They just let you blow your foot off. All static variables can be directly
 initialized at runtime, so it's easy to use variables before they're actually
 initialized. I don't know how they decide what order to run static
 constructors in, but AFAIK, it never worries about circular dependencies.
 We're only running into this problem beacuse we're trying to provide higher
 safety and better guarantees with regards to when and how variables are
 initialized.


I see.

-- 
/Jacob Carlborg
Nov 30 2012
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 11/29/2012 01:17 PM, Jonathan M Davis wrote:

 In the past when I've brought up similar solutions, he's been completely
opposed to them.
 ...


It is not a solution, it is a workaround.
Nov 29 2012
next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, November 29, 2012 23:28:07 Timon Gehr wrote:

 On 11/29/2012 01:17 PM, Jonathan M Davis wrote:

 In the past when I've brought up similar solutions, he's been completely
 opposed to them. ...


 
 It is not a solution, it is a workaround.


What do you mean? The runtime sees circular dependencies between modules even 
when there's no actual circular dependency between static constructors. We 
need to fix that. One way is to just make the runtime not care, which wouldn't 
be particularly safe. Another is to explicitly tell it that there are no such 
dependencies. I don't see how that's not a solution. And unless someone can 
come up with a way for the runtime to somehow determine on its own that 
there's no actual, circular dependency, I don't see how anything better could 
be done.

Granted, I think that having a single pragma for the whole module like Walter 
is suggesting (as opposed to marking static constructors individually) is a 
bad idea because of the silent breakage that it can cause, but the basic idea 
is solid.

- Jonathan M Davis
Nov 29 2012
prev sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 11/29/12 23:34, Jonathan M Davis wrote:

 On Thursday, November 29, 2012 23:28:07 Timon Gehr wrote:

 On 11/29/2012 01:17 PM, Jonathan M Davis wrote:

 In the past when I've brought up similar solutions, he's been completely
 opposed to them. ...


 It is not a solution, it is a workaround.


 
 What do you mean? The runtime sees circular dependencies between modules even 
 when there's no actual circular dependency between static constructors. We 
 need to fix that. One way is to just make the runtime not care, which wouldn't 
 be particularly safe. Another is to explicitly tell it that there are no such 
 dependencies. I don't see how that's not a solution. And unless someone can 
 come up with a way for the runtime to somehow determine on its own that 
 there's no actual, circular dependency, I don't see how anything better could 
 be done.


It's relatively easy for the /compiler/ to figure it out; it's just that
implementing a simple user-provided flag requires the least amount of work.
What Walter suggested can be tweaked to be sane (per-ctor flag) and will
still be useful if/when the compiler becomes smarter (think lazily initted
module fields).
For the compiler to check if the value of every imported symbol accessed
inside a mod-ctor can be evaluated at compile-time (if you encounter a case
where this is not true it means there (potentially) is a true dependency and
the ctors should be ordered) would require more work.

artur
Nov 30 2012
prev sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Thu, 29 Nov 2012 12:39:19 +0100
"Paulo Pinto" <pjmlp progtools.org> wrote:


 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei 
 Alexandrescu wrote:

 On 11/28/12 9:34 PM, Walter Bright wrote:

 For discussion:


 [snip]

 I'd say we better finish const, immutable, and shared first.

 Andrei


 
 +1
 
 Fully agree.
 
 Cyclic imports are a minor nuisance that can be easily solvable 
 with better code architecture.
 


Or just lazy initialization:

    Foo f = blah();

-->

    private Foo _f;
    private bool fInited=false;
     property Foo f()
    {
        if(!fInited)
        {
            _f = blah();
            fInited = true;
        }

        return _f;
    }

And that boilerplate's trivially wrapped up with a mixin.
Dec 07 2012
prev sibling next sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 11/29/12 03:34, Walter Bright wrote:

 Proposed Solution:
 
 Add a pragma,
 
     pragma(cyclic_imports);
 
 This can appear anywhere in a module, and applies globally to that module.
 It means that static constructors from imports that are not part of the cycle
 are run first, and that the static constructor for this module may be run
before
 the static constructors of other modules that are part of the cycle.


Bad name. Something like "module_ctors_unordered" would be better (I don't
like that one either, but this way it's at least clear what it does).

The equivalent attribute version could be

   module mymodule  unordered_ctors;

which is better, but still has the problem that adding another module ctor
can result is silent breakage. These things can be mixed in, so something
as innocent looking as

   import some_lib;
   [...]
   mixin blah;

can already be a bug. Hence:

   static this()  unordered { /*whatever*/ }

and then either enforce that all mod-ctors have this attribute (otherwise
every ctor in that moduile gets treated as ordered), or split them into
two sets (better, but larger change; backward compatible, just requires
a new enough runtime to be used). 

Of course it could be also done as

    static this() pragma(unordered) { /*whatever*/ }

but this wouldn't really be better, and would require language change (the
fact that you cannot attach a pragma to /anything/ is already a problem,
eg for gcc-specific attributes).

" unordered" could even be inferred, but I'm not sure how often that would
help in practice. At least w/o making module-level imports invisible inside
module ctors (which would make the deps explicit).

artur
Nov 28 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, November 29, 2012 07:36:41 Artur Skawina wrote:

 On 11/29/12 03:34, Walter Bright wrote:

 Proposed Solution:
 
 Add a pragma,
 
     pragma(cyclic_imports);
 
 This can appear anywhere in a module, and applies globally to that module.
 It means that static constructors from imports that are not part of the
 cycle are run first, and that the static constructor for this module may
 be run before the static constructors of other modules that are part of
 the cycle.


 Bad name. Something like "module_ctors_unordered" would be better (I don't
 like that one either, but this way it's at least clear what it does).


no_cyclic_imports would probably be better given that you're trying to get 
around a cyclic import, but I don't see what's unclear about cyclic_imports 
given that that's exactly what the runtime complains about when this problem 
occurs.


 but still has the problem that adding another module ctor
 can result is silent breakage.


That's actually a really good argument IMHO for having to put the pragma or 
attribute on every single static constructor in a module. True, it may be a 
bit annoying, but it would avoid silent breakage.

- Jonathan M Davis
Nov 28 2012
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-11-29 03:34, Walter Bright wrote:


 Add a pragma,

      pragma(cyclic_imports);

 This can appear anywhere in a module, and applies globally to that module.
 It means that static constructors from imports that are not part of the
 cycle
 are run first, and that the static constructor for this module may be
 run before
 the static constructors of other modules that are part of the cycle.


I would think that "cyclic_imports" sounds like the static constructors 
are part of the cycle.

-- 
/Jacob Carlborg
Nov 28 2012
prev sibling next sibling parent "Max Samukha" <maxsamukha gmail.com> writes:
On Thursday, 29 November 2012 at 02:34:11 UTC, Walter Bright 
wrote:

 For discussion:

 Cyclical Imports

 Problem:

 ---- a.d ----
     module a;
     import b;
     static this () { ... }
 ---- b.d ----
     module b;
     import a;
     static this() { ... }
 -------------

 Static constructors for a module are only run after static 
 constructors
 for all its imports are run. Circular imports, such as the 
 above, are
 detected at run time and the program is aborted.

 This can in general be solved by moving the static 
 constructor(s) into
 a third module, c.d, which does not import a or b. But, people 
 find this
 to be unnatural.


It is natural. It just doesn't cover important use cases.


 Proposed Solution:

 Add a pragma,

     pragma(cyclic_imports);

 This can appear anywhere in a module, and applies globally to 
 that module.
 It means that static constructors from imports that are not 
 part of the cycle
 are run first, and that the static constructor for this module 
 may be run before
 the static constructors of other modules that are part of the 
 cycle.

 If any static constructors in such a module with the pragma have
 the  safe attribute, that is a compile time error.


Can we have that implemented in a branch and see how it goes?
Nov 29 2012
prev sibling next sibling parent reply Manfred Nowak <svv1999 hotmail.com> writes:
Walter Bright wrote:


 It means that [...] the static constructor for this
 module may be run before


This is sufficient only for a simple cycle without any branches 
or a trivial clique like the one shown. Please recall, that in a 
clique every member is connected to every other member---in this 
case by an import statement.

It is already not sufficient for a simple clique consisting of 
three modules, because now orderinh would be given for the 
remaining two modules.

And it is ambiguous if both of the modules in the example given 
are marked with that pragma.

In general a topological sorting has to be specified for all 
strongly connected components of the graph of imports.

This cannot be done within the module, because this would bind 
the module to that special strongly connected component in that 
special set of  modules. Which in turn would destroy reusability 
of the module for other programming tasks.

Therefore: marking with a pragma isn't a fix for the depicted 
problem.

-manfred
Nov 29 2012
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/29/2012 11:58 PM, Manfred Nowak wrote:

 This is sufficient only for a simple cycle without any branches
 or a trivial clique like the one shown. Please recall, that in a
 clique every member is connected to every other member---in this
 case by an import statement.

 It is already not sufficient for a simple clique consisting of
 three modules, because now orderinh would be given for the
 remaining two modules.


If you have:

    a imports b,c
    b imports a,c
    c imports a,b

then two of those will need the pragma, that is correct. I don't see an 
issue with that, because there are two cycles.


 And it is ambiguous if both of the modules in the example given
 are marked with that pragma.


If:

    a imports b
    b imports a

and a has the pragma, then the order of construction is a,b. If both 
have the pragma, then the order is a,b or b,a, yes, it is ambiguous, but 
it is not an error. One gets picked arbitrarily.



 In general a topological sorting has to be specified for all
 strongly connected components of the graph of imports.


I believe that the pragma does that.


 This cannot be done within the module, because this would bind
 the module to that special strongly connected component in that
 special set of  modules. Which in turn would destroy reusability
 of the module for other programming tasks.


I just don't see the problem.
Nov 29 2012
parent Manfred Nowak <svv1999 hotmail.com> writes:
Walter Bright wrote:


 yes, it is ambiguous, but it is not an error.
 One gets picked arbitrarily.


I did not see that it would not be erroneous. But if it is true, 
then I do not see a sense in manually adding pragmas: they can 
be assumed to exist.

-manfred
Nov 29 2012
prev sibling next sibling parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:k96hj2$2lus$1 digitalmars.com...

 For discussion:

 Cyclical Imports

 Problem:


 Proposed Solution:

 Add a pragma,

     pragma(cyclic_imports);

 This can appear anywhere in a module, and applies globally to that module.
 It means that static constructors from imports that are not part of the 
 cycle
 are run first, and that the static constructor for this module may be run 
 before
 the static constructors of other modules that are part of the cycle.


I don't think this is sufficient.  Imagine a group of modules that really 
_do_ have a cyclic dependency, and a mixin that adds an independent static 
this.  Ideally you'd be able to mark the mixed-in constructor as independent 
without tainting the whole module.

So just make the pragma apply to declarations, you either mark specific 
functions (which can then be mixed in) or put `pragma(...):` at the top of 
your module and you get your behaviour.
Nov 29 2012
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/30/2012 12:09 AM, Daniel Murphy wrote:

 I don't think this is sufficient.  Imagine a group of modules that really
 _do_ have a cyclic dependency, and a mixin that adds an independent static
 this.  Ideally you'd be able to mark the mixed-in constructor as independent
 without tainting the whole module.

 So just make the pragma apply to declarations, you either mark specific
 functions (which can then be mixed in) or put `pragma(...):` at the top of
 your module and you get your behaviour.


It is possible for each static constructor to specify independently of 
the other static constructors which imports must be constructed first. 
But do we really want to go that far?
Nov 29 2012
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/30/2012 9:43 AM, Walter Bright wrote:

 It is possible for each static constructor to specify independently of
 the other static constructors which imports must be constructed first.
 But do we really want to go that far?


One way to do that might be to borrow syntax from classes:

    static this() : std.stdio, a, c
    {
        ...
    }

and the  this static constructor only requires that modules std.stdio, 
a, and c be constructed first.

    static this() : void
    {
        ...
    }

means it has no dependencies on other imports.

    static this()
    {
       ...
    }

has the current behavior (all imported modules must be constructed first).
Nov 29 2012
parent reply "foobar" <foo bar.com> writes:
On Thursday, 29 November 2012 at 23:02:17 UTC, Walter Bright 
wrote:

 On 11/30/2012 9:43 AM, Walter Bright wrote:

 It is possible for each static constructor to specify 
 independently of
 the other static constructors which imports must be 
 constructed first.
 But do we really want to go that far?


 One way to do that might be to borrow syntax from classes:

    static this() : std.stdio, a, c
    {
        ...
    }

 and the  this static constructor only requires that modules 
 std.stdio, a, and c be constructed first.

    static this() : void
    {
        ...
    }

 means it has no dependencies on other imports.

    static this()
    {
       ...
    }

 has the current behavior (all imported modules must be 
 constructed first).


Why not simplify?

static this()
{
import std.stdio, a, c; // existing syntax
    ...
}

static this()
{ // no imports -> no dependencies
    ...
}

The current behavior should just be dropped.
Nov 30 2012
next sibling parent reply "Tove" <tove fransson.se> writes:
On Friday, 30 November 2012 at 14:09:48 UTC, foobar wrote:

 Why not simplify?

 static this()
 {
 import std.stdio, a, c; // existing syntax
    ...
 }

 static this()
 { // no imports -> no dependencies
    ...
 }

 The current behavior should just be dropped.


+2
Simple & Elegant.
Nov 30 2012
parent "Regan Heath" <regan netmail.co.nz> writes:
On Fri, 30 Nov 2012 14:29:19 -0000, Tove <tove fransson.se> wrote:


 On Friday, 30 November 2012 at 14:09:48 UTC, foobar wrote:

 Why not simplify?

 static this()
 {
 import std.stdio, a, c; // existing syntax
    ...
 }

 static this()
 { // no imports -> no dependencies
    ...
 }

 The current behavior should just be dropped.


 +2
 Simple & Elegant.


-2
Confusing.  What is the scope of the import?  How does it interact with  
imports above/below the static this?

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/
Nov 30 2012
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/30/12 9:09 AM, foobar wrote:

 On Thursday, 29 November 2012 at 23:02:17 UTC, Walter Bright wrote:

 On 11/30/2012 9:43 AM, Walter Bright wrote:

 It is possible for each static constructor to specify independently of
 the other static constructors which imports must be constructed first.
 But do we really want to go that far?


 One way to do that might be to borrow syntax from classes:

 static this() : std.stdio, a, c




[snip]

 Why not simplify?


[snip]

Why not further simplify?

static this()
{
     // JUST AS BEFORE
     ...
}

There is no need to redundantly specify what modules are used because... 
well they are right there in the body of the static constructor.

* no extra syntax
* no change in the symbol visibility rules (why are symbols invisible by 
default in static cdtors?)
* no change to the manual
* no breakage of existing code (only code that was broken will be accepted)
* no acceptance of actual circular dependencies go through compilation

This would be purely an improvement to the implementation that would 
allow more correct programs to compile. It's a removal of limitation - 
the best kind of language change there ever is.

We should have a "bootcamp" area with small compiler and library 
projects (such as this after we reach consensus). People who are 
interested in helping D, whether or not they've done it before, could 
find this area things that are well defined and will definitely be 
accepted if properly executed.


Andrei
Nov 30 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/29/12 5:43 PM, Walter Bright wrote:

 On 11/30/2012 12:09 AM, Daniel Murphy wrote:

 I don't think this is sufficient. Imagine a group of modules that really
 _do_ have a cyclic dependency, and a mixin that adds an independent
 static
 this. Ideally you'd be able to mark the mixed-in constructor as
 independent
 without tainting the whole module.

 So just make the pragma apply to declarations, you either mark specific
 functions (which can then be mixed in) or put `pragma(...):` at the
 top of
 your module and you get your behaviour.


 It is possible for each static constructor to specify independently of
 the other static constructors which imports must be constructed first.
 But do we really want to go that far?


I think we either do it right or leave it as it is. It's not like 
there's no workaround so if we take a stand here we better have 
something compelling.

Andrei
Nov 29 2012
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 30 November 2012 at 01:07:57 UTC, Andrei Alexandrescu 
wrote:

 I think we either do it right or leave it as it is. It's not 
 like there's no workaround so if we take a stand here we better 
 have something compelling.

 Andrei


Finally some sanity here !
Nov 29 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, November 29, 2012 20:07:57 Andrei Alexandrescu wrote:

 I think we either do it right or leave it as it is. It's not like
 there's no workaround so if we take a stand here we better have
 something compelling.


I think that an attribute per static constructor indicating that it had no 
circular dependencies would solve the problem just fine (putting it on the 
module is problematic, because new static constructors which _do_ have actual 
circular dependencies could be added), but we certainly want to be sure of 
this before doing it.

And if we want to focus on shared or whatever now because it's higher 
priority, that's fine, but we do need to move forward with outstanding issues 
like these, and I do think that really need to solve this particular problem 
rather than considering the workarounds to be okay. A situation where static 
constructors effectively must be shunned because of the problems that they 
cause is definitely problematic considering how many features require them 
(e.g. a const or immutable static variable). And that's what we have right 
now.

- Jonathan M Davis
Nov 29 2012
prev sibling parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Friday, 30 November 2012 at 01:07:57 UTC, Andrei Alexandrescu 
wrote:

 On 11/29/12 5:43 PM, Walter Bright wrote:

 On 11/30/2012 12:09 AM, Daniel Murphy wrote:

 I don't think this is sufficient. Imagine a group of modules 
 that really
 _do_ have a cyclic dependency, and a mixin that adds an 
 independent
 static
 this. Ideally you'd be able to mark the mixed-in constructor 
 as
 independent
 without tainting the whole module.

 So just make the pragma apply to declarations, you either 
 mark specific
 functions (which can then be mixed in) or put `pragma(...):` 
 at the
 top of
 your module and you get your behaviour.


 It is possible for each static constructor to specify 
 independently of
 the other static constructors which imports must be 
 constructed first.
 But do we really want to go that far?


 I think we either do it right or leave it as it is. It's not 
 like there's no workaround so if we take a stand here we better 
 have something compelling.

 Andrei


+1

FWIW, I think this proposal sounds like a massive hack. Not a fan.
Nov 30 2012
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/30/2012 9:05 PM, Peter Alexander wrote:

 On Friday, 30 November 2012 at 01:07:57 UTC, Andrei Alexandrescu wrote:

 I think we either do it right or leave it as it is. It's not like
 there's no workaround so if we take a stand here we better have
 something compelling.

 Andrei


 +1

 FWIW, I think this proposal sounds like a massive hack. Not a fan.


Andrei has a point.
Nov 30 2012
prev sibling next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, November 30, 2012 10:27:31 Artur Skawina wrote:

 On 11/29/12 23:34, Jonathan M Davis wrote:

 On Thursday, November 29, 2012 23:28:07 Timon Gehr wrote:

 On 11/29/2012 01:17 PM, Jonathan M Davis wrote:

 In the past when I've brought up similar solutions, he's been completely
 opposed to them. ...


 
 It is not a solution, it is a workaround.


 
 What do you mean? The runtime sees circular dependencies between modules
 even when there's no actual circular dependency between static
 constructors. We need to fix that. One way is to just make the runtime
 not care, which wouldn't be particularly safe. Another is to explicitly
 tell it that there are no such dependencies. I don't see how that's not a
 solution. And unless someone can come up with a way for the runtime to
 somehow determine on its own that there's no actual, circular dependency,
 I don't see how anything better could be done.


 
 It's relatively easy for the /compiler/ to figure it out; it's just that
 implementing a simple user-provided flag requires the least amount of work.
 What Walter suggested can be tweaked to be sane (per-ctor flag) and will
 still be useful if/when the compiler becomes smarter (think lazily initted
 module fields).
 For the compiler to check if the value of every imported symbol accessed
 inside a mod-ctor can be evaluated at compile-time (if you encounter a case
 where this is not true it means there (potentially) is a true dependency and
 the ctors should be ordered) would require more work.


It can't be evaluated at compile time because of .di files. The compiler 
doesn't necessarily have all of the source to work with - including the static 
constructors - and if it doesn't have that, it can't do it. If anything figures 
this out automatically, it has to be the runtime. If we can do that, great, 
but it means that it'll have actually have to look at individual symbols 
rather than just at the module level like it's doing now, and right now, we 
have nothing even close to that. Regardless, while the compiler may be able
to provide additional information to the runtime, it's still the runtime that
needs to figure this out and not the compiler.

- Jonathan M Davis
Nov 30 2012
parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 30 November 2012 at 20:40:23 UTC, Jonathan M Davis 
wrote:

 It can't be evaluated at compile time because of .di files. The 
 compiler
 doesn't necessarily have all of the source to work with - 
 including the static
 constructors - and if it doesn't have that, it can't do it. If 
 anything figures
 this out automatically, it has to be the runtime. If we can do 
 that, great,
 but it means that it'll have actually have to look at 
 individual symbols
 rather than just at the module level like it's doing now, and 
 right now, we
 have nothing even close to that. Regardless, while the compiler 
 may be able
 to provide additional information to the runtime, it's still 
 the runtime that
 needs to figure this out and not the compiler.


I'd bet you can solve most cases anyway.
Nov 30 2012
prev sibling next sibling parent reply Artur Skawina <art.08.09 gmail.com> writes:
On 11/30/12 21:40, Jonathan M Davis wrote:

 On Friday, November 30, 2012 10:27:31 Artur Skawina wrote:

 On 11/29/12 23:34, Jonathan M Davis wrote:

 On Thursday, November 29, 2012 23:28:07 Timon Gehr wrote:

 On 11/29/2012 01:17 PM, Jonathan M Davis wrote:

 In the past when I've brought up similar solutions, he's been completely
 opposed to them. ...


 It is not a solution, it is a workaround.


 What do you mean? The runtime sees circular dependencies between modules
 even when there's no actual circular dependency between static
 constructors. We need to fix that. One way is to just make the runtime
 not care, which wouldn't be particularly safe. Another is to explicitly
 tell it that there are no such dependencies. I don't see how that's not a
 solution. And unless someone can come up with a way for the runtime to
 somehow determine on its own that there's no actual, circular dependency,
 I don't see how anything better could be done.


 It's relatively easy for the /compiler/ to figure it out; it's just that
 implementing a simple user-provided flag requires the least amount of work.
 What Walter suggested can be tweaked to be sane (per-ctor flag) and will
 still be useful if/when the compiler becomes smarter (think lazily initted
 module fields).
 For the compiler to check if the value of every imported symbol accessed
 inside a mod-ctor can be evaluated at compile-time (if you encounter a case
 where this is not true it means there (potentially) is a true dependency and
 the ctors should be ordered) would require more work.


 
 It can't be evaluated at compile time because of .di files. The compiler 


It can. The case where all definitions aren't available is of course a potential
source of cycles and means that the ctors have to be run in order. There's no
way around that, and it's where the attribute (nee pragma) helps, because it can
be attached to the declaration, when the programmer thinks he knows better.

Andrei's naive suggestion would only handle the trivial toy example case, but
fail for much of real code, that needs to access statically known properties of
other modules. D has enough features that work for five-liners, but are unusable
for real work.

The important points are that:

1) a ctor that does not access any symbol from another module is not treated
   as dependent on that module.
2) accessing just /types/ defined in another module works. No RT dep here.
3) reading known initialized constant data works. That's const/immutable/enum -
   again, those can never become a RT dep.
4) calling side-effect free code that does not depend on non-local state works.
   This is why an is-it-ctfeable check works - it will catch even the indirect
deps.
   Yeah, it's conservative, but is has to be. A kind of
'pure-but-w/o-external-refs"
   thing would help further, but that can be added incrementally and may not
even
   be necessary.   


 have nothing even close to that. Regardless, while the compiler may be able
 to provide additional information to the runtime, it's still the runtime that
 needs to figure this out and not the compiler.


No. Would, at least, require going from per-module to per-symbol which is a
*much*
larger change than was proposed here. I don't even want to think about the
(runtime)
cost.

artur
Dec 01 2012
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 1 December 2012 at 10:54:37 UTC, Artur Skawina wrote:

 3) reading known initialized constant data works. That's 
 const/immutable/enum -
    again, those can never become a RT dep.


const or immutable can be instanciated by another static ctor.


 4) calling side-effect free code that does not depend on 
 non-local state works.
    This is why an is-it-ctfeable check works - it will catch 
 even the indirect deps.
    Yeah, it's conservative, but is has to be. A kind of 
 'pure-but-w/o-external-refs"
    thing would help further, but that can be added 
 incrementally and may not even
    be necessary.


pure is probably enough. CTFEable is more restrictive.


 have nothing even close to that. Regardless, while the 
 compiler may be able
 to provide additional information to the runtime, it's still 
 the runtime that
 needs to figure this out and not the compiler.




That is an undemonstrated assertion (and I think it is false).


 No. Would, at least, require going from per-module to 
 per-symbol which is a *much*
 larger change than was proposed here. I don't even want to 
 think about the (runtime)
 cost.


This reasonning is based on the assertion above, so is 
meaningless until the assertion is proven to be true.
Dec 01 2012
parent reply Artur Skawina <art.08.09 gmail.com> writes:
On 12/01/12 12:29, deadalnix wrote:

 On Saturday, 1 December 2012 at 10:54:37 UTC, Artur Skawina wrote:

 3) reading known initialized constant data works. That's const/immutable/enum -
    again, those can never become a RT dep.


 
 const or immutable can be instanciated by another static ctor.


The key words are "known" and "initialized". IOW

   const int a = 42;

doesn't add a dep, but

  const int a;

does. 

/Modifying/ initialized const data from a mod ctor is (and should be) illegal.



 4) calling side-effect free code that does not depend on non-local state works.
    This is why an is-it-ctfeable check works - it will catch even the indirect
deps.
    Yeah, it's conservative, but is has to be. A kind of
'pure-but-w/o-external-refs"
    thing would help further, but that can be added incrementally and may not
even
    be necessary.


 
 pure is probably enough. CTFEable is more restrictive.


D's pure isn't enough, because it will allow accesses to external immutable
data - which can be modified by a mod ctor. But like i said - this might not be
a problem in practice, as it's just about the cross-module non-ctfeable, but
truly pure function calls inside mod ctors - is this really something that
occurs
often enough?


 have nothing even close to that. Regardless, while the compiler may be able
 to provide additional information to the runtime, it's still the runtime that
 needs to figure this out and not the compiler.




 
 That is an undemonstrated assertion (and I think it is false).
 

 No. Would, at least, require going from per-module to per-symbol which is a
*much*
 larger change than was proposed here. I don't even want to think about the
(runtime)
 cost.


 
 This reasonning is based on the assertion above, so is meaningless until the
assertion is proven to be true.


I'm not sure what assertion you think is false, but theoretically it /is/
possible
to figure out the deps at runtime, even w/o any hints. Think valgrind. But it's
a very bad idea, with a significant runtime cost (not as bad as valgrind since
you
can figure out some things statically, but determining the order alone would
already
be too slow for larger projects with many symbols and deps.)

artur
Dec 01 2012
parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 1 December 2012 at 12:08:10 UTC, Artur Skawina wrote:

 The key words are "known" and "initialized". IOW

    const int a = 42;

 doesn't add a dep, but

   const int a;

 does.

 /Modifying/ initialized const data from a mod ctor is (and 
 should be) illegal.


That make sense.


 D's pure isn't enough, because it will allow accesses to 
 external immutable
 data - which can be modified by a mod ctor. But like i said - 
 this might not be
 a problem in practice, as it's just about the cross-module 
 non-ctfeable, but
 truly pure function calls inside mod ctors - is this really 
 something that occurs
 often enough?


Ho yes, this make sense too. CTFEable is still too restrictive 
but pure isn't enough.


 I'm not sure what assertion you think is false, but 
 theoretically it /is/ possible
 to figure out the deps at runtime, even w/o any hints. Think 
 valgrind. But it's
 a very bad idea, with a significant runtime cost (not as bad as 
 valgrind since you
 can figure out some things statically, but determining the 
 order alone would already
 be too slow for larger projects with many symbols and deps.)


The assertion is that it must be done at runtime. It is obviously 
doable at runtime, but complicated. I think that most of it can 
be done at compile time.
Dec 01 2012
prev sibling parent "Jason House" <jason.james.house gmail.com> writes:
On Thursday, 29 November 2012 at 02:34:11 UTC, Walter Bright 
wrote:

 For discussion:

 Cyclical Imports

 Problem:

 ---- a.d ----
     module a;
     import b;
     static this () { ... }
 ---- b.d ----
     module b;
     import a;
     static this() { ... }
 -------------

 Static constructors for a module are only run after static 
 constructors
 for all its imports are run. Circular imports, such as the 
 above, are
 detected at run time and the program is aborted.

 This can in general be solved by moving the static 
 constructor(s) into
 a third module, c.d, which does not import a or b. But, people 
 find this
 to be unnatural.

 Proposed Solution:

 Add a pragma,

     pragma(cyclic_imports);

 This can appear anywhere in a module, and applies globally to 
 that module.
 It means that static constructors from imports that are not 
 part of the cycle
 are run first, and that the static constructor for this module 
 may be run before
 the static constructors of other modules that are part of the 
 cycle.

 If any static constructors in such a module with the pragma have
 the  safe attribute, that is a compile time error.


While more complex, would it be possible to declare which 
imported modules can be constructed after the current one? That 
allows catching any other unintended cyclic imports. For example:

static this {
   pragma(cyclic_import, b);
   // code that does not depend on
   // static constructor(s) in module b
}
Dec 01 2012