www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - bug in foreach continue

reply Hussien <SH glory.com> writes:
foreach (y; aliasSeqOf!["a", "b", "c"])
{
static if (y == "a") { }
else
  pragma(msg, y);
}

works but

foreach (y; aliasSeqOf!["a", "b", "c"])
{
static if (y == "a") continue
pragma(msg, y);
}

fails.

Seems like continue needs to be static aware.

This leads to subtle bugs where one thinks the continue 
statement(like in in a runtime foreach loop, should jump to to 
the next iteration, but it doesn't.

The runtime versions of the code work identically, and so should 
the static versions.
Mar 16 2017
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 17 March 2017 at 01:34:52 UTC, Hussien wrote:

 Seems like continue needs to be static aware.


That's not a bug, pragma is triggered when the code is 
*compiled*, not when it is run. The code is compiled, even if it 
is skipped over by a continue.
Mar 16 2017
parent reply Hussien <SH glory.com> writes:
On Friday, 17 March 2017 at 01:41:47 UTC, Adam D. Ruppe wrote:

 On Friday, 17 March 2017 at 01:34:52 UTC, Hussien wrote:

 Seems like continue needs to be static aware.


 That's not a bug, pragma is triggered when the code is 
 *compiled*, not when it is run. The code is compiled, even if 
 it is skipped over by a continue.


My point is, that it shouldnt! a static if should have a static 
continue in some way. It is confusing to be thinking in terms of 
the compile time code and (static if) and use a ambiguous 
statement(continue) and it not be compile time.

Either one needs a static version of continue or a warning when 
used inside a static if.

As is, continue doesn't even apply to the static case, so it 
shouldn't be allowed. A static foreach with a dynamic continue is 
illogical yet no warning is emitted.
Mar 16 2017
parent reply Jonathan M Davis via Digitalmars-d-learn writes:
On Friday, March 17, 2017 01:55:19 Hussien via Digitalmars-d-learn wrote:

 On Friday, 17 March 2017 at 01:41:47 UTC, Adam D. Ruppe wrote:

 On Friday, 17 March 2017 at 01:34:52 UTC, Hussien wrote:

 Seems like continue needs to be static aware.


 That's not a bug, pragma is triggered when the code is
 *compiled*, not when it is run. The code is compiled, even if
 it is skipped over by a continue.


 My point is, that it shouldnt! a static if should have a static
 continue in some way. It is confusing to be thinking in terms of
 the compile time code and (static if) and use a ambiguous
 statement(continue) and it not be compile time.

 Either one needs a static version of continue or a warning when
 used inside a static if.

 As is, continue doesn't even apply to the static case, so it
 shouldn't be allowed. A static foreach with a dynamic continue is
 illogical yet no warning is emitted.


I tend to agree with this. If the foreach is static, and continue and break
are just going to be ignored, then they should just be illegal. Allowing
them is just going to confuse people. Now, making it so that they actually
work statically has some interesting possibilities, but that would fall
apart as soon as you have any constructs that would use continue or break
(e.g. a loop or switch statement) inside the static foreach, and it might
break code in rare cases. So, we're probably better off just making them
illegal. But having them be legal just seems disingenious, since they don't
do anything.

- Jonathan M Davis
Mar 17 2017
parent reply Michael <michael toohuman.io> writes:
On Friday, 17 March 2017 at 11:30:48 UTC, Jonathan M Davis wrote:

 On Friday, March 17, 2017 01:55:19 Hussien via 
 Digitalmars-d-learn wrote:

 I tend to agree with this. If the foreach is static, and 
 continue and break are just going to be ignored, then they 
 should just be illegal. Allowing them is just going to confuse 
 people. Now, making it so that they actually work statically 
 has some interesting possibilities, but that would fall apart 
 as soon as you have any constructs that would use continue or 
 break (e.g. a loop or switch statement) inside the static 
 foreach, and it might break code in rare cases. So, we're 
 probably better off just making them illegal. But having them 
 be legal just seems disingenious, since they don't do anything.

 - Jonathan M Davis


What exactly IS happening in the case of a continue in a 
static-if? I could sort of imagine that maybe if you were 
expecting the loop to be unrolled, that you then have a continue 
statement in the correct part of the unrolled loop. But I take it 
this isn't what's happening?
Mar 17 2017
parent reply Jonathan M Davis via Digitalmars-d-learn writes:
On Friday, March 17, 2017 11:53:41 Michael via Digitalmars-d-learn wrote:

 On Friday, 17 March 2017 at 11:30:48 UTC, Jonathan M Davis wrote:

 On Friday, March 17, 2017 01:55:19 Hussien via
 Digitalmars-d-learn wrote:

 I tend to agree with this. If the foreach is static, and
 continue and break are just going to be ignored, then they
 should just be illegal. Allowing them is just going to confuse
 people. Now, making it so that they actually work statically
 has some interesting possibilities, but that would fall apart
 as soon as you have any constructs that would use continue or
 break (e.g. a loop or switch statement) inside the static
 foreach, and it might break code in rare cases. So, we're
 probably better off just making them illegal. But having them
 be legal just seems disingenious, since they don't do anything.

 - Jonathan M Davis


 What exactly IS happening in the case of a continue in a
 static-if? I could sort of imagine that maybe if you were
 expecting the loop to be unrolled, that you then have a continue
 statement in the correct part of the unrolled loop. But I take it
 this isn't what's happening?


Okay. I completely misunderstood what was happening based on the previous
posts. For instance, from this test

import std.meta;
import std.stdio;

void main()
{
    foreach(str; ["hello", "world", "foo", "bar"])
    {
        foreach(i; AliasSeq!(0, 1, 2, 3, 4))
        {
            static if(i / 2 == 0)
                writefln("%s: %s", i, str);
            else
            {
                writefln("%s: skipping %s", i, str);
                break;
            }
        }
        writeln();
    }
}

it looks like break and continue _are_ used at compile time, since it
prints

0: hello
1: hello
2: skipping hello

0: world
1: world
2: skipping world

0: foo
1: foo
2: skipping foo

0: bar
1: bar
2: skipping bar

whereas if the break were just compiled in to be run at runtime, you never
would have seen any of the strings past "hello" in the outer loop get
printed. So, the issue is _not_ that continue and break are always treated
as runtime constructs or outright ignored. And if we have

import std.meta;
import std.stdio;

void main()
{
    foreach(str; AliasSeq!("a", "b", "c"))
    {
        static if(str == "a")
            continue;
        writeln(str);
    }
}

it prints

b
c

because when the loop is unrolled, wirteln isn't compiled in (and in fact,
if you compile with -w, the code won't compile at all, because writeln is
unreachable in the "a" iteration).

On the other hand, if we have

import std.meta;
import std.stdio;

void main()
{
    foreach(str; AliasSeq!("a", "b", "c"))
    {
        static if(str == "a")
            continue;
        pragma(msg, str);
    }
}

then we get

a
b
c

which is what the OP was complaining about. Now, as to whether that's a bug
or not, I don't know. pragma(msg, ...) prints whenever it's compiled, and as
shown by the compilation error when you compile in the -w case, the code
after the static if _is_ compiled in. It just ends up not being run.
Certainly, the continue and break are _not_ skipped like I originally
understood to be the case. So, that's not the bug. If there _is_ a bug, it's
that the rest of the code in the loop after the static if is compiled after
the continue, and I don't know that that is a bug. Remember that this is
basically doing loop unrolling, which is not necessarily the same thing as
you would expect from a "static foreach." It's not like we're building a
string mixin here. It's more like the same code is being copy-pasted over
and over but potentially with some tweaks on a given iteration.

So, I'm not at all convinced that this is a bug, but I do agree that the
semantics are a bit wonky. Regardless, based on what happens with the
writeln case, it's clearly the case that you should be using an else branch
if you want to cleanly use a continue in a foreach like this.

- Jonathan M Davis
Mar 17 2017
next sibling parent reply Hussien <SH glory.com> writes:
On Friday, 17 March 2017 at 13:10:23 UTC, Jonathan M Davis wrote:

 On Friday, March 17, 2017 11:53:41 Michael via 
 Digitalmars-d-learn wrote:

 On Friday, 17 March 2017 at 11:30:48 UTC, Jonathan M Davis 
 wrote:

 On Friday, March 17, 2017 01:55:19 Hussien via 
 Digitalmars-d-learn wrote:

 I tend to agree with this. If the foreach is static, and 
 continue and break are just going to be ignored, then they 
 should just be illegal. Allowing them is just going to 
 confuse people. Now, making it so that they actually work 
 statically has some interesting possibilities, but that 
 would fall apart as soon as you have any constructs that 
 would use continue or break (e.g. a loop or switch 
 statement) inside the static foreach, and it might break 
 code in rare cases. So, we're probably better off just 
 making them illegal. But having them be legal just seems 
 disingenious, since they don't do anything.

 - Jonathan M Davis


 What exactly IS happening in the case of a continue in a 
 static-if? I could sort of imagine that maybe if you were 
 expecting the loop to be unrolled, that you then have a 
 continue statement in the correct part of the unrolled loop. 
 But I take it this isn't what's happening?


 Okay. I completely misunderstood what was happening based on 
 the previous posts. For instance, from this test

 import std.meta;
 import std.stdio;

 void main()
 {
     foreach(str; ["hello", "world", "foo", "bar"])
     {
         foreach(i; AliasSeq!(0, 1, 2, 3, 4))
         {
             static if(i / 2 == 0)
                 writefln("%s: %s", i, str);
             else
             {
                 writefln("%s: skipping %s", i, str);
                 break;
             }
         }
         writeln();
     }
 }

 it looks like break and continue _are_ used at compile time, 
 since it prints

 0: hello
 1: hello
 2: skipping hello

 0: world
 1: world
 2: skipping world

 0: foo
 1: foo
 2: skipping foo

 0: bar
 1: bar
 2: skipping bar

 whereas if the break were just compiled in to be run at 
 runtime, you never would have seen any of the strings past 
 "hello" in the outer loop get printed. So, the issue is _not_ 
 that continue and break are always treated as runtime 
 constructs or outright ignored. And if we have


Yes, but you have a nested foreach loop. One runtime and one 
compile time. The break goes with the runtime loop... but NORMAL 
programming logic tells us that the break goes with the loop that 
it exists in.


     foreach(str; ["hello", "world", "foo", "bar"])
     {
         foreach(i; AliasSeq!(0, 1, 2, 3, 4))
         {
             static if(i / 2 == 0)
                 writefln("%s: %s", i, str);
             else
             {
                 writefln("%s: skipping %s", i, str);
                 break;
             }
         }
         writeln();
     }



reduces too


     foreach(str; ["hello", "world", "foo", "bar"])
     {


           writefln("%s: %s", 0, str);
           writefln("%s: %s", 1, str);
           writefln("%s: skipping %s", 2, str);
           break;

           writefln("%s: skipping %s", 3, str);
           break;

           writefln("%s: %s", 4, str);
           break;


         writeln();
     }


And I'm sure this is not what you intended(The first break is 
good enough). Not only does the unrolled loop not make sense, the 
standard logic any normal programmer without knowing the 
ambiguity of foreach loop in D would assume that the break in the 
inner most loop is for the inner most foreach.

This can create very complex and subtle bugs. It gets worse the 
more complex the mixing of the runtime and compile time 
programming is.

Static foreach loops should use static break and static 
continue... "Birds of a feather flock together"...

If one needs to use a runtime break in a compile time loop, then 
some special mechanism could be added to disambiguate the two.

Mixing compile time and runtime identifiers is a very bad idea! 
It creates very subtle ambiguities that will cause very serious 
bugs in the future when D becomes more widely used. At that 
point, it will probably be too late to be changed(as it will then 
break code that took in to account the ambiguity).

One can say that all break's and continues are runtime... which 
is how it currently is. That is fine, but a warning should be 
issued because it goes against meta programming logic. When I 
think of meta programming, I don't start thinking using a 
completely different set of rules... especially since most of the 
syntax and semantics is between runtime and compile time is 
identical.








 import std.meta;
 import std.stdio;

 void main()
 {
     foreach(str; AliasSeq!("a", "b", "c"))
     {
         static if(str == "a")
             continue;
         writeln(str);
     }
 }

 it prints

 b
 c

 because when the loop is unrolled, wirteln isn't compiled in 
 (and in fact, if you compile with -w, the code won't compile at 
 all, because writeln is unreachable in the "a" iteration).

 On the other hand, if we have

 import std.meta;
 import std.stdio;

 void main()
 {
     foreach(str; AliasSeq!("a", "b", "c"))
     {
         static if(str == "a")
             continue;
         pragma(msg, str);
     }
 }

 then we get

 a
 b
 c

 which is what the OP was complaining about. Now, as to whether 
 that's a bug or not, I don't know. pragma(msg, ...) prints 
 whenever it's compiled, and as shown by the compilation error 
 when you compile in the -w case, the code after the static if 
 _is_ compiled in. It just ends up not being run. Certainly, the 
 continue and break are _not_ skipped like I originally 
 understood to be the case. So, that's not the bug. If there 
 _is_ a bug, it's that the rest of the code in the loop after 
 the static if is compiled after the continue, and I don't know 
 that that is a bug. Remember that this is basically doing loop 
 unrolling, which is not necessarily the same thing as you would 
 expect from a "static foreach." It's not like we're building a 
 string mixin here. It's more like the same code is being 
 copy-pasted over and over but potentially with some tweaks on a 
 given iteration.

 So, I'm not at all convinced that this is a bug, but I do agree 
 that the semantics are a bit wonky. Regardless, based on what 
 happens with the writeln case, it's clearly the case that you 
 should be using an else branch if you want to cleanly use a 
 continue in a foreach like this.

 - Jonathan M Davis


A bug or not is irrelevant. There is an issue with the design of 
for loops between runtime and compile time versions. There is 
overlap in syntax and it creates ambiguity and this leads to REAL 
bugs... sometimes very hard to track down. When someone is 
scanning a complex piece of code and they run across something 
similar to the above. They are not going to think of the continue 
and break's as being dynamic and having complex fall through 
behavior.

They are going to expect them to continue or break out of the 
inner most loop that they are called from. That is the definition 
of them:

"A break statement results in the termination of the statement to 
which it applies ( switch , for , do , or while ). A continue 
statement is used to end the current loop iteration and return 
control to the loop statement."

"The break statement "jumps out" of a loop.
The continue statement "jumps over" one iteration in the loop."


It's bad enough foreach is ambiguous... we can't now if it's 
static or not unless we analyze the object it loops over.

A better syntax would be to introduce a static syntax for static 
constructs:

$foreach = static/compile time foreach
$continue = static/compile time continue
$break = static/compile time break
$if = static/compile time if
$while
$do
void $foo(int) = static/compile time function. (not necessarily 
needed because of CTFE but it is explicit.


Such a syntax(not saying that $ should be used, just had to use 
something for demonstration purposes) is not ambiguous. When 
someone sees $continue, they know EXACTLY what it is doing.

What has been done is this:

Black:

1.
of the very darkest color owing to the absence of or complete 
absorption of light; the opposite of white.
"black smoke"
synonyms:	dark, pitch-black, jet-black, coal-black, ebony, sable, 
inky
"a black horse"
antonyms:	white

     (of the sky or night) completely dark owing to nonvisibility 
of the sun, moon, or stars.
     "the sky was moonless and black"
     synonyms:	unlit, dark, starless, moonless, wan; More
     literarytenebrous, Stygian
     "a black night"
     antonyms:	clear, bright
     deeply stained with dirt.
     "his clothes were absolutely black"
     (of a plant or animal) dark in color as distinguished from a 
lighter variety.
     "Japanese black pine"
     synonyms:	dark, pitch-black, jet-black, coal-black, ebony, 
sable, inky
     "a black horse"
     antonyms:	white
     (of coffee or tea) served without milk or cream.
     of or denoting the suits spades and clubs in a deck of cards.
     (of a ski run) of the highest level of difficulty, as 
indicated by black markers positioned along it.

2.
of any human group having dark-colored skin, especially of 
African or Australian Aboriginal ancestry.


And what will eventually have to happen is that the term black 
will have to be banned as a racist word... simply because of the 
ambiguity. It is like all words, people are confused by them 
because of their different meanings. It's best to be explicit as 
necessary, and in this case, I think D is not explicit enough. 
The convenience factor created by ambiguity will cause more 
problems than it solves down the road for a select group of 
people.

A non-ambiguous syntax could be added on top of what currently 
exists and eventually the old syntax could be depreciated... If 
continue/break are the only problems then some other, simpler 
solution, could be found. I think a warning would be good enough.

e.g., "Warning: continue used in static foreach loop."

which brings attention to the issue. Since no one likes warnings, 
a pragma(-warn, C34324) could be used(or whatever D's mechanism 
is to specific appropriate warnings).
Mar 17 2017
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 17 March 2017 at 13:53:58 UTC, Hussien wrote:

 Yes, but you have a nested foreach loop. One runtime and one 
 compile time. The break goes with the runtime loop... but 
 NORMAL programming logic tells us that the break goes with the 
 loop that it exists in.


It did. It broke the loop counting the numbers, but continued the 
loop of strings normally.


People have requested a `static foreach` before, but that request 
has not yet been granted.


 They are going to expect them to continue or break out of the 
 inner most loop that they are called from.


That is exactly what they do, and exactly what happened in all 
the examples posted.
Mar 17 2017
parent reply Hussien <SH glory.com> writes:
On Friday, 17 March 2017 at 14:27:25 UTC, Adam D. Ruppe wrote:

 On Friday, 17 March 2017 at 13:53:58 UTC, Hussien wrote:

 Yes, but you have a nested foreach loop. One runtime and one 
 compile time. The break goes with the runtime loop... but 
 NORMAL programming logic tells us that the break goes with the 
 loop that it exists in.


 It did. It broke the loop counting the numbers, but continued 
 the loop of strings normally.


 People have requested a `static foreach` before, but that 
 request has not yet been granted.


 They are going to expect them to continue or break out of the 
 inner most loop that they are called from.


 That is exactly what they do, and exactly what happened in all 
 the examples posted.


You're failing to see what I am talking about. I understand the 
compiled code is correct and functions as it should.

What I am talking about is

how the statement

static if (x) continue;
pragma(msg, "called");

vs

static if (x) { } else
pragma(msg, "not called");


They are both semantically equivalent

e.g.,

if (x) continue; foo();

vs

if (x) {} else { foo(); }

Both do the same thing in loops when used in the context I 
originally gave. e.g., using a continue to skip over elements of 
an interation. Works great in normally programming!

BUT in static programming, the continue FUNCTIONS as a runtime 
component... just as you say. BUT that is not how one interprets 
it on the static level.

The logic for


if (x) continue; foo();

vs

if (x) {} else { foo(); }

and the logic for


static if (x) continue; foo();

vs

static if (x) {} else { foo(); }


should be identical. Just because we are programming on two 
different levels doesn't change the logic(or shouldn't).

The problem, of course, is the continue does exactly what you 
claim it does.

But when I am meta programming I am thinking in terms of meta 
programming. I program exactly the same way as I do in runtime 
programming except for the obvious shifts that are required. But 
foreach is a foreach. In runtime it works on runtime stuff and at 
compile time it works on compile time stuff.

Effectively I have

runtime_foreach and static_foreach in my head and I magically 
disambiguate when required.

But there is no such concept for continue. It is impossible to 
"continue" or even "break" a static for each loop. That is what I 
am addressing.

SO, because I had a mental glitch and I used continue "thinking" 
it would work for the static case, e.g., I assumed it was a 
static_continue... when no such thing exists, the code compiled 
and did something I did not expect!

YES, It was my fault! BUT the point is that D has that ambiguity. 
AND because the flow structures for runtime and compile time are 
virtually identical, the mistake is easy to make! Even if not, if 
the mistake does occur, it will be a difficult thing to track 
down in real world cases that are complex. That is the only point 
I am making! It has nothing to do with what D actually does, but 
how it does it.

A continue or break should create a warning when used in the 
context of a static foreach. OR change the freaken syntax of 
static semantics vs runtime. Making them identical is 
ridiculously and only leads to these kinds of problems. Sure, it 
works in 99% of the cases, but so will a new syntax.

My point is not what D does specifically in compiling source 
code, or that it does it right or wrong. I am talking about the 
interpretation by a human of the code. 

be created programming language... but it doesn't mean that 
humans will be able to interpret it properly. I am talking about 
how humans interpret things.

Do you really think that a non-D programmer that sees the code

foreach(x; [1,2,3])
foreach(y; aliasSeqOf![1,2,3])
static if (y == 1) continue;
pragma(msg, y)

is going to be able to interpret the code correctly?

Because if they use a rewrite rule, which is GENERALLY VALID in 
all programming languages:

foreach(x; [1,2,3])
foreach(y; aliasSeqOf![1,2,3])
static if (y == 1) {} else
{
pragma(msg, y)
}

(again, this should be the same as the first case, because we can 
do that at runtime and it is valid... people write code like that)

Yet one gets a different result.

Hence, if they are programming some NASA rocket, and make that 
mistake and the rocket goes left instead of dropping it's 
boosters, then what? Who's fault? Is it D's fault for creating 
ambiguity between static and runtime programming? Of course it 
is, because It sets the foundation for what is to come.

Again, my point is not about D compiling the above code, but 
about the syntax/grammar/language issue it creates conceptually. 
You may not have that problem because you are familiar enough 
with D's meta programming but to someone that isn't, they have to 
go on what they know, and there is a problem conceptually they 
will have to learn about. It shouldn't be that way. It is a 
problem of D treating apples as oranges. foreach != foreach... 
does that make sense? Sure, if you already learned, but if you 
are learning or have a brain fart, too bad for you...

This is D's approach: "Meta programming - The same as 
programming... well almost". The almost part is what gets people 
killed.
Mar 17 2017
next sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 17 March 2017 at 15:14:08 UTC, Hussien wrote:

 What I am talking about is


If you want to add a new feature, `static foreach`, that has 
static continue and static break, I can get behind that, but 
that's a new feature, not a bug in the existing feature. You 
think it is something it isn't.

`static if` is unique in D. Nothing else works like it. There is 
no static switch, no static foreach. Regular switch and foreach 
can work on compile time values, but they don't actually change 
their behavior like static if does, they work the same as always. 
The best you get is that the current value of the foreach can 
inherit the compile-time staticness of what it is looping, 
allowing you to use it in a few more places (notably in static 
if), but the loop construct itself is the same.

That's it. It does not do special flow control, it does not 
change scopes or compile requirements.

A new `static foreach` that DOES do those things, and works in 
places `static if` currently works like outside functions, would 
be a cool new feature. But until that's added, stop thinking of 
"static foreach" as being a special thing in the language now. It 
isn't there. It is just regular foreach and thinking of it as 
that, always, will make things a lot easier.


Then you can join the calls for a new `static foreach` that 
explicitly IS compile time! And we'll be on the same page there.
Mar 17 2017
prev sibling parent reply "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Fri, Mar 17, 2017 at 03:14:08PM +0000, Hussien via Digitalmars-d-learn wrote:
[...]

 What I am talking about is
 
 how the statement
 
 static if (x) continue;
 pragma(msg, "called");
 
 vs
 
 static if (x) { } else
 pragma(msg, "not called");
 
 
 They are both semantically equivalent


This appears to be yet another case of the term "compile-time" causing
confusion, because it's actually an ambiguous term.  I actually think
it's a bad term that should be replaced by something more precise; its
ambiguity has caused confusion on the part of D learners more often than
not.

There are actually (at least) TWO distinct phases of compilation that
are conventionally labelled "compile time":

1) Template expansion / AST manipulation, and:

2) CTFE (compile-time function evaluation).

Not clearly understanding the distinction between the two often leads to
confusion and frustration at why the compiler isn't doing "what I want".

This confusion is confounded by the fact that these two do mutually
interact in any non-trivial metaprogramming D code, often in rather
complex ways. But the fundamental thing to understand here is:

	Template expansion / AST manipulation must be completed *before*
	CTFE can run.

Basically, CTFE is essentially a D interpreter embedded inside the
compiler, that simulates computation at runtime.  As such, it requires
the AST to be fully compiled, as in, the compiler is ready to generate
object code for it, before CTFE can work.  This is because it makes no
sense to generate code on an incomplete / partial AST.

Furthermore, once a piece of code has made it to the CTFE stage, its AST
has already been processed, and it's now compiled into an internal
representation (analogous to bytecode), so AST-manipulating constructs
no longer make any sense.  In the CTFE stage, there is no such thing as
an AST anymore.

Constructs like `static if` and `pragma(msg)` belong to the AST
manipulation stage. `static if` changes the effective AST that the
compiler sees when it's generating code, and pragma(msg) is essentially
a debugging construct that says "print this message if this part of the
AST makes it past the template expansion phase". This is why code like
this one doesn't do what you might think it does:

	int ctfeFunc(bool b) {
		if (b) {
			pragma(msg, "hello");
			return 1;
		} else {
			pragma(msg, "goodbye");
			return 2;
		}
	}

The compiler will print *both* "hello" and "goodbye", because the
pragma(msg) is evaluated *when the AST of this function is processed*.
The variable 'b' doesn't even exist at that point, because ASTs don't
have the concept of a 'variable' -- 'b' is just an identifier node in
the tree. And 'if' is just another node in the tree that represents a
control-flow directive.  Change that to 'static if', however, and the
compiler sees it differently:

	int ctfeFunc(bool b) {
		static if (b) {
			pragma(msg, "hello");
			return 1;
		} else {
			pragma(msg, "goodbye");
			return 2;
		}
	}

This code won't work, because 'b' is a variable, but variables don't
exist at the AST manipulation stage. So you have to turn 'b' into a
so-called "compile-time argument" (boy I hate that term "compile-time",
it totally is not precise enough to convey the meaning here):

	int ctfeFunc(bool b)() {
		static if (b) {
			pragma(msg, "hello");
			return 1;
		} else {
			pragma(msg, "goodbye");
			return 2;
		}
	}

Do not be deceived by the appearances; the story here is far more
involved than "since static if is a `compile-time` construct, obviously
it needs the condition to be made from `compile-time` arguments". You
have to understand that when you move 'b' into the so-called
compile-time parameter list, you're essentially saying "here's a
template that, given a boolean value, produces an AST tree according to
the following pattern".  If you instantiate ctfeFunc!true, then what
effectively happens is that the compiler sees this AST:

	int ctfeFunc!true() {
		return 1;
	}

So it prints "hello". Note that the else branch is NOT EVEN SEEN past
the AST stage. It practically doesn't exist at that point. The
pragma(msg) is also not seen past that point: it is consumed in the AST
manipulation stage. The compiler sees the static if, evaluates the
condition, and basically prunes the else branch off, then sees the
pragma(msg) and emits the message (i.e., acknowledging "yes this part of
the AST makes it past the AST manipulation stage"). The pragma(msg) is
pruned from the AST after that.

By the time CTFE runs on this function, all the CTFE interpreter sees is
"return 1;".  It doesn't see the "static if" nor the "pragma(msg)",
because those things are AST manipulation constructs; past the AST stage
such things don't exist anymore.


What complicates this 2-stage picture, though, and probably what doesn't
help the confusion with the term "compile-time", is that the compiler is
smart enough to perform CTFE on-demand. Meaning that it's valid to do
this:

	int ctfeFunc(bool b) { /* N.B.: "runtime" parameter! */
		if (b) return 1;
		else return 2;
	}

	enum b = true;
	static if (ctfeFunc(b) == 1)
		struct S { int x; }
	else
		struct S { int y; }

Note that the static if here has a condition that depends on the output
of a CTFE function. This appears be a reversal of the AST manipulation /
CTFE stages, but strictly speaking that's actually not the case.  What
actually happens is:

1) The compiler sees the declaration of ctfeFunc(), produces an AST for it.
   Note that this means the AST of ctfeFunc gets finalised here -- any
   static if's, templates, etc., are expanded into the "final" AST for
   this function.

2) Then the compiler sees the static if, and enters the AST phase *for
   this part of the code*. It says, I need to know the value of
   ctfeFunc(b) in order to know which struct declaration to use -- so it
   *emits code* for ctfeFunc(), and then runs the CTFE engine on the
   resulting code.  Then it takes the output of that evaluation to make
   the decision.

The key point here is that ctfeFunc has *already* passed the AST
manipulation stage while the struct declaration is still in the AST
stage.  The compiler is clever enough to sequence the processing of
these two parts of the code so that it's able to compute the static if
condition.  But the principle of AST manipulation coming before CTFE
still applies -- it's not possible for the CTFE engine to interpret
ctfeFunc if its AST is still being manipulated (because if the AST is
not finalized yet, there isn't any code to interpret!), and it's not
possible for the AST manipulation stage to read the value of a CTFE
computation that's being run on the same piece of code (because CTFE
can't run in the first place while the AST is still being manipulated,
and if the code is already ready for CTFE to interpret, that means the
AST has already been finalized, you can't change it anymore).

Coming back to your loop:

	static if (x) continue;
	foo();

In the AST manipulation stage, the compiler evaluates x, and if x is
true, then the effective AST is:

	continue;
	foo();

By the time it gets to CTFE, it has already "forgotten" that there was
such a thing as a static if in the source code. So it will interpret the
"continue" unconditionally.

Whereas if you wrote:

	static if (x) {} else { foo(); }

in the AST manipulation stage, if x is true, then the effective AST is:

	{}

and if x is false, then the effective AST is:

	foo();

which is what you intended.


T

-- 
A mathematician is a device for turning coffee into theorems. -- P. Erdos
Mar 17 2017
next sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 03/17/2017 12:05 PM, H. S. Teoh via Digitalmars-d-learn wrote:


 1) Template expansion / AST manipulation, and:

 2) CTFE (compile-time function evaluation).

 Not clearly understanding the distinction between the two often leads to
 confusion and frustration at why the compiler isn't doing "what I want".


I always find your posts very informative and very well articulated. 
Where is the book already? :) Or blog posts?

Ali
Mar 17 2017
next sibling parent reply "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Fri, Mar 17, 2017 at 02:52:39PM -0700, Ali Çehreli via Digitalmars-d-learn
wrote:

 On 03/17/2017 12:05 PM, H. S. Teoh via Digitalmars-d-learn wrote:
 

 1) Template expansion / AST manipulation, and:
 
 2) CTFE (compile-time function evaluation).
 
 Not clearly understanding the distinction between the two often
 leads to confusion and frustration at why the compiler isn't doing
 "what I want".


 
 I always find your posts very informative and very well articulated.
 Where is the book already? :) Or blog posts?


[...]

Haha, I don't think I'm up for writing a book... and I don't really keep
a blog either.  But perhaps a writeup on wiki.dlang.org is in order.

This particular topic, I think, is something somebody *should* write
about, because it seems to trip up newbies quite often.  Maybe sometimes
even seasoned D users get tripped up, too (though they tend to know
better where the problem is and how to work around it).  And this isn't
the first time I wrote about it.  I guess it's time to consolidate what
I've written into a wiki article.


T

-- 
Life begins when you can spend your spare time programming instead of watching
television. -- Cal Keegan
Mar 18 2017
parent Mike Parker <aldacron gmail.com> writes:
On Sunday, 19 March 2017 at 00:18:15 UTC, H. S. Teoh wrote:


 Haha, I don't think I'm up for writing a book... and I don't 
 really keep a blog either.  But perhaps a writeup on 
 wiki.dlang.org is in order.

 This particular topic, I think, is something somebody *should* 
 write about, because it seems to trip up newbies quite often.  
 Maybe sometimes even seasoned D users get tripped up, too 
 (though they tend to know better where the problem is and how 
 to work around it).  And this isn't the first time I wrote 
 about it.  I guess it's time to consolidate what I've written 
 into a wiki article.


Ahem. https://dlang.org/blog/

I'm always ready to talk about guest posts at aldacron gmail.com 
:-)
Mar 18 2017
prev sibling parent Jonathan M Davis via Digitalmars-d-learn writes:
On Saturday, March 18, 2017 17:18:15 H. S. Teoh via Digitalmars-d-learn 
wrote:

 On Fri, Mar 17, 2017 at 02:52:39PM -0700, Ali Çehreli via Digitalmars-d-


learn wrote:

 On 03/17/2017 12:05 PM, H. S. Teoh via Digitalmars-d-learn wrote:

 1) Template expansion / AST manipulation, and:

 2) CTFE (compile-time function evaluation).

 Not clearly understanding the distinction between the two often
 leads to confusion and frustration at why the compiler isn't doing
 "what I want".


 I always find your posts very informative and very well articulated.
 Where is the book already? :) Or blog posts?


 [...]

 Haha, I don't think I'm up for writing a book... and I don't really keep
 a blog either.  But perhaps a writeup on wiki.dlang.org is in order.

 This particular topic, I think, is something somebody *should* write
 about, because it seems to trip up newbies quite often.  Maybe sometimes
 even seasoned D users get tripped up, too (though they tend to know
 better where the problem is and how to work around it).  And this isn't
 the first time I wrote about it.  I guess it's time to consolidate what
 I've written into a wiki article.


For the most part, I understand compile-time vs runtime quite well - in
particular, I don't find templates vs CTFE to be all that complicated once
you get into it. I find it to usually be straightforward and obvious, but
there are corner cases where it gets harder to understand, and in
particular, foreach with compile-time constructs gets weird enough that - to
me at least - it stops being straightforward. So, to a great extent, with
foreach and compile-time constructs, my understanding is primarily limited
to what I need to work in order to do the stuff that I do. And I'm obviously
not a D newbie. For newbies, the whole thing is going to be far harder to
understand, and for whatever reason, the fact that a function run with CTFE
can't use the template parameters beyond what can be done at runtime throws
off a lot of people.

- Jonathan M Davis
Mar 18 2017
prev sibling next sibling parent Hussien <SH glory.com> writes:
On Friday, 17 March 2017 at 19:05:20 UTC, H. S. Teoh wrote:

 On Fri, Mar 17, 2017 at 03:14:08PM +0000, Hussien via 
 Digitalmars-d-learn wrote: [...]

 [...]


 This appears to be yet another case of the term "compile-time" 
 causing confusion, because it's actually an ambiguous term.  I 
 actually think it's a bad term that should be replaced by 
 something more precise; its ambiguity has caused confusion on 
 the part of D learners more often than not.

 [...]


Thanks for agreeing with me ;)
Mar 17 2017
prev sibling next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 17 March 2017 at 19:05:20 UTC, H. S. Teoh wrote:

 There are actually (at least) TWO distinct phases of 
 compilation that are conventionally labelled "compile time":

 1) Template expansion / AST manipulation, and:

 2) CTFE (compile-time function evaluation).

 [ ... ]
 	Template expansion / AST manipulation must be completed 
 *before*
 	CTFE can run.


Only the templates that the ctfe relies on.

 [ ... ]
 This is because it makes no sense to generate code on an 
 incomplete / partial AST.


This is not exactly true whenever you use ctfe to generate a 
source string that you later mix-in. You are working with a 
partial ast.

 Furthermore, once a piece of code has made it to the CTFE 
 stage, its AST has already been processed, and it's now 
 compiled into an internal representation (analogous to 
 bytecode), so AST-manipulating constructs no longer make any 
 sense.


Yes.


 In the CTFE stage, there is no such thing as an AST anymore.


There is an AST. It's just already processed.
Mar 21 2017
prev sibling parent reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
On Friday, 17 March 2017 at 19:05:20 UTC, H. S. Teoh wrote:

 There are actually (at least) TWO distinct phases of 
 compilation that are conventionally labelled "compile time":

 1) Template expansion / AST manipulation, and:

 2) CTFE (compile-time function evaluation).


This was an awesome explanation, please do put it on the wiki or 
as a pull request for the website.
Mar 22 2017
parent reply "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Wed, Mar 22, 2017 at 11:12:06PM +0000, Jesse Phillips via
Digitalmars-d-learn wrote:

 On Friday, 17 March 2017 at 19:05:20 UTC, H. S. Teoh wrote:

 There are actually (at least) TWO distinct phases of compilation
 that are conventionally labelled "compile time":
 
 1) Template expansion / AST manipulation, and:
 
 2) CTFE (compile-time function evaluation).


 
 This was an awesome explanation, please do put it on the wiki or as a
 pull request for the website.


I'm still working on that. :-)  But the article is turning out to be
quite a bit longer than I anticipated -- there is just so much I want to
say!  The draft is on the wiki but I don't really want to post the link
just yet because it still needs lots of editing before it's ready for
public consumption.


T

-- 
Dogs have owners ... cats have staff. -- Krista Casada
Mar 22 2017
next sibling parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 03/22/2017 05:39 PM, H. S. Teoh via Digitalmars-d-learn wrote:


 the article is turning out to be quite a bit longer


I said "book"; didn't I? :)

Ali
Mar 22 2017
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 23 March 2017 at 00:39:56 UTC, H. S. Teoh wrote:

 I'm still working on that. :-)


Hey, can you at least put scare quotes around "static foreach" 
each time it is used? There's no such thing as static foreach and 
while we might call foreach over a typetuple "static", it isn't a 
precise term and invites additional confusion.
Mar 23 2017
parent "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Thu, Mar 23, 2017 at 05:11:49PM +0000, Adam D. Ruppe via Digitalmars-d-learn
wrote:

 On Thursday, 23 March 2017 at 00:39:56 UTC, H. S. Teoh wrote:

 I'm still working on that. :-)


 
 Hey, can you at least put scare quotes around "static foreach" each
 time it is used? There's no such thing as static foreach and while we
 might call foreach over a typetuple "static", it isn't a precise term
 and invites additional confusion.


Done.

I've gotten the draft into a semi-acceptable state, though I'm still not
fully satisfied with it just yet.  But perhaps now is a good time to
solicit input from forum members, so here's the link to the draft:

	https://wiki.dlang.org/User:Quickfur/Compile-time_vs._compile-time


T

-- 
It always amuses me that Windows has a Safe Mode during bootup. Does that mean
that Windows is normally unsafe?
Mar 24 2017
prev sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 17 March 2017 at 13:10:23 UTC, Jonathan M Davis wrote:

 it looks like break and continue _are_ used at compile time, 
 since it prints


They are working exactly the same way as any other loop. The fact 
that it is unrolled and the dead code removed from the binary is 
an implementation detail that doesn't change the semantics of the 
loop, just like any other loop unroll optimization.

`continue` is more like `goto` than it is like `static if`, even 
when looping over AliasSeq.


 whereas if the break were just compiled in to be run at 
 runtime, you never would have seen any of the strings past 
 "hello" in the outer loop get printed.


No, they did a normal break at runtime. But they are breaking the 
INNER loop, so the outer loop continues. The generated code 
simply removed the dead portions since the optimizer realized 
there was no point carrying it around.

You are totally overthinking this, loops still work the same way 
regardless of what they are iterating over.

The only special thing about an AliasSeq thing is that the 
current value is available at compile time, so it is legal to use 
in more places (then the compiler implementation generates the 
code differently to make it happen, but that's the same idea as 
passing -funroll-loops or -O3 and getting different code, they 
don't change how break works.)


 Remember that this is basically doing loop unrolling, which is 
 not necessarily the same thing as you would expect from a 
 "static foreach." It's not like we're building a string mixin 
 here.


Yes.
Mar 17 2017
prev sibling parent Daniel Kozak via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
Dne 17.3.2017 v 02:34 Hussien via Digitalmars-d-learn napsal(a):


 foreach (y; aliasSeqOf!["a", "b", "c"])
 {
 static if (y == "a") { }
 else
  pragma(msg, y);
 }

 works but

 foreach (y; aliasSeqOf!["a", "b", "c"])
 {
 static if (y == "a") continue
 pragma(msg, y);
 }

 fails.

 Seems like continue needs to be static aware.

 This leads to subtle bugs where one thinks the continue statement(like 
 in in a runtime foreach loop, should jump to to the next iteration, 
 but it doesn't.

 The runtime versions of the code work identically, and so should the 
 static versions.


No it is not a bug, it works as I would expected:
import std.meta;
import std.stdio;

void main()
{
     foreach (y; aliasSeqOf!(["a", "b", "c"]))
     {
         static if (y == "a") continue;
         else writeln(y);
         pragma(msg, y);
     }
}
Mar 16 2017