www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - About foreach loops

reply bearophile <bearophileHUGS lycos.com> writes:
This post is about some characteristics of the D foreach that has nagged me for
some time.

This is a little Python 2.6 program:


for i in xrange(10):
    i += 1
    print i,


Its output shows that you are allowed to modify the iteration variable
(contents of the iteration name), but the iteration goes on with no change:
1 2 3 4 5 6 7 8 9 10


Similar code in D using foreach shows a different story:

import std.stdio;
void main() {
    foreach (i; 0 .. 10) {
        i += 1;
        write(i, " ");
    }
}

The output:
1 3 5 7 9 


In my opinion this is a bit bug-prone because in real code there is some risk
of modifying the iteration variable "i" by mistake. (Note: here I am not
talking about D for() loops. They are OK, their semantics is transparent
enough. foreach() loops are less transparent and they *look* higher level than
for() loops). I'd like the iteration variable to act as being a copy of the
true loop variable as in Python. If this is a bit bad for foreach performance,
then I'd like the compiler to forbid the mutation of the foreach iteration
variable inside the foreach body. Is this even possible?


Currently you can't solve the problem adding a const(int) to the iteration
variable:

import std.stdio;
void main() {
    foreach (const(int) i; 0 .. 10) { // line 3
        write(i, " ");
    }
}


DMD gives:
test.d(3): Error: variable test.main.i cannot modify const


This is a related but different thing:
http://d.puremagic.com/issues/show_bug.cgi?id=5255

Bye,
bearophile
Jun 14 2011
next sibling parent reply Caligo <iteronvexor gmail.com> writes:
I think D is fine and you may be confusing index with element.

The equivalence of your Python example in D is this:

  foreach(e; [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]){
    e += 1;
    write(e, " ");
  }

or this:

  foreach(e; take(recurrence!("a[n]+1")(0), 10)){
    e += 1;
    write(e, " ");
  }

and they both output:
1 2 3 4 5 6 7 8 9 10



This:

  foreach(i; 0..10){
    i += 1;
    write(i, " ");
  }

is the same as this:

  for(int i = 0; i < 10; ++i){
    i += 1;
    write(i, " ");
  }

and they both output:
1 3 5 7 9


This might make it clear:

  foreach(i, e; [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]){
    i += 1;
    writeln(i, " ", e);
  }

which outputs:
1 0
3 2
5 4
7 6
9 8
Jun 14 2011
next sibling parent Jesse Phillips <jessekphillips+D gmail.com> writes:
Nice! I've always expected bearophile's behavior and end up switching to a for
loop when wanting to modify the index.

Caligo Wrote:


 I think D is fine and you may be confusing index with element.
 
 The equivalence of your Python example in D is this:
 
   foreach(e; [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]){
     e += 1;
     write(e, " ");
   }
 
 or this:
 
   foreach(e; take(recurrence!("a[n]+1")(0), 10)){
     e += 1;
     write(e, " ");
   }
 
 and they both output:
 1 2 3 4 5 6 7 8 9 10
 
 
 
 This:
 
   foreach(i; 0..10){
     i += 1;
     write(i, " ");
   }
 
 is the same as this:
 
   for(int i = 0; i < 10; ++i){
     i += 1;
     write(i, " ");
   }
 
 and they both output:
 1 3 5 7 9
 
 
 This might make it clear:
 
   foreach(i, e; [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]){
     i += 1;
     writeln(i, " ", e);
   }
 
 which outputs:
 1 0
 3 2
 5 4
 7 6
 9 8
Jun 14 2011
prev sibling next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Caligo:


 I think D is fine and you may be confusing index with element.


Unfortunately I think what's confused here is the design of foreach.



 The equivalence of your Python example in D is this:
 
   foreach(e; [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]){


Nope, xrange is lazy.



   foreach(e; take(recurrence!("a[n]+1")(0), 10)){


There's std.range.iota for this :-)



 This:
 
   foreach(i; 0..10){
     i += 1;
     write(i, " ");
   }
 
 is the same as this:
 
   for(int i = 0; i < 10; ++i){
     i += 1;
     write(i, " ");
   }


The problem is this equivalence is hidden. foreach() loops look higher level
than for loops. So programmers expect this higher level look to be associated
with a higher level semantics too. This is why I think currently they are a bit
bug-prone. Modifying the loop variable of a foreach loop is code smell, if I
see it in code I fix it in some way, using a copy of the loop variable, or I
replace the foreach loop with a for loop. So I'd like the compiler to "ignore"
or probably better refuse such modifications to the foreach loop variable, if
possible.

Bye,
bearophile
Jun 14 2011
prev sibling parent Ali =?iso-8859-1?q?=C7ehreli?= <acehreli yahoo.com> writes:
On Tue, 14 Jun 2011 23:45:11 -0500, Caligo wrote:


 This might make it clear:
 
   foreach(i, e; [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]){
     i += 1;
     writeln(i, " ", e);
   }
 
 which outputs:
 1 0
 3 2
 5 4
 7 6
 9 8


I think that example strengthens bearophile's point of view. I would 
expect 'e' to provide access to all of the elements regardless of whether 
we asked for the extra index information or not.

The fact that modifying the extra information changes the elements that 
get visited is confusing.

Despite 'e' being a copy of the element, 'i's being a reference to the 
actual implementation variable is extra confusing. :)

Ali
Jun 14 2011
prev sibling next sibling parent reply KennyTM~ <kennytm gmail.com> writes:
On Jun 15, 11 10:20, bearophile wrote:

 This post is about some characteristics of the D foreach that has nagged me
for some time.

 This is a little Python 2.6 program:


 for i in xrange(10):
      i += 1
      print i,


 Its output shows that you are allowed to modify the iteration variable
(contents of the iteration name), but the iteration goes on with no change:
 1 2 3 4 5 6 7 8 9 10


 Similar code in D using foreach shows a different story:

 import std.stdio;
 void main() {
      foreach (i; 0 .. 10) {
          i += 1;
          write(i, " ");
      }
 }

 The output:
 1 3 5 7 9


 In my opinion this is a bit bug-prone because in real code there is some risk
of modifying the iteration variable "i" by mistake. (Note: here I am not
talking about D for() loops. They are OK, their semantics is transparent
enough. foreach() loops are less transparent and they *look* higher level than
for() loops). I'd like the iteration variable to act as being a copy of the
true loop variable as in Python. If this is a bit bad for foreach performance,
then I'd like the compiler to forbid the mutation of the foreach iteration
variable inside the foreach body. Is this even possible?


 Currently you can't solve the problem adding a const(int) to the iteration
variable:

 import std.stdio;
 void main() {
      foreach (const(int) i; 0 .. 10) { // line 3
          write(i, " ");
      }
 }


 DMD gives:
 test.d(3): Error: variable test.main.i cannot modify const


 This is a related but different thing:
 http://d.puremagic.com/issues/show_bug.cgi?id=5255

 Bye,
 bearophile


Perhaps the `foreach` loop

     foreach (i; a .. b)
        loop_body;

should be rewritten into

     for (auto __foreachtemp1234 = a;
          __foreachtemp1234 < b;
          ++ __foreachtemp1234)
     {
        const i = __foreachtmp1234;
        loop_body;
     }

this would prevent the loop index be modified inside the loop body.

Similarly, the indiced array loop

     foreach (i, elem; arr)
        loop_body;

should be rewritten into

     for (size_t __foreachtemp5678 = 0, __foreachtemp5679 = a.length;
          __foreachtemp5678 < __foreachtemp5679;
          ++ __foreachtemp5678)
     {
        const i = __foreachtemp5678;
        auto elem = arr[__foreachtemp5678];
        loop_body;
     }

bearophile, have you filed an enhancement request for this?
Jun 15 2011
parent bearophile <bearophileHUGS lycos.com> writes:
KennyTM~:


 bearophile, have you filed an enhancement request for this?


I haven't, I'm looking for opinions first, because foreach is a basic D
construct.

Bye,
bearophile
Jun 15 2011
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 14 Jun 2011 22:20:45 -0400, bearophile <bearophileHUGS lycos.com>  
wrote:


 This post is about some characteristics of the D foreach that has nagged  
 me for some time.

 This is a little Python 2.6 program:


 for i in xrange(10):
     i += 1
     print i,


 Its output shows that you are allowed to modify the iteration variable  
 (contents of the iteration name), but the iteration goes on with no  
 change:
 1 2 3 4 5 6 7 8 9 10


 Similar code in D using foreach shows a different story:

 import std.stdio;
 void main() {
     foreach (i; 0 .. 10) {
         i += 1;
         write(i, " ");
     }
 }

 The output:
 1 3 5 7 9


I think this is a bug.  Either i should be const during loop iteration, or  
an additional temporary should be manufactured.

It is always possible to get the above behavior using a for loop.  foreach  
should be sane.


 In my opinion this is a bit bug-prone because in real code there is some  
 risk of modifying the iteration variable "i" by mistake. (Note: here I  
 am not talking about D for() loops. They are OK, their semantics is  
 transparent enough. foreach() loops are less transparent and they *look*  
 higher level than for() loops). I'd like the iteration variable to act  
 as being a copy of the true loop variable as in Python. If this is a bit  
 bad for foreach performance, then I'd like the compiler to forbid the  
 mutation of the foreach iteration variable inside the foreach body. Is  
 this even possible?


 Currently you can't solve the problem adding a const(int) to the  
 iteration variable:

 import std.stdio;
 void main() {
     foreach (const(int) i; 0 .. 10) { // line 3
         write(i, " ");
     }
 }


 DMD gives:
 test.d(3): Error: variable test.main.i cannot modify const


generating an additional temporary variable should solve this.  Looks like  
the way to go.

-Steve
Jun 15 2011
next sibling parent reply Caligo <iteronvexor gmail.com> writes:
You can create a temporary if you like:

  foreach(i; 0..10){
    int ii = i + 1;
    writeln(ii, " ");
  }

Which outputs:
1 2 3 4 5 6 7 8 9 10


The problem with trying to "fix" foreach is that it would create
problems of its own.  The following code would not behave correctly:

  foreach(i; 0..10){
    if(i & 1)
      i += 1;
    writeln(i, " is even.");
  }

not to mention all the code that's already using foreach.
Jun 15 2011
next sibling parent reply KennyTM~ <kennytm gmail.com> writes:
On Jun 15, 11 22:35, Caligo wrote:

 You can create a temporary if you like:

    foreach(i; 0..10){
      int ii = i + 1;
      writeln(ii, " ");
    }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

    foreach(i; 0..10){
      if(i&  1)
        i += 1;
      writeln(i, " is even.");
    }

 not to mention all the code that's already using foreach.


If the code rely on the 'implicit ref' behavior of 'i', the code is 
doing it wrong. The spec never mentioned what happened when the variable 
is modified in a ForeachRangeStatement. I believe it is more sane if 
modifying 'i' is disallowed unless 'ref' is used.
Jun 15 2011
parent reply Caligo <iteronvexor gmail.com> writes:
On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~ <kennytm gmail.com> wrote:

 On Jun 15, 11 22:35, Caligo wrote:

 You can create a temporary if you like:

 =A0 foreach(i; 0..10){
 =A0 =A0 int ii =3D i + 1;
 =A0 =A0 writeln(ii, " ");
 =A0 }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own. =A0The following code would not behave correctly:

 =A0 foreach(i; 0..10){
 =A0 =A0 if(i& =A01)
 =A0 =A0 =A0 i +=3D 1;
 =A0 =A0 writeln(i, " is even.");
 =A0 }

 not to mention all the code that's already using foreach.


 If the code rely on the 'implicit ref' behavior of 'i', the code is doing=


 it

 wrong. The spec never mentioned what happened when the variable is modifi=


ed

 in a ForeachRangeStatement. I believe it is more sane if modifying 'i' is
 disallowed unless 'ref' is used.



This should be a compile time error:

  foreach(i; 0..10){
    i +=3D 1;
    write(i, " ");
  }

This should be legal.
  foreach(ref i; 0..10){
    i +=3D 1;
    write(i, " ");
  }

Is that the general consensus here?
Jun 15 2011
next sibling parent reply KennyTM~ <kennytm gmail.com> writes:
On Jun 15, 11 23:23, Caligo wrote:

 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com>  wrote:

 On Jun 15, 11 22:35, Caligo wrote:

 You can create a temporary if you like:

    foreach(i; 0..10){
      int ii = i + 1;
      writeln(ii, " ");
    }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

    foreach(i; 0..10){
      if(i&    1)
        i += 1;
      writeln(i, " is even.");
    }

 not to mention all the code that's already using foreach.


 If the code rely on the 'implicit ref' behavior of 'i', the code is doing it
 wrong. The spec never mentioned what happened when the variable is modified
 in a ForeachRangeStatement. I believe it is more sane if modifying 'i' is
 disallowed unless 'ref' is used.



 This should be a compile time error:

    foreach(i; 0..10){
      i += 1;
      write(i, " ");
    }

 This should be legal.
    foreach(ref i; 0..10){
      i += 1;
      write(i, " ");
    }

 Is that the general consensus here?


I'm +1 on this.
Jun 15 2011
parent Daniel Gibson <metalcaedes gmail.com> writes:
Am 15.06.2011 17:29, schrieb KennyTM~:

 On Jun 15, 11 23:23, Caligo wrote:

 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com>  wrote:

 On Jun 15, 11 22:35, Caligo wrote:

 You can create a temporary if you like:

    foreach(i; 0..10){
      int ii = i + 1;
      writeln(ii, " ");
    }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

    foreach(i; 0..10){
      if(i&    1)
        i += 1;
      writeln(i, " is even.");
    }

 not to mention all the code that's already using foreach.


 If the code rely on the 'implicit ref' behavior of 'i', the code is
 doing it
 wrong. The spec never mentioned what happened when the variable is
 modified
 in a ForeachRangeStatement. I believe it is more sane if modifying
 'i' is
 disallowed unless 'ref' is used.



 This should be a compile time error:

    foreach(i; 0..10){
      i += 1;
      write(i, " ");
    }

 This should be legal.
    foreach(ref i; 0..10){
      i += 1;
      write(i, " ");
    }

 Is that the general consensus here?


 
 I'm +1 on this.


Actually I thought it'd already be like this (or i being a copy).

So +1 from me as well.
Jun 15 2011
prev sibling next sibling parent reply so <so so.so> writes:
On Wed, 15 Jun 2011 18:23:55 +0300, Caligo <iteronvexor gmail.com> wrote:


 This should be a compile time error:

   foreach(i; 0..10){
     i += 1;
     write(i, " ");
   }

 This should be legal.
   foreach(ref i; 0..10){
     i += 1;
     write(i, " ");
   }

 Is that the general consensus here?


They both should be legal, the first one is value, the second one is  
reference.
Jun 15 2011
parent reply Caligo <iteronvexor gmail.com> writes:
On Wed, Jun 15, 2011 at 10:34 AM, so <so so.so> wrote:

 On Wed, 15 Jun 2011 18:23:55 +0300, Caligo <iteronvexor gmail.com> wrote:


 This should be a compile time error:

 =A0foreach(i; 0..10){
 =A0 =A0i +=3D 1;
 =A0 =A0write(i, " ");
 =A0}

 This should be legal.
 =A0foreach(ref i; 0..10){
 =A0 =A0i +=3D 1;
 =A0 =A0write(i, " ");
 =A0}

 Is that the general consensus here?


 They both should be legal, the first one is value, the second one is
 reference.


A reference to what exactly?

Does this make any sense:
  for(ref int i; i < 10; ++i){
    i +=3D 1;
    write(i, " ");
  }

It doesn't even compile.

'foreach(i, 0..10){ }' is syntactic sugar for 'for(int i =3D 0; i < 10;
++i){ }', is it not?
Jun 15 2011
parent so <so so.so> writes:
On Wed, 15 Jun 2011 19:08:29 +0300, Caligo <iteronvexor gmail.com> wrote:


 A reference to what exactly?

 Does this make any sense:
   for(ref int i; i < 10; ++i){
     i += 1;
     write(i, " ");
   }

 It doesn't even compile.


It is same as the other argument, the element, it should work the same way.


 'foreach(i, 0..10){ }' is syntactic sugar for 'for(int i = 0; i < 10;
 ++i){ }', is it not?


We don't know, and we don't need to know. It might be a wrapper over  
"while" too, since its a different entity it might have its own rules. And  
having element and the element index share same rules would be both  
consistent and the right thing to do.
Jun 15 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 15 Jun 2011 11:23:55 -0400, Caligo <iteronvexor gmail.com> wrote:


 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~ <kennytm gmail.com> wrote:

 On Jun 15, 11 22:35, Caligo wrote:

 You can create a temporary if you like:

   foreach(i; 0..10){
     int ii = i + 1;
     writeln(ii, " ");
   }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

   foreach(i; 0..10){
     if(i&  1)
       i += 1;
     writeln(i, " is even.");
   }

 not to mention all the code that's already using foreach.


 If the code rely on the 'implicit ref' behavior of 'i', the code is  
 doing it
 wrong. The spec never mentioned what happened when the variable is  
 modified
 in a ForeachRangeStatement. I believe it is more sane if modifying 'i'  
 is
 disallowed unless 'ref' is used.



 This should be a compile time error:

   foreach(i; 0..10){
     i += 1;
     write(i, " ");
   }


No, it should be legal and write 1 2 3 4 5 6 7 8 9 10


 This should be legal.
   foreach(ref i; 0..10){
     i += 1;
     write(i, " ");
   }


If this is legal, it should write 1 3 5 7 9

I have no problem with it being legal.

-Steve
Jun 15 2011
prev sibling parent reply KennyTM~ <kennytm gmail.com> writes:
On Jun 15, 11 23:23, Caligo wrote:

 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com>  wrote:

 On Jun 15, 11 22:35, Caligo wrote:

 You can create a temporary if you like:

    foreach(i; 0..10){
      int ii = i + 1;
      writeln(ii, " ");
    }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

    foreach(i; 0..10){
      if(i&    1)
        i += 1;
      writeln(i, " is even.");
    }

 not to mention all the code that's already using foreach.


 If the code rely on the 'implicit ref' behavior of 'i', the code is doing it
 wrong. The spec never mentioned what happened when the variable is modified
 in a ForeachRangeStatement. I believe it is more sane if modifying 'i' is
 disallowed unless 'ref' is used.



 This should be a compile time error:

    foreach(i; 0..10){
      i += 1;
      write(i, " ");
    }

 This should be legal.
    foreach(ref i; 0..10){
      i += 1;
      write(i, " ");
    }

 Is that the general consensus here?


A simple patch for this:

https://github.com/kennytm/dmd/compare/master...const_foreach_range
Jun 15 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/15/11 11:51 AM, KennyTM~ wrote:

 On Jun 15, 11 23:23, Caligo wrote:

 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com> wrote:

 On Jun 15, 11 22:35, Caligo wrote:

 You can create a temporary if you like:

 foreach(i; 0..10){
 int ii = i + 1;
 writeln(ii, " ");
 }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own. The following code would not behave correctly:

 foreach(i; 0..10){
 if(i& 1)
 i += 1;
 writeln(i, " is even.");
 }

 not to mention all the code that's already using foreach.


 If the code rely on the 'implicit ref' behavior of 'i', the code is
 doing it
 wrong. The spec never mentioned what happened when the variable is
 modified
 in a ForeachRangeStatement. I believe it is more sane if modifying
 'i' is
 disallowed unless 'ref' is used.



 This should be a compile time error:

 foreach(i; 0..10){
 i += 1;
 write(i, " ");
 }

 This should be legal.
 foreach(ref i; 0..10){
 i += 1;
 write(i, " ");
 }

 Is that the general consensus here?


 A simple patch for this:

 https://github.com/kennytm/dmd/compare/master...const_foreach_range


Does that take care of change through an alias, e.g.

int * p = i;
++*p;


Andrei
Jun 15 2011
parent reply KennyTM~ <kennytm gmail.com> writes:
On Jun 16, 11 00:53, Andrei Alexandrescu wrote:

 On 6/15/11 11:51 AM, KennyTM~ wrote:

 On Jun 15, 11 23:23, Caligo wrote:

 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com> wrote:

 On Jun 15, 11 22:35, Caligo wrote:

 You can create a temporary if you like:

 foreach(i; 0..10){
 int ii = i + 1;
 writeln(ii, " ");
 }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own. The following code would not behave correctly:

 foreach(i; 0..10){
 if(i& 1)
 i += 1;
 writeln(i, " is even.");
 }

 not to mention all the code that's already using foreach.


 If the code rely on the 'implicit ref' behavior of 'i', the code is
 doing it
 wrong. The spec never mentioned what happened when the variable is
 modified
 in a ForeachRangeStatement. I believe it is more sane if modifying
 'i' is
 disallowed unless 'ref' is used.



 This should be a compile time error:

 foreach(i; 0..10){
 i += 1;
 write(i, " ");
 }

 This should be legal.
 foreach(ref i; 0..10){
 i += 1;
 write(i, " ");
 }

 Is that the general consensus here?


 A simple patch for this:

 https://github.com/kennytm/dmd/compare/master...const_foreach_range


 Does that take care of change through an alias, e.g.

 int * p = i;
 ++*p;


 Andrei



Error: cannot implicitly convert expression (i) of type const(int) to int*



What the patch does is simply rewrite

     foreach (key; lwr .. upr) body;

into

     foreach (auto __iterator1234 = lwr, auto __limit1235 = upr;
              __iterator1234 < __limit1235;
              __iterator1234++)
     {
        const key = __iterator1234;
        body;
     }

so as long as the 'const' system in D has no bug, the compiler can deny 
all unintended mutation in the loop.
Jun 15 2011
next sibling parent KennyTM~ <kennytm gmail.com> writes:
On Jun 16, 11 02:14, KennyTM~ wrote:

 On Jun 16, 11 00:53, Andrei Alexandrescu wrote:

 On 6/15/11 11:51 AM, KennyTM~ wrote:

 On Jun 15, 11 23:23, Caligo wrote:

 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com> wrote:

 On Jun 15, 11 22:35, Caligo wrote:

 You can create a temporary if you like:

 foreach(i; 0..10){
 int ii = i + 1;
 writeln(ii, " ");
 }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own. The following code would not behave correctly:

 foreach(i; 0..10){
 if(i& 1)
 i += 1;
 writeln(i, " is even.");
 }

 not to mention all the code that's already using foreach.


 If the code rely on the 'implicit ref' behavior of 'i', the code is
 doing it
 wrong. The spec never mentioned what happened when the variable is
 modified
 in a ForeachRangeStatement. I believe it is more sane if modifying
 'i' is
 disallowed unless 'ref' is used.



 This should be a compile time error:

 foreach(i; 0..10){
 i += 1;
 write(i, " ");
 }

 This should be legal.
 foreach(ref i; 0..10){
 i += 1;
 write(i, " ");
 }

 Is that the general consensus here?


 A simple patch for this:

 https://github.com/kennytm/dmd/compare/master...const_foreach_range


 Does that take care of change through an alias, e.g.

 int * p = i;
 ++*p;


 Andrei



 Error: cannot implicitly convert expression (i) of type const(int) to int*



 What the patch does is simply rewrite

 foreach (key; lwr .. upr) body;

 into

 foreach (auto __iterator1234 = lwr, auto __limit1235 = upr;
 __iterator1234 < __limit1235;
 __iterator1234++)
 {
 const key = __iterator1234;
 body;
 }

 so as long as the 'const' system in D has no bug, the compiler can deny
 all unintended mutation in the loop.


Actually I think you meant 'int* p = &i;'. Well that is detected too.

Error: cannot implicitly convert expression (& i) of type const(int)* to 
int*
Jun 15 2011
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/15/11 1:14 PM, KennyTM~ wrote:

 On Jun 16, 11 00:53, Andrei Alexandrescu wrote:

 On 6/15/11 11:51 AM, KennyTM~ wrote:

 On Jun 15, 11 23:23, Caligo wrote:

 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com> wrote:

 On Jun 15, 11 22:35, Caligo wrote:

 You can create a temporary if you like:

 foreach(i; 0..10){
 int ii = i + 1;
 writeln(ii, " ");
 }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own. The following code would not behave correctly:

 foreach(i; 0..10){
 if(i& 1)
 i += 1;
 writeln(i, " is even.");
 }

 not to mention all the code that's already using foreach.


 If the code rely on the 'implicit ref' behavior of 'i', the code is
 doing it
 wrong. The spec never mentioned what happened when the variable is
 modified
 in a ForeachRangeStatement. I believe it is more sane if modifying
 'i' is
 disallowed unless 'ref' is used.



 This should be a compile time error:

 foreach(i; 0..10){
 i += 1;
 write(i, " ");
 }

 This should be legal.
 foreach(ref i; 0..10){
 i += 1;
 write(i, " ");
 }

 Is that the general consensus here?


 A simple patch for this:

 https://github.com/kennytm/dmd/compare/master...const_foreach_range


 Does that take care of change through an alias, e.g.

 int * p = i;
 ++*p;


 Andrei



 Error: cannot implicitly convert expression (i) of type const(int) to int*



 What the patch does is simply rewrite

 foreach (key; lwr .. upr) body;

 into

 foreach (auto __iterator1234 = lwr, auto __limit1235 = upr;
 __iterator1234 < __limit1235;
 __iterator1234++)
 {
 const key = __iterator1234;
 body;
 }

 so as long as the 'const' system in D has no bug, the compiler can deny
 all unintended mutation in the loop.


I understand, thanks. One issue we need to be careful about is 
performance. You need to ensure that the code ultimately produced is not 
impacted at all.

Also - for now please limit this to built-in numbers only (it won't work 
for pointers), and replace const with immutable.


Thanks,

Andrei
Jun 15 2011
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 6/15/11 1:14 PM, KennyTM~ wrote:

 On Jun 16, 11 00:53, Andrei Alexandrescu wrote:

 On 6/15/11 11:51 AM, KennyTM~ wrote:

 On Jun 15, 11 23:23, Caligo wrote:

 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com> wrote:

 On Jun 15, 11 22:35, Caligo wrote:

 You can create a temporary if you like:

 foreach(i; 0..10){
 int ii = i + 1;
 writeln(ii, " ");
 }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own. The following code would not behave correctly:

 foreach(i; 0..10){
 if(i& 1)
 i += 1;
 writeln(i, " is even.");
 }

 not to mention all the code that's already using foreach.


 If the code rely on the 'implicit ref' behavior of 'i', the code is
 doing it
 wrong. The spec never mentioned what happened when the variable is
 modified
 in a ForeachRangeStatement. I believe it is more sane if modifying
 'i' is
 disallowed unless 'ref' is used.



 This should be a compile time error:

 foreach(i; 0..10){
 i += 1;
 write(i, " ");
 }

 This should be legal.
 foreach(ref i; 0..10){
 i += 1;
 write(i, " ");
 }

 Is that the general consensus here?


 A simple patch for this:

 https://github.com/kennytm/dmd/compare/master...const_foreach_range


 Does that take care of change through an alias, e.g.

 int * p = i;
 ++*p;


 Andrei



 Error: cannot implicitly convert expression (i) of type const(int) to int*



 What the patch does is simply rewrite

 foreach (key; lwr .. upr) body;

 into

 foreach (auto __iterator1234 = lwr, auto __limit1235 = upr;
 __iterator1234 < __limit1235;
 __iterator1234++)
 {
 const key = __iterator1234;
 body;
 }

 so as long as the 'const' system in D has no bug, the compiler can deny
 all unintended mutation in the loop.


key should not be const imho.

1. const pollution.
2. does not work satisfactory with pointers/user defined structs with
indirections.

3? Does not work with structs with a postblit:
tt.d(16): Error: function tt.S.__cpctor (ref S p) is not callable using argument
types (S) const
Is that a compiler bug or a limitation of the language? Is there any way to
define
a const postblit operator? If yes, see 1.



Timon
Jun 15 2011
parent KennyTM~ <kennytm gmail.com> writes:
On Jun 16, 11 03:20, Timon Gehr wrote:

 On 6/15/11 1:14 PM, KennyTM~ wrote:

 On Jun 16, 11 00:53, Andrei Alexandrescu wrote:

 On 6/15/11 11:51 AM, KennyTM~ wrote:

 On Jun 15, 11 23:23, Caligo wrote:

 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com>  wrote:

 On Jun 15, 11 22:35, Caligo wrote:

 You can create a temporary if you like:

 foreach(i; 0..10){
 int ii = i + 1;
 writeln(ii, " ");
 }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own. The following code would not behave correctly:

 foreach(i; 0..10){
 if(i&  1)
 i += 1;
 writeln(i, " is even.");
 }

 not to mention all the code that's already using foreach.


 If the code rely on the 'implicit ref' behavior of 'i', the code is
 doing it
 wrong. The spec never mentioned what happened when the variable is
 modified
 in a ForeachRangeStatement. I believe it is more sane if modifying
 'i' is
 disallowed unless 'ref' is used.



 This should be a compile time error:

 foreach(i; 0..10){
 i += 1;
 write(i, " ");
 }

 This should be legal.
 foreach(ref i; 0..10){
 i += 1;
 write(i, " ");
 }

 Is that the general consensus here?


 A simple patch for this:

 https://github.com/kennytm/dmd/compare/master...const_foreach_range


 Does that take care of change through an alias, e.g.

 int * p = i;
 ++*p;


 Andrei



 Error: cannot implicitly convert expression (i) of type const(int) to int*



 What the patch does is simply rewrite

 foreach (key; lwr .. upr) body;

 into

 foreach (auto __iterator1234 = lwr, auto __limit1235 = upr;
 __iterator1234<  __limit1235;
 __iterator1234++)
 {
 const key = __iterator1234;
 body;
 }

 so as long as the 'const' system in D has no bug, the compiler can deny
 all unintended mutation in the loop.


 key should not be const imho.


Then the compiler will not complain about mutating the copy of index. 
This may silently break code that rely on the 'ref' behavior (there are 
2 foreach loops in Phobos that does this, one in std.path and another in 
std.xml). This can be a good or bad thing. I'm agnostic on this.


 1. const pollution.


?


 2. does not work satisfactory with pointers/user defined structs with
indirections.


Yes.


 3? Does not work with structs with a postblit:
 tt.d(16): Error: function tt.S.__cpctor (ref S p) is not callable using
argument
 types (S) const
 Is that a compiler bug or a limitation of the language? Is there any way to
define
 a const postblit operator? If yes, see 1.


http://d.puremagic.com/issues/show_bug.cgi?id=4867


 Timon
Jun 15 2011
prev sibling parent reply Don <nospam nospam.com> writes:
KennyTM~ wrote:

 On Jun 16, 11 00:53, Andrei Alexandrescu wrote:

 On 6/15/11 11:51 AM, KennyTM~ wrote:

 On Jun 15, 11 23:23, Caligo wrote:

 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com> wrote:

 On Jun 15, 11 22:35, Caligo wrote:

 You can create a temporary if you like:

 foreach(i; 0..10){
 int ii = i + 1;
 writeln(ii, " ");
 }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own. The following code would not behave correctly:

 foreach(i; 0..10){
 if(i& 1)
 i += 1;
 writeln(i, " is even.");
 }

 not to mention all the code that's already using foreach.


 If the code rely on the 'implicit ref' behavior of 'i', the code is
 doing it
 wrong. The spec never mentioned what happened when the variable is
 modified
 in a ForeachRangeStatement. I believe it is more sane if modifying
 'i' is
 disallowed unless 'ref' is used.



 This should be a compile time error:

 foreach(i; 0..10){
 i += 1;
 write(i, " ");
 }

 This should be legal.
 foreach(ref i; 0..10){
 i += 1;
 write(i, " ");
 }

 Is that the general consensus here?


 A simple patch for this:

 https://github.com/kennytm/dmd/compare/master...const_foreach_range


 Does that take care of change through an alias, e.g.

 int * p = i;
 ++*p;


 Andrei


 
 
 Error: cannot implicitly convert expression (i) of type const(int) to int*
 
 
 
 What the patch does is simply rewrite
 
     foreach (key; lwr .. upr) body;
 
 into
 
     foreach (auto __iterator1234 = lwr, auto __limit1235 = upr;
              __iterator1234 < __limit1235;
              __iterator1234++)
     {
        const key = __iterator1234;
        body;
     }
 
 so as long as the 'const' system in D has no bug, the compiler can deny 
 all unintended mutation in the loop.
 


The alternative would be to create __iterator1234 as const, and then 
cast away const in the ++ step. Ugly, but would eliminate the 
performance cost for structs.
Jun 15 2011
parent reply bearophile <bearophileHUGS lycos.com> writes:
Don:


 The alternative would be to create __iterator1234 as const, and then 
 cast away const in the ++ step. Ugly, but would eliminate the 
 performance cost for structs.


Isn't it undefined in D to modify a const?

And Andrei suggests built-ins only:
http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=138718
(I think for the x..y foreach range syntax only.)

Bye,
bearophile
Jun 15 2011
next sibling parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On Wed, 15 Jun 2011 22:40:24 +0200, bearophile <bearophileHUGS lycos.com>  
wrote:


 Don:


 The alternative would be to create __iterator1234 as const, and then
 cast away const in the ++ step. Ugly, but would eliminate the
 performance cost for structs.


 Isn't it undefined in D to modify a const?


I believe this is only because the source may be immutable, and thus
reside in ROM or otherwise be optimized away in places. Casting away
const for a variable you have full control of, should not pose
problems.

-- 
   Simen
Jun 16 2011
prev sibling next sibling parent Don <nospam nospam.com> writes:
bearophile wrote:

 Don:
 

 The alternative would be to create __iterator1234 as const, and then 
 cast away const in the ++ step. Ugly, but would eliminate the 
 performance cost for structs.


 
 Isn't it undefined in D to modify a const?


Yes, but that's in the user world. This is in the compiler world.
Lowering frequently creates constructs which aren't well-formed D code.
For example, the compiler creates variables in comma expressions, which 
isn't legal D (it won't even compile).
Jun 17 2011
prev sibling parent reply Jesse Phillips <jessekphillips+d gmail.com> writes:
On Wed, 15 Jun 2011 16:40:24 -0400, bearophile wrote:


 Isn't it undefined in D to modify a const?


Yes, but unlike what every redditer is say, that doesn't mean the 
compiler can do whatever it wants. It means you can't define its 
behavior. Go ahead try.

Throws an exception, nope, can't do that no information at run-time and 
const is already gone from a cast during compile-time.
Modifies value, nope, might not be possible.
Does nothing, nope, same as throwing exception.
Jun 17 2011
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
Jesse Phillips wrote:

 On Wed, 15 Jun 2011 16:40:24 -0400, bearophile wrote:


 Isn't it undefined in D to modify a const?


 Yes, but unlike what every redditer is say, that doesn't mean the
 compiler can do whatever it wants. It means you can't define its
 behavior. Go ahead try.
 [snip.]


How do "cannot do what it wants" and "the behavior cannot be defined" not
contradict each other?

It may work as you would expect in some cases but not in others. "The compiler
can
do what it wants". The behavior will depend on what the compiler wants. The way
out would be to just define the behavior of casting away const and mutating the
memory to do the expected thing in case it is not typed as immutable somewhere
in
the program. Would that kill any opportunities for optimization? I don't think
so.


Timon
Jun 17 2011
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
Jesse Phillips wrote:

 On Wed, 15 Jun 2011 16:40:24 -0400, bearophile wrote:


 Isn't it undefined in D to modify a const?


 Yes, but unlike what every redditer is say, that doesn't mean the
 compiler can do whatever it wants. It means you can't define its
 behavior. Go ahead try.
 [snip.]


I tried:
void main(){
    const c=128;
    auto p = &c;
    *cast(int*)p = 64;
    assert(p==&c);
    assert(*p!=c);
    //dmd 2.053: PASS
}

void main(){
    for(const i=0;i<32;++*cast(int*)&i){}
    assert(0); // 'pass' ;)
}

Another compiler might decide to fail both assertions or do some funny stuff.

Timon
Jun 17 2011
prev sibling parent reply Jesse Phillips <jessekphillips+D gmail.com> writes:
Timon Gehr Wrote:


 Jesse Phillips wrote:

 On Wed, 15 Jun 2011 16:40:24 -0400, bearophile wrote:


 Isn't it undefined in D to modify a const?


 Yes, but unlike what every redditer is say, that doesn't mean the
 compiler can do whatever it wants. It means you can't define its
 behavior. Go ahead try.
 [snip.]


 
 How do "cannot do what it wants" and "the behavior cannot be defined" not
 contradict each other?


Because the compiler can't identify when it happens. Try making the compiler
format your hard drive when you modify const. It can't do what it wants because
it can't define what it will do!


 It may work as you would expect in some cases but not in others. "The compiler
can
 do what it wants". The behavior will depend on what the compiler wants. The way
 out would be to just define the behavior of casting away const


Casting away const is defined. But once you do, the compiler can no longer
identify when you are modifying its value. 

void main(){
    const a = 53;
    auto pa = &a;
    auto pb = cast(int*) pa;
    assert(*pb == *pa); // Always true, all implementations
}


 and mutating the
 memory to do the expected thing in case it is not typed as immutable somewhere
in
 the program.


This is not possible. You don't know where the variable resides at compile
time. The compiler doesn't know where in memory the values are stored and it
isn't allowed to add code which checks where the variable is stored. This means
if the variable is in modifiable memory then when you modify it, it will do
exactly as you expect, even if it comes from an immutable declaration.

Your examples are flawed, declaring a variable with const is equivalent to
declaring it immutable, the compiler is allowed to place it in read only memory
(or at least that is what most people expect, I don't really know if the spec
says anything on it).

void main(){
    int a = 53;
    const int* pa = &a;
    auto pb = cast(int*) pa;
    assert(*pb == *pa);
    *pb = 3;
    assert(*pb == *pa);
}

This will always be true on every implementation.
Jun 17 2011
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
Jesse Phillips wrote:

 Timon Gehr Wrote:


 Jesse Phillips wrote:

 On Wed, 15 Jun 2011 16:40:24 -0400, bearophile wrote:


 Isn't it undefined in D to modify a const?


 Yes, but unlike what every redditer is say, that doesn't mean the
 compiler can do whatever it wants. It means you can't define its
 behavior. Go ahead try.
 [snip.]


 How do "cannot do what it wants" and "the behavior cannot be defined" not
 contradict each other?


 Because the compiler can't identify when it happens. Try making the compiler


format your hard drive when you modify const. It can't do what it wants because
it
can't define what it will do!

Uh, you could modify DMD to do just that... Left as an exercise. The point is,
the
resulting compiler would be compliant with the D spec. Ergo a D compiler can do
just what it wants in case it has compile code with undefined behavior. And how
"can't [the compiler] define what it will do"? It is generating the resulting
executable after all. The compiler is *omnipotent*!


 It may work as you would expect in some cases but not in others. "The compiler
can
 do what it wants". The behavior will depend on what the compiler wants. The way
 out would be to just define the behavior of casting away const


 Casting away const is defined. But once you do, the compiler can no longer


identify when you are modifying its value.

 void main(){
     const a = 53;
     auto pa = &a;
     auto pb = cast(int*) pa;
     assert(*pb == *pa); // Always true, all implementations


Yes, I'm perfectly fine with that. See below to learn why it is irrelevant.


 and mutating the
 memory to do the expected thing in case it is not typed as immutable somewhere
in
 the program.


 This is not possible. You don't know where the variable resides


I will now do the same thing you did: Yes I know where it resides, I can
disassemble the resulting executable and there is the address, plain as day!

JK, I guess you did not realize there was a "...and mutating the memory" part in
my statement? =)


 at compile time. The compiler doesn't know where in memory the values are
stored


and it isn't allowed to add

 code which checks where the variable is stored. This means if the variable is
in


modifiable memory then when you modify it, it will do exactly as you expect,
even
if it comes > from an immutable declaration.

I think you did not understand the point. The compiler does not have to check
anything (and why do you think it is not allowed to?), to provide that
guarantee.
It just has to act as (following you example below) DMD does at the moment. But
the spec has to reflect this, otherwise, a complying compiler can do anything.


 Your examples are flawed, declaring a variable with const is equivalent to


declaring it immutable, the compiler is allowed to place it in read only memory
(or at least that

 is what most people expect, I don't really know if the spec says anything on
it).


My examples are not flawed, that is what the discussion was about in the
beginning: rewriting the foreach range loop into the second example. Then
bearophile protested.


 void main(){
     int a = 53;
     const int* pa = &a;
     auto pb = cast(int*) pa;
     assert(*pb == *pa);
     *pb = 3;
     assert(*pb == *pa);
 }

 This will always be true on every implementation.


No. Not as long as the behavior is undefined.

All existing implementations, true. But all existing implementations use the
same
front-end so that is no argument at all. It is certainly not true for the
general
case of 'all [possible] implementations'. As long as the spec does not define
what
happens if you modify const, _the compiler can do anything!_. Does not mean that
it will do anything specific, such as that what you'd like it to do, just that
theoretically, it *could* format your hard drive, call your grandma, or steal
your
identity without breaking language imposed rules.

You may want to have a look at the excellent series about the topic located at
http://blog.regehr.org/archives/213

When I am talking about undefined behavior, I mean this.

Cheers,
- Timon
Jun 17 2011
parent reply Jesse Phillips <jessekphillips+D gmail.com> writes:
Timon Gehr Wrote:


 I will now do the same thing you did: Yes I know where it resides, I can
 disassemble the resulting executable and there is the address, plain as day!


Not possible, the executable can not contain an address because nothing is in
memory to have an address.


 JK, I guess you did not realize there was a "...and mutating the memory" part
in
 my statement? =)


I did see that but you must first understand that casting away const is
defined. Once this is done though the compiler can no longer identify when
something that was once const is now being modified.

The behavior of modifying an int* is also defined, and when you cast away const
to get an int* you maybe modifying what use to be a const variables, but the
compiler does not know that.
 

 at compile time. The compiler doesn't know where in memory the values are
stored


 I think you did not understand the point. The compiler does not have to check
 anything (and why do you think it is not allowed to?), to provide that
guarantee.
 It just has to act as (following you example below) DMD does at the moment. But
 the spec has to reflect this, otherwise, a complying compiler can do anything.


No it can not provide that guarantee at no cost. It is allowed to do
optimizations on const variables, I can not tell what optimizations the
compiler could preform as I don't think there are any at this time, but why
restrict future geniuses?

Sorry shouldn't have brought up extra code, if you can come up with some code
or design that can identify when a const value has changed, within the current
guidelines of the spec the more power to you.


 My examples are not flawed, that is what the discussion was about in the
 beginning: rewriting the foreach range loop into the second example. Then
 bearophile protested.


This is an implementation detail of foreach that can be made to work with DMD.

Though this does bring up a point you aren't trying to make. There are places
where the compiler can statically identify you are modifying const and put in
any code it wants, but it still can't define what happens when you modify const.


 

 void main(){
     int a = 53;
     const int* pa = &a;
     auto pb = cast(int*) pa;
     assert(*pb == *pa);
     *pb = 3;
     assert(*pb == *pa);
 }

 This will always be true on every implementation.


 
 No. Not as long as the behavior is undefined.


Yes even if it is undefined.
 

 As long as the spec does not define what
 happens if you modify const, _the compiler can do anything!_. Does not mean
that
 it will do anything specific, such as that what you'd like it to do, just that
 theoretically, it *could* format your hard drive, call your grandma, or steal
your
 identity without breaking language imposed rules.


You are correct, that it could theoretically do all those things and still be
valid to the spec. But it can't be done, not without breaking some other aspect
of the spec.
 

 You may want to have a look at the excellent series about the topic located at
 http://blog.regehr.org/archives/213
 
 When I am talking about undefined behavior, I mean this.


Yes I've read his posts. And many of them the compiler knows what is happening
or the hardware is even able to detect the behavior. But in the case of
modifying const, you can't identify it to make it defined.

I haven't been touching on the main reason it isn't defined, could be an
immutable value. You are just asking for defining when it does point to a
mutable value. It is like trying to define gravity as the force that keeps you
on the earth. It does do a good job of that but only because of the properties
of gravity.
Jun 17 2011
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
Jesse Phillips wrote:

 Timon Gehr Wrote:


 I will now do the same thing you did: Yes I know where it resides, I can
 disassemble the resulting executable and there is the address, plain as day!


 Not possible, the executable can not contain an address because nothing is in


memory to have an address.

Uh... You don't actually know the internals of executables, do you? Because if
you
do, please clarify your statement.


 JK, I guess you did not realize there was a "...and mutating the memory" part
in
 my statement? =)


 I did see that but you must first understand that casting away const is
defined.


Once this is done though the compiler can no longer identify when something that
was once

 const is now being modified.


I understand that.


 The behavior of modifying an int* is also defined, and when you cast away const


to get an int* you maybe modifying what use to be a const variables, but the
compiler does not > know that.

Nope. As soon as you invoke undefined behavior, even the behavior of modifying
an
int* is not defined anymore. Encountering undefined behavior makes your whole
program nonsensical. It does not matter if the compiler knows that. It could
know
if its developer wanted it to.


 at compile time. The compiler doesn't know where in memory the values are
stored


 I think you did not understand the point. The compiler does not have to check
 anything (and why do you think it is not allowed to?), to provide that
guarantee.
 It just has to act as (following you example below) DMD does at the moment. But
 the spec has to reflect this, otherwise, a complying compiler can do anything.


 No it can not provide that guarantee at no cost. It is allowed to do


optimizations on const variables, I can not tell what optimizations the compiler
could preform as I don't > think there are any at this time, but why restrict
future geniuses?

You cannot perform optimizations on const variables, because they don't give
more
guarantees than mutable variables. They only impose restrictions. It is as
simple
as that. If you want guarantees and optimizations, use immutable variables.


 Sorry shouldn't have brought up extra code, if you can come up with some code
or


design that can identify when a const value has changed, within the current
guidelines of the > spec the more power to you.

You can perform a runtime-check. But you don't need to to give the guarantee
that
casting away const from a reference to mutable memory will give you a valid
reference through which that memory can be modified.


 My examples are not flawed, that is what the discussion was about in the
 beginning: rewriting the foreach range loop into the second example. Then
 bearophile protested.


 This is an implementation detail of foreach that can be made to work with DMD.


I think it is not worth the special casing, but sure.


 Though this does bring up a point you aren't trying to make. There are places


where the compiler can statically identify you are modifying const and put in
any
code it wants, > but it still can't define what happens when you modify const.

Sure, because the const could reference immutable memory.


 void main(){
     int a = 53;
     const int* pa = &a;
     auto pb = cast(int*) pa;
     assert(*pb == *pa);
     *pb = 3;
     assert(*pb == *pa);
 }

 This will always be true on every implementation.


 No. Not as long as the behavior is undefined.


 Yes even if it is undefined.


defined behavior <=> it must always behave that way on every implementation.
undefined behavior <=> the behavior could be anything, but it might also be what
you expected.

There is nothing more to it.


 As long as the spec does not define what
 happens if you modify const, _the compiler can do anything!_. Does not mean
that
 it will do anything specific, such as that what you'd like it to do, just that
 theoretically, it *could* format your hard drive, call your grandma, or steal
your
 identity without breaking language imposed rules.


 You are correct, that it could theoretically do all those things and still be


valid to the spec. But it can't be done, not without breaking some other aspect
of
the spec.

So it is theoretically possible, yet impossible? This fails to sound like a fair
point. Which aspect would be broken?


 You may want to have a look at the excellent series about the topic located at
 http://blog.regehr.org/archives/213

 When I am talking about undefined behavior, I mean this.


 Yes I've read his posts. And many of them the compiler knows what is happening


or the hardware is even able to detect the behavior. But in the case of
modifying
const, you

 can't identify it to make it defined.


You don't need to identify it to make the behavior defined just if it is
actually
pointing to mutable memory. The compiler does not know what is going on, it is
simply ignoring it to generate faster code within the spec.


 I haven't been touching on the main reason it isn't defined, could be an


immutable value. You are just asking for defining when it does point to a
mutable
value. It is like

 trying to define gravity as the force that keeps you on the earth. It does do a


good job of that but only because of the properties of gravity.

Your analogy has nothing to do with the issue. A better one.

You have two classes of items.

1. Tea Cups. Dropping them to the floor will break them.
2. The magical flubber slime ball from outer space (TMFSBFOS). Dropping it to
the
floor will have undefined behavior.

Now you have a set of equal boxes, each containing either a Tea Cup or the
magical
flubber slime ball from outer space. You have no way to know what it is.

Now, a quick wrap-up of our points:
Timon: Now what happens if you drop one of these boxes?
Jesse: The behavior is undefined, because it could contain the magical flubber
slime ball from outer space!
Timon: And if there is a tea cup in there?
Jesse: The behavior is undefined, because it could contain the magical flubber
slime ball from outer space!
Timon: Well, no it is a tea cup. What happens if you drop a tea cup?
Jesse: It breaks.
Timon: Yes, and if it is in a box?
Jesse: The behavior is still undefined!
Timon: No. The tea cup breaks.
Jesse: Yes that is what happens, always happened and will always happen in all
circumstances, yet the behavior is undefined. Also, white is black.


Cheers,
-Timon
Jun 17 2011
parent Jesse Phillips <jessekphillips+d gmail.com> writes:
On Sat, 18 Jun 2011 02:21:19 +0000, Timon Gehr wrote:


 You have two classes of items.
 
 1. Tea Cups. Dropping them to the floor will break them. 2. The magical
 flubber slime ball from outer space (TMFSBFOS). Dropping it to the floor
 will have undefined behavior.
 
 Now you have a set of equal boxes, each containing either a Tea Cup or
 the magical flubber slime ball from outer space. You have no way to know
 what it is.
 ...
 Cheers,
 -Timon


I really like the analogy you have set up. So let us define what happens 
when we drop the box.

My position is that you can not define the resulting behavior.

You claim that if the Tea Cup is in the box, the Tea Cup will break.

I can agree that this will be what happens, but I can not define for you 
what dropping the box will do.

But if the box were to be dropped with a Tea Cup was inside, and the Tea 
Cup turned into a whale, then it would still fulfill the spec, yet I can 
not provide for you a way that dropping a boxed Tea Cup could result in 
anything but a broken Tea Cup.
Jun 19 2011
prev sibling next sibling parent David Nadlinger <see klickverbot.at> writes:
On 6/15/11 4:35 PM, Caligo wrote:

 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

    foreach(i; 0..10){
      if(i&  1)
        i += 1;
      writeln(i, " is even.");
    }


But in my opinion it should not be legal in the first place anyway, as 
we already have »continue« to do this in a very clear way:

---
foreach (i; 0 .. 10) {
   if (i & 1) continue;
   writeln(i, " is even.");
}
---

David
Jun 15 2011
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 15 Jun 2011 10:35:33 -0400, Caligo <iteronvexor gmail.com> wrote:


 You can create a temporary if you like:

   foreach(i; 0..10){
     int ii = i + 1;
     writeln(ii, " ");
   }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

   foreach(i; 0..10){
     if(i & 1)
       i += 1;
     writeln(i, " is even.");
   }


That code relies on undocumented behavior.  It's likely to fail on some  
other D compiler.

We can't worry about existing code that uses undocumented "features" of  
dmd.  It's utilizing the knowledge of how D rewrites the foreach  
statement, not how foreach is documented to work.

Besides, I think this case is very uncommon.  A better implementation for  
such a thing would be:

foreach(i;  0..5)
    writeln(i * 2, " is even.");

-Steve
Jun 15 2011
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
Steven Schveighoffer wrote:

 On Wed, 15 Jun 2011 10:35:33 -0400, Caligo <iteronvexor gmail.com> wrote:


 You can create a temporary if you like:

   foreach(i; 0..10){
     int ii = i + 1;
     writeln(ii, " ");
   }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

   foreach(i; 0..10){
     if(i & 1)
       i += 1;
     writeln(i, " is even.");
   }


 That code relies on undocumented behavior.  It's likely to fail on some
 other D compiler.

 We can't worry about existing code that uses undocumented "features" of
 dmd.  It's utilizing the knowledge of how D rewrites the foreach
 statement, not how foreach is documented to work.
 [snip.]


Actually that *is* exactly as it is documented to work.
Any code exploiting it relies on documented behavior.

See TDPL p. 74:

foreach(<symbol>; <expression1> .. <expression2>) <statement>

is rewritten to:

{
    auto __n = <expression2>;
    auto symbol = true ? <expression1> : <expression2>;
    for(; <symbol> < __n; ++<symbol>) <statement>
}

This change would introduce an inconsistency with TDPL.


Timon
Jun 15 2011
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 15 Jun 2011 13:09:40 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:


 Steven Schveighoffer wrote:

 On Wed, 15 Jun 2011 10:35:33 -0400, Caligo <iteronvexor gmail.com>  
 wrote:


 You can create a temporary if you like:

   foreach(i; 0..10){
     int ii = i + 1;
     writeln(ii, " ");
   }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

   foreach(i; 0..10){
     if(i & 1)
       i += 1;
     writeln(i, " is even.");
   }


 That code relies on undocumented behavior.  It's likely to fail on some
 other D compiler.

 We can't worry about existing code that uses undocumented "features" of
 dmd.  It's utilizing the knowledge of how D rewrites the foreach
 statement, not how foreach is documented to work.
 [snip.]


 Actually that *is* exactly as it is documented to work.
 Any code exploiting it relies on documented behavior.

 See TDPL p. 74:

 foreach(<symbol>; <expression1> .. <expression2>) <statement>

 is rewritten to:

 {
     auto __n = <expression2>;
     auto symbol = true ? <expression1> : <expression2>;
     for(; <symbol> < __n; ++<symbol>) <statement>
 }

 This change would introduce an inconsistency with TDPL.


I think it would be worth it.

In any instance of foreach *except* this one, rebinding the value does not  
change the iteration.  It doesn't fit with all other cases of foreach.

-Steve
Jun 15 2011
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
Steven Schveighoffer wrote:

 On Wed, 15 Jun 2011 13:09:40 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:


 Steven Schveighoffer wrote:

 On Wed, 15 Jun 2011 10:35:33 -0400, Caligo <iteronvexor gmail.com>
 wrote:


 You can create a temporary if you like:

   foreach(i; 0..10){
     int ii = i + 1;
     writeln(ii, " ");
   }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

   foreach(i; 0..10){
     if(i & 1)
       i += 1;
     writeln(i, " is even.");
   }


 That code relies on undocumented behavior.  It's likely to fail on some
 other D compiler.

 We can't worry about existing code that uses undocumented "features" of
 dmd.  It's utilizing the knowledge of how D rewrites the foreach
 statement, not how foreach is documented to work.
 [snip.]


 Actually that *is* exactly as it is documented to work.
 Any code exploiting it relies on documented behavior.

 See TDPL p. 74:

 foreach(<symbol>; <expression1> .. <expression2>) <statement>

 is rewritten to:

 {
     auto __n = <expression2>;
     auto symbol = true ? <expression1> : <expression2>;
     for(; <symbol> < __n; ++<symbol>) <statement>
 }

 This change would introduce an inconsistency with TDPL.


 I think it would be worth it.

 In any instance of foreach *except* this one, rebinding the value does not
 change the iteration.  It doesn't fit with all other cases of foreach.

 -Steve


Well, it does make sense but we have to be careful here. If foreach range
statements get less efficient underway, it is not worth it.
I am perfectly fine with foreach(i;0..n){} just being syntactic sugar for
for(int
i=0;i<n;i++){}.

Timon
Jun 15 2011
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 15 Jun 2011 13:56:04 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:


 Steven Schveighoffer wrote:

 On Wed, 15 Jun 2011 13:09:40 -0400, Timon Gehr <timon.gehr gmx.ch>  
 wrote:


 Steven Schveighoffer wrote:

 On Wed, 15 Jun 2011 10:35:33 -0400, Caligo <iteronvexor gmail.com>
 wrote:


 You can create a temporary if you like:

   foreach(i; 0..10){
     int ii = i + 1;
     writeln(ii, " ");
   }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

   foreach(i; 0..10){
     if(i & 1)
       i += 1;
     writeln(i, " is even.");
   }


 That code relies on undocumented behavior.  It's likely to fail on  
 some
 other D compiler.

 We can't worry about existing code that uses undocumented "features"  
 of
 dmd.  It's utilizing the knowledge of how D rewrites the foreach
 statement, not how foreach is documented to work.
 [snip.]


 Actually that *is* exactly as it is documented to work.
 Any code exploiting it relies on documented behavior.

 See TDPL p. 74:

 foreach(<symbol>; <expression1> .. <expression2>) <statement>

 is rewritten to:

 {
     auto __n = <expression2>;
     auto symbol = true ? <expression1> : <expression2>;
     for(; <symbol> < __n; ++<symbol>) <statement>
 }

 This change would introduce an inconsistency with TDPL.


 I think it would be worth it.

 In any instance of foreach *except* this one, rebinding the value does  
 not
 change the iteration.  It doesn't fit with all other cases of foreach.

 -Steve


 Well, it does make sense but we have to be careful here. If foreach range
 statements get less efficient underway, it is not worth it.
 I am perfectly fine with foreach(i;0..n){} just being syntactic sugar  
 for for(int
 i=0;i<n;i++){}.


I've seen several cases where the syntactic sugar does not work.

two things:

if the compiler detects you never change i, then it can eliminate the  
temporary variable.
We should favor the case which is most useful, not the one which is  
simplest to implement.  If you want surprising behavior, use a for loop.

When I see foreach(i; xxx), I think i should take on every value contained  
in xxx, whether that be a range, array, or something else.  Being able to  
change the elements iterated during the loop is surprising to say the  
least.

Also, think about this:

foreach(i; 0..10)
{
    i -= 2;
}

This is a loop that counts *down* to int.min from -1.  The idea that this  
should be *expected* behavior is just ludicrous to me.

-Steve
Jun 15 2011
parent Timon Gehr <timon.gehr gmx.ch> writes:
Steven Schveighoffer wrote:

 I've seen several cases where the syntactic sugar does not work.

 two things:

 if the compiler detects you never change i, then it can eliminate the
 temporary variable.


It could. Does DMD do that? Let's see:
import std.stdio;

void main(){
    foreach(_i;0..100){
        writeln(i);
    }
}

void main(){
    foreach(_i;0..100){
        const i=_i;
        writeln(i);
    }
}

dmd test
1 mov    -0x8(%rbp),%edi

2 mov    -0x8(%rbp),%ecx
2 mov    %rcx,%rdi

dmd test -O

Identical. Nice. Now for something more complex:

import std.stdio;

struct S{
    int i;
    this(this){writeln("postblit");}
    void opUnary(string op:"++")(){++i;}
}

void main(){
    foreach(_i;S(0)..S(2)){
        //auto i=_i;
        alias _i i;
    }
}

dmd test -O

With the alias i:

With the auto i:
postblit
postblit


=> Not optimized away, even though it could remove the whole loop.


 We should favor the case which is most useful, not the one which is
 simplest to implement.  If you want surprising behavior, use a for loop.


Also if I want guaranteed efficient looping. :o)


 When I see foreach(i; xxx), I think i should take on every value contained
 in xxx, whether that be a range, array, or something else.  Being able to
 change the elements iterated during the loop is surprising to say the
 least.

 Also, think about this:

 foreach(i; 0..10)
 {
     i -= 2;
 }

 This is a loop that counts *down* to int.min from -1.  The idea that this
 should be *expected* behavior is just ludicrous to me.

 -Steve


Absolutely. Regarding consistency, it is pretty clear that it should be changed.


Timon
Jun 15 2011
prev sibling parent so <so so.so> writes:
 I think this is a bug.  Either i should be const during loop iteration,  
 or an additional temporary should be manufactured.

 It is always possible to get the above behavior using a for loop.   
 foreach should be sane.


I agree, you'd expect it to be passed by value. As mentioned reference  
must be stated explicitly.

foreach(ref i...)
Jun 15 2011
prev sibling next sibling parent reply Kagamin <spam here.lot> writes:
bearophile Wrote:


 In my opinion this is a bit bug-prone because in real code there is some risk
of modifying the iteration variable "i" by mistake. (Note: here I am not
talking about D for() loops. They are OK, their semantics is transparent
enough. foreach() loops are less transparent and they *look* higher level than
for() loops). I'd like the iteration variable to act as being a copy of the
true loop variable as in Python.


What if user wants foreach(i;0..10) to be a shortcut for for(i=0;i<10;i++) and
still modify the variable with the exception that bounds are evaluated only
once? :)
Well, yeah, this makes little sense.


 If this is a bit bad for foreach performance, then I'd like the compiler to
forbid the mutation of the foreach iteration variable inside the foreach body.
Is this even possible?


is iota a solution here?
Jun 15 2011
parent Daniel Gibson <metalcaedes gmail.com> writes:
Am 15.06.2011 16:37, schrieb Kagamin:

 bearophile Wrote:
 

 In my opinion this is a bit bug-prone because in real code there is some risk
of modifying the iteration variable "i" by mistake. (Note: here I am not
talking about D for() loops. They are OK, their semantics is transparent
enough. foreach() loops are less transparent and they *look* higher level than
for() loops). I'd like the iteration variable to act as being a copy of the
true loop variable as in Python.


 
 What if user wants foreach(i;0..10) to be a shortcut for for(i=0;i<10;i++) and
still modify the variable with the exception that bounds are evaluated only
once? :)
 Well, yeah, this makes little sense.


maybe with foreach(ref i;0..10) ?


 

 If this is a bit bad for foreach performance, then I'd like the compiler to
forbid the mutation of the foreach iteration variable inside the foreach body.
Is this even possible?


 
 is iota a solution here?
Jun 15 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-06-15 04:20, bearophile wrote:

 This post is about some characteristics of the D foreach that has nagged me
for some time.

 This is a little Python 2.6 program:


 for i in xrange(10):
      i += 1
      print i,


 Its output shows that you are allowed to modify the iteration variable
(contents of the iteration name), but the iteration goes on with no change:
 1 2 3 4 5 6 7 8 9 10


 Similar code in D using foreach shows a different story:

 import std.stdio;
 void main() {
      foreach (i; 0 .. 10) {
          i += 1;
          write(i, " ");
      }
 }

 The output:
 1 3 5 7 9


 In my opinion this is a bit bug-prone because in real code there is some risk
of modifying the iteration variable "i" by mistake. (Note: here I am not
talking about D for() loops. They are OK, their semantics is transparent
enough. foreach() loops are less transparent and they *look* higher level than
for() loops). I'd like the iteration variable to act as being a copy of the
true loop variable as in Python. If this is a bit bad for foreach performance,
then I'd like the compiler to forbid the mutation of the foreach iteration
variable inside the foreach body. Is this even possible?


 Currently you can't solve the problem adding a const(int) to the iteration
variable:

 import std.stdio;
 void main() {
      foreach (const(int) i; 0 .. 10) { // line 3
          write(i, " ");
      }
 }


 DMD gives:
 test.d(3): Error: variable test.main.i cannot modify const


 This is a related but different thing:
 http://d.puremagic.com/issues/show_bug.cgi?id=5255

 Bye,
 bearophile


I agree with this. The iteration variable should not be changeable by 
default and to get the current behavior one could add "ref".

-- 
/Jacob Carlborg
Jun 17 2011
parent bearophile <bearophileHUGS lycos.com> writes:
Jacob Carlborg:


 I agree with this. The iteration variable should not be changeable by 
 default and to get the current behavior one could add "ref".


Thank you for all the answers. It seems most people agree with this idea (about
90%+ comments are positive). I think Andrei is not against this idea, if
limited to normal numbers. I'd like to know if Walter is OK with this little
change in D.

Bye,
bearophile
Jun 17 2011