digitalmars.D - The future of foreach
- Janice Caron (70/70) Dec 23 2007 Walter has stated many times that foreach is a good thing because it
- downs (3/21) Dec 23 2007 :)
- Bill Baxter (4/32) Dec 23 2007 Just two arrays downs? C'mon man! Don't you have a variadic template
- downs (3/8) Dec 23 2007 No, not _yet_. Gimme a minute. :)
- downs (50/50) Dec 23 2007 Finally. This took entirely too long.
- bearophile (72/73) Dec 23 2007 I am developing a large functional lib too ;-) I'll add something like t...
- bearophile (12/13) Dec 23 2007 Fixed, instead of size_t id, ref size_t:
- downs (8/16) Dec 23 2007 Yay! :D More functional for D is always good.
- bearophile (49/54) Dec 23 2007 Some things are better functional, others are better not, IMHO.
- Hxal (7/106) Dec 23 2007 What you're suggesting is perfectly possible to do without any language ...
- Janice Caron (17/18) Dec 23 2007 Since what I'm suggesting /is/ a language change, it clearly isn't.
- Don Clugston (5/25) Dec 25 2007 I don't think that's the main reason. A more fundumental problem is that...
- Janice Caron (2/4) Dec 25 2007 Cool!
- Hxal (6/10) Dec 25 2007 A dedicated optimization pass for functions taking delegates that is abl...
- Jascha Wetzel (20/24) Dec 23 2007 Induction variable analysis and reduction in strength also work for
- Craig Black (9/16) Dec 23 2007 Once we have better support for structs, we can have struct iterators.
- BCS (16/18) Dec 23 2007 If you can be totally sure of the amount of stack space that will be use...
- Paul Anderson (3/6) Dec 24 2007 Technically, the plural would be "opApply operations".
- BCS (3/11) Dec 26 2007 no, going that way it would be opApply functions or methods, or maybe ca...
- Paul Anderson (4/18) Dec 27 2007 You're right, of course. I was only trying to make the point that for so...
- BCS (6/12) Dec 27 2007 singular (?)
Walter has stated many times that foreach is a good thing because it expresses the programmer's intent, and leaves the optimisation down to the compiler. (Should it use pointers? Should it use indeces? etc.) I agree with him. However, it is sadly flawed in that you can't iterate through two collections in lockstep. I'm sure that many suggestions have been proposed in the past to work around this limitation, but the bottom line has always been that we're stuck with opApply(), and opApply() cannot be made to loop through two things at once. So... I'd like to suggest a /gradual/ change. It seems to me that this would work without really hurting anything, and programmers could get used to new idioms a little bit at a time. STEP ONE - Make it work for built-in arrays /only/ This one seems pretty straightforward. For built-in arrays, we allow people to do this: int[] a, b, c; foreach(ref x;a)(y;b)(z;c) { x = y * z; } This should present the compiler with no difficulty, because we're /only/ talking about builtin arrays here, and so there's no opApply() to worry about. This will also give us coders a chance to play with it and get used to the idiom. At this point, /some/ structs and classes will be able to add their own elementwise features simply by providing a function which returns an array. For example: Vector!(10,int) a,b,c; foreach(ref x;a.toArray)(y;b.toArray)(z;c.toArray) { x = y * z; } It's not perfect (yet), but it's a step in the right direction. STEP TWO - Allow foreach to recurse into multidimensional arrays This is a pretty nice one. int[][] a; foreach(int[] x;a) { /*elements of a*/ } foreach(int x; a) { /* elements of elements of a*/ } Now we'll be able to add elementwise features to even more structs and classes. For example: Matrix!(10,10,int) a,b,c; foreach(ref int x;a.toArray)(int y;b.toArray)(int z;c.toArray) { x = y * z; } (Yes, I'm aware that that's not doing matrix multiplication, but apparently there is a need to do this). Again, it's not perfect (yet), but it's moving just a little bit closer. STEP THREE - Extend these features to "array-like types". If we consider an "array-like type" to be any class or struct which implements: opIndex() opIndexAssign() length() and/or ptr() end() (with the latter two returning iterators), then I see no reason why these features couldn't also be made to work with arbitrary collections. The rule would be: (1) if we implement opIndex(), opIndexAssign() and length(), use those, else (2) if we implement ptr() and end(), use those, else (3) if we implement opApply(), use that (with all the old limitations), else (4) compile-time error Of course, we don't have iterators yet, so I should probably have added step 2.5, finish implementiing iterators. We already have /most/ of what iterators need: opEquals(), opPostInc(), opPostDec() and opStar() (hopefully to be renamed opDeref()). I think we're still missing opStarAssign() / opDerefAssign(), but once that's in place we'd be good to go. Once step three is in place, structs and classes will no longer need a toArray() function, and (better still) the mechanism will work even for collections which /can't/ return an array, such as linked lists. At this point we'll be able to do List!(Widget) a,b,c; foreach(ref x;a)(y;b)(z;c) { a = b.someFunction(c); } Thoughts?
Dec 23 2007
Or just do this:module lockstep; import std.stdio; struct _lockstep(T) { T[] a, b; int opApply(int delegate(size_t id, ref T, ref T) dg) { foreach (id, entry; a) if (auto res=dg(id, entry, b[id])) return res; return 0; } int opApply(int delegate(ref T, ref T) dg) { foreach (id, entry; a) if (auto res=dg(entry, b[id])) return res; return 0; } } _lockstep!(T) lockstep(T)(T[] a, T[] b) { _lockstep!(T) res; res.a=a; res.b=b; return res; } void main() { foreach (entry1, entry2; lockstep([1, 2, 3], [4, 5, 6])) writefln(entry1, " - ", entry2); }:) --downs
Dec 23 2007
downs wrote:Or just do this:Just two arrays downs? C'mon man! Don't you have a variadic template version up your sleeves somewhere? --bbmodule lockstep; import std.stdio; struct _lockstep(T) { T[] a, b; int opApply(int delegate(size_t id, ref T, ref T) dg) { foreach (id, entry; a) if (auto res=dg(id, entry, b[id])) return res; return 0; } int opApply(int delegate(ref T, ref T) dg) { foreach (id, entry; a) if (auto res=dg(entry, b[id])) return res; return 0; } } _lockstep!(T) lockstep(T)(T[] a, T[] b) { _lockstep!(T) res; res.a=a; res.b=b; return res; } void main() { foreach (entry1, entry2; lockstep([1, 2, 3], [4, 5, 6])) writefln(entry1, " - ", entry2); }:) --downs
Dec 23 2007
Bill Baxter wrote:Just two arrays downs? C'mon man! Don't you have a variadic template version up your sleeves somewhere? --bbNo, not _yet_. Gimme a minute. :) --downs
Dec 23 2007
Finally. This took entirely too long. I'll save myself the hassle of indenting it. Have fun! :) --downs module lockstep; import std.stdio; template Tuple(T...) { alias T Tuple; } template ElemType(T) { static assert(false); } template ElemType(T: T[]) { alias T ElemType; } template Unstatic(T...) { static if (T.length) { alias Tuple!(ElemType!(T[0])[], Unstatic!(T[1..$])) Unstatic; } else alias Tuple!() Unstatic; } template RefParams(int LEN) { static if (LEN>1) { const int minus1=LEN-1; const string def=RefParams!(LEN-1).def~ ", ref ElemType!(T["~minus1.stringof~"])"; const string call=RefParams!(LEN-1).call~ ", a["~minus1.stringof~"][id]"; } else { static assert(LEN); const string def="ref ElemType!(T[0])"; const string call="a[0][id]"; } } struct _lockstep(T...) { T a; mixin(" int opApply(int delegate(ref size_t id, "~RefParams!(T.length).def~") dg) { foreach (id, bogus; a[0]) if (auto res=dg(id, "~RefParams!(T.length).call~")) return res; return 0; } int opApply(int delegate("~RefParams!(T.length).def~") dg) { foreach (id, ref entry; a[0]) if (auto res=dg("~RefParams!(T.length).call~")) return res; return 0; } "); } _lockstep!(Unstatic!(T)) lockstep(T...)(T p) { _lockstep!(Unstatic!(T)) res; foreach (id, entry; p) res.a[id]=p[id]; return res; } void main() { foreach (entry1, entry2, entry3; lockstep([1, 2, 3], [4, 5, 6], [7, 8, 9])) writefln(entry1, " - ", entry2, " - ", entry3); }
Dec 23 2007
downs Wrote:Finally. This took entirely too long.I am developing a large functional lib too ;-) I'll add something like this too (I already have two zip-like thingies there, but they aren't lazy). This is my first try at a solution, but it has a bug still, the index i gives problems still (in your original 2-element solution too, I think). If you want you can fix the problem. import std.stdio; static import std.metastrings; /// Like ArrayType, but it goes down just 1 level. template ArrayType1(T: T[]) { alias T ArrayType1; } /// ... template Lets2(string txt, int n) { static if (n > 0) const Lets2 = Lets2!(txt, n-1) ~ std.metastrings.Format!(txt, std.metastrings.ToString!(n-1), std.metastrings.ToString!(n-1)); else const Lets2 = ""; } // ------------------------------------ template SeriesGen1S(string txt, string separator, int max, int min=0) { static if (min > max) const SeriesGen1S = ""; else static if (min == max) const SeriesGen1S = std.metastrings.Format!(txt, std.metastrings.ToString!(max)); else const SeriesGen1S = SeriesGen1S!(txt, separator, max-1, min) ~ separator ~ std.metastrings.Format!(txt, std.metastrings.ToString!(max)); } private struct _xzip(TyArrays...) { mixin( Lets2!("alias ArrayType1!(TyArrays[%s]) T%s;\n", TyArrays.length) ); mixin( Lets2!("T%s[] a%s;\n", TyArrays.length) ); int len = 0; mixin(" int opApply(int delegate(" ~ SeriesGen1S!("ref T%s", ", ", TyArrays.length-1) ~ ") dg) { foreach (size_t id, entry; a0[0 .. len])" ~ "if (auto res = dg(entry, "~ SeriesGen1S!("a%s[id]", ", ", TyArrays.length-1, 1) ~ ")) return res; return 0; } "); mixin(" int opApply(int delegate(size_t id, " ~ SeriesGen1S!("ref T%s", ", ", TyArrays.length-1) ~ ") dg) { foreach (size_t id, entry; a0[0 .. len])" ~ "if (auto res = dg(id, entry, "~ SeriesGen1S!("a%s[id]", ", ", TyArrays.length-1, 1) ~ ")) return res; return 0; } "); } _xzip!(TyArrays) xzip(TyArrays...)(TyArrays arrays) { int lenmin = arrays[0].length; foreach(arr; arrays[1 .. $]) if (arr.length < lenmin) lenmin = arr.length; mixin("auto iter = _xzip!(TyArrays)(" ~ SeriesGen1S!("arrays[%s]", ", ", TyArrays.length-1) ~ ");"); iter.len = lenmin; return iter; } void main() { foreach (x, y; xzip([1, 2, 3, 4], [4.1, 5.1, 6.1])) writefln(x, " - ", y); writefln(); foreach (x, y, z; xzip([1, 2, 3, 4], [4.1, 5.1, 6.1], "abcd")) writefln(x, " - ", y, " - ", z); //foreach (i, x, y; xzip([1, 2, 3, 4], [4.1, 5.1, 6.1])) // BUG // writefln(i, ": ", x, " - ", y); } Bye, bearophile
Dec 23 2007
bearophile:but it has a bug still, the index i gives problems still (in your original 2-element solution too, I think). If you want you can fix the problem.Fixed, instead of size_t id, ref size_t: mixin(" int opApply(int delegate(ref size_t, " ~ SeriesGen1S!("ref T%s", ", ", TyArrays.length-1) ~ ") dg) { foreach (size_t id, entry; a0[0 .. len])" ~ "if (auto res = dg(id, entry, "~ SeriesGen1S!("a%s[id]", ", ", TyArrays.length-1, 1) ~ ")) return res; return 0; } "); Bye, bearophile
Dec 23 2007
bearophile wrote:downs Wrote:Yay! :D More functional for D is always good. I think the problem is that arbitrary lazy zip over foreachable things requires the use of Stackthreads or similar to work, which adds a speed hit.Finally. This took entirely too long.I am developing a large functional lib too ;-) I'll add something like this too (I already have two zip-like thingies there, but they aren't lazy).This is my first try at a solution, but it has a bug still, the index i gives problems still (in your original 2-element solution too, I think). If you want you can fix the problem. [snip lots of code]Ugh. Thanks for making me experience what other people feel when they read my code .. Just kidding. Good work, even though I don't fully understand it yet :D BTW, metastrings looks seriously cute. I'll have to learn that. --downs
Dec 23 2007
downs:Yay! :D More functional for D is always good.Some things are better functional, others are better not, IMHO. If you try to follow the two last chapters of "The little Schemer" you learn when FP becomes just a good way to twist your brain.I think the problem is that arbitrary lazy zip over foreachable things requires the use of Stackthreads or similar to work, which adds a speed hit.My code can be improved a bit to make it work when the first parameter is any iterable object too.Ugh. Thanks for making me experience what other people feel when they read my code ..I know mixins aren't much readable... :o)Just kidding. Good work, even though I don't fully understand it yet :DIf you have questions just ask. And you want to look the whole functional lib ask. It's open source after all... And this gives a better error message when you give a single array: _Xzip!(TyArrays) xzip(TyArrays...)(TyArrays arrays) { static assert(TyArrays.length >= 2, "xzip() accepts only 2 or more arrays."); int lenmin = arrays[0].length; foreach(arr; arrays[1 .. $]) if (arr.length < lenmin) lenmin = arr.length; mixin("auto iter = _Xzip!(TyArrays)(" ~ SeriesGen1S!("arrays[%s]", ", ", TyArrays.length-1) ~ ");"); iter.len = lenmin; return iter; } private struct _Xzip(TyArrays...) { static assert(TyArrays.length >= 2, "_Xzip accepts only 2 or more arrays."); mixin( Lets2!("alias ArrayType1!(TyArrays[%s]) T%s;\n", TyArrays.length) ); mixin( Lets2!("T%s[] a%s;\n", TyArrays.length) ); int len = 0; static if (TyArrays.length >= 2) { mixin(" int opApply(int delegate(" ~ SeriesGen1S!("ref T%s", ", ", TyArrays.length-1) ~ ") dg) { foreach (size_t id, entry; a0[0 .. len])" ~ "if (auto res = dg(entry, "~ SeriesGen1S!("a%s[id]", ", ", TyArrays.length-1, 1) ~ ")) return res; return 0; } "); mixin(" int opApply(int delegate(ref size_t, " ~ SeriesGen1S!("ref T%s", ", ", TyArrays.length-1) ~ ") dg) { foreach (size_t id, entry; a0[0 .. len])" ~ "if (auto res = dg(id, entry, "~ SeriesGen1S!("a%s[id]", ", ", TyArrays.length-1, 1) ~ ")) return res; return 0; } "); } } Limitations: - xzip([]), zip([],[]), etc don't work. - it doesn't work on a single array, like xzip([1, 2, 3]). - So far it only works on static and dynamic arrays (no AAs or iterable objects). Bye, bearophile
Dec 23 2007
Janice Caron Wrote:Walter has stated many times that foreach is a good thing because it expresses the programmer's intent, and leaves the optimisation down to the compiler. (Should it use pointers? Should it use indeces? etc.) I agree with him. However, it is sadly flawed in that you can't iterate through two collections in lockstep. I'm sure that many suggestions have been proposed in the past to work around this limitation, but the bottom line has always been that we're stuck with opApply(), and opApply() cannot be made to loop through two things at once. So... I'd like to suggest a /gradual/ change. It seems to me that this would work without really hurting anything, and programmers could get used to new idioms a little bit at a time. STEP ONE - Make it work for built-in arrays /only/ This one seems pretty straightforward. For built-in arrays, we allow people to do this: int[] a, b, c; foreach(ref x;a)(y;b)(z;c) { x = y * z; } This should present the compiler with no difficulty, because we're /only/ talking about builtin arrays here, and so there's no opApply() to worry about. This will also give us coders a chance to play with it and get used to the idiom. At this point, /some/ structs and classes will be able to add their own elementwise features simply by providing a function which returns an array. For example: Vector!(10,int) a,b,c; foreach(ref x;a.toArray)(y;b.toArray)(z;c.toArray) { x = y * z; } It's not perfect (yet), but it's a step in the right direction. STEP TWO - Allow foreach to recurse into multidimensional arrays This is a pretty nice one. int[][] a; foreach(int[] x;a) { /*elements of a*/ } foreach(int x; a) { /* elements of elements of a*/ } Now we'll be able to add elementwise features to even more structs and classes. For example: Matrix!(10,10,int) a,b,c; foreach(ref int x;a.toArray)(int y;b.toArray)(int z;c.toArray) { x = y * z; } (Yes, I'm aware that that's not doing matrix multiplication, but apparently there is a need to do this). Again, it's not perfect (yet), but it's moving just a little bit closer. STEP THREE - Extend these features to "array-like types". If we consider an "array-like type" to be any class or struct which implements: opIndex() opIndexAssign() length() and/or ptr() end() (with the latter two returning iterators), then I see no reason why these features couldn't also be made to work with arbitrary collections. The rule would be: (1) if we implement opIndex(), opIndexAssign() and length(), use those, else (2) if we implement ptr() and end(), use those, else (3) if we implement opApply(), use that (with all the old limitations), else (4) compile-time error Of course, we don't have iterators yet, so I should probably have added step 2.5, finish implementiing iterators. We already have /most/ of what iterators need: opEquals(), opPostInc(), opPostDec() and opStar() (hopefully to be renamed opDeref()). I think we're still missing opStarAssign() / opDerefAssign(), but once that's in place we'd be good to go. Once step three is in place, structs and classes will no longer need a toArray() function, and (better still) the mechanism will work even for collections which /can't/ return an array, such as linked lists. At this point we'll be able to do List!(Widget) a,b,c; foreach(ref x;a)(y;b)(z;c) { a = b.someFunction(c); } Thoughts?What you're suggesting is perfectly possible to do without any language changes. See ZipIterator in http://zygfryd.net/hg/jive/file/tip/jive/iterators.d ; while you can't iterate two opApply-exposing objects without using inefficient buffering you can always implement some init()/next() protocol in your types and make a special case for it in iterator adaptors like zip/lockstep. (In the example such a special case is implemented for TangoIterator).
Dec 23 2007
On 12/23/07, Hxal <hxal freenode.d.channel> wrote:What you're suggesting is perfectly possible to do without any language changes.Since what I'm suggesting /is/ a language change, it clearly isn't. /Obviously/ one can work around the inconsistencies. It's not hard. For arrays, for example, one can simply write: for (int i=0; i<a.length; ++i) { a[i] = b[i] * c[i]; } (...and in fact, I'll bet good money that that's what most people do). The point is, that might not be the most efficient way of doing it. To get the most efficient method, you really want to let the compiler, not the programmer, choose the "how". That is, after all, the very motivation behind foreach in the first place. I'm not the first to suggest this. I have it on good authority that Andrei suggested the syntax for foreach with multiple loops long before this. But it never got implemented, because Walter couldn't figure out how to make it interact with opApply. This is a suggestion about how to achieve that. The suggestion is, specifically, a language change - *so that the compiler can decide* how best to do it.
Dec 23 2007
Janice Caron wrote:On 12/23/07, Hxal <hxal freenode.d.channel> wrote:I don't think that's the main reason. A more fundumental problem is that requiring a call to opApply on each foreach iteration is a massive performance hit. This is a problem even in a normal use of foreach. Andrei is currently trying to come up with a solution.What you're suggesting is perfectly possible to do without any language changes.Since what I'm suggesting /is/ a language change, it clearly isn't. /Obviously/ one can work around the inconsistencies. It's not hard. For arrays, for example, one can simply write: for (int i=0; i<a.length; ++i) { a[i] = b[i] * c[i]; } (...and in fact, I'll bet good money that that's what most people do). The point is, that might not be the most efficient way of doing it. To get the most efficient method, you really want to let the compiler, not the programmer, choose the "how". That is, after all, the very motivation behind foreach in the first place. I'm not the first to suggest this. I have it on good authority that Andrei suggested the syntax for foreach with multiple loops long before this. But it never got implemented, because Walter couldn't figure out how to make it interact with opApply.
Dec 25 2007
On 12/25/07, Don Clugston <dac nospam.com.au> wrote:Andrei is currently trying to come up with a solution.Cool!
Dec 25 2007
Don Clugston Wrote:I don't think that's the main reason. A more fundumental problem is that requiring a call to opApply on each foreach iteration is a massive performance hit. This is a problem even in a normal use of foreach. Andrei is currently trying to come up with a solution.A dedicated optimization pass for functions taking delegates that is able to inline them for delegates known at compile time would be nice. Especially if it worked for nested delegates such as those used in iterator adaptors. But that might be hard to do with virtual opApply in classes. (PS. opApply gets called once, its delegate argument gets called for each iteration)
Dec 25 2007
Janice Caron wrote:Walter has stated many times that foreach is a good thing because it expresses the programmer's intent, and leaves the optimisation down to the compiler. (Should it use pointers? Should it use indeces? etc.)...Thoughts?Induction variable analysis and reduction in strength also work for foreach ( i; arr1 ) arr2 = arr1[i] + arr3[i]*arr4[i]; no need for special syntax. Instead, foreach ( i; arr1 ) ( j; arr3 ) ( k; arr4 ) arr2 = arr1[i]+arr3[j]*arr4[k]; is ambiguous wrt. the loop condition, since i.g. arr1.length != arr3.length, etc. "Recursing" into multidim. arrays is in fact nesting foreach loops. There is no generality gained. IVA and Loop Invariant Code Motion will behave equally well with manually nested foreach loops. The syntactic effect can be achieved with opApply. Generalizing iteration to any type providing opIndex, etc. won't work because the mere existence of these operators won't tell the compiler how to generate indices. That's what opApply is for. It would work for continuous iterators (like .ptr() and .end()), but then again, that's what opApply is for.
Dec 23 2007
Once step three is in place, structs and classes will no longer need a toArray() function, and (better still) the mechanism will work even for collections which /can't/ return an array, such as linked lists. At this point we'll be able to do List!(Widget) a,b,c; foreach(ref x;a)(y;b)(z;c) { a = b.someFunction(c); } Thoughts?Once we have better support for structs, we can have struct iterators. Iterators are superior to the current approach because they are more efficient. Iterating through multiple collections in lockstep would be possible using a specialized iterator. If anyone wants me to lay out a detailed design I can do it, but otherwise I won't waste my time. As far as letting the compiler decide the most efficient way to do it, I don't think that it's necessary for collections. Struct iterators would be very efficient. -Craig
Dec 23 2007
Reply to Janice,Thoughts?If you can be totally sure of the amount of stack space that will be used you can do 2+ opApplies (*) at the same time. 1) Run the first opApply 2) Push a bunch of stuff on the stack (to make some room) 3) call the other opApply 4) call the real action delegate 5) return from the action delegate called just after step 1 6) when you get back to that delegate then return from the action delegate called after step 3 7) loop at step 4 To make it work you need to have low level access the the stack and frame pointers. I'm not suggesting it be done (it's a bit to low level) but it just struck me that it could be done. *) what is the plural of "opApply"; "opApplies" or "opApplys"?
Dec 23 2007
BCS Wrote:*) what is the plural of "opApply"; "opApplies" or "opApplys"?Technically, the plural would be "opApply operations". Paul
Dec 24 2007
Reply to Paul,BCS Wrote:no, going that way it would be opApply functions or methods, or maybe calls. But I think there is only one operation.*) what is the plural of "opApply"; "opApplies" or "opApplys"?Technically, the plural would be "opApply operations". Paul
Dec 26 2007
BCS Wrote:Reply to Paul,You're right, of course. I was only trying to make the point that for some nouns (usually names) it is difficult to form plurals. They are invariant -- Wait! that's it! Instead of "const" or "manifest" or "pure" or "final" we can call manifest constants "non-pluralizable"! It's so simple! Why didn't anyone think of this before! PaulBCS Wrote:no, going that way it would be opApply functions or methods, or maybe calls. But I think there is only one operation.*) what is the plural of "opApply"; "opApplies" or "opApplys"?Technically, the plural would be "opApply operations". Paul
Dec 27 2007
Reply to Paul,Wait! that's it! Instead of "const" or "manifest" or "pure" or "final" we can call manifest constants "non-pluralizable"! It's so simple! Why didn't anyone think of this before! Paulsingular (?) or how about inline with scope(exit) uses: cingular(tm) But actually that doesn't really cover it, how about "value" (oops I think I just broke my self imposed ambivalence to the whole const thing.)
Dec 27 2007