www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - .array changes the order

reply xtreak <tir.karthi gmail.com> writes:
I came across the issue where using .array after .joiner caused 
the changes to the output. The program is at 
https://dpaste.dzfl.pl/0885ba2eddb4 . I tried to debug through 
the output but I couldn't get the exact issue. It will be helpful 
if someone confirms this as a bug.
May 12 2016
parent reply thedeemon <dlang thedeemon.com> writes:
On Thursday, 12 May 2016 at 09:44:39 UTC, xtreak wrote:

 I came across the issue where using .array after .joiner caused 
 the changes to the output. The program is at 
 https://dpaste.dzfl.pl/0885ba2eddb4 . I tried to debug through 
 the output but I couldn't get the exact issue. It will be 
 helpful if someone confirms this as a bug.


It's not a bug per se, just probably documentation being not 
clear enough. Essentially it's the same issue as with 
File.byLine: a range that mutates its state, so it's ok to 
consume lazily but if you just store references (as array() does) 
you get references to the same mutated value as result.

Quick fix:

   r.chunks(2).map!permutations.joiner.map!array.array.writeln;

i.e. copy each permutation to separate array before it's gone.
May 12 2016
parent reply xtreak <tir.karthi gmail.com> writes:
On Thursday, 12 May 2016 at 10:02:46 UTC, thedeemon wrote:

 On Thursday, 12 May 2016 at 09:44:39 UTC, xtreak wrote:

 I came across the issue where using .array after .joiner 
 caused the changes to the output. The program is at 
 https://dpaste.dzfl.pl/0885ba2eddb4 . I tried to debug through 
 the output but I couldn't get the exact issue. It will be 
 helpful if someone confirms this as a bug.


 It's not a bug per se, just probably documentation being not 
 clear enough. Essentially it's the same issue as with 
 File.byLine: a range that mutates its state, so it's ok to 
 consume lazily but if you just store references (as array() 
 does) you get references to the same mutated value as result.

 Quick fix:

   r.chunks(2).map!permutations.joiner.map!array.array.writeln;

 i.e. copy each permutation to separate array before it's gone.


Thanks a lot. Can you kindly elaborate a little more on 
File.byLine with an example of the scenario so that I don't get 
bitten by it. File.byLine.array works as expected for me. A 
little more explanation on the permutations will also be helpful 
since joiner.map!array converts the subranges to arrays, so does 
joiner work by mutating state if so how does it do.
May 12 2016
next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 5/12/16 6:17 AM, xtreak wrote:

 On Thursday, 12 May 2016 at 10:02:46 UTC, thedeemon wrote:

 On Thursday, 12 May 2016 at 09:44:39 UTC, xtreak wrote:

 I came across the issue where using .array after .joiner caused the
 changes to the output. The program is at
 https://dpaste.dzfl.pl/0885ba2eddb4 . I tried to debug through the
 output but I couldn't get the exact issue. It will be helpful if
 someone confirms this as a bug.


 It's not a bug per se, just probably documentation being not clear
 enough. Essentially it's the same issue as with File.byLine: a range
 that mutates its state, so it's ok to consume lazily but if you just
 store references (as array() does) you get references to the same
 mutated value as result.

 Quick fix:

   r.chunks(2).map!permutations.joiner.map!array.array.writeln;

 i.e. copy each permutation to separate array before it's gone.


 Thanks a lot. Can you kindly elaborate a little more on File.byLine with
 an example of the scenario so that I don't get bitten by it.
 File.byLine.array works as expected for me. A little more explanation on
 the permutations will also be helpful since joiner.map!array converts
 the subranges to arrays, so does joiner work by mutating state if so how
 does it do.


File.byLine.array will create an array of references to the same buffer, 
which is rewritten as byLine iterates. There can be cases where it 
works, but there are definitely those for which it won't work.

Basically, you are keeping references to the buffer for buffered i/o.

I think there is a byLineCopy which works.

-Steve
May 12 2016
prev sibling parent reply thedeemon <dlang thedeemon.com> writes:
On Thursday, 12 May 2016 at 10:17:19 UTC, xtreak wrote:


 Thanks a lot. Can you kindly elaborate a little more on 
 File.byLine with an example of the scenario so that I don't get 
 bitten by it. File.byLine.array works as expected for me. A 
 little more explanation on the permutations will also be 
 helpful since joiner.map!array converts the subranges to 
 arrays, so does joiner work by mutating state if so how does it 
 do.


With some ranges the value they return by .front is only valid 
until the next call to popFront(). For example, File.byLine() 
reuses its buffer so after popFront() is called the buffer 
contains different data and if you had references to the contents 
of previously returned value they become invalid. This is why 
byLineCopy() was added. In the same vein permutations() returns a 
range that has a mutable array of indices inside, which changes 
every time popFront() is called, and since every value returned 
by its .front refers to that indices array, if you collect such 
permutations they will all use the same array of indices and show 
the same order of elements, the same permutation.
Because of this mutable nature of some ranges it's important to 
understand in which order calls to .front and .popFront() happen. 
The array() function calls popFront on its input in  a loop, 
consuming the mutable ranges, while map() does not. So in
  r.map!f.array
function f will receive different valid values, before they get 
invalidated in the loop of array(). But if they contain 
references to something mutable, it makes sense to make copies 
before the thing they refer to mutates.
May 12 2016
parent xtreak <tir.karthi gmail.com> writes:
On Thursday, 12 May 2016 at 13:43:10 UTC, thedeemon wrote:

 On Thursday, 12 May 2016 at 10:17:19 UTC, xtreak wrote:


 Thanks a lot. Can you kindly elaborate a little more on 
 File.byLine with an example of the scenario so that I don't 
 get bitten by it. File.byLine.array works as expected for me. 
 A little more explanation on the permutations will also be 
 helpful since joiner.map!array converts the subranges to 
 arrays, so does joiner work by mutating state if so how does 
 it do.


 With some ranges the value they return by .front is only valid 
 until the next call to popFront(). For example, File.byLine() 
 reuses its buffer so after popFront() is called the buffer 
 contains different data and if you had references to the 
 contents of previously returned value they become invalid. This 
 is why byLineCopy() was added. In the same vein permutations() 
 returns a range that has a mutable array of indices inside, 
 which changes every time popFront() is called, and since every 
 value returned by its .front refers to that indices array, if 
 you collect such permutations they will all use the same array 
 of indices and show the same order of elements, the same 
 permutation.
 Because of this mutable nature of some ranges it's important to 
 understand in which order calls to .front and .popFront() 
 happen. The array() function calls popFront on its input in  a 
 loop, consuming the mutable ranges, while map() does not. So in
  r.map!f.array
 function f will receive different valid values, before they get 
 invalidated in the loop of array(). But if they contain 
 references to something mutable, it makes sense to make copies 
 before the thing they refer to mutates.


Thanks a lot for the info that really clears a lot of things for 
me :) I see from the source that byLineCopy makes a .dup on each 
line and cached at 
https://github.com/dlang/phobos/blob/master/std/stdio.d#L2029-L2034 with
gotFront when front is called a lot of times.  I could see a function called
emplaceref being used in array. Any ideas on that please.
May 12 2016