www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Fix transposed ranges

reply Cym13 <cpicard openmailbox.org> writes:
Hi, I've got the following code that takes a list of files as 
argument and xor them together (demo example sufficient for that 
discussion).

     import std.stdio;
     import std.array;
     import std.range;
     import std.algorithm;

     auto rawContent(string path) {
         return File(path).byChunk(16384).joiner;
     }

     void main(string[] args) {
         args[1..$]
             .map!rawContent
             .array
             .transposed
             .map!(bytes => bytes.fold!((a, b) => a^b))
             .writeln;
     }

This works but compiles with deprecations:

/usr/include/dlang/dmd/std/algorithm/iteration.d(663): 
Deprecation: function `std.range.Transposed!(Result[], 
cast(TransverseOptions)0).Transposed.save` is deprecated - This 
function is incorrect and will be removed November 2018. See the 
docs for more details.
/usr/include/dlang/dmd/std/algorithm/iteration.d(663): 
Deprecation: function `std.range.Transposed!(Result[], 
cast(TransverseOptions)0).Transposed.save` is deprecated - This 
function is incorrect and will be removed November 2018. See the 
docs for more details.

What do you think the best course of action would be to make that 
code resilient to the modification of transposed? I'd very *very* 
much like to keep a straight UFCS line, there's really nothing 
complicated enough with this operation to warrant writting 
boilerplate, yet the straightforward solution isn't 
future-proof...

Any idea?
Mar 30 2018
parent reply Cym13 <cpicard openmailbox.org> writes:
On Friday, 30 March 2018 at 20:43:09 UTC, Cym13 wrote:

 Hi, I've got the following code that takes a list of files as 
 argument and xor them together (demo example sufficient for 
 that discussion).

 [...]


Forgot to mention but I'm also quite annoyed at the need for that 
".array" because "transposed" requires the RoR to be assignable. 
That kills the laziness. I'm very much open to suggestions 
regarding that point.
Mar 30 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/30/18 4:45 PM, Cym13 wrote:

 On Friday, 30 March 2018 at 20:43:09 UTC, Cym13 wrote:

 Hi, I've got the following code that takes a list of files as argument 
 and xor them together (demo example sufficient for that discussion).

 [...]


 
 Forgot to mention but I'm also quite annoyed at the need for that 
 ".array" because "transposed" requires the RoR to be assignable. That 
 kills the laziness. I'm very much open to suggestions regarding that point.


1. The .save deprecation may not affect you. It's probably being used by 
map or fold but may not need to be. Once it's removed, you may just see 
the warning go away, and everything still works just fine. I'm not 100% 
sure on this, as I don't know where it's being used.

2. The array is necessary, as map is lazy. what you want is a range of 
the first byte of each file, then a range of the second byte of each 
file, etc. mapping to a byte array can't possibly do this, because what 
would happen is that map would re-open the file, re-read it's contents, 
and then give you the *second* byte. This is horribly inefficient.

But you can probably reduce the memory requirements by streaming each 
file's bytes as you need it. Unfortunately, I don't see a 'byByte' 
method on File, so you may have to look elsewhere for that.

-Steve
Apr 02 2018
parent reply Cym13 <cpicard openmailbox.org> writes:
On Monday, 2 April 2018 at 18:33:25 UTC, Steven Schveighoffer 
wrote:

 On 3/30/18 4:45 PM, Cym13 wrote:

 On Friday, 30 March 2018 at 20:43:09 UTC, Cym13 wrote:

 Hi, I've got the following code that takes a list of files as 
 argument and xor them together (demo example sufficient for 
 that discussion).

 [...]


 
 Forgot to mention but I'm also quite annoyed at the need for 
 that ".array" because "transposed" requires the RoR to be 
 assignable. That kills the laziness. I'm very much open to 
 suggestions regarding that point.


 1. The .save deprecation may not affect you. It's probably 
 being used by map or fold but may not need to be. Once it's 
 removed, you may just see the warning go away, and everything 
 still works just fine. I'm not 100% sure on this, as I don't 
 know where it's being used.


That's good to hear, although I don't realy like warnings on 
which I have no control.


 2. The array is necessary, as map is lazy. what you want is a 
 range of the first byte of each file, then a range of the 
 second byte of each file, etc. mapping to a byte array can't 
 possibly do this, because what would happen is that map would 
 re-open the file, re-read it's contents, and then give you the 
 *second* byte. This is horribly inefficient.


While I agree that using an array is ugly, and that I want ranges 
of first byte (which is why I'm using transposed in the first 
place), transposed just doesn't let me work with the result of 
map itself. I suppose it's because its signature stipulates 
hasAssignableElements. I feel like I'm missing something there 
but I can't see what.


 But you can probably reduce the memory requirements by 
 streaming each file's bytes as you need it. Unfortunately, I 
 don't see a 'byByte' method on File, so you may have to look 
 elsewhere for that.

 -Steve
Apr 02 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/2/18 3:24 PM, Cym13 wrote:

 On Monday, 2 April 2018 at 18:33:25 UTC, Steven Schveighoffer wrote:

 On 3/30/18 4:45 PM, Cym13 wrote:

 On Friday, 30 March 2018 at 20:43:09 UTC, Cym13 wrote:

 Hi, I've got the following code that takes a list of files as 
 argument and xor them together (demo example sufficient for that 
 discussion).

 [...]


 Forgot to mention but I'm also quite annoyed at the need for that 
 ".array" because "transposed" requires the RoR to be assignable. That 
 kills the laziness. I'm very much open to suggestions regarding that 
 point.


 1. The .save deprecation may not affect you. It's probably being used 
 by map or fold but may not need to be. Once it's removed, you may just 
 see the warning go away, and everything still works just fine. I'm not 
 100% sure on this, as I don't know where it's being used.


 
 That's good to hear, although I don't realy like warnings on which I 
 have no control.


Well, it's tough, because you can compose ranges in infinite ways. All 
you need to generate the warning is some code like this:

static if(is(typeof(r.save)) r.save;

And now, even if you're not supposed to use it, it gets used. But if 
there is a fallback implementation when save *isn't* available, then it 
will still work when the deprecation becomes an error.

Caveat here is that I don't know if this is a case in your example. But 
if it is working how you expect (and I believe it does), then it should 
continue to work. If not, file an issue when that happens.


 2. The array is necessary, as map is lazy. what you want is a range of 
 the first byte of each file, then a range of the second byte of each 
 file, etc. mapping to a byte array can't possibly do this, because 
 what would happen is that map would re-open the file, re-read it's 
 contents, and then give you the *second* byte. This is horribly 
 inefficient.


 
 While I agree that using an array is ugly, and that I want ranges of 
 first byte (which is why I'm using transposed in the first place), 
 transposed just doesn't let me work with the result of map itself. I 
 suppose it's because its signature stipulates hasAssignableElements. I 
 feel like I'm missing something there but I can't see what.


Let's simplify it into an array of arrays:

int[][] x = [[1,2,3],[4,5,6],[7,8,9]];

In order to progress the transposed wrapper, you need to "popFront" each 
element in the array. Which means you need to traverse the array, 
getting each element, getting the front of *that* element, and then 
popping it off and assigning it back. In other words, after one 
popFront, it should look like this:

[[2,3],[5,6],[8,9]]

If you can't assign the range back to itself, then it can't be made to 
progress -- you would simply just get the first front over and over again.

The fact that you have to reassign the elements belies why save can 
never be made to work -- you have to overwrite the original!

-Steve
Apr 02 2018
parent Cym13 <cpicard openmailbox.org> writes:
On Monday, 2 April 2018 at 19:45:31 UTC, Steven Schveighoffer 
wrote:

 On 4/2/18 3:24 PM, Cym13 wrote:

 [...]


 Well, it's tough, because you can compose ranges in infinite 
 ways. All you need to generate the warning is some code like 
 this:

 [...]


That makes sense, thanks.
Apr 02 2018