www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Treating a slice as an InputRange

reply unleashy <lakeavenger gmail.com> writes:
Hello,

I'm writing a small parser for a specific binary format. The 
format is parsed by way of a sequence of functions, each 
deserializing a small portion of the format into a D type such as 
int, double, string, etc., operating on a single InputRange. The 
problem is, if I pass a slice to each of these functions, the 
slice doesn't get mutated by the popFront function used in the 
functions, so something like this:

ubyte[] slice;
...
int a = readInt(slice);
double b = readDouble(slice);

Ends up failing, because readInt properly reads an int, but the 
slice is not mutated "outside" of itself, which means readDouble 
will not read from where readInt stopped, and instead read from 
the start of the slice, and failing because there is an int 
encoded there and not a double. Both functions' signatures are 
like this:

T readXXX(Range)(auto ref Range range) if (isInputRange!Range)

So my question is, is there a way to treat a slice strictly as an 
InputRange, so that it is mutated no matter what? Or is there 
another way to do what I'm trying to do?

I've worked around it using a "wrapper" InputRange struct, but I 
feel like there must be another way.

Thanks!
Nov 15 2017
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, November 15, 2017 20:53:39 unleashy via Digitalmars-d-learn 
wrote:

 Hello,

 I'm writing a small parser for a specific binary format. The
 format is parsed by way of a sequence of functions, each
 deserializing a small portion of the format into a D type such as
 int, double, string, etc., operating on a single InputRange. The
 problem is, if I pass a slice to each of these functions, the
 slice doesn't get mutated by the popFront function used in the
 functions, so something like this:

 ubyte[] slice;
 ...
 int a = readInt(slice);
 double b = readDouble(slice);

 Ends up failing, because readInt properly reads an int, but the
 slice is not mutated "outside" of itself, which means readDouble
 will not read from where readInt stopped, and instead read from
 the start of the slice, and failing because there is an int
 encoded there and not a double. Both functions' signatures are
 like this:

 T readXXX(Range)(auto ref Range range) if (isInputRange!Range)

 So my question is, is there a way to treat a slice strictly as an
 InputRange, so that it is mutated no matter what? Or is there
 another way to do what I'm trying to do?

 I've worked around it using a "wrapper" InputRange struct, but I
 feel like there must be another way.


Typically, functions that operate on ranges either consume them, return a
new range that wrapped the one passed in, or return the original range with
some number of elements popped off. If you specifically want a function to
accept a range and mutate it without returning it, then it should take its
argument by ref. Having it take auto ref is actually quite odd, since that
means that the behavior can depend on whether an lvalue or rvalue is passed
in.

If you can't alter the function's signature, then you could always use
std.range.RefRange to wrap it, but it's better to change the function
signature if you can.

- Jonathan M Davis
Nov 15 2017
parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 11/15/2017 01:02 PM, Jonathan M Davis wrote:

 On Wednesday, November 15, 2017 20:53:39 unleashy via Digitalmars-d-learn



 ubyte[] slice;
 ...
 int a = readInt(slice);
 double b = readDouble(slice);

 Ends up failing, because readInt properly reads an int, but the
 slice is not mutated "outside" of itself




That should work:

import std.range;

T read(T, Range)(auto ref Range range) if (isInputRange!Range) {
     range.popFront();
     return 42;
}

unittest {
     ubyte[] slice = [ 1, 2 ];
     read!int(slice);
     assert(slice == [2]);
}

void main() {
}


 If you specifically want a function to
 accept a range and mutate it without returning it, then it should 


take its

 argument by ref.


Agreed.

Ali
Nov 15 2017
prev sibling next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 11/15/17 3:53 PM, unleashy wrote:


 So my question is, is there a way to treat a slice strictly as an 
 InputRange, so that it is mutated no matter what? Or is there another 
 way to do what I'm trying to do?


I'd model your functions after the std.conv.parse functions:

https://dlang.org/phobos/std_conv.html#parse

This always takes the range by reference, and takes the type to parse 
via a template parameter (more idiomatic D to have parse!int than parseInt).

-Steve
Nov 15 2017
prev sibling parent reply unleashy <lakeavenger gmail.com> writes:
Thanks for the insights everyone, it really helped!

I actually discovered that it wasn't working because one of the 
parsing functions used `std.range.take` and, since I was giving 
it a slice, `take` decided to save the fwdrange instead of 
mutating it. I realised the `take` call was 100% useless, so I 
removed it and it works perfectly now, and refactored to be more 
idiomatic :)

On Wednesday, 15 November 2017 at 21:02:35 UTC, Jonathan M Davis 
wrote:

 If you specifically want a function to accept a range and 
 mutate it without returning it, then it should take its 
 argument by ref. Having it take auto ref is actually quite odd, 
 since that means that the behavior can depend on whether an 
 lvalue or rvalue is passed in.


I was under the impression that templated parameters needed `auto 
ref` to work as `ref` properly. Good to know that's not true.

On Wednesday, 15 November 2017 at 21:11:16 UTC, Steven 
Schveighoffer wrote:

I'd model your functions after the std.conv.parse functions:

https://dlang.org/phobos/std_conv.html#parse

This always takes the range by reference, and takes the type to 
parse via a template parameter (more idiomatic D to have 
parse!int than parseInt).


Yes, that is much more idiomatic than what I was going for. I've 
been writing some Java, so I guess it got to my head :)

On Wednesday, 15 November 2017 at 21:14:04 UTC, Ali Çehreli wrote:

That should work:

import std.range;

T read(T, Range)(auto ref Range range) if (isInputRange!Range) {
    range.popFront();
    return 42;
}

unittest {
    ubyte[] slice = [ 1, 2 ];
    read!int(slice);
    assert(slice == [2]);
}

void main() {
}


That is exactly what I was looking for, thanks!
Nov 15 2017
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, November 15, 2017 22:48:12 unleashy via Digitalmars-d-learn 
wrote:

 On Wednesday, 15 November 2017 at 21:02:35 UTC, Jonathan M Davis
 wrote:

 If you specifically want a function to accept a range and
 mutate it without returning it, then it should take its
 argument by ref. Having it take auto ref is actually quite odd,
 since that means that the behavior can depend on whether an
 lvalue or rvalue is passed in.


 I was under the impression that templated parameters needed `auto
 ref` to work as `ref` properly. Good to know that's not true.


What auto ref does is make it so that the parameter is infered as ref if the
argument is an lvalue and infered as non-ref if it's an rvalue. That way,
lvalues get passed by references, and rvalues get moved. It's really not the
sort of thing you use when you intend to mutate the parameter. It's either
used with the intent of avoiding copying (similar to when you use const T&
in C++), or it's used to forward the refness of the argument (e.g. that's
important with something like emplace, which forwards the arguments to the
constructor of the type being constructed).

Because auto ref generates different template instantiations based on the
refness of the type, it only works with templated functions, but ref in
general doesn't function any differently with templates than it does with
non-templates.

- Jonathan M Davis
Nov 16 2017