• Justin Whear (31/31) Jan 03 2014 I've run into a design issue surrounding ranges and am looking for advic...
• monarch_dodra (21/78) Jan 03 2014 Actaully, dynamic arrays *are* output ranges. Each "put" places
• monarch_dodra (3/7) Jan 03 2014 To add to that, it's *kind* of like an input/output stream, that
• Jacob Carlborg (8/26) Jan 04 2014 If you're satisfied with just using arrays and not a general output
• Justin Whear (6/47) Jan 04 2014 I guess my larger concern is providing an interface which avoids

Justin Whear <justin economicmodeling.com> writes:

I've run into a design issue surrounding ranges and am looking for advice 
on the best way to proceed.  To illustrate the issue, consider the 
Shapefile format: a 100 byte header followed by variable-length records.  
The tricky bit is that the header includes a field which contains the 
total length of the file (as measured in 16-bit words, curiously).  The 
header must be written first, but the total length of the file isn't 
known until all the records have been encoded.  When writing to a File 
this isn't a problem: write 100 bytes of padding, write the records, use 
rewind(), and write the proper header.  It's in the context of an 
OutputRange that I don't know how to proceed.  Consider the most flexible 
range type: the array.  An array is not an OutputRange, so it needs to be 
wrapped in something like std.array.Appender.  Ideally I could save off 
the initial state of the range, write a bogus header, write the records, 
then jump back and write the proper header.  Unfortunately, Appender is 
not a ForwardRange, nor does it appear that the field of OutputRanges 
which are also ForwardRanges has been explored.  I'm using the excellent 
read, write, and append functions from std.bitmanip, so write() would fit 
the bill if only Appender supported slicing.

My current solution is require the user to construct the ShapeWriter 
output range (which supports `put(Shape)`) over two separate output 
ranges of ubyte: one for the header and a second for the records, then 
delay writing to the header range until the record range is complete.  
This is both needlessly complex and it leaves the proper combination of 
the two to the user, making ShapeWriter a very leaky abstraction.

So, for particular questions:
1) Am I missing something in Phobos that would provide an OutputRange of 
ubytes while also providing ForwardRange/slicing capabilities?
2) I know the relationship between streams and ranges has been discussed 
at least once before; is the concept of rewinding and overwriting simply 
incompatible with OutputRange in general?

Justin

"monarch_dodra" <monarchdodra gmail.com> writes:

On Friday, 3 January 2014 at 22:56:41 UTC, Justin Whear wrote:
 I've run into a design issue surrounding ranges and am looking 
 for advice
 on the best way to proceed.  To illustrate the issue, consider 
 the
 Shapefile format: a 100 byte header followed by variable-length 
 records.
 The tricky bit is that the header includes a field which 
 contains the
 total length of the file (as measured in 16-bit words, 
 curiously).  The
 header must be written first, but the total length of the file 
 isn't
 known until all the records have been encoded.  When writing to 
 a File
 this isn't a problem: write 100 bytes of padding, write the 
 records, use
 rewind(), and write the proper header.  It's in the context of 
 an
 OutputRange that I don't know how to proceed.  Consider the 
 most flexible
 range type: the array.  An array is not an OutputRange, so it 
 needs to be
 wrapped in something like std.array.Appender.

Actaully, dynamic arrays *are* output ranges. Each "put" places 
the element at the front of the range, and the range is then 
pop'ed front.

Its' not an "expanding" output range, rather, a "fillable" output 
range, so not really something that would fit your need.

 Ideally I could save off
 the initial state of the range, write a bogus header, write the 
 records,
 then jump back and write the proper header.  Unfortunately, 
 Appender is
 not a ForwardRange, nor does it appear that the field of 
 OutputRanges
 which are also ForwardRanges has been explored.  I'm using the 
 excellent
 read, write, and append functions from std.bitmanip, so write() 
 would fit
 the bill if only Appender supported slicing.

 My current solution is require the user to construct the 
 ShapeWriter
 output range (which supports `put(Shape)`) over two separate 
 output
 ranges of ubyte: one for the header and a second for the 
 records, then
 delay writing to the header range until the record range is 
 complete.
 This is both needlessly complex and it leaves the proper 
 combination of
 the two to the user, making ShapeWriter a very leaky 
 abstraction.

 So, for particular questions:
 1) Am I missing something in Phobos that would provide an 
 OutputRange of
 ubytes while also providing ForwardRange/slicing capabilities?

<pedantic>ubyte[] is such a range</pedantic>. It won't do what 
you want though.

 2) I know the relationship between streams and ranges has been 
 discussed
 at least once before; is the concept of rewinding and 
 overwriting simply
 incompatible with OutputRange in general?

 Justin

I think the biggest issue is that there is actaully 0 relation 
between input ranges and output ranges. The two concepts are 
completely orthogonal.

In particular, arguably, input ranges that are also output ranges 
are confusing,  since rather than "growing as you add items to 
their tip (what you'd want)", they instead "shrink as you 
overwrite their front, until they are empty/full".

When using put(range, item), the "put" primitive of "range" takes 
precedence over "front". You could design a range that uses that, 
which would look a bit more like what you want, but there is 
nothing that exists in phobos that does this that I know of 
anyways.

It's a bit of a mess (IMO).

"monarch_dodra" <monarchdodra gmail.com> writes:

On Friday, 3 January 2014 at 23:14:31 UTC, monarch_dodra wrote:
 In particular, arguably, input ranges that are also output 
 ranges are confusing,  since rather than "growing as you add 
 items to their tip (what you'd want)", they instead "shrink as 
 you overwrite their front, until they are empty/full".

To add to that, it's *kind* of like an input/output stream, that 
you can only write to, after you have read it...

Jacob Carlborg <doob me.com> writes:

On 2014-01-03 23:56, Justin Whear wrote:
 I've run into a design issue surrounding ranges and am looking for advice
 on the best way to proceed.  To illustrate the issue, consider the
 Shapefile format: a 100 byte header followed by variable-length records.
 The tricky bit is that the header includes a field which contains the
 total length of the file (as measured in 16-bit words, curiously).  The
 header must be written first, but the total length of the file isn't
 known until all the records have been encoded.  When writing to a File
 this isn't a problem: write 100 bytes of padding, write the records, use
 rewind(), and write the proper header.  It's in the context of an
 OutputRange that I don't know how to proceed.  Consider the most flexible
 range type: the array.  An array is not an OutputRange, so it needs to be
 wrapped in something like std.array.Appender.  Ideally I could save off
 the initial state of the range, write a bogus header, write the records,
 then jump back and write the proper header.  Unfortunately, Appender is
 not a ForwardRange, nor does it appear that the field of OutputRanges
 which are also ForwardRanges has been explored.  I'm using the excellent
 read, write, and append functions from std.bitmanip, so write() would fit
 the bill if only Appender supported slicing.

If you're satisfied with just using arrays and not a general output 
range you can do something similar to what you did with a file. Add the 
100 extra bytes for the header to Appender, then add the contents. 
Extract the array used for backing in the Appender and fill the proper 
header.

-- 
/Jacob Carlborg

"Justin Whear" <justin economicmodeling.com> writes:

On Saturday, 4 January 2014 at 11:22:03 UTC, Jacob Carlborg wrote:
 On 2014-01-03 23:56, Justin Whear wrote:
 I've run into a design issue surrounding ranges and am looking 
 for advice
 on the best way to proceed.  To illustrate the issue, consider 
 the
 Shapefile format: a 100 byte header followed by 
 variable-length records.
 The tricky bit is that the header includes a field which 
 contains the
 total length of the file (as measured in 16-bit words, 
 curiously).  The
 header must be written first, but the total length of the file 
 isn't
 known until all the records have been encoded.  When writing 
 to a File
 this isn't a problem: write 100 bytes of padding, write the 
 records, use
 rewind(), and write the proper header.  It's in the context of 
 an
 OutputRange that I don't know how to proceed.  Consider the 
 most flexible
 range type: the array.  An array is not an OutputRange, so it 
 needs to be
 wrapped in something like std.array.Appender.  Ideally I could 
 save off
 the initial state of the range, write a bogus header, write 
 the records,
 then jump back and write the proper header.  Unfortunately, 
 Appender is
 not a ForwardRange, nor does it appear that the field of 
 OutputRanges
 which are also ForwardRanges has been explored.  I'm using the 
 excellent
 read, write, and append functions from std.bitmanip, so 
 write() would fit
 the bill if only Appender supported slicing.

 If you're satisfied with just using arrays and not a general 
 output range you can do something similar to what you did with 
 a file. Add the 100 extra bytes for the header to Appender, 
 then add the contents. Extract the array used for backing in 
 the Appender and fill the proper header.

I guess my larger concern is providing an interface which avoids 
allocation.  In that case, having an overload which accepts an 
array for writing into and signals if it was too small could 
work.  It is a bit disappointing to lose the nice generic 
interface which would really give the user control.