www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - array operations and ranges

reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
Array operations are super cool, and I'm using ranges (which kinda
look and feel like arrays) more and more these days, but I can't help
but feel like their incompatibility with the standard array operations
is a massive loss.

Let's say I want to assign one range to another: b[] = a[];
It's not clear to me why this should fall down if I want to apply a
lazy operation for instance: b[] = a.map!(e=>e*2)[];
... or something to that effect.

I find that my lazy ranges often end up on the stack, but I can't
assign/initialise directly: float[] a = b.transform[];
Instead I need to: float[] a;  b.transform.copy(a[]);

The fact that they don't mix with array expressions or operators means
as soon as a lazy range finds it wants to enter existing array code,
it needs to be converted to a series of map()'s.

There must be years of thoughts and work on this sort of thing?
It seems arrays and ranges are unnecessarily distanced from
eachother... what are the reasons for this?
Apr 26 2015
next sibling parent reply "Laeeth Isharc" <nospamlaeeth nospam.laeeth.com> writes:
On Sunday, 26 April 2015 at 10:17:59 UTC, Manu wrote:

 Array operations are super cool, and I'm using ranges (which 
 kinda
 look and feel like arrays) more and more these days, but I 
 can't help
 but feel like their incompatibility with the standard array 
 operations
 is a massive loss.

 Let's say I want to assign one range to another: b[] = a[];
 It's not clear to me why this should fall down if I want to 
 apply a
 lazy operation for instance: b[] = a.map!(e=>e*2)[];
 ... or something to that effect.

 I find that my lazy ranges often end up on the stack, but I 
 can't
 assign/initialise directly: float[] a = b.transform[];
 Instead I need to: float[] a;  b.transform.copy(a[]);

 The fact that they don't mix with array expressions or 
 operators means
 as soon as a lazy range finds it wants to enter existing array 
 code,
 it needs to be converted to a series of map()'s.

 There must be years of thoughts and work on this sort of thing?
 It seems arrays and ranges are unnecessarily distanced from
 eachother... what are the reasons for this?


Vlad Levenfeld has been doing some interesting work on just this 
sort of thing.  I will see if he is around.
Apr 26 2015
parent reply "Vlad Levenfeld" <vlevenfeld gmail.com> writes:
 On Sunday, 26 April 2015 at 10:17:59 UTC, Manu wrote:

 I find that my lazy ranges often end up on the stack, but I 
 can't
 assign/initialise directly: float[] a = b.transform[];
 Instead I need to: float[] a;  b.transform.copy(a[]);




To enable the first line, builtin arrays would need to be able to 
recognize arbitrary range types and do the right thing. You can 
always do this:

   float[] a = b[].transform.array;

I've got a lib to enable this syntax:

   Array!float a = b[].transform_1;
   a[i..j] = c[x..y].transform_2;

for arbitrary user-defined or composed n-dimensional range types.

here: https://github.com/evenex/autodata

where ranges are made more interoperable by common mixin 
templates which also cut down on a lot of boilerplate.

(see examples in:
https://github.com/evenex/autodata/tree/master/source/spaces)
Apr 26 2015
parent reply "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Sunday, 26 April 2015 at 18:48:15 UTC, Vlad Levenfeld wrote:

 I've got a lib to enable this syntax:

   Array!float a = b[].transform_1;
   a[i..j] = c[x..y].transform_2;


Phobos containers already support the first line, and it would be 
a natural extension to make them support the second.
Apr 26 2015
parent reply "Vlad Levenfeld" <vlevenfeld gmail.com> writes:
 Phobos containers already support the first line, and it would 
 be a natural extension to make them support the second.


Sure, it's not complicated. It's something I had done in this 
other code and showing for example.
Apr 26 2015
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 27 April 2015 at 15:58, Vlad Levenfeld via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 Phobos containers already support the first line, and it would be a
 natural extension to make them support the second.



 Sure, it's not complicated. It's something I had done in this other code and
 showing for example.



Yeah, see I don't feel making a simple thing like an array into
something more complex by wrapping it in templates is ever a good
thing to do.
I just think it's a missed opportunity that the compiler doesn't
support any of this in the language.

It would appear at face value to be a great opportunity for lowering.
Assignment can lower to .copy(), operators can lower to map!(...)

I can tell you, if I tried to explain to my colleagues that we should
wrap an array in a template so assignment works, they would laugh at
me, then ridicule me, and then they would dismiss D. Better to say
it's not supported than to show them that approach.
Apr 26 2015
next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 27 April 2015 at 06:52:11 UTC, Manu wrote:

 On 27 April 2015 at 15:58, Vlad Levenfeld via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 Phobos containers already support the first line, and it 
 would be a
 natural extension to make them support the second.



 Sure, it's not complicated. It's something I had done in this 
 other code and
 showing for example.



 Yeah, see I don't feel making a simple thing like an array into
 something more complex by wrapping it in templates is ever a 
 good
 thing to do.
 I just think it's a missed opportunity that the compiler doesn't
 support any of this in the language.

 It would appear at face value to be a great opportunity for 
 lowering.
 Assignment can lower to .copy(), operators can lower to 
 map!(...)


It's similar to foreach, which already recognizes input ranges, 
so why not...
Apr 27 2015
prev sibling next sibling parent "Vlad Levenfeld" <vlevenfeld gmail.com> writes:
On Monday, 27 April 2015 at 06:52:11 UTC, Manu wrote:

 On 27 April 2015 at 15:58, Vlad Levenfeld via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 Phobos containers already support the first line, and it 
 would be a
 natural extension to make them support the second.



 Sure, it's not complicated. It's something I had done in this 
 other code and
 showing for example.



 Yeah, see I don't feel making a simple thing like an array into
 something more complex by wrapping it in templates is ever a 
 good
 thing to do.
 I just think it's a missed opportunity that the compiler doesn't
 support any of this in the language.

 It would appear at face value to be a great opportunity for 
 lowering.
 Assignment can lower to .copy(), operators can lower to 
 map!(...)

 I can tell you, if I tried to explain to my colleagues that we 
 should
 wrap an array in a template so assignment works, they would 
 laugh at
 me, then ridicule me, and then they would dismiss D. Better to 
 say
 it's not supported than to show them that approach.


Yeah, it is an excessive solution. I guess just wanted to talk 
about my thing and didn't think it through. I think laughing at 
and ridiculing are the same thing though. Anyway, lowering to 
copy built-in to the language, at least, would be pretty great. 
Especially if it worked like foreach and picked up local symbols, 
so that custom copy and map implementations could be used 
wherever they were defined.
Apr 27 2015
prev sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 27 April 2015 at 06:52:11 UTC, Manu wrote:

 On 27 April 2015 at 15:58, Vlad Levenfeld via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 Phobos containers already support the first line, and it 
 would be a
 natural extension to make them support the second.



 Sure, it's not complicated. It's something I had done in this 
 other code and
 showing for example.



 Yeah, see I don't feel making a simple thing like an array into
 something more complex by wrapping it in templates is ever a 
 good
 thing to do.
 I just think it's a missed opportunity that the compiler doesn't
 support any of this in the language.

 It would appear at face value to be a great opportunity for 
 lowering.
 Assignment can lower to .copy(), operators can lower to 
 map!(...)


builtin slicesopSliceAssign (and opSliceOpAssign) understanding 
ranges as source operands is a good idea. I might even see if I 
can implement it.

Lowering array operations to lazy ranges seems like a huge can of 
worms. Not so keen. Lazy array ops are great, but I don't see it 
working out as a builtin feature unless it had it's own syntax.
Apr 27 2015
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 28 April 2015 at 06:42, John Colvin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 27 April 2015 at 06:52:11 UTC, Manu wrote:

 On 27 April 2015 at 15:58, Vlad Levenfeld via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 Phobos containers already support the first line, and it would be a
 natural extension to make them support the second.




 Sure, it's not complicated. It's something I had done in this other code
 and
 showing for example.




 Yeah, see I don't feel making a simple thing like an array into
 something more complex by wrapping it in templates is ever a good
 thing to do.
 I just think it's a missed opportunity that the compiler doesn't
 support any of this in the language.

 It would appear at face value to be a great opportunity for lowering.
 Assignment can lower to .copy(), operators can lower to map!(...)



 builtin slicesopSliceAssign (and opSliceOpAssign) understanding ranges as
 source operands is a good idea. I might even see if I can implement it.

 Lowering array operations to lazy ranges seems like a huge can of worms. Not
 so keen. Lazy array ops are great, but I don't see it working out as a
 builtin feature unless it had it's own syntax.


The only reason I considered it workable was because an operator
expression is generally meaningless without an assignment to the left;
"a[] = b[]*2;", so the mul could conceivably be expressed in a lazy
sense, since it will be evaluated immediately by the assignment.

Array operations passed as arguments to functions are already not
allowed in the usual case, so I think it would be fine in this
instance... it's impossibly for an array operation to have any meaning
without a final assignment (I think?)
Apr 27 2015
prev sibling next sibling parent "Vlad Levenfeld" <vlevenfeld gmail.com> writes:
Manu, I just saw your other post clarifying the code was float[N] 
a = ..., not float[] a. That changes things a bit.

I just implemented a static array type in the lib (1-d only for 
now) which can do the following:

unittest {
	import std.range: only;

	StaticArray!(int, 2) x;

	assert (x[] == [0, 0]);

	x[0..2] = only (5, 6);

	assert (x[] == [5, 6]);

	x[] += 5;

	assert (x[] == [10, 11]);

	x[0..1] -= 5;

	assert (x[] == [5, 11]);

	StaticArray!(int, 4) y = only (1,2,3,4);

	assert (y[] == [1, 2, 3, 4]);

	auto z = only (9,8,7).static_array!([3]);

	assert (z[] == [9, 8, 7]);
}


Assertions are thrown if the assigned range doesn't match the 
static array length.

https://github.com/evenex/autodata/blob/master/source/spaces/array.d
Apr 26 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/26/2015 3:17 AM, Manu via Digitalmars-d wrote:

 Array operations are super cool, and I'm using ranges (which kinda
 look and feel like arrays) more and more these days, but I can't help
 but feel like their incompatibility with the standard array operations
 is a massive loss.

 Let's say I want to assign one range to another: b[] = a[];
 It's not clear to me why this should fall down if I want to apply a
 lazy operation for instance: b[] = a.map!(e=>e*2)[];
 ... or something to that effect.

 I find that my lazy ranges often end up on the stack, but I can't
 assign/initialise directly: float[] a = b.transform[];
 Instead I need to: float[] a;  b.transform.copy(a[]);

 The fact that they don't mix with array expressions or operators means
 as soon as a lazy range finds it wants to enter existing array code,
 it needs to be converted to a series of map()'s.

 There must be years of thoughts and work on this sort of thing?
 It seems arrays and ranges are unnecessarily distanced from
 eachother... what are the reasons for this?


I think you've got the start of some dazz ideas. I want to think about this
more.
Apr 30 2015
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 1 May 2015 at 15:10, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 4/26/2015 3:17 AM, Manu via Digitalmars-d wrote:

 There must be years of thoughts and work on this sort of thing?
 It seems arrays and ranges are unnecessarily distanced from
 eachother... what are the reasons for this?



 I think you've got the start of some dazz ideas. I want to think about this
 more.


Cool. I'm finding myself using ranges more and more in my code these
days. My current projects are almost exclusively range based code.
The most problem cases are:
  float[100] stackArray = range.map!(e => e*2)[]; // <- I often have a
lazy operation that needs to be resolved to the stack. It's sad to
break this on to another line and use .copy()

Other cases come when I have a standard array operation, for instance:
  a[] = b[]*2;
I often want to apply an operation that is expressed via a lazy range,
ie, map!() or something. As soon as you do that, you need to rearrange
the code:
  a[] = b.amplitude[]*2; // <- where amplitude is a function that
returns map!(e=>20*log10(abs(e)))
Or should it be:
  a[] = b[].amplitude*2;// ?
Either way, it must be refactored to keep the compiler happy:
  b[].amplitude.map!(e=>e*2).copy(a);

I think that's a lot less easy to follow. I would try to avoid that
compared to the line above, but that creates a mutual exclusion with
the use of ranges and simple, readable code.


My experience is revealing to me that about 60% of my lines (in
range-related code) are map!(e=>...), and that's a lot of boilerplate
surrounding '...', obscuring the actual operation.
I also find the paren nesting gets pretty deep (for instance, the map
above) and the code is harder to follow than a traditional foreach
loop.
If we can do normal array operations on ranges, and have them lower to
sensible things (like map), I think that sugar will make range
programming a lot more succinct.

Perhaps my 'amplitude' function above could just be a scalar function?
I have one of them too, there are duplicates of most things because I
need to supply a range based version in parallel to the scalar
versions :/
Using UFCS; range.scalarFunction could be recognised as a
range-vs-scalar operation, the same as range*2. Ie, if there are no
proper overload/template matches, the compiler could try and lower to
map(e=>e.scalarFunction), assuming ElementType!Range is compatible
with the scalar function. That would eliminate most instances of
map!(), which are the highest contributor to pointless visual noise in
my current code.

Anyway, I think there's a lot of low-hanging fruit in this area, and I
think range based programming is proving to be one of the most
compelling (and surprisingly awesome) parts of D. With a little polish
it could become a major selling point.
May 02 2015