www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - static array is not a range

reply Alexibu <alex sunopti.com> writes:
It looks like isInputRange is false for arrays with fixed length 
by design.

I can do:

```d
float[4] arr;
foreach(x;arr)
    writefln("%s",x)
```
but not :

```d
arr.each!(a => a.writefln("%s",a));
```
Is there a good reason for this ?
It took my a long time to figure out.
Jan 09
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, January 9, 2024 3:11:35 AM MST Alexibu via Digitalmars-d-learn 
wrote:

 It looks like isInputRange is false for arrays with fixed length
 by design.

 I can do:

 ```d
 float[4] arr;
 foreach(x;arr)
     writefln("%s",x)
 ```
 but not :

 ```d
 arr.each!(a => a.writefln("%s",a));
 ```
 Is there a good reason for this ?
 It took my a long time to figure out.


How would it even be possible for a static array to be a range? It has a
fixed length. For a type to work as a range, it needs to be possible to pop
elements off of it, which you can't do with a static array. Input ranges
must have front, popFront, and empty. Dynamic arrays have that from
std.range.primitivies via UFCS (Universal Function Call Syntax), and that
works, because it's possible to shrink a dynamic array, but it won't work
with a static array, because its size will be fixed.

Now, what you can do is slice a static array to get a dynamic array which
refers to the static array. And since dynamic arrays work as ranges, you can
use that with range-based functions. That being said, you do then have to be
careful about the dynamic array (or any ranges which wrap it) escaping from
the scope where the static array is, because if the static array goes out of
scope and is destroyed, then any dynamic arrays referring to it will be
referring to invalid memory, and you'll get undefined behavior. So, while
slicing static arrays can be very useful, it needs to be done with caution.

- Jonathan M Davis
Jan 09
parent reply Alexibu <alex sunopti.com> writes:
On Tuesday, 9 January 2024 at 10:44:34 UTC, Jonathan M Davis 
wrote:

 How would it even be possible for a static array to be a range? 
 It has a fixed length. For a type to work as a range, it needs 
 to be possible to pop elements off of it, which you can't do 
 with a static array. Input ranges must have front, popFront, 
 and empty. Dynamic arrays have that from std.range.primitivies 
 via UFCS (Universal Function Call Syntax), and that works, 
 because it's possible to shrink a dynamic array, but it won't 
 work with a static array, because its size will be fixed.

 Now, what you can do is slice a static array to get a dynamic 
 array which refers to the static array. And since dynamic 
 arrays work as ranges, you can use that with range-based 
 functions. That being said, you do then have to be careful 
 about the dynamic array (or any ranges which wrap it) escaping 
 from the scope where the static array is, because if the static 
 array goes out of scope and is destroyed, then any dynamic 
 arrays referring to it will be referring to invalid memory, and 
 you'll get undefined behavior. So, while slicing static arrays 
 can be very useful, it needs to be done with caution.

 - Jonathan M Davis


All good information, thanks.
I suppose I use ranges as things that can be arguments to 
algorithms in std.algorithm and std.range.

Although there is no state in the static array itself as you 
point out, couldn't we have a temporary input range created and 
then the compiler can elide it into whatever the foreach loop 
does.
So all the range algorithms could auto convert a static array 
into a range backed by the static array ?

Something like this : (although written by someone more competent)

```d

struct TempRange(X)
{
    x[n] * array;
    size_t i;
    this(static_array a)
    {
       array = a;
       i = 0;
    }
    X popFront() { return array[i]; }
    bool empty() { return i == array.length;}
}


R each(R,F)(R r,F f)
static if (isInputRange!R)
{
    normal implementation
}else if (isStaticArray!R)
{
    return TempRange(r).each(f);
}
```
Jan 09
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, January 9, 2024 4:13:23 AM MST Alexibu via Digitalmars-d-learn 
wrote:

 On Tuesday, 9 January 2024 at 10:44:34 UTC, Jonathan M Davis

 wrote:

 How would it even be possible for a static array to be a range?
 It has a fixed length. For a type to work as a range, it needs
 to be possible to pop elements off of it, which you can't do
 with a static array. Input ranges must have front, popFront,
 and empty. Dynamic arrays have that from std.range.primitivies
 via UFCS (Universal Function Call Syntax), and that works,
 because it's possible to shrink a dynamic array, but it won't
 work with a static array, because its size will be fixed.

 Now, what you can do is slice a static array to get a dynamic
 array which refers to the static array. And since dynamic
 arrays work as ranges, you can use that with range-based
 functions. That being said, you do then have to be careful
 about the dynamic array (or any ranges which wrap it) escaping
 from the scope where the static array is, because if the static
 array goes out of scope and is destroyed, then any dynamic
 arrays referring to it will be referring to invalid memory, and
 you'll get undefined behavior. So, while slicing static arrays
 can be very useful, it needs to be done with caution.

 - Jonathan M Davis


 All good information, thanks.
 I suppose I use ranges as things that can be arguments to
 algorithms in std.algorithm and std.range.

 Although there is no state in the static array itself as you
 point out, couldn't we have a temporary input range created and
 then the compiler can elide it into whatever the foreach loop
 does.
 So all the range algorithms could auto convert a static array
 into a range backed by the static array ?

 Something like this : (although written by someone more competent)

 ```d

 struct TempRange(X)
 {
     x[n] * array;
     size_t i;
     this(static_array a)
     {
        array = a;
        i = 0;
     }
     X popFront() { return array[i]; }
     bool empty() { return i == array.length;}
 }


 R each(R,F)(R r,F f)
 static if (isInputRange!R)
 {
     normal implementation
 }else if (isStaticArray!R)
 {
     return TempRange(r).each(f);
 }
 ```


If you want a range backed by a static array, simply slice the static array
to get a dynamic array. e.g.

int[5] a = [1, 2, 3, 4, 5];
int[] arr = a[];

However, it's not something that should be done automatically, because
having any kind of pointer or reference to a static array poses the risk of
leaking a pointer or reference to the stack - i.e. the exact same problem
that you get when taking the address of a local variable. The scope
attribute has a limited ability to track escaping references (and DIP 1000
increases those abilities), but ultimately, if you're doing stuff like
passing a dynamic array that's a slice of a static array to range-based
functions, there's a decent chance that the compiler will not be able to
properly detect whether any references to the static array actually escape
(which with DIP 1000 tends to mean errors about not being allowed to do
stuff, because the compiler can't prove that what you're doing won't escape
any references). If you're careful, you can slice a static array and pass
the resulting dynamic array to a range-based function, and it'll work just
fine, but you have to be very careful that no references / pointers to the
static array escape, or you're going to end up referring to memory that used
to be static array but is no longer, which would be a serious problem.

Any user-defined type that you created which was a pointer to a static array
would have the same problem as slicing the static array. If anything, you'd
basically just be implementing a more limited form of D's dynamic arrays
with such a type. Fundamentally, there really isn't a fully safe way to pass
around a pointer to a static array without risking escaping references - not
unless the compiler is smart enough to fully determine whether a reference
might escape, and it's quite difficult for the compiler to be that smart -
particularly when calling functions where the compiler can't necessarily see
the source code.

Ultimately, you really don't want anything to automatically slice a static
array or take its address, because you're risking undefined behavior from
references that escape.

Static arrays are nice in that they provide a way to have an array of
elements without allocating anything on the heap, but if you're going to
start passing them around, pretty quickly, you want a dynamic array that
refers to memory on the heap and not a static array. Slicing static arrays
does provide a middle ground, but it's not completely safe to do so and
really can't be, so having it be done implicitly for you is pretty much just
asking for bugs.

Unfortunately, if you pass a static array to a function that explicitly
takes a dynamic array of the type you get when slicing the static array, the
static array will be sliced automatically for you (which was a design
decision that was made for convenience many years ago without taking into
account how error-prone it is), but DIP 1000 tries to fix that by making it
so that slicing a static array is always scope, meaning that passing a
static array to a function that takes a dynamic array won't work
automatically unless the function paramter is also scope.

Regardless, range-based functions are templated, so they take static arrays
as static arrays and never slice them automatically. So, if you want to pass
a static array to a range-based function, you'll need to slice it explicitly
- and then be careful to make sure that no references to that static array
outlive the static array.

- Jonathan M Davis
Jan 09
prev sibling parent reply bachmeier <no spam.net> writes:
On Tuesday, 9 January 2024 at 10:11:35 UTC, Alexibu wrote:

 It looks like isInputRange is false for arrays with fixed 
 length by design.

 I can do:

 ```d
 float[4] arr;
 foreach(x;arr)
    writefln("%s",x)
 ```
 but not :

 ```d
 arr.each!(a => a.writefln("%s",a));
 ```
 Is there a good reason for this ?
 It took my a long time to figure out.


Jonathan's been giving you good general information about this. 
I'm curious about your partial example. If I fix the writefln 
call, it works.

```
import std;
float[4] arr;
void main() {
   arr[0] = 1;
   arr[1] = 2;
   arr[2] = 3;
   arr[3] = 4;
   arr.each!(a => "%s".writefln(a));
}
```
Jan 09
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, January 9, 2024 6:22:24 AM MST bachmeier via Digitalmars-d-learn 
wrote:

 On Tuesday, 9 January 2024 at 10:11:35 UTC, Alexibu wrote:

 It looks like isInputRange is false for arrays with fixed
 length by design.

 I can do:

 ```d
 float[4] arr;
 foreach(x;arr)

    writefln("%s",x)

 ```
 but not :

 ```d
 arr.each!(a => a.writefln("%s",a));
 ```
 Is there a good reason for this ?
 It took my a long time to figure out.


 Jonathan's been giving you good general information about this.
 I'm curious about your partial example. If I fix the writefln
 call, it works.

 ```
 import std;
 float[4] arr;
 void main() {
    arr[0] = 1;
    arr[1] = 2;
    arr[2] = 3;
    arr[3] = 4;
    arr.each!(a => "%s".writefln(a));
 }
 ```



From the looks of it, each is explicitly designed to work with anything that


can be iterated with foreach rather than just ranges. So, unlike a normal
range-based functions, it will work directly with a static array (and should
take it by ref to avoid copying) - at least so long as the function that
it's given compiles properly. Personally, I'd just use a foreach loop and
don't see much point in each at all, but looking at its implementation, it
does look like the OP should be able to use it with static arrays in spite
of the fact that they're not ranges

- Jonathan M Davis
Jan 09
prev sibling parent reply Alexibu <alex sunopti.com> writes:
On Tuesday, 9 January 2024 at 13:22:24 UTC, bachmeier wrote:

 On Tuesday, 9 January 2024 at 10:11:35 UTC, Alexibu wrote:

 It looks like isInputRange is false for arrays with fixed 
 length by design.

 I can do:

 ```d
 float[4] arr;
 foreach(x;arr)
    writefln("%s",x)
 ```
 but not :

 ```d
 arr.each!(a => a.writefln("%s",a));
 ```
 Is there a good reason for this ?
 It took my a long time to figure out.


 Jonathan's been giving you good general information about this. 
 I'm curious about your partial example. If I fix the writefln 
 call, it works.

 ```
 import std;
 float[4] arr;
 void main() {
   arr[0] = 1;
   arr[1] = 2;
   arr[2] = 3;
   arr[3] = 4;
   arr.each!(a => "%s".writefln(a));
 }
 ```


You're right. My original problem was with the map algorithm, and 
I changed the example to make it simpler thinking all range 
algorithms would be the same.
If each works, I can't see why map filter etc can't work 
consistently where they only need an input range.
```d
auto line = arr.filter!(a > 0).map!(a => 
a.to!string).joiner("\t").text;
```
Should be fine because each result range is passed on the stack 
to the next algorithm, and then at the end the text (or array) 
algorithm doesn't return a range.
Also this should be fine because the ranges are all used on the 
stack.
```d
float[6] array;
string[] result;
auto line = arr.filter!(a > 0).map!(a => a.to!string).each(a => 
result ~= a);
return result;
```

If someone keeps ranges around longer than the static array then 
there are the problems Jonathan is describing.
```d
float[6] array;
auto filtered = arr.filter!(a > 0);
return filtered;
```
I wonder if the compiler could tell if you are only using the 
range as a temporary argument as opposed to assigning it to a 
variable ? Is there that rvalue lvalue distinction in D ?

Obviously anything that adds or removes values won't work, and as 
Jonathan points out returning a slice from find etc would be a 
slice to a static array which could cause problems.

I don't actually use static arrays very often but they can be 
convienient, especially if you were trying to convert maths or 
scientific code from something like matlab or numpy.
Jan 09
parent Siarhei Siamashka <siarhei.siamashka gmail.com> writes:
On Tuesday, 9 January 2024 at 21:30:06 UTC, Alexibu wrote:

 If each works, I can't see why map filter etc can't work 
 consistently where they only need an input range.
 ```d
 auto line = arr.filter!(a > 0).map!(a => 
 a.to!string).joiner("\t").text;
 ```
 Should be fine because each result range is passed on the stack 
 to the next algorithm, and then at the end the text (or array) 
 algorithm doesn't return a range. Also this should be fine 
 because the ranges are all used on the stack.

 [...]

 I wonder if the compiler could tell if you are only using the 
 range as a temporary argument as opposed to assigning it to a 
 variable ?


You can do it like this:
```D
import std;
void main()  trusted {
   float[6] arr = [1, 2, 3, 4, 5, 6];
   auto line = arr[].filter!(a => a > 0).map!(a => 
a.to!string).joiner("\t").text;
   writeln(line);
}
```

The `arr[]` creates a temporary slice. This currently has to be 
run as ` trusted` and you also need to be sure that the temporary 
slice does not escape the scope.
Jan 10