• Dennis (19/19) Feb 07 2020 If I have an input range with element type `int[3]`, how do I
• Steven Schveighoffer (22/42) Feb 07 2020 This is correct. Consider that you did not mark x as ref. So what x
• Dennis (4/6) Feb 07 2020 Thanks. I was hoping something like that existed in Phobos, but I
• nullptr (3/23) Feb 07 2020 Depending on how your range is structured, it might be possible
• Dennis (19/21) Feb 07 2020 That's a good one. I can't make front() return by ref, but I can
• Steven Schveighoffer (5/32) Feb 07 2020 S.popFront is not @safe, and S is not a template. So no inferrence.
• Dennis (17/19) Feb 07 2020 Oops, minimized a bit too much. Corrected test case:
• nullptr (21/36) Feb 07 2020 ```
• ag0aep6g (22/47) Feb 07 2020 That shouldn't compile. You have found a hole in DIP 1000.
• Steven Schveighoffer (8/60) Feb 07 2020 The original code is not invalid though. f is not valid, and that is a
• ag0aep6g (54/60) Feb 07 2020 Or DIP 1000 just doesn't allow that kind of code. @safe (including DIP
• ag0aep6g (10/52) Feb 07 2020 I went a bit overboard with that second reduction. The original struct
• ag0aep6g (4/26) Feb 08 2020 My attempt at lifting this limitation:
• Steven Schveighoffer (8/26) Feb 08 2020 Again, the limitations of dip1000 are apparent. Why can't I mark p as
• ag0aep6g (6/9) Feb 08 2020 I 100% agree that DIP 1000 is hard to reason about. It's pretty limited
• Steven Schveighoffer (16/27) Feb 08 2020 I unfortunately don't. I suspect that making scope a type constructor
• Steven Schveighoffer (10/35) Feb 07 2020 I tried the AST button. for the front function I got:

Dennis <dkorpel gmail.com> writes:

If I have an input range with element type `int[3]`, how do I 
easily turn it into a range of `int` so I can map it?
If it were an int[3][] I could simply cast it to an int[] before 
mapping, but I don't want to eagerly turn it into an array.
I thought of doing this:
```
range.map!(x => x[]).joiner.map!(x => x*2);
```

But it gives:
Error: returning x[] escapes a reference to parameter x, perhaps 
annotate with return

I tried doing:
```
map!((return x) => x[]) // same error
map!((return ref x) => x[]) // does not match, map.front is not 
an lvalue
```

I can easily work around it with some more code, but I wonder if 
someone knows an easy solution.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 2/7/20 3:13 PM, Dennis wrote:
 If I have an input range with element type `int[3]`, how do I easily 
 turn it into a range of `int` so I can map it?
 If it were an int[3][] I could simply cast it to an int[] before 
 mapping, but I don't want to eagerly turn it into an array.
 I thought of doing this:
 ```
 range.map!(x => x[]).joiner.map!(x => x*2);
 ```
 
 But it gives:
 Error: returning x[] escapes a reference to parameter x, perhaps 
 annotate with return

This is correct. Consider that you did not mark x as ref. So what x 
actually is is a function local. If you returned x[], you just returned 
a reference to a local, which is about to go out of scope.

 I tried doing:
 ```
 map!((return x) => x[]) // same error
 map!((return ref x) => x[]) // does not match, map.front is not an lvalue
 ```
 
 I can easily work around it with some more code, but I wonder if someone 
 knows an easy solution.

This means your actual input range is not an array (map should forward 
the lvalueness of it). This means you can't just slice, as you will 
again return references to the local stack frame.

The only solution I can provide is to wrap the static array into a range 
(maybe something like this exists in Phobos?):

struct SARange(T, size_t N)
{
    T[N] storage;
    size_t elem;
    auto front() { return storage[elem]; }
    void popFront() { ++elem; }
    bool empty() { return elem >= N; }
}

auto asRange(T : E[N], E, size_t N)(T val) {
    return SARange!(E, N)(val);
}

range.map!(x => x.asRange).joiner.map!(x => x*2);

-Steve

Dennis <dkorpel gmail.com> writes:

On Friday, 7 February 2020 at 20:31:47 UTC, Steven Schveighoffer 
wrote:
 The only solution I can provide is to wrap the static array 
 into a range (maybe something like this exists in Phobos?):

Thanks. I was hoping something like that existed in Phobos, but I 
can't find anything.

nullptr <null p.tr> writes:

On Friday, 7 February 2020 at 20:13:57 UTC, Dennis wrote:
 If I have an input range with element type `int[3]`, how do I 
 easily turn it into a range of `int` so I can map it?
 If it were an int[3][] I could simply cast it to an int[] 
 before mapping, but I don't want to eagerly turn it into an 
 array.
 I thought of doing this:
 ```
 range.map!(x => x[]).joiner.map!(x => x*2);
 ```

 But it gives:
 Error: returning x[] escapes a reference to parameter x, 
 perhaps annotate with return

 I tried doing:
 ```
 map!((return x) => x[]) // same error
 map!((return ref x) => x[]) // does not match, map.front is not 
 an lvalue
 ```

 I can easily work around it with some more code, but I wonder 
 if someone knows an easy solution.

Depending on how your range is structured, it might be possible 
to just mark front as returning by ref to make this work.

Dennis <dkorpel gmail.com> writes:

On Friday, 7 February 2020 at 20:55:14 UTC, nullptr wrote:
 Depending on how your range is structured, it might be possible 
 to just mark front as returning by ref to make this work.

That's a good one. I can't make front() return by ref, but I can 
make front a member variable of the range struct. Only problem:
 safe function ... cannot call  system function 
std.algorithm.iteration.joiner!(...).joiner

I don't know why. I don't have time to delve into this at the 
moment but if anyone wants to try, here's a minimal testcase:

```
import std;

struct S {
     int[3] front = [10, 20, 30];
     bool empty = false;
     void popFront() {empty = true;}
}

void main()  safe {
     S.init.map!((return ref x) => x[]).joiner.writeln;
}
```
flags: -dip1000 -dip25

Steven Schveighoffer <schveiguy gmail.com> writes:

On 2/7/20 4:17 PM, Dennis wrote:
 On Friday, 7 February 2020 at 20:55:14 UTC, nullptr wrote:
 Depending on how your range is structured, it might be possible to 
 just mark front as returning by ref to make this work.

 
 That's a good one. I can't make front() return by ref, but I can make 
 front a member variable of the range struct. Only problem:
  safe function ... cannot call  system function 
 std.algorithm.iteration.joiner!(...).joiner
 
 I don't know why. I don't have time to delve into this at the moment but 
 if anyone wants to try, here's a minimal testcase:
 
 ```
 import std;
 
 struct S {
      int[3] front = [10, 20, 30];
      bool empty = false;
      void popFront() {empty = true;}
 }
 
 void main()  safe {
      S.init.map!((return ref x) => x[]).joiner.writeln;
 }
 ```
 flags: -dip1000 -dip25
 


S.popFront is not  safe, and S is not a template. So no inferrence.

But I did that, and it still doesn't compile. Not sure why not. Another 
fun dip1000 problem...

-Steve

Dennis <dkorpel gmail.com> writes:

On Friday, 7 February 2020 at 21:40:36 UTC, Steven Schveighoffer 
wrote:
 S.popFront is not  safe, and S is not a template. So no 
 inferrence.

Oops, minimized a bit too much. Corrected test case:
```
import std;

struct S {
 safe:
      int[3] front = [10, 20, 30];
      bool empty = false;
      void popFront() {empty = true;}
}

void main()  safe {
      S.init.map!((return ref x) => x[]).joiner.writeln;
}
```
It indeed still errors with -dip1000, but without -dip1000 it 
compiles now interestingly.

nullptr <null p.tr> writes:

On Friday, 7 February 2020 at 22:55:29 UTC, Dennis wrote:
 Oops, minimized a bit too much. Corrected test case:
 ```
 import std;

 struct S {
  safe:
      int[3] front = [10, 20, 30];
      bool empty = false;
      void popFront() {empty = true;}
 }

 void main()  safe {
      S.init.map!((return ref x) => x[]).joiner.writeln;
 }
 ```
 It indeed still errors with -dip1000, but without -dip1000 it 
 compiles now interestingly.

```
import std;

struct SomeRange
{
     int[3] val;

     enum empty = false;

     auto popFront()  safe {}

     ref auto front()  safe
     {
         return val;
     }
}

void main()  safe
{
     SomeRange().take(10).map!((return ref x) => 
x[]).joiner.writeln;
}
```

I don't know how applicable this is to your use case, but this 
code will compile and run under -dip1000.

ag0aep6g <anonymous example.com> writes:

On 08.02.20 00:10, nullptr wrote:
 ```
 import std;
 
 struct SomeRange
 {
      int[3] val;
 
      enum empty = false;
 
      auto popFront()  safe {}
 
      ref auto front()  safe
      {
          return val;
      }
 }
 
 void main()  safe
 {
      SomeRange().take(10).map!((return ref x) => x[]).joiner.writeln;
 }
 ```
 
 I don't know how applicable this is to your use case, but this code will 
 compile and run under -dip1000.

That shouldn't compile. You have found a hole in DIP 1000.

----
struct SomeRange
{
     int[3] val = [10, 20, 30];
     ref auto front()  safe { return val; }
}

int[] f()  safe
{
     SomeRange sr;
//     return sr.val[]; /* error, as expected */
     return sr.front[]; /* no error, but escapes reference to local */
}

void main()  safe
{
     auto x = f();
     import std.stdio;
     writeln(x); /* Prints garbage. */
}
----

I'm too lazy right now to check if it's already in Bugzilla.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 2/7/20 6:30 PM, ag0aep6g wrote:
 On 08.02.20 00:10, nullptr wrote:
 ```
 import std;

 struct SomeRange
 {
      int[3] val;

      enum empty = false;

      auto popFront()  safe {}

      ref auto front()  safe
      {
          return val;
      }
 }

 void main()  safe
 {
      SomeRange().take(10).map!((return ref x) => x[]).joiner.writeln;
 }
 ```

 I don't know how applicable this is to your use case, but this code 
 will compile and run under -dip1000.

 
 That shouldn't compile. You have found a hole in DIP 1000.
 
 ----
 struct SomeRange
 {
      int[3] val = [10, 20, 30];
      ref auto front()  safe { return val; }
 }
 
 int[] f()  safe
 {
      SomeRange sr;
 //     return sr.val[]; /* error, as expected */
      return sr.front[]; /* no error, but escapes reference to local */
 }
 
 void main()  safe
 {
      auto x = f();
      import std.stdio;
      writeln(x); /* Prints garbage. */
 }
 ----
 
 I'm too lazy right now to check if it's already in Bugzilla.

The original code is not invalid though. f is not valid, and that is a 
bug, but the original code posted by nullptr should be fine by memory 
safety standards.

If there is no possible way to do the original code with dip1000 
attributes, then that's a bug with dip1000's design (would not be 
surprised).

-Steve

ag0aep6g <anonymous example.com> writes:

On 08.02.20 01:17, Steven Schveighoffer wrote:
 The original code is not invalid though. f is not valid, and that is a 
 bug, but the original code posted by nullptr should be fine by memory 
 safety standards.

Maybe. But then it should also work with `int[3] front;`.

 If there is no possible way to do the original code with dip1000 
 attributes, then that's a bug with dip1000's design (would not be 
 surprised).

Or DIP 1000 just doesn't allow that kind of code.  safe (including DIP 
1000) is not supposed to allow all de-facto safe programs.

Simplified, we're looking at this:

----
struct Joiner
{
     int[3] _items;
     int[] _current;
}
void main()  safe
{
     Joiner j;
     j._current = j._items[];
}
----

I.e., a self-referential struct. Or most fundamentally:

----
struct Joiner
{
     Joiner* p;
}
void main()  safe
{
     Joiner j;
     j.p = &j; /* error */
}
----

`dmd -dip1000` complains about the marked line:

     Error: reference to local variable j assigned to non-scope j.p

What if I mark `j` as `scope`? Then I should be able to assign a 
reference-to-local to `j.p`. Indeed, the error goes away, but another 
takes its place:

     Error: cannot take address of scope local j in  safe function main

Right. That can't be allowed, because `scope` gives only one level of 
protection, and `&j` would need two (one for `j` itself and one for the 
thing it points at).

If that code were allowed, you could do this:

----
struct Joiner
{
     Joiner* p;
}
Joiner g;
void main()  safe
{
     scope Joiner j;
     ()  trusted { j.p = &j; } (); /* pretend it's allowed */
     g = *j.p; /* dereference and copy */
}
----

Returning a copy of a dereferenced `scope` pointer is always allowed, 
because `scope` only provides one level of protection.

ag0aep6g <anonymous example.com> writes:

On 08.02.20 02:38, ag0aep6g wrote:
 Simplified, we're looking at this:
 
 ----
 struct Joiner
 {
      int[3] _items;
      int[] _current;
 }
 void main()  safe
 {
      Joiner j;
      j._current = j._items[];
 }
 ----
 
 I.e., a self-referential struct. Or most fundamentally:
 
 ----
 struct Joiner
 {
      Joiner* p;
 }
 void main()  safe
 {
      Joiner j;
      j.p = &j; /* error */
 }
 ----
 
 `dmd -dip1000` complains about the marked line:
 
      Error: reference to local variable j assigned to non-scope j.p
 
 What if I mark `j` as `scope`? Then I should be able to assign a 
 reference-to-local to `j.p`. Indeed, the error goes away, but another 
 takes its place:
 
      Error: cannot take address of scope local j in  safe function main
 
 Right. That can't be allowed, because `scope` gives only one level of 
 protection, and `&j` would need two (one for `j` itself and one for the 
 thing it points at).

I went a bit overboard with that second reduction. The original struct 
is self-referential, but it's not recursive. The errors are the same for 
the first reduction. But it's not as clear that they're necessary.

In the first reduction, `j` might be `scope`, but `j._items` has no 
indirections. `scope` doesn't actually mean anything for it. It's just 
an `int[3]` on the stack. So taking its address could be allowed.

But (the current implementation of) DIP 1000 is apparently too 
conservative for that. It seems to treat pointers into a struct the same 
as pointers to the whole thing.

ag0aep6g <anonymous example.com> writes:

On 08.02.20 07:27, ag0aep6g wrote:
 On 08.02.20 02:38, ag0aep6g wrote:
 Simplified, we're looking at this:

 ----
 struct Joiner
 {
      int[3] _items;
      int[] _current;
 }
 void main()  safe
 {
      Joiner j;
      j._current = j._items[];
 }
 ----


[...]
 In the first reduction, `j` might be `scope`, but `j._items` has no 
 indirections. `scope` doesn't actually mean anything for it. It's just 
 an `int[3]` on the stack. So taking its address could be allowed.
 
 But (the current implementation of) DIP 1000 is apparently too 
 conservative for that. It seems to treat pointers into a struct the same 
 as pointers to the whole thing.

My attempt at lifting this limitation:

https://github.com/dlang/dmd/pull/10773

Steven Schveighoffer <schveiguy gmail.com> writes:

On 2/7/20 8:38 PM, ag0aep6g wrote:

 If that code were allowed, you could do this:
 
 ----
 struct Joiner
 {
      Joiner* p;
 }
 Joiner g;
 void main()  safe
 {
      scope Joiner j;
      ()  trusted { j.p = &j; } (); /* pretend it's allowed */
      g = *j.p; /* dereference and copy */
 }
 ----
 
 Returning a copy of a dereferenced `scope` pointer is always allowed, 
 because `scope` only provides one level of protection.

Again, the limitations of dip1000 are apparent. Why can't I mark p as 
only pointing at scope data? And in which cases do I need to do this (in 
the true Joiner code)?

This kind of stuff is so difficult to reason about and develop as a 
library that people will just end up removing dip1000 from their 
compilation.

-Steve

ag0aep6g <anonymous example.com> writes:

On 08.02.20 15:57, Steven Schveighoffer wrote:
 This kind of stuff is so difficult to reason about and develop as a 
 library that people will just end up removing dip1000 from their 
 compilation.

I 100% agree that DIP 1000 is hard to reason about. It's pretty limited 
by design, and the implementation has so many bugs. If anyone has a 
better design (and implementation), I'd be all for that.

About just ditching the compiler switch: Then you can't even take the 
address of a local. Also, it's going to become the default eventually.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 2/8/20 1:20 PM, ag0aep6g wrote:
 On 08.02.20 15:57, Steven Schveighoffer wrote:
 This kind of stuff is so difficult to reason about and develop as a 
 library that people will just end up removing dip1000 from their 
 compilation.

 
 I 100% agree that DIP 1000 is hard to reason about. It's pretty limited 
 by design, and the implementation has so many bugs. If anyone has a 
 better design (and implementation), I'd be all for that.

I unfortunately don't. I suspect that making scope a type constructor 
might help, but I don't know enough about how dip1000 works to know for 
sure.

I like the idea of dip1000 inferring most things. But the end result of 
joiner just getting inferred  system is so difficult to figure out why.

I think before dip1000 is the default, we need some way to say, I expect 
inference to infer these calls as safe. And it should tell you why it can't.

 
 About just ditching the compiler switch: Then you can't even take the 
 address of a local. Also, it's going to become the default eventually.

Then the user will just switch to  trusted. I mean to have to figure out 
what joiner is actually doing, in conjunction with map, like you did, is 
so convoluted, nobody is going to spend that time. The easier thing is 
to give up and switch to system where you have to.

I don't feel dip1000 is ready for the default mode until we have better 
ways to diagnose its issues, and concrete responses to things that 
really should be safe but aren't.

-Steve

Steven Schveighoffer <schveiguy gmail.com> writes:

On 2/7/20 6:10 PM, nullptr wrote:

 ```
 import std;
 
 struct SomeRange
 {
      int[3] val;
 
      enum empty = false;
 
      auto popFront()  safe {}
 
      ref auto front()  safe
      {
          return val;
      }
 }
 
 void main()  safe
 {
      SomeRange().take(10).map!((return ref x) => x[]).joiner.writeln;
 }
 ```
 
 I don't know how applicable this is to your use case, but this code will 
 compile and run under -dip1000.

I tried the AST button. for the front function I got:

return pure nothrow  nogc ref  safe int[3] front() return

Note the duplicate return attributes. When I type that in, it doesn't 
compile, the return at the front is invalid.

When I remove that return, I get a safety error. So something is weird 
about how the attributes are inferred. Either there is some magic 
attribute not being printed, or an attribute that can't be specified 
manually, or it's a bug.

-Steve