www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Why can't I give a function's return type the scope storage class?

reply Meta <jared771 gmail.com> writes:
Am I wrong in thinking that this is something one would want to 
do? It seems like it would be useful for the callee to enforce 
that its return value is assigned to a scope variable.

Currently, it seems like there is some sort of inference of scope 
on local variables:

struct Test
{
     int n = 3;

      safe
     int* test() return
     {
         return &n;
     }
}

int* gn;

void main()  safe
{
     Test t;
     int* n = t.test();
     gn = n; //Error: scope variable n assigned to gn with longer 
lifetime
}

Which is fine, but I'd like to be able to explicitly say, as part 
of a function's contract, that its result may not be escaped from 
the caller's scope, and not have to rely on the compiler's 
inference. The following doesn't work, however, or doesn't work 
as I'd like it to:

      safe
     scope int* test() return //Compiles, but doesn't work
     {
         return &n;
     }

      safe
     scope(int*) test() return //Does not compile; syntax error
     {
         return &n;
     }
Mar 15
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/15/2019 10:29 PM, Meta wrote:
 Am I wrong in thinking that this is something one would want to do? It seems 
 like it would be useful for the callee to enforce that its return value is 
 assigned to a scope variable.
Scope on function return values comes from the scope of any arguments passed to the function marked as 'return scope'. Scope on a function return with no such arguments is currently meaningless in D. To add such a feature, there woul'd need to be compelling use cases.
Mar 16
next sibling parent Olivier FAURE <couteaubleu gmail.com> writes:
On Saturday, 16 March 2019 at 17:59:41 UTC, Walter Bright wrote:
 Scope on function return values comes from the scope of any 
 arguments passed to the function marked as 'return scope'. 
 Scope on a function return with no such arguments is currently 
 meaningless in D.

 To add such a feature, there woul'd need to be compelling use 
 cases.
Wait, what? You can apply scope to return values? I thought in the example above, `scope` applied to the `this` parameter, the same way `return` did?
Mar 16
prev sibling parent reply Meta <jared771 gmail.com> writes:
On Saturday, 16 March 2019 at 17:59:41 UTC, Walter Bright wrote:
 On 3/15/2019 10:29 PM, Meta wrote:
 Am I wrong in thinking that this is something one would want 
 to do? It seems like it would be useful for the callee to 
 enforce that its return value is assigned to a scope variable.
Scope on function return values comes from the scope of any arguments passed to the function marked as 'return scope'. Scope on a function return with no such arguments is currently meaningless in D. To add such a feature, there woul'd need to be compelling use cases.
So to be clear, you *can* add scope to the return type of a function, and it will propagate the shortest lifetime among any parameters marked with `return` or `return scope` to the return value? I don't think this is mentioned anywhere in the docs, which is what confused me. I thought that scope on the return type was either a no-op, or that it applied to the function (despite being on the left-hand side). I *think* I can get the effect that I want, given that's how it works. I'll play around a bit and report back. Also, I'd like to echo the request that you document this stuff. What's already there is fairly useful, but obviously it's not complete.
Mar 18
next sibling parent reply Meta <jared771 gmail.com> writes:
On Monday, 18 March 2019 at 17:20:37 UTC, Meta wrote:
 On Saturday, 16 March 2019 at 17:59:41 UTC, Walter Bright wrote:
 On 3/15/2019 10:29 PM, Meta wrote:
 Am I wrong in thinking that this is something one would want 
 to do? It seems like it would be useful for the callee to 
 enforce that its return value is assigned to a scope variable.
Scope on function return values comes from the scope of any arguments passed to the function marked as 'return scope'. Scope on a function return with no such arguments is currently meaningless in D. To add such a feature, there woul'd need to be compelling use cases.
So to be clear, you *can* add scope to the return type of a function, and it will propagate the shortest lifetime among any parameters marked with `return` or `return scope` to the return value? I don't think this is mentioned anywhere in the docs, which is what confused me. I thought that scope on the return type was either a no-op, or that it applied to the function (despite being on the left-hand side). I *think* I can get the effect that I want, given that's how it works. I'll play around a bit and report back. Also, I'd like to echo the request that you document this stuff. What's already there is fairly useful, but obviously it's not complete.
As an addendum, are these three member function declarations equivalent (disregard the fact that this would normally cause a duplicate function definition error)? struct Test { int n = 3; safe int* test() return { return &n; } safe scope int* test() return { return &n; } safe scope int* test() return scope { return &n; } }
Mar 18
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/18/2019 10:22 AM, Meta wrote:
 struct Test
 {
      int n = 3;
 
       safe
      int* test() return
      {
          return &n;
      }
 
       safe
      scope int* test() return
      {
          return &n;
      }
 
       safe
      scope int* test() return scope
      {
          return &n;
      }
 }
 
An explicit scope doesn't apply to the returned value, it attaches to the implicit `this` argument.
Mar 18
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/18/2019 10:20 AM, Meta wrote:
 So to be clear, you *can* add scope to the return type of a function, and it 
 will propagate the shortest lifetime among any parameters marked with `return` 
 or `return scope` to the return value?
No. Adding scope as the storage class of the function means it is attached to the 'this' parameter, if any. Any 'return scope' parameter to the function will constrict the lifetime of the return value to that of the smallest lifetime of such arguments.
 Also, I'd like to echo the request that you document this stuff. What's
already 
 there is fairly useful, but obviously it's not complete.
It can be improved. But my focus at the moment is getting Phobos to compile with -dip1000.
Mar 18
next sibling parent reply Meta <jared771 gmail.com> writes:
On Tuesday, 19 March 2019 at 02:03:12 UTC, Walter Bright wrote:
 On 3/18/2019 10:20 AM, Meta wrote:
 So to be clear, you *can* add scope to the return type of a 
 function, and it will propagate the shortest lifetime among 
 any parameters marked with `return` or `return scope` to the 
 return value?
No. Adding scope as the storage class of the function means it is attached to the 'this' parameter, if any.
Ah, I misinterpreted what you meant when you said "Scope on function return values comes from the scope of any arguments passed to the function marked as 'return scope'." I pictured this: scope T* doSomething(return scope T*); Where the `scope` is attached to the T* return value, not to the function, but I think what you meant was this: T* doSomething(return scope T*); scope val = doSomething(someOtherVal); Right?
 Any 'return scope' parameter to the function will constrict the 
 lifetime of the return value to that of the smallest lifetime 
 of such arguments.
Yes, I've got it now. Thank you.
 Also, I'd like to echo the request that you document this 
 stuff. What's already there is fairly useful, but obviously 
 it's not complete.
It can be improved. But my focus at the moment is getting Phobos to compile with -dip1000.
Once I get a feel for DIP1000, I can hopefully help with that a bit.
Mar 18
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/18/2019 7:22 PM, Meta wrote:
 I think what you meant was this:
 
 T* doSomething(return scope T*);
 
 scope val = doSomething(someOtherVal);
 
 Right?
Yes, except it is unnecessary to mark val as scope. The compiler will attach scope to it automatically if someOtherVal is scope.
Mar 18
parent reply Meta <jared771 gmail.com> writes:
Okay, one more question. I might be misunderstanding how DIP1000 
works, but I cannot figure out how to copy data owned by an inner 
scope to an outer scope. I know that `scope` is supposed to 
prevent this, but in my case it's perfectly fine, as I am 
transferring ownership of the data from the inner scope to a data 
store in the outer scope (e.g., copying some data in a 
deeper-nested scope into an outer array). However, I can't seem 
to get the compiler to understand this transfer of ownership (I 
also tried moving values with std.algorithm.move).

My example code is a bit too long to post here, but it can be 
found at https://run.dlang.io/is/e6Lc15. The main crux of this 
issue is:

 safe
Queue!Data copyToQueue(DataRange data)
{
     Queue!Data output;
     while (!data.empty)
     {
         auto d = data.front;
         data.popFront();
         import std.random: uniform;
         if (uniform(0, 100) < 50)
         {
             output.push(d);
         }
     }
     return output;
}

And the definition of Queue!T.push is:

      safe
     void push(return scope T val) scope
     {
         ptr += 1;
         if (ptr >= store.length && ptr < size_t.max)
         {
             store.length = store.length == 0
                 ? 8
                 : store.length * growthFactor;
         }
         //FIXME: either I don't understand DIP1000
         //well enough, or this should compile
         //Workaround: surround it with an  trusted lambda
         store[ptr] = val;
     }

 From my understanding of DIP1000, annotating `val` with `return 
scope` in `Queue.push` tells the compiler "this value can only 
escape by being returned from the function, or being copied into 
the first parameter" (in this case, the first parameter being the 
`this` reference). Then, marking `Queue.push` as `scope` also 
tells the compiler "no references to `this` may escape from this 
function".

In this case, the Queue I am copying data into is in an outer 
scope, and the data I'm copying is `scope` (see the code at my 
link). The problem is that the compiler doesn't like this; it's 
telling me "Error: scope variable val assigned to non-scope 
this.store[this.ptr]". This is confusing, though, because I 
thought that's exactly why there's this special proviso for 
`return` and `return scope`:

"If the function returns void, and the first parameter is ref or 
out, then all subsequent return ref parameters are considered as 
being assigned to the first parameter for lifetime checking. The 
this reference parameter to a struct non-static member function 
is considered the first parameter."

As long as I'm not escaping the `this` reference from 
`Queue.push`, shouldn't the lifetime analysis all check out?
Mar 18
parent ag0aep6g <anonymous example.com> writes:
On 19.03.19 04:48, Meta wrote:
 My example code is a bit too long to post here, but it can be found at 
 https://run.dlang.io/is/e6Lc15. The main crux of this issue is:
 
  safe
 Queue!Data copyToQueue(DataRange data)
 {
      Queue!Data output;
      while (!data.empty)
      {
          auto d = data.front;
          data.popFront();
          import std.random: uniform;
          if (uniform(0, 100) < 50)
          {
              output.push(d);
          }
      }
      return output;
 }
 
 And the definition of Queue!T.push is:
 
       safe
      void push(return scope T val) scope
      {
          ptr += 1;
          if (ptr >= store.length && ptr < size_t.max)
          {
              store.length = store.length == 0
                  ? 8
                  : store.length * growthFactor;
          }
          //FIXME: either I don't understand DIP1000
          //well enough, or this should compile
          //Workaround: surround it with an  trusted lambda
          store[ptr] = val;
      }
I think it boils down to the following. This works: struct Q { string s; void push(return scope string v) scope safe { this.s = v; } } This doesn't: struct Q { string[] s; void push(return scope string v) scope safe { this.s.length = this.s.length + 1; this.s[$ - 1] = v; } } The reason is that `scope` is not transitive. The array `this.s` is `scope`, but an element like `this.s[$ - 1]` isn't. The compiler considers the elements to have infinite lifetime. Here's another example that illustrates this. When you've got a `scope Q`, the compiler won't let you return `q.s`, but it doesn't mind you returning an element of it: struct Q { string[] s; } string[] f(scope Q q) { return q.s; /* Error: scope variable q may not be returned */ } string g(scope Q q) { return q.s[0]; /* no error, the element is not `scope` */ }
Mar 19
prev sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 19 March 2019 at 02:03:12 UTC, Walter Bright wrote:
 [snip]

 It can be improved. But my focus at the moment is getting 
 Phobos to compile with -dip1000.
BTW, did you see this: https://atilaoncode.blog/2019/03/13/issues-dip1000-cant-yet-catch/
Mar 18
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/18/2019 7:28 PM, jmh530 wrote:
 BTW, did you see this:
 https://atilaoncode.blog/2019/03/13/issues-dip1000-cant-yet-catch/
C++ has no protection for its library types, except by convention. Translating them by rote into D leaves those problems intact. They need to be redesigned with D's mechanisms in mind for them to be memory safe. Leaking pointers to its internals is exactly that sort of issue.
Mar 18
parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 19 March 2019 at 03:16:04 UTC, Walter Bright wrote:
 On 3/18/2019 7:28 PM, jmh530 wrote:
 BTW, did you see this:
 https://atilaoncode.blog/2019/03/13/issues-dip1000-cant-yet-catch/
C++ has no protection for its library types, except by convention. Translating them by rote into D leaves those problems intact. They need to be redesigned with D's mechanisms in mind for them to be memory safe. Leaking pointers to its internals is exactly that sort of issue.
I didn't translate anything, I wrote it from scratch. C++ doesn't even have slices! The fact is that DIP1000 didn't prevent me from writing safe code where a pointer dangled. The point of my blog was "how can we improve D to disallow anyone else from writing code like this by making it fail to compile with -dip1000?". It's got nothing to do with C++, the comparison was with Rust, where one can't write the faulty code. Given the focus on memory safety in D without having to rely on the GC, I think this should be given serious consideration.
Mar 19
next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 19.03.19 11:19, Atila Neves wrote:
 On Tuesday, 19 March 2019 at 03:16:04 UTC, Walter Bright wrote:
 On 3/18/2019 7:28 PM, jmh530 wrote:
 BTW, did you see this:
 https://atilaoncode.blog/2019/03/13/issues-dip1000-cant-yet-catch/
[...]
 The fact is that DIP1000 didn't prevent me from writing  safe code where 
 a pointer dangled.
I don't think that's what happens. As far as I see, you get a dangling pointer, because you've got bad ` trusted` code in automem [1]: () trusted { _allocator.expandArray(mutableElements, delta.toSizeT); }(); By default, `_allocator` is `GCAllocator` (via some obfuscations). `expandArray` calls the allocator's `reallocate` method. `GCAllocator.reallocate` is not memory safe. The documentation says [2]: "The deallocate and reallocate methods are system because they may move memory around, leaving dangling pointers in user code" [1] https://github.com/atilaneves/automem/blob/4d8e8800b27ac7e92ed066237fd1359f59116fc5/source/automem/vector.d#L441 [2] https://dlang.org/phobos/std_experimental_allocator_gc_allocator.html#.GCAllocator.reallocate
Mar 19
next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 19 March 2019 at 14:28:00 UTC, ag0aep6g wrote:
 On 19.03.19 11:19, Atila Neves wrote:
 [...]
[...]
 [...]
I don't think that's what happens. As far as I see, you get a dangling pointer, because you've got bad ` trusted` code in automem [1]: [...]
Very good point. I'm going to have to stroke my beard whilst looking upwards for a bit now.
Mar 19
prev sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 19 March 2019 at 14:28:00 UTC, ag0aep6g wrote:
 
 [snip]
 By default, `_allocator` is `GCAllocator` (via some 
 obfuscations). `expandArray` calls the allocator's `reallocate` 
 method. `GCAllocator.reallocate` is not memory safe. The 
 documentation says [2]:
So I suppose the question then becomes can something like `reallocate` be made safe with DIP1000?
Mar 19
parent Olivier FAURE <couteaubleu gmail.com> writes:
On Tuesday, 19 March 2019 at 20:10:50 UTC, jmh530 wrote:
 So I suppose the question then becomes can something like 
 `reallocate` be made  safe with DIP1000?
Probably not.
Mar 20
prev sibling parent Olivier FAURE <couteaubleu gmail.com> writes:
On Tuesday, 19 March 2019 at 10:19:49 UTC, Atila Neves wrote:
 The fact is that DIP1000 didn't prevent me from writing  safe 
 code where a pointer dangled. The point of my blog was "how can 
 we improve D to disallow anyone else from writing code like 
 this by making it fail to compile with -dip1000?". It's got 
 nothing to do with C++, the comparison was with Rust, where one 
 can't write the faulty code.
Hum, that's a little inaccurate. The part of your code that creates the dangling is marked as trusted, which makes... aaaaand I've been ninja'ed. But yeah, the problem isn't that -dip1000 allows unsafe code, it's that there is no way to do what automem does with -dip1000 without using trusted at some point. Language-level reference counting could fix that and is on W&A's roadmap, but that's another can of worms.
Mar 19