digitalmars.D - Why can't I give a function's return type the scope storage class?
- Meta (37/37) Mar 15 2019 Am I wrong in thinking that this is something one would want to
- Walter Bright (5/8) Mar 16 2019 Scope on function return values comes from the scope of any arguments pa...
- Olivier FAURE (5/11) Mar 16 2019 Wait, what?
- Meta (13/23) Mar 18 2019 So to be clear, you *can* add scope to the return type of a
- Meta (23/48) Mar 18 2019 As an addendum, are these three member function declarations
- Walter Bright (3/26) Mar 18 2019 An explicit scope doesn't apply to the returned value, it attaches to th...
- Walter Bright (7/12) Mar 18 2019 No. Adding scope as the storage class of the function means it is attach...
- Meta (14/29) Mar 18 2019 Ah, I misinterpreted what you meant when you said "Scope on
- Walter Bright (3/10) Mar 18 2019 Yes, except it is unnecessary to mark val as scope. The compiler will at...
- jmh530 (3/6) Mar 18 2019 BTW, did you see this:
- Walter Bright (5/7) Mar 18 2019 C++ has no protection for its library types, except by convention. Trans...
- Atila Neves (11/19) Mar 19 2019 I didn't translate anything, I wrote it from scratch. C++ doesn't
- ag0aep6g (15/21) Mar 19 2019 I don't think that's what happens. As far as I see, you get a dangling
- Atila Neves (4/12) Mar 19 2019 Very good point.
- jmh530 (3/9) Mar 19 2019 So I suppose the question then becomes can something like
- Olivier FAURE (2/4) Mar 20 2019 Probably not.
- Olivier FAURE (9/15) Mar 19 2019 Hum, that's a little inaccurate. The part of your code that
Am I wrong in thinking that this is something one would want to do? It seems like it would be useful for the callee to enforce that its return value is assigned to a scope variable. Currently, it seems like there is some sort of inference of scope on local variables: struct Test { int n = 3; safe int* test() return { return &n; } } int* gn; void main() safe { Test t; int* n = t.test(); gn = n; //Error: scope variable n assigned to gn with longer lifetime } Which is fine, but I'd like to be able to explicitly say, as part of a function's contract, that its result may not be escaped from the caller's scope, and not have to rely on the compiler's inference. The following doesn't work, however, or doesn't work as I'd like it to: safe scope int* test() return //Compiles, but doesn't work { return &n; } safe scope(int*) test() return //Does not compile; syntax error { return &n; }
Mar 15 2019
On 3/15/2019 10:29 PM, Meta wrote:Am I wrong in thinking that this is something one would want to do? It seems like it would be useful for the callee to enforce that its return value is assigned to a scope variable.Scope on function return values comes from the scope of any arguments passed to the function marked as 'return scope'. Scope on a function return with no such arguments is currently meaningless in D. To add such a feature, there woul'd need to be compelling use cases.
Mar 16 2019
On Saturday, 16 March 2019 at 17:59:41 UTC, Walter Bright wrote:Scope on function return values comes from the scope of any arguments passed to the function marked as 'return scope'. Scope on a function return with no such arguments is currently meaningless in D. To add such a feature, there woul'd need to be compelling use cases.Wait, what? You can apply scope to return values? I thought in the example above, `scope` applied to the `this` parameter, the same way `return` did?
Mar 16 2019
On Saturday, 16 March 2019 at 17:59:41 UTC, Walter Bright wrote:On 3/15/2019 10:29 PM, Meta wrote:So to be clear, you *can* add scope to the return type of a function, and it will propagate the shortest lifetime among any parameters marked with `return` or `return scope` to the return value? I don't think this is mentioned anywhere in the docs, which is what confused me. I thought that scope on the return type was either a no-op, or that it applied to the function (despite being on the left-hand side). I *think* I can get the effect that I want, given that's how it works. I'll play around a bit and report back. Also, I'd like to echo the request that you document this stuff. What's already there is fairly useful, but obviously it's not complete.Am I wrong in thinking that this is something one would want to do? It seems like it would be useful for the callee to enforce that its return value is assigned to a scope variable.Scope on function return values comes from the scope of any arguments passed to the function marked as 'return scope'. Scope on a function return with no such arguments is currently meaningless in D. To add such a feature, there woul'd need to be compelling use cases.
Mar 18 2019
On Monday, 18 March 2019 at 17:20:37 UTC, Meta wrote:On Saturday, 16 March 2019 at 17:59:41 UTC, Walter Bright wrote:As an addendum, are these three member function declarations equivalent (disregard the fact that this would normally cause a duplicate function definition error)? struct Test { int n = 3; safe int* test() return { return &n; } safe scope int* test() return { return &n; } safe scope int* test() return scope { return &n; } }On 3/15/2019 10:29 PM, Meta wrote:So to be clear, you *can* add scope to the return type of a function, and it will propagate the shortest lifetime among any parameters marked with `return` or `return scope` to the return value? I don't think this is mentioned anywhere in the docs, which is what confused me. I thought that scope on the return type was either a no-op, or that it applied to the function (despite being on the left-hand side). I *think* I can get the effect that I want, given that's how it works. I'll play around a bit and report back. Also, I'd like to echo the request that you document this stuff. What's already there is fairly useful, but obviously it's not complete.Am I wrong in thinking that this is something one would want to do? It seems like it would be useful for the callee to enforce that its return value is assigned to a scope variable.Scope on function return values comes from the scope of any arguments passed to the function marked as 'return scope'. Scope on a function return with no such arguments is currently meaningless in D. To add such a feature, there woul'd need to be compelling use cases.
Mar 18 2019
On 3/18/2019 10:22 AM, Meta wrote:struct Test { int n = 3; safe int* test() return { return &n; } safe scope int* test() return { return &n; } safe scope int* test() return scope { return &n; } }An explicit scope doesn't apply to the returned value, it attaches to the implicit `this` argument.
Mar 18 2019
On 3/18/2019 10:20 AM, Meta wrote:So to be clear, you *can* add scope to the return type of a function, and it will propagate the shortest lifetime among any parameters marked with `return` or `return scope` to the return value?No. Adding scope as the storage class of the function means it is attached to the 'this' parameter, if any. Any 'return scope' parameter to the function will constrict the lifetime of the return value to that of the smallest lifetime of such arguments.Also, I'd like to echo the request that you document this stuff. What's already there is fairly useful, but obviously it's not complete.It can be improved. But my focus at the moment is getting Phobos to compile with -dip1000.
Mar 18 2019
On Tuesday, 19 March 2019 at 02:03:12 UTC, Walter Bright wrote:On 3/18/2019 10:20 AM, Meta wrote:Ah, I misinterpreted what you meant when you said "Scope on function return values comes from the scope of any arguments passed to the function marked as 'return scope'." I pictured this: scope T* doSomething(return scope T*); Where the `scope` is attached to the T* return value, not to the function, but I think what you meant was this: T* doSomething(return scope T*); scope val = doSomething(someOtherVal); Right?So to be clear, you *can* add scope to the return type of a function, and it will propagate the shortest lifetime among any parameters marked with `return` or `return scope` to the return value?No. Adding scope as the storage class of the function means it is attached to the 'this' parameter, if any.Any 'return scope' parameter to the function will constrict the lifetime of the return value to that of the smallest lifetime of such arguments.Yes, I've got it now. Thank you.Once I get a feel for DIP1000, I can hopefully help with that a bit.Also, I'd like to echo the request that you document this stuff. What's already there is fairly useful, but obviously it's not complete.It can be improved. But my focus at the moment is getting Phobos to compile with -dip1000.
Mar 18 2019
On 3/18/2019 7:22 PM, Meta wrote:I think what you meant was this: T* doSomething(return scope T*); scope val = doSomething(someOtherVal); Right?Yes, except it is unnecessary to mark val as scope. The compiler will attach scope to it automatically if someOtherVal is scope.
Mar 18 2019
Okay, one more question. I might be misunderstanding how DIP1000 works, but I cannot figure out how to copy data owned by an inner scope to an outer scope. I know that `scope` is supposed to prevent this, but in my case it's perfectly fine, as I am transferring ownership of the data from the inner scope to a data store in the outer scope (e.g., copying some data in a deeper-nested scope into an outer array). However, I can't seem to get the compiler to understand this transfer of ownership (I also tried moving values with std.algorithm.move). My example code is a bit too long to post here, but it can be found at https://run.dlang.io/is/e6Lc15. The main crux of this issue is: safe Queue!Data copyToQueue(DataRange data) { Queue!Data output; while (!data.empty) { auto d = data.front; data.popFront(); import std.random: uniform; if (uniform(0, 100) < 50) { output.push(d); } } return output; } And the definition of Queue!T.push is: safe void push(return scope T val) scope { ptr += 1; if (ptr >= store.length && ptr < size_t.max) { store.length = store.length == 0 ? 8 : store.length * growthFactor; } //FIXME: either I don't understand DIP1000 //well enough, or this should compile //Workaround: surround it with an trusted lambda store[ptr] = val; } From my understanding of DIP1000, annotating `val` with `return scope` in `Queue.push` tells the compiler "this value can only escape by being returned from the function, or being copied into the first parameter" (in this case, the first parameter being the `this` reference). Then, marking `Queue.push` as `scope` also tells the compiler "no references to `this` may escape from this function". In this case, the Queue I am copying data into is in an outer scope, and the data I'm copying is `scope` (see the code at my link). The problem is that the compiler doesn't like this; it's telling me "Error: scope variable val assigned to non-scope this.store[this.ptr]". This is confusing, though, because I thought that's exactly why there's this special proviso for `return` and `return scope`: "If the function returns void, and the first parameter is ref or out, then all subsequent return ref parameters are considered as being assigned to the first parameter for lifetime checking. The this reference parameter to a struct non-static member function is considered the first parameter." As long as I'm not escaping the `this` reference from `Queue.push`, shouldn't the lifetime analysis all check out?
Mar 18 2019
On 19.03.19 04:48, Meta wrote:My example code is a bit too long to post here, but it can be found at https://run.dlang.io/is/e6Lc15. The main crux of this issue is: safe Queue!Data copyToQueue(DataRange data) { Queue!Data output; while (!data.empty) { auto d = data.front; data.popFront(); import std.random: uniform; if (uniform(0, 100) < 50) { output.push(d); } } return output; } And the definition of Queue!T.push is: safe void push(return scope T val) scope { ptr += 1; if (ptr >= store.length && ptr < size_t.max) { store.length = store.length == 0 ? 8 : store.length * growthFactor; } //FIXME: either I don't understand DIP1000 //well enough, or this should compile //Workaround: surround it with an trusted lambda store[ptr] = val; }I think it boils down to the following. This works: struct Q { string s; void push(return scope string v) scope safe { this.s = v; } } This doesn't: struct Q { string[] s; void push(return scope string v) scope safe { this.s.length = this.s.length + 1; this.s[$ - 1] = v; } } The reason is that `scope` is not transitive. The array `this.s` is `scope`, but an element like `this.s[$ - 1]` isn't. The compiler considers the elements to have infinite lifetime. Here's another example that illustrates this. When you've got a `scope Q`, the compiler won't let you return `q.s`, but it doesn't mind you returning an element of it: struct Q { string[] s; } string[] f(scope Q q) { return q.s; /* Error: scope variable q may not be returned */ } string g(scope Q q) { return q.s[0]; /* no error, the element is not `scope` */ }
Mar 19 2019
On Tuesday, 19 March 2019 at 02:03:12 UTC, Walter Bright wrote:[snip] It can be improved. But my focus at the moment is getting Phobos to compile with -dip1000.BTW, did you see this: https://atilaoncode.blog/2019/03/13/issues-dip1000-cant-yet-catch/
Mar 18 2019
On 3/18/2019 7:28 PM, jmh530 wrote:BTW, did you see this: https://atilaoncode.blog/2019/03/13/issues-dip1000-cant-yet-catch/C++ has no protection for its library types, except by convention. Translating them by rote into D leaves those problems intact. They need to be redesigned with D's mechanisms in mind for them to be memory safe. Leaking pointers to its internals is exactly that sort of issue.
Mar 18 2019
On Tuesday, 19 March 2019 at 03:16:04 UTC, Walter Bright wrote:On 3/18/2019 7:28 PM, jmh530 wrote:I didn't translate anything, I wrote it from scratch. C++ doesn't even have slices! The fact is that DIP1000 didn't prevent me from writing safe code where a pointer dangled. The point of my blog was "how can we improve D to disallow anyone else from writing code like this by making it fail to compile with -dip1000?". It's got nothing to do with C++, the comparison was with Rust, where one can't write the faulty code. Given the focus on memory safety in D without having to rely on the GC, I think this should be given serious consideration.BTW, did you see this: https://atilaoncode.blog/2019/03/13/issues-dip1000-cant-yet-catch/C++ has no protection for its library types, except by convention. Translating them by rote into D leaves those problems intact. They need to be redesigned with D's mechanisms in mind for them to be memory safe. Leaking pointers to its internals is exactly that sort of issue.
Mar 19 2019
On 19.03.19 11:19, Atila Neves wrote:On Tuesday, 19 March 2019 at 03:16:04 UTC, Walter Bright wrote:[...]On 3/18/2019 7:28 PM, jmh530 wrote:BTW, did you see this: https://atilaoncode.blog/2019/03/13/issues-dip1000-cant-yet-catch/The fact is that DIP1000 didn't prevent me from writing safe code where a pointer dangled.I don't think that's what happens. As far as I see, you get a dangling pointer, because you've got bad ` trusted` code in automem [1]: () trusted { _allocator.expandArray(mutableElements, delta.toSizeT); }(); By default, `_allocator` is `GCAllocator` (via some obfuscations). `expandArray` calls the allocator's `reallocate` method. `GCAllocator.reallocate` is not memory safe. The documentation says [2]: "The deallocate and reallocate methods are system because they may move memory around, leaving dangling pointers in user code" [1] https://github.com/atilaneves/automem/blob/4d8e8800b27ac7e92ed066237fd1359f59116fc5/source/automem/vector.d#L441 [2] https://dlang.org/phobos/std_experimental_allocator_gc_allocator.html#.GCAllocator.reallocate
Mar 19 2019
On Tuesday, 19 March 2019 at 14:28:00 UTC, ag0aep6g wrote:On 19.03.19 11:19, Atila Neves wrote:Very good point. I'm going to have to stroke my beard whilst looking upwards for a bit now.[...][...][...]I don't think that's what happens. As far as I see, you get a dangling pointer, because you've got bad ` trusted` code in automem [1]: [...]
Mar 19 2019
On Tuesday, 19 March 2019 at 14:28:00 UTC, ag0aep6g wrote:[snip] By default, `_allocator` is `GCAllocator` (via some obfuscations). `expandArray` calls the allocator's `reallocate` method. `GCAllocator.reallocate` is not memory safe. The documentation says [2]:So I suppose the question then becomes can something like `reallocate` be made safe with DIP1000?
Mar 19 2019
On Tuesday, 19 March 2019 at 20:10:50 UTC, jmh530 wrote:So I suppose the question then becomes can something like `reallocate` be made safe with DIP1000?Probably not.
Mar 20 2019
On Tuesday, 19 March 2019 at 10:19:49 UTC, Atila Neves wrote:The fact is that DIP1000 didn't prevent me from writing safe code where a pointer dangled. The point of my blog was "how can we improve D to disallow anyone else from writing code like this by making it fail to compile with -dip1000?". It's got nothing to do with C++, the comparison was with Rust, where one can't write the faulty code.Hum, that's a little inaccurate. The part of your code that creates the dangling is marked as trusted, which makes... aaaaand I've been ninja'ed. But yeah, the problem isn't that -dip1000 allows unsafe code, it's that there is no way to do what automem does with -dip1000 without using trusted at some point. Language-level reference counting could fix that and is on W&A's roadmap, but that's another can of worms.
Mar 19 2019