digitalmars.D - Tell us your DIP1000 woes
- Mike Parker (22/22) Aug 25 DIP1000 keeps rearing its head in discussions touching on safety
- Richard (Rikki) Andrew Cattermole (27/27) Aug 25 I'll include the items I said at the meeting just so we can hopefully
- jmh530 (13/17) Aug 27 I think it would probably be useful to separate woes with DIP
- Steven Schveighoffer (10/14) Aug 25 Here, I just made a PR to mysql-native, with dip1000 enabled. The
- Timon Gehr (34/39) Aug 25 diff --git a/source/mysql/protocol/comms.d b/source/mysql/protocol/comms...
- Steven Schveighoffer (6/13) Aug 27 Huh, this actually was it. I fully expected a deeper rabbit hole.
- Elias Batek (15/18) Aug 25 My first attempt at adding DIP1000-compatibility to
- Elias Batek (22/25) Aug 25 One of the most common issues I run into when talking about
- Walter Bright (14/14) Aug 26 I found the following inconsistency:
- Richard (Rikki) Andrew Cattermole (18/40) Aug 26 There is a fourth option:
- Walter Bright (2/5) Aug 26 It not only appears to be, it is true that trusted functions are not val...
- Dom DiSc (11/17) Aug 26 But their API should be checked. That would actually make
- Timon Gehr (16/30) Aug 26 This is exactly how it should work.
- Nick Treleaven (13/17) Aug 26 Yes, though the compiler could actually error on the return
- Walter Bright (16/16) Aug 26 That is correct, I was just thinking of using @trusted to avoid checks, ...
- Juraj (13/22) Aug 26 I am probably adding to the noise, but I think this is relevant.
- Dennis (44/47) Aug 26 1. `scope` on a struct variable makes all fields `scope`. This
- Dukc (16/44) Aug 26 That `scope` can get inferred to a local variable due to a LATER
- Juraj (35/39) Aug 26 As stated in my previous reply, I dont have problems with DIP1000
- Paul Backus (24/28) Aug 27 Here's an issue I ran into when working on my proof-of-concept
- Paul Backus (47/51) Aug 27 In addition to inference, it is vitally important to have
- Paul Backus (28/32) Aug 28 Here's an issue that stems from a fundamental shortcoming of
- Richard (Rikki) Andrew Cattermole (34/75) Aug 28 This is a good example of what I mean by multiple outputs are not suppor...
- Richard (Rikki) Andrew Cattermole (54/54) Aug 29 I've been thinking about swap/move functions some more for my own propos...
- Richard (Rikki) Andrew Cattermole (5/7) Aug 29 To clarify, "invalidate the original input variable" means that from the...
- Atila Neves (4/9) Aug 29 Interesting, thank you. I went and looked at how Rust fixes this
- Paul Backus (7/18) Aug 29 To be clear, the issue here is not that the *implementation*
- H. S. Teoh (20/27) Aug 29 Exactly; this is why 'scope' is an inadequate solution to the wrong
- Atila Neves (2/16) Aug 30 Ah. Yes, that's bad.
- IchorDev (14/18) Aug 29 A problem I’ve often encountered with the design of DIP1000 is
- IchorDev (8/16) Aug 29 Just wanted to add that I do think `scope` is a fantastic and
DIP1000 keeps rearing its head in discussions touching on safety in our monthly meetings. So we recently held a planning session focused exclusively on DIP1000. Two things came out of it. First, it's pretty clear that inference by default is necessary to make DIP1000 easier to use. It's not clear at the moment how this can be achieved, but it's something that we need to work out. Second, we'd like to get a number of examples of problems people have had with using DIP1000 that haven't shown up in Bugzilla. Walter wants to be as certain as he can whether such issues are fixable or if the design is fundamentally flawed. If you've had problems using DIP1000, please post here with a description of what you encountered and any details about it you think we may find helpful. Please provide example code where possible. As an example of the sort of thing we're looking for, someone in the meeting brought up the problems encountered with trying to add DIP1000 support to vibe.d. **Please do not reply to posts in this thread unless you have additional relevant information regarding the problem described.** We want to collect examples, and any side discussions will just be noise. Thanks!
Aug 25
I'll include the items I said at the meeting just so we can hopefully minimize extras. From meeting: - Reference counted types (not in language) get affected by scope, like any other pointer contained struct. Which is a problem since it has runtime lifetime, not a CT analyzed one. - Only one output is described, (returned value or this pointer), this means ref and out parameters aren't outputs and therefore cannot be safe. - The abomination that is scope return + return scope (this is easier to resolve in terms of new UDA's, but does not solve communicability of behavior). Other ones: - It does not model owners and borrows, this isn't in its scope, it also means in practice it has to try and act as a guarantee both up the call stack and down it, which is not what such escape analysis is designed for. Which makes it a lot harder to use. - Does not attempt at modelling globals including TLS, so with safe code you can absolutely escape memory from the thread completely unintentionally. As someone who has over 120k LOC fully annotated DIP1000 code, my only conclusion is that DIP1000 has some serious scope of proposal level issues and cannot be rectified. This has been my conclusion *after* trying to fix it for a few years and having no proposals that could. Depending upon how DConf goes, I might be looking at dumping it, because as-is it has too many scope-level holes that will not be fixed. But in saying that, Dennis I sincerely hope you can do some serious magic here because it'll be impressive!
Aug 25
jmh530 <john.michael.hall gmail.com> On Sunday, 25 August 2024 at 16:23:40 UTC, Richard (Rikki) Andrew Cattermole wrote:[snip] - The abomination that is scope return + return scope (this is easier to resolve in terms of new UDA's, but does not solve communicability of behavior).I think it would probably be useful to separate woes with DIP 1000 into woes with `scope` parameters and woes with `return scope`/`ref return scope` parameters. When it comes to mental complexity, I see that as less of an issue for `scope` parameters than for `return scope` or `ref return scope` parameters There might be some outstanding issues with `scope`, like what you and Dennis bring up, but ultimately I think what `return scope` is doing is kind of like Rust lifetimes-lite. It solves a particular use case, but isn't sophisticated enough for more complicated ones.
Aug 27
On Sunday, 25 August 2024 at 13:10:22 UTC, Mike Parker wrote:If you've had problems using DIP1000, please post here with a description of what you encountered and any details about it you think we may find helpful. Please provide example code where possible.Here, I just made a PR to mysql-native, with dip1000 enabled. The last time I went down this rabbit hole, it was clear this is unworkable. You are welcome to submit PRs to my PR to show me how to fix it, please explain how everything works. I don't have time to play around with dip1000 normally -- it generally just results in bizarre and unfixable errors. https://github.com/mysql-d/mysql-native/pull/283 -Steve
Aug 25
On 8/25/24 19:02, Steven Schveighoffer wrote:I don't have time to play around with dip1000 normally -- it generally just results in bizarre and unfixable errors. https://github.com/mysql-d/mysql-native/pull/283diff --git a/source/mysql/protocol/comms.d b/source/mysql/protocol/comms.d index ec2a3ea..0238926 100644 --- a/source/mysql/protocol/comms.d +++ b/source/mysql/protocol/comms.d -689,7 +689,7 package(mysql) ubyte[] getPacket(Connection conn) return packet; } -package(mysql) void send(MySQLSocket _socket, const(ubyte)[] packet) +package(mysql) void send(MySQLSocket _socket, scope const(ubyte)[] packet) in { assert(packet.length > 4); // at least 1 byte more than header diff --git a/source/mysql/protocol/sockets.d b/source/mysql/protocol/sockets.d index a582fc9..7da43d7 100644 --- a/source/mysql/protocol/sockets.d +++ b/source/mysql/protocol/sockets.d -39,7 +39,7 interface MySQLSocket safe: void close(); property bool connected() const; - void read(ubyte[] dst); + void read(scope ubyte[] dst); void write(const scope ubyte[] bytes); void acquire(); -78,7 +78,7 class MySQLSocketPhobos : MySQLSocket return socket.isAlive; } - void read(ubyte[] dst) + void read(scope ubyte[] dst) { // Note: I'm a little uncomfortable with this line as it doesn't // (and can't) update Connection._open. Not sure what can be done,
Aug 25
On Sunday, 25 August 2024 at 20:02:01 UTC, Timon Gehr wrote:On 8/25/24 19:02, Steven Schveighoffer wrote:Huh, this actually was it. I fully expected a deeper rabbit hole. Now I have to figure out how to only turn on dip1000 for certain versions of D, as older compilers didn't have proper marking on std.socket... -SteveI don't have time to play around with dip1000 normally -- it generally just results in bizarre and unfixable errors. https://github.com/mysql-d/mysql-native/pull/283diff --git a/source/mysql/protocol/comms.d
Aug 27
On Sunday, 25 August 2024 at 13:10:22 UTC, Mike Parker wrote:As an example of the sort of thing we're looking for, someone in the meeting brought up the problems encountered with trying to add DIP1000 support to vibe.d.My first attempt at adding DIP1000-compatibility to oceandrift/http was a disaster. An hour or two of attribute tuning in a whack-a-mole fashion during which I was asking myself why I had to do this manually as I was mostly just making the changes suggested by the compiler. It all had an end when I ran into an presumably `return ref return scope` case; something that isn’t doable in D. The second attempt was a bit more successful. I’ve committed a scarily big number of patches to a feature branch, pushed it to GitHub and let it be: <https://github.com/oceandrift/http/commit/996b743f1a7b5f4799dcde802971bfeb5a675cf1> My third attempt was more pragmatic and meant “not doing anything that would make me bother with DIP1000’s scope gulasch myself”. I was quite happy with the end result.
Aug 25
On Sunday, 25 August 2024 at 13:10:22 UTC, Mike Parker wrote:If you've had problems using DIP1000, please post here with a description of what you encountered and any details about it you think we may find helpful.One of the most common issues I run into when talking about DIP1000 is that DIP1000 is actually two things: 1. Plugging holes in ` safe`. 2. Providing a mechanism the return/ref/scope mechanism re-enables most of the sort of code that is flagged ` system` after implementing the former. Furthermore, I have yet to meet anyone who tells me that adding 2-word attributes (`scope return` vs `return scope`) to the soup was a great choice. I see a major comprehensibility problem with these. It’s also a bit funny that all of those things are still referred to as “DIP1000”. If one takes just a slight glimpse at the document of the same name, it clearly says “Superseded”. I do understand why the article “Coralling Wild Pointers With ref return scope” explains most of the concepts on raw pointers. But on the other hand I’m not sure how useful those are to the unsuspecting users who run into DIP1000 violations with static arrays or refs. Or they even just wanted to use `std.getopt` in ` safe` code and ended up dealing with DIP1000. That’s what made me turn on the compiler switch for first time.
Aug 25
I found the following inconsistency:
```
safe ref int abc(int i) { return ghi(i); }
trusted ref int ghi(ref int r) { return r; }
safe int* foo(int i) { return bar(&i); } // Error: reference to local variable
`i` assigned to non-scope parameter `p` calling `bar`
trusted int* bar(int* p) { return p; }
```
Compiling with -dip1000 gives the error indicated. Note that it did not give an
error for the semantically equivalent call to ghi(i).
Possible resolutions:
1. give an error for both
2. give no error for both, as trusted implementations are the user's problem
3. disallow taking the address of a local variable in safe code
Aug 26
On 26/08/2024 8:17 PM, Walter Bright wrote:
I found the following inconsistency:
```
safe ref int abc(int i) { return ghi(i); }
trusted ref int ghi(ref int r) { return r; }
safe int* foo(int i) { return bar(&i); } // Error: reference to local
variable `i` assigned to non-scope parameter `p` calling `bar`
trusted int* bar(int* p) { return p; }
```
Compiling with -dip1000 gives the error indicated. Note that it did not
give an error for the semantically equivalent call to ghi(i).
Possible resolutions:
1. give an error for both
2. give no error for both, as trusted implementations are the user's
problem
3. disallow taking the address of a local variable in safe code
There is a fourth option:
Infer for `` trusted`` functions, but don't validate.
Support describing the empty escape set separately from wanting it inferred.
If set, trust it, otherwise infer.
Will infer:
```d
trusted ref int ghi(ref int r) { return r; }
```
Will use annotation:
```d
trusted ref int ghi( escapevia(return) ref int r) { return r; }
trusted ref int ghi( escapevia() ref int r) { return r; }
```
First and second will error, but third won't.
This appears to be an interaction between DIP1000 and the fact that
`` trusted`` functions have an interface that is required to be
`` safe`` but not its body validated.
Aug 26
On 8/26/2024 1:33 AM, Richard (Rikki) Andrew Cattermole wrote:This appears to be an interaction between DIP1000 and the fact that `` trusted`` functions have an interface that is required to be `` safe`` but not its body validated.It not only appears to be, it is true that trusted functions are not validated.
Aug 26
On Monday, 26 August 2024 at 18:54:26 UTC, Walter Bright wrote:On 8/26/2024 1:33 AM, Richard (Rikki) Andrew Cattermole wrote:But their API should be checked. That would actually make trusted *functions* (not trusted blocks) much more useful. That would make for a useful difference between safe and trusted - one is completely checked, for the other only the parameters and return values are checked. These checks would make it much more plausible why safe functions should be allowed to call trusted functions but not system functions. Having a (guaranteed) safe interface makes trusted functions a whole lot more trustworthy.This appears to be an interaction between DIP1000 and the fact that `` trusted`` functions have an interface that is required to be `` safe`` but not its body validated.It not only appears to be, it is true that trusted functions are not validated.
Aug 26
On 8/26/24 10:17, Walter Bright wrote:I found the following inconsistency: ``` safe ref int abc(int i) { return ghi(i); } trusted ref int ghi(ref int r) { return r; } safe int* foo(int i) { return bar(&i); } // Error: reference to local variable `i` assigned to non-scope parameter `p` calling `bar` trusted int* bar(int* p) { return p; } ``` ...This is exactly how it should work. What's inconsistent here is your code. The first ` trusted` function is invalid. The second ` trusted` function is fine because the parameter is not `scope`. The second function can even be ` safe`. The first memory safety bug is therefore not caught (because the bug is in the ` trusted` function), while the second one is caught (because the bug is in the ` safe` function).Compiling with -dip1000 gives the error indicated. Note that it did not give an error for the semantically equivalent call to ghi(i).If you want to attribute blame for a bug, not only the semantics matter, also the contracts at function interfaces matter. This is exactly why DIP1000 comes with annotations to make the intent clear and checkable modularly. There is no analogy between raw pointers and `ref`, there is an analogy between `scope` pointers and `ref`. (If you want to continue this discussion, we should probably take it out of the thread.)
Aug 26
On Monday, 26 August 2024 at 09:31:18 UTC, Timon Gehr wrote:On 8/26/24 10:17, Walter Bright wrote:Yes, though the compiler could actually error on the return statement for `ghi` (because a trusted function must have a safe interface). That would be consistent with erroring when returning a local variable as `ref`: ```d trusted ref int h() { int i; return i; // Error: returning `i` escapes a reference to local variable `i` } ```trusted ref int ghi(ref int r) { return r; }What's inconsistent here is your code. The first ` trusted` function is invalid.
Aug 26
That is correct, I was just thinking of using trusted to avoid checks, rather
than ensuring safe code.
The reason inference is not done automatically for all functions is because:
```
int* foo(int*);
```
```
int* foo(int* p) { return p; }
```
are different functions.
Inference affects the name mangling of D functions. So, for D functions, the
program will fail to link with inferred attributes.
But with C and C++ linkage, the name mangling isn't reflected in the symbol
name.
I suppose it's a problem we can live with.
Attribute inference isn't perfect, either, but it may work well enough to make
dip1000 more palatable.
Aug 26
On Sunday, 25 August 2024 at 13:10:22 UTC, Mike Parker wrote:**Please do not reply to posts in this thread unless you have additional relevant information regarding the problem described.** We want to collect examples, and any side discussions will just be noise. Thanks!I am probably adding to the noise, but I think this is relevant. I can not be the only one, that can not provide a bug report or a code snipped because I just lost interest in pursuing DIP1000 compatibility years ago. In the end the whole thing just feels off, and is one of my major concerns about going ` safe` by default. In all seriousness I am not willing to look at a function signature full of `return` keywords and figuring out what it means. More so, not gonna spend my energy writing such thing.First, it's pretty clear that inference by default is necessary to make DIP1000 easier to use. It's not clear at the moment how this can be achieved, but it's something that we need to work out.In my opinion, without this, there will just not be enough users. (I written a bigger rant, but lets keep it simple) Juraj
Aug 26
On Sunday, 25 August 2024 at 13:10:22 UTC, Mike Parker wrote:Second, we'd like to get a number of examples of problems people have had with using DIP1000 that haven't shown up in Bugzilla.1. `scope` on a struct variable makes all fields `scope`. This came up when making a tokenizer range: When parsing a stack-allocated string, the resulting tokens also become `scope` even when they don't refer to the input string. ```D class Token {} struct Tokenizer { char[] input; Token front; } Token getToken() safe { char[5] input = "1 + 2"; auto range = Tokenizer(input[]); // range as a whole becomes `scope` here return range.front; // Unnecessary error that `range.front` is `scope` } ``` 2. Scope inference on assignment doesn't remember the original lifetime, but shortens it to the assigned variable. ```D struct S { string name; void doSomething() scope; } void f(scope S s) { string prev = s.name; // prev simply becomes `scope`, doesn't remember it came from `s` s.doSomething(); s.name = prev; // Unnecessary error: `prev` assigned to s with longer lifetime } ``` 3. Lack of transitive scope I.e. you can't have an array of stack allocated strings, or a hashmap using scope keys/values. There are workarounds. For example, in my regex match function, I return indices of matches instead of slices of the input string. This does make the API a lot less ergonomic though.
Aug 26
Mike Parker kirjoitti 25.8.2024 klo 16.10:DIP1000 keeps rearing its head in discussions touching on safety in our monthly meetings. So we recently held a planning session focused exclusively on DIP1000. Two things came out of it. First, it's pretty clear that inference by default is necessary to make DIP1000 easier to use. It's not clear at the moment how this can be achieved, but it's something that we need to work out. Second, we'd like to get a number of examples of problems people have had with using DIP1000 that haven't shown up in Bugzilla. Walter wants to be as certain as he can whether such issues are fixable or if the design is fundamentally flawed. If you've had problems using DIP1000, please post here with a description of what you encountered and any details about it you think we may find helpful. Please provide example code where possible. As an example of the sort of thing we're looking for, someone in the meeting brought up the problems encountered with trying to add DIP1000 support to vibe.d. **Please do not reply to posts in this thread unless you have additional relevant information regarding the problem described.** We want to collect examples, and any side discussions will just be noise. Thanks!That `scope` can get inferred to a local variable due to a LATER assignment to it leads to pretty confusing situations. I think only the initialisation should infer. I'm not sure if it's worth a change anymore though, as it'll lead to more breakage. Maybe over an edition switch. I still think `scope`/`return scope` inference should only happen on a ` safe` function (whether explicitly safe or inferred as so). In ` trusted` / ` system` code it is critical you don't have those attributes added and removed under your nose or in an implementation-defined manner. Yet it [can happen](https://forum.dlang.org/post/edtbjavjzkwogvutxpho forum.dlang.org). Design by introspection also doesn't [fully work with](https://issues.dlang.org/show_bug.cgi?id=24003) `scope` / `return scope`, as I discovered when I tried to make a [Phobos function DIP1000 - compilant](https://issues.dlang.org/show_bug.cgi?id=23300). Dennis did make a pull request to enable this but it stalled.
Aug 26
On Sunday, 25 August 2024 at 13:10:22 UTC, Mike Parker wrote:Second, we'd like to get a number of examples of problems people have had with using DIP1000 that haven't shown up in Bugzilla. Walter wants to be as certain as he can whether such issues are fixable or if the design is fundamentally flawed.As stated in my previous reply, I dont have problems with DIP1000 as I just ignore it, because of bad ergonomics. But I felt guilty for bashing something I haven't used in years, so here is first thing I tried. All my shenanigans are catched without `-dip1000`. (I over-use ` safe` just to be sure) ```d safe: struct S { int x; S* s_ptr_0; S* s_ptr_1; } S* g_ptr = null; S* goo(return ref S sp) safe => &sp; S foo(int v) safe { S s0 = S(333); S s1 = S(v); void ez_escape() safe { auto res = goo(s1); g_ptr = res; } ez_escape(); () safe { S* moo(return ref S sp) safe => &sp; s1.s_ptr_0 = moo(s0); }(); () safe { S* moo(S* sp) safe => sp; s1.s_ptr_1 = moo(&s0); }(); return s1; } ```
Aug 26
On Sunday, 25 August 2024 at 13:10:22 UTC, Mike Parker wrote:If you've had problems using DIP1000, please post here with a description of what you encountered and any details about it you think we may find helpful. Please provide example code where possible.Here's an issue I ran into when working on my proof-of-concept allocator/container library. When writing exception-safe code, a common pattern is to save the original value of a variable before performing an operation that might throw, and then restore that value if the operation fails. Unfortunately, it is impossible to do this in ` safe` code if the variable you want to save and restore is `scope`, because the compiler will treat the saved copy as having a shorter lifetime than the original. For example, this code fails to compile when using `-preview=dip1000`: safe void example(ref scope int[] var) { int[] savedValue = var; scope (failure) var = savedValue; // Do stuff with a that might throw... } ...with the following error message: onlineapp.d(4): Error: scope variable `savedValue` assigned to `ref` variable `var` with longer lifetime Ideally, the compiler should be able to infer that `savedValue` has the *exact same* lifetime as `var` in this code, and allow the assignment.
Aug 27
On Sunday, 25 August 2024 at 13:10:22 UTC, Mike Parker wrote:First, it's pretty clear that inference by default is necessary to make DIP1000 easier to use. It's not clear at the moment how this can be achieved, but it's something that we need to work out.In addition to inference, it is vitally important to have detailed error messages showing exactly where and why inference failed. Here's an illustrative example: int* global; void foo()(int* p) { global = p; } void bar()() { int n; foo(&n); } safe void main() { bar(); } Currently, compiling this program with `-preview=dip1000` gives the following error message: example.d(16): Error: ` safe` function `D main` cannot call ` system` function `example.bar!().bar` example.d(11): which wasn't inferred ` safe` because of: example.d(11): reference to local variable `n` assigned to non-scope parameter `p` calling `foo` The problem with this message is that it does not explain *why* `p` is a non-scope parameter. In order to get that information, we have to modify the definition of `foo` and add `scope` manually: -void foo()(int* p) +void foo()(scope int* p) When we do, we get an error message that points to the *real* source of the problem: example2.d(16): Error: ` safe` function `D main` cannot call ` system` function `example2.bar!().bar` example2.d(3): which calls `example2.foo!().foo` example2.d(5): which wasn't inferred ` safe` because of: example2.d(5): scope variable `p` assigned to global variable `global` In order for normal D programmers to have any hope of using DIP 1000 successfully, all of the relevant information *must* be present in the original error message. Users must not be required to modify their code (or, worse, the code of libraries they depend on) in order to reveal this information.
Aug 27
On Sunday, 25 August 2024 at 13:10:22 UTC, Mike Parker wrote:Second, we'd like to get a number of examples of problems people have had with using DIP1000 that haven't shown up in Bugzilla. Walter wants to be as certain as he can whether such issues are fixable or if the design is fundamentally flawed.Here's an issue that stems from a fundamental shortcoming of DIP1000's design: you can't write a ` safe` `swap` function. Consider the following example: safe void swap(ref /* ??? */ int* a, ref return scope int* b); safe unittest { int local; static int global; // Ensure both pointers have identical lexical scopes static struct Pair { int* a, b; } auto p1 = Pair(&local, &local); auto p2 = Pair(&global, &global); auto p3 = Pair(&local, &global); auto p4 = Pair(&global, &local); swap(p1.a, p1.b); swap(p2.a, p2.b); static assert(!__traits(compiles, swap(p3.a, p3.b))); static assert(!__traits(compiles, swap(p4.a, p4.b))); } A correct, ` safe` function signature for `swap` should be able to pass this unit test. However, under the DIP1000 system, there is no possible combination of attributes for the parameter `a` that will not cause one of the test cases to fail. - `ref int* a` causes `swap(p1.a, p1.b)` to fail. - Both `ref scope int* a` and `ref return scope int* a` cause `swap(p3.a, p3.b)` to compile when it shouldn't.
Aug 28
On 29/08/2024 2:48 AM, Paul Backus wrote:On Sunday, 25 August 2024 at 13:10:22 UTC, Mike Parker wrote:This is a good example of what I mean by multiple outputs are not supported. Both parameters are outputs, not just the first one. ``` onlineapp.d(9): Error: scope variable `temp` assigned to `ref` variable `a` with longer lifetime onlineapp.d(25): Error: scope variable `p1` assigned to non-scope parameter `a` calling `swap` onlineapp.d(8): which is not `scope` because of `b = a` ``` ```d safe void swap(ref /* ??? */ int* a, ref return scope int* b) { int* temp = b; b = a; a = temp; } safe unittest { int local; static int global; // Ensure both pointers have identical lexical scopes static struct Pair { int* a, b; } auto p1 = Pair(&local, &local); auto p2 = Pair(&global, &global); auto p3 = Pair(&local, &global); auto p4 = Pair(&global, &local); swap(p1.a, p1.b); swap(p2.a, p2.b); static assert(!__traits(compiles, swap(p3.a, p3.b))); static assert(!__traits(compiles, swap(p4.a, p4.b))); } ``` The line ``b = a;`` should also be erroring, but because ``b`` isn't seen as an error, its accepted.Second, we'd like to get a number of examples of problems people have had with using DIP1000 that haven't shown up in Bugzilla. Walter wants to be as certain as he can whether such issues are fixable or if the design is fundamentally flawed.Here's an issue that stems from a fundamental shortcoming of DIP1000's design: you can't write a ` safe` `swap` function. Consider the following example: safe void swap(ref /* ??? */ int* a, ref return scope int* b); safe unittest { int local; static int global; // Ensure both pointers have identical lexical scopes static struct Pair { int* a, b; } auto p1 = Pair(&local, &local); auto p2 = Pair(&global, &global); auto p3 = Pair(&local, &global); auto p4 = Pair(&global, &local); swap(p1.a, p1.b); swap(p2.a, p2.b); static assert(!__traits(compiles, swap(p3.a, p3.b))); static assert(!__traits(compiles, swap(p4.a, p4.b))); } A correct, ` safe` function signature for `swap` should be able to pass this unit test. However, under the DIP1000 system, there is no possible combination of attributes for the parameter `a` that will not cause one of the test cases to fail. - `ref int* a` causes `swap(p1.a, p1.b)` to fail. - Both `ref scope int* a` and `ref return scope int* a` cause `swap(p3.a, p3.b)` to compile when it shouldn't.
Aug 28
I've been thinking about swap/move functions some more for my own proposal.
The following applies to swap just as much as move, but I'll use move as
its simpler to understand here.
Lets say you don't have owner escape analysis (so just DIP1000).
And you have some stack memory that then gets borrowed from.
You can corrupt the borrowed memory, by calling a function like move.
Simply because a dependency exists of the thing being moved/swapped.
```d
int* value = ...;
int** ptr = &value; // ptr depends upon value
int* another = move(value);
// ugh oh, ptr is now corrupted
```
Now lets say you are able to annotate the escape sets fully (DIP1000 can
do move).
```d
T move(T)(ref return T input);
```
Thing is, this signature just says the parameter contributes towards the
return, it doesn't consume and invalidate the original input variable.
So a solution is to throw a new attribute into the mix like
`` escapeas(return)``. To show that it'll consume the input and it will
go to a specific variable.
```d
void swap(T)(ref escapeas(input2) T input1, ref escapeas(input1) T
input2);
```
Ok great, you modeled the escapes, congratulations.
Or you could use owner escape analysis to say "if have borrow, make
effectively const" there cannot be assigned to.
Boom, neither swap nor move could modify the original and it all works.
So there are three scenarios here that I can think of:
1. Attribute the exact variable it'll escape to, to know that it will
invalidate the input, preventing any relationships from it from being
corrupted
2. Make these functions `` system`` and say its a rare enough thing that
it doesn't need to be made `` safe``
3. Use owner escape analysis to provide the effectively const guarantees
to prevent the mutation
Full example of a move in `` safe`` code, that can corrupt memory (null,
wrong length, wrong value ext.):
```d
safe:
void main() {
int*[1] value;
int** ptr = &value[0];
int* another = move(value[0]);
}
T move(T)(scope ref return T input) {
T temp = input;
input = T.init;
return temp;
}
```
Aug 29
On 29/08/2024 8:53 PM, Richard (Rikki) Andrew Cattermole wrote:Thing is, this signature just says the parameter contributes towards the return, it doesn't consume and invalidate the original input variable.To clarify, "invalidate the original input variable" means that from the caller function, after the function call the argument variables will no longer be the values that the compiler will be expecting. Potentially causing memory corruption that would otherwise have been caught.
Aug 29
On Wednesday, 28 August 2024 at 14:48:04 UTC, Paul Backus wrote:On Sunday, 25 August 2024 at 13:10:22 UTC, Mike Parker wrote:Interesting, thank you. I went and looked at how Rust fixes this and as I expected it's with an `unsafe` block: https://doc.rust-lang.org/1.80.1/src/core/mem/mod.rs.html#728[...]Here's an issue that stems from a fundamental shortcoming of DIP1000's design: you can't write a ` safe` `swap` function. [...]
Aug 29
On Thursday, 29 August 2024 at 16:10:37 UTC, Atila Neves wrote:On Wednesday, 28 August 2024 at 14:48:04 UTC, Paul Backus wrote:To be clear, the issue here is not that the *implementation* requires `unsafe`/` trusted`. The issue is that under the current system, it is impossible to give `swap` a ` safe` *function signature* that accepts `scope` arguments. That's why I left the body of the `swap` function out of my example. It's completely irrelevant.On Sunday, 25 August 2024 at 13:10:22 UTC, Mike Parker wrote:Interesting, thank you. I went and looked at how Rust fixes this and as I expected it's with an `unsafe` block: https://doc.rust-lang.org/1.80.1/src/core/mem/mod.rs.html#728[...]Here's an issue that stems from a fundamental shortcoming of DIP1000's design: you can't write a ` safe` `swap` function. [...]
Aug 29
On Thu, Aug 29, 2024 at 07:38:50PM +0000, Paul Backus via Digitalmars-d wrote: [...]To be clear, the issue here is not that the *implementation* requires `unsafe`/` trusted`. The issue is that under the current system, it is impossible to give `swap` a ` safe` *function signature* that accepts `scope` arguments. That's why I left the body of the `swap` function out of my example. It's completely irrelevant.Exactly; this is why 'scope' is an inadequate solution to the wrong problem. The whole discussion about lifetimes in D sounds like D really, badly, wants a Rust-like lifetime system, but, not wanting to ape Rust, which is considered too complex, we're trying instead to shoehorn various simple but incomplete solutions to try to pretend to solve the original problem. I appreciate Walter's efforts to find simple solutions to problems -- that's the quality of a worthy engineer -- but some problems are simply inherently complex, and any solution that's simpler than the minimum complexity simply cannot be made to work, no matter how hard you try. More and more, lifetime tracking is appearing to be one of these problems. Any solution less complex than Rust's seems doomed to fail on some obscure corner case that will nevertheless compromise the entire solution. T -- Nearly all men can stand adversity, but if you want to test a man's character, give him power. -- Abraham Lincoln
Aug 29
On Thursday, 29 August 2024 at 19:38:50 UTC, Paul Backus wrote:On Thursday, 29 August 2024 at 16:10:37 UTC, Atila Neves wrote:Ah. Yes, that's bad.On Wednesday, 28 August 2024 at 14:48:04 UTC, Paul Backus wrote:To be clear, the issue here is not that the *implementation* requires `unsafe`/` trusted`. The issue is that under the current system, it is impossible to give `swap` a ` safe` *function signature* that accepts `scope` arguments. That's why I left the body of the `swap` function out of my example. It's completely irrelevant.[...]Interesting, thank you. I went and looked at how Rust fixes this and as I expected it's with an `unsafe` block: https://doc.rust-lang.org/1.80.1/src/core/mem/mod.rs.html#728
Aug 30
On Sunday, 25 August 2024 at 13:10:22 UTC, Mike Parker wrote:Second, we'd like to get a number of examples of problems people have had with using DIP1000 that haven't shown up in Bugzilla. Walter wants to be as certain as he can whether such issues are fixable or if the design is fundamentally flawed.A problem I’ve often encountered with the design of DIP1000 is that factory functions returning `scope` variables isn’t really possible. I often want to construct something in a function and then return it, which allows me to make my code more modular. Technically you can do this with a `ref` parameter, but then you have to always remember to create the variable and then populate it with the factory function… and then I remember that I can just turn DIP1000 off, so I do. It’s great that I *could* put a class instance on the stack with `scope`, but I don’t want to deal with not even being able to return it. In short, my experience with DIP1000 `scope` is like if `pure` was upgraded to the final boss of D; declaring ‘thou shalt write a program that never returns anything!’
Aug 29
On Thursday, 29 August 2024 at 21:30:31 UTC, IchorDev wrote:On Sunday, 25 August 2024 at 13:10:22 UTC, Mike Parker wrote:Just wanted to add that I do think `scope` is a fantastic and powerful tool for library APIs to indicate that they’re not going to hold on to your data, meaning you don’t have to worry about it being modified from outside. I think any replacement to DIP1000’s `scope` should provide some way to make the same guarantee about whether a parameter is retained or not. (And obviously it should be inferred by default!)Second, we'd like to get a number of examples of problems people have had with using DIP1000 that haven't shown up in Bugzilla. Walter wants to be as certain as he can whether such issues are fixable or if the design is fundamentally flawed.In short, my experience with DIP1000 `scope` is like if `pure` was upgraded to the final boss of D; declaring ‘thou shalt write a program that never returns anything!’
Aug 29