www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - On Borrow Checking

reply Walter Bright <newshound2 digitalmars.com> writes:
I caveat my remarks with although I've have studed the Rust specification, I 
have not written a line of Rust code.

I was quite intrigued with the borrow checker, and set about learning about it. 
While D cannot be retrofitted with a borrow checker, it can be enhanced with
it. 
A borrow checker has nothing tying it to the Rust syntax, so it should work.

So I implemented a borrow checker for D, and it is enabled by adding the
` live` 
annotation for a function, which turns on the borrow checker for that function. 
There are no syntax or semantic changes to the language, other than laying on a 
borrow checker.

Yes, it does data flow analysis, has semantic scopes, yup. It issues errors in 
the right places, although the error messages are rather basic.

In my personal coding style, I have gravitated towards following the borrow 
checker rules. I like it. But it doesn't work for everything.

It reminds me of OOP. OOP was sold as the answer to every programming problem. 
Many OOP languages appeared. But, eventually, things died down and OOP became 
just another tool in the toolbox. D and C++ support OOP, too.

I predict that over time the borrow checker will become just another tool in
the 
toolbox, and it'll be used for algorithms and data structures where it makes 
sense, and other methods will be used where it doesn't.

I've been around to see a lot of fashions in programming, which is most likely 
why D is a bit of a polyglot language :-/





The language can nail that down for you (D does). What's left are memory 
allocation errors. Garbage collection fixes that.
Apr 29
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 4/29/25 19:12, Walter Bright wrote:

 I caveat my remarks with although I've have studed the Rust 
 specification, I have not written a line of Rust code.
 
 I was quite intrigued with the borrow checker, and set about learning 
 about it. While D cannot be retrofitted with a borrow checker, it can be 
 enhanced with it. A borrow checker has nothing tying it to the Rust 
 syntax, so it should work.
 
 So I implemented a borrow checker for D, and it is enabled by adding the 
 ` live` annotation for a function, which turns on the borrow checker for 
 that function. There are no syntax or semantic changes to the language, 
 other than laying on a borrow checker.
 
 Yes, it does data flow analysis, has semantic scopes, yup. It issues 
 errors in the right places, although the error messages are rather basic.
 
 In my personal coding style, I have gravitated towards following the 
 borrow checker rules. I like it. But it doesn't work for everything.
 
 It reminds me of OOP. OOP was sold as the answer to every programming 
 problem. Many OOP languages appeared. But, eventually, things died down 
 and OOP became just another tool in the toolbox. D and C++ support OOP, 
 too.
 
 I predict that over time the borrow checker will become just another 
 tool in the toolbox, and it'll be used for algorithms and data 
 structures where it makes sense, and other methods will be used where it 
 doesn't.
 
 I've been around to see a lot of fashions in programming, which is most 
 likely why D is a bit of a polyglot language :-/
 



 (use arrays and ref's instead).
 
 The language can nail that down for you (D does). What's left are memory 
 allocation errors. Garbage collection fixes that.


I am fully on board with your high-level fundamentals here.

However, I do not agree that ` live` really lives up to them. It's a 
borrow-checking-inspired linting tool for primarily ` system` code. It 
is not the kind of innovative tool people are likely to expect to find 
in the toolbox going forward.

At least my expectation would be that this is a borrow checker that 
plays nice with a language supporting memory safe mutable aliasing while 
providing memory safety guarantees for ` nogc` code when it is used.
Apr 29
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 29 April 2025 at 20:40:51 UTC, Timon Gehr wrote:

 On 4/29/25 19:12, Walter Bright wrote:

 [...]


 I am fully on board with your high-level fundamentals here.

 However, I do not agree that ` live` really lives up to them. 
 It's a borrow-checking-inspired linting tool for primarily 
 ` system` code. It is not the kind of innovative tool people 
 are likely to expect to find in the toolbox going forward.


Hmm, for some reason I thought  live only worked with  safe code, 
like DIP 1000, but I when I scan over the  live documentation I 
don’t see any reference to  safe.
Apr 29
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/29/2025 3:46 PM, jmh530 wrote:

 Hmm, for some reason I thought  live only worked with  safe code, like DIP
1000, 
 but I when I scan over the  live documentation I don’t see any reference to
 safe.


It works with both.
Apr 29
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/29/2025 1:40 PM, Timon Gehr wrote:

 However, I do not agree that ` live` really lives up to them. It's a 
 borrow-checking-inspired linting tool for primarily ` system` code. It is not 
 the kind of innovative tool people are likely to expect to find in the toolbox 
 going forward.


I'm aware that you and I disagree on the details! But I'm reluctant to invest 
more time on it given the decided general lack of interest in it.

P.S. maybe we can have a great discussion on it at Dconf! I was very pleased 
with how our rvalue discussion went last time.



 At least my expectation would be that this is a borrow checker that plays nice 
 with a language supporting memory safe mutable aliasing while providing memory 
 safety guarantees for ` nogc` code when it is used.
Apr 29
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 29 April 2025 at 23:40:56 UTC, Walter Bright wrote:

 On 4/29/2025 1:40 PM, Timon Gehr wrote:

 However, I do not agree that ` live` really lives up to them. 
 It's a borrow-checking-inspired linting tool for primarily 
 ` system` code. It is not the kind of innovative tool people 
 are likely to expect to find in the toolbox going forward.


 I'm aware that you and I disagree on the details! But I'm 
 reluctant to invest more time on it given the decided general 
 lack of interest in it.

 [snip]


I have flashbacks of the recent D hacker news thread where you 
said marketing isn't your strong suit...

You wrote a blog post on ownership and borrowing back in 2019 
(https://dlang.org/blog/2019/07/15/ownership-and-borrowing-in-d/), but it has
just a brief mention of  live. Would you want to expand your original post with
an emphasis on  live, why to use it, and what it gets you?

Otherwise to your point about lack of interest, my recollection 
is that there has been a lot of discussion over the years about 
safe reference-counting, maybe and allocator-aware container 
libraries (I assume this is possible but limited in ways that 
people aren't happy about). And my sense is that DIP 1000 and 
 live haven't quite gotten us to the place where people want the 
language to be. That could be a driver to the lack of interest.

There are only so many hours in the day, and I'm sure people 
would prefer you polish some other features before focusing on 
this. But I don't think it is unwise to be thinking about these 
issues.
Apr 29
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/29/2025 6:26 PM, jmh530 wrote:

 You wrote a blog post on ownership and borrowing back in 2019 
 (https://dlang.org/blog/2019/07/15/ownership-and-borrowing-in-d/), but it has 
 just a brief mention of  live. Would you want to expand your original post
with 
 an emphasis on  live, why to use it, and what it gets you?
 
 Otherwise to your point about lack of interest, my recollection is that there 
 has been a lot of discussion over the years about safe reference-counting,
maybe 
 and allocator-aware container libraries (I assume this is possible but limited 
 in ways that people aren't happy about). And my sense is that DIP 1000 and
 live 
 haven't quite gotten us to the place where people want the language to be.
That 
 could be a driver to the lack of interest.


We spent a considerable time thinking about safe reference-counting. We could 
not find a way around its considerable performance penalties, and its unsafety, 
and so abandoned it.
Apr 29
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 30/04/2025 2:58 PM, Walter Bright wrote:

 On 4/29/2025 6:26 PM, jmh530 wrote:

 You wrote a blog post on ownership and borrowing back in 2019 
 (https://dlang.org/blog/2019/07/15/ownership-and-borrowing-in-d/), but 
 it has just a brief mention of  live. Would you want to expand your 
 original post with an emphasis on  live, why to use it, and what it 
 gets you?

 Otherwise to your point about lack of interest, my recollection is 
 that there has been a lot of discussion over the years about safe 
 reference-counting, maybe and allocator-aware container libraries (I 
 assume this is possible but limited in ways that people aren't happy 
 about). And my sense is that DIP 1000 and  live haven't quite gotten 
 us to the place where people want the language to be. That could be a 
 driver to the lack of interest.


 
 We spent a considerable time thinking about safe reference-counting. We 
 could not find a way around its considerable performance penalties, and 
 its unsafety, and so abandoned it.


I'm still trying to solve it with my DFA work, however I do have a 
backup plan.

1. Ban all pointers. System handles and with that coroutines/fibers are 
all  trusted and  system so it doesn't effect them. Anything else should 
be using the GC and therefore unaffected by this restriction.

2. Let the backend handle optimizations. Reference counting primitives 
are improving (at least in LLVM). Worst case scenario, we have the exact 
same performance as we do now, but with a better user experience.
Apr 29
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/29/2025 8:25 PM, Richard (Rikki) Andrew Cattermole wrote:

 1. Ban all pointers. System handles and with that coroutines/fibers are all 
  trusted and  system so it doesn't effect them. Anything else should be using 
 the GC and therefore unaffected by this restriction.


Memory unsafety happens when there's a pointer into the reference counted
object 
that is a direct pointer.



 2. Let the backend handle optimizations. Reference counting primitives are 
 improving (at least in LLVM). Worst case scenario, we have the exact same 
 performance as we do now, but with a better user experience.


Performance problems come from:

1. using a mutex (not an issue for thread local data)

2. needing an exception handler for the decrement

3. increment

4. decrement-test-free
Apr 30
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 01/05/2025 10:53 AM, Walter Bright wrote:

 On 4/29/2025 8:25 PM, Richard (Rikki) Andrew Cattermole wrote:

 1. Ban all pointers. System handles and with that coroutines/fibers 
 are all  trusted and  system so it doesn't effect them. Anything else 
 should be using the GC and therefore unaffected by this restriction.


 
 Memory unsafety happens when there's a pointer into the reference 
 counted object that is a direct pointer.


Right, if you don't have the escape analysis to protect the owner, 
banning all pointers surrounding the RC type solves it inherently.

I don't like the idea of banning all pointers, but it does solve the 
problem for system handles since they are not regular code. Which is the 
purpose of having RC in the language (for my motivation).


 2. Let the backend handle optimizations. Reference counting primitives 
 are improving (at least in LLVM). Worst case scenario, we have the 
 exact same performance as we do now, but with a better user experience.


 
 Performance problems come from:
 
 1. using a mutex (not an issue for thread local data)
 
 2. needing an exception handler for the decrement
 
 3. increment
 
 4. decrement-test-free


Why are you treating these as new costs? I don't understand.
They are already being paid for with copy constructors and destructors.

But if you are want to remove it, it can be solved.
My plan was to solve this with my DFA framework.

However, for a dedicated solution:

1. Put the add call into the called function as part of the calling 
convention (extremely important).

2. If a variable declaration has a RC type, flag the function 
declaration as containing RC during semantic 3.

3. If a function contains RC, call optimization DFA iff optimizations 
are turned on (speed is not a concern if optimizations are turned off).

4. Use a stack, a nice simple one.

4.1. When you see an add, add the call to the stack.

4.2. When you see end of scope, remove all add's in stack corresponding 
to that stack.

4.2.1. If a removed add does not have the flag that it is required, 
disable it and its corresponding sub.

4.3. If mutation event occurs on a variable, find its last add in stack, 
flag it as requried.

4.3.1. A mutation event is anything that could mutate the variable, i.e. 
ref, out, taking a pointer, assignment.

5. Do not codegen any expression that has been marked as disabled.

You do not need control flow block handling for this.
If it cannot disable calls, its because they are needed.


```d
void func(RC input) {
     input.add; // elidable
     scope(exit)
		input.sub; // elidable

     if (condition) {
         input.sub;
         input = RC(...);
         input.add;
     }
}
```
Apr 30
next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 01/05/2025 2:28 PM, Richard (Rikki) Andrew Cattermole wrote:

 ```d
 void func(RC input) {
      input.add; // elidable
      scope(exit)
          input.sub; // elidable
 
      if (condition) {
          input.sub;
          input = RC(...);
          input.add;
      }
 }
 ```


Ok that example is bad, no sub oops.

I came up with this optimization solution a long time ago and may have 
forgot a few things.
Apr 30
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/30/2025 7:51 PM, Richard (Rikki) Andrew Cattermole wrote:

 I came up with this optimization solution a long time ago and may have forgot
a 
 few things.


No prob. Happens to me all the time!
Apr 30
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/30/2025 7:28 PM, Richard (Rikki) Andrew Cattermole wrote:

 Why are you treating these as new costs? I don't understand.
 They are already being paid for with copy constructors and destructors.


Pointers and references are used to avoid needing copy/destroy



 However, for a dedicated solution:


It's an interesting approach. I recommend hand-coding it, and testing the 
performance.
Apr 30
parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 01/05/2025 6:08 PM, Walter Bright wrote:

     However, for a dedicated solution:
 
 It's an interesting approach. I recommend hand-coding it, and testing 
 the performance.


I am unconvinced that such test code would be worth trying to benchmark.

On the newer x86 cpu's we're talking 1-8uops for atomic operations.

And test code tends not to reflect the problem domain, which for system 
handles involves a lot of costly syscalls and IPC, that make atomics and 
with that system lock basically free.

In saying that, unnecessary work is still unnecessary and removing it is 
a good thing. However I won't miss the optimizations, but I do miss not 
having classes that are RC (going with struct wrapper can result in CT 
errors).
May 01
prev sibling parent Derek Fawcus <dfawcus+dlang employees.org> writes:
On Wednesday, 30 April 2025 at 01:26:14 UTC, jmh530 wrote:

 And my sense is that DIP 1000 and  live haven't quite gotten us 
 to the place where people want the language to be. That could 
 be a driver to the lack of interest.


I'm playing with DIP 1000, mainly because I'm also playing with 
 safe and want to be able to pass pointers to local variables in 
to functions.  (This is using DIP1000, DIP1021 and preview=in 
together with  safe.)

Note I want to use pointers, not references so as to improve 
local reasoning about the code.
Apr 30
prev sibling parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 30/04/2025 11:40 AM, Walter Bright wrote:

 I'm aware that you and I disagree on the details! But I'm reluctant to 
 invest more time on it given the decided general lack of interest in it.


I've been suspecting for a couple of months, that you've been feeling 
like that you've done your time on it, which this confirms.

If we're getting a different DFA based language feature solution, it'll 
be because somebody else implemented it.

It won't be come as any surprise that the result of this has been me 
implementing what I've been calling the fast DFA :)
Apr 29
prev sibling next sibling parent reply Luna <luna foxgirls.gay> writes:
On Tuesday, 29 April 2025 at 17:12:41 UTC, Walter Bright wrote:

 I caveat my remarks with although I've have studed the Rust 
 specification, I have not written a line of Rust code.

 I was quite intrigued with the borrow checker, and set about 
 learning about it. While D cannot be retrofitted with a borrow 
 checker, it can be enhanced with it. A borrow checker has 
 nothing tying it to the Rust syntax, so it should work.

 So I implemented a borrow checker for D, and it is enabled by 
 adding the ` live` annotation for a function, which turns on 
 the borrow checker for that function. There are no syntax or 
 semantic changes to the language, other than laying on a borrow 
 checker.

 Yes, it does data flow analysis, has semantic scopes, yup. It 
 issues errors in the right places, although the error messages 
 are rather basic.

 In my personal coding style, I have gravitated towards 
 following the borrow checker rules. I like it. But it doesn't 
 work for everything.

 It reminds me of OOP. OOP was sold as the answer to every 
 programming problem. Many OOP languages appeared. But, 
 eventually, things died down and OOP became just another tool 
 in the toolbox. D and C++ support OOP, too.

 I predict that over time the borrow checker will become just 
 another tool in the toolbox, and it'll be used for algorithms 
 and data structures where it makes sense, and other methods 
 will be used where it doesn't.

 I've been around to see a lot of fashions in programming, which 
 is most likely why D is a bit of a polyglot language :-/




 pointer arithmetic (use arrays and ref's instead).

 The language can nail that down for you (D does). What's left 
 are memory allocation errors. Garbage collection fixes that.


My personal opinion on the matter has softened a bit over time. I 
do think  live could overall be a nice addition to the language. 
Much like how  nogc is an opt in, there are cases where opting in 
to borrow checking would be interesting. So as an opt-in 
attribute I’d not mind it being in the language; I’m only opposed 
to it being enforced. My strong feelings in the past was mainly 
down to that. Didn’t want to end up having it forced on me 
everywhere. As long as dlang keeps its own identity and doesn’t 
try to become rust, I at least think it’d be an ok change 
nowadays.
Apr 29
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/29/2025 5:24 PM, Luna wrote:

 My personal opinion on the matter has softened a bit over time. I do think
 live 
 could overall be a nice addition to the language. Much like how  nogc is an
opt 
 in, there are cases where opting in to borrow checking would be interesting.
So 
 as an opt-in attribute I’d not mind it being in the language; I’m only
opposed 
 to it being enforced. My strong feelings in the past was mainly down to that. 
 Didn’t want to end up having it forced on me everywhere. As long as dlang
keeps 
 its own identity and doesn’t try to become rust, I at least think it’d be
an ok 
 change nowadays.


I agree that  live should be opt-in.
Apr 30
prev sibling next sibling parent a11e99z <a a.com> writes:
On Tuesday, 29 April 2025 at 17:12:41 UTC, Walter Bright wrote:

 I caveat my remarks with although I've have studed the Rust 
 specification, I have not written a line of Rust code.


https://github.com/rust-lang/polonius

https://gcc.gnu.org/pipermail/gcc-patches/2025-March/677874.html

from this news in Russian:
https://www.opennet.ru/opennews/art.shtml?num=62924
May 01
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On Wed, 30 Apr 2025 at 03:16, Walter Bright via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 I caveat my remarks with although I've have studed the Rust specification,
 I
 have not written a line of Rust code.

 I was quite intrigued with the borrow checker, and set about learning
 about it.
 While D cannot be retrofitted with a borrow checker, it can be enhanced
 with it.
 A borrow checker has nothing tying it to the Rust syntax, so it should
 work.

 So I implemented a borrow checker for D, and it is enabled by adding the
 ` live`
 annotation for a function, which turns on the borrow checker for that
 function.
 There are no syntax or semantic changes to the language, other than laying
 on a
 borrow checker.

 Yes, it does data flow analysis, has semantic scopes, yup. It issues
 errors in
 the right places, although the error messages are rather basic.

 In my personal coding style, I have gravitated towards following the
 borrow
 checker rules. I like it. But it doesn't work for everything.

 It reminds me of OOP. OOP was sold as the answer to every programming
 problem.
 Many OOP languages appeared. But, eventually, things died down and OOP
 became
 just another tool in the toolbox. D and C++ support OOP, too.

 I predict that over time the borrow checker will become just another tool
 in the
 toolbox, and it'll be used for algorithms and data structures where it
 makes
 sense, and other methods will be used where it doesn't.

 I've been around to see a lot of fashions in programming, which is most
 likely
 why D is a bit of a polyglot language :-/


 errors is

 variables.


 The language can nail that down for you (D does). What's left are memory
 allocation errors. Garbage collection fixes that.


On a slight tangent; why do I need to attribute the function at all for it
to check `scope` args don't escape?
I do want to add `scope` to pointers and ref arguments, but I don't see any
reason that I should have to attribute `live`... why isn't `scope` enough
to enable escape analysis?
May 02
next sibling parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 02/05/2025 7:44 PM, Manu wrote:

 On a slight tangent; why do I need to attribute the function at all for 
 it to check `scope` args don't escape?
 I do want to add `scope` to pointers and ref arguments, but I don't see 
 any reason that I should have to attribute `live`... why isn't `scope` 
 enough to enable escape analysis?


DIP1000 was the escape analysis meant to be enabled by scope.

And it can't be turned on.
May 02
prev sibling next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Friday, 2 May 2025 at 07:44:52 UTC, Manu wrote:

 On a slight tangent; why do I need to attribute the function at 
 all for it check `scope` args don't escape?


You don't need  live for that, only  safe and -preview=dip1000. 
 live / -preview=dip1021 checks for mutable aliasing, which is 
something added on top of `scope` escape analysis.
May 02
parent reply Manu <turkeyman gmail.com> writes:
On Fri, 2 May 2025 at 23:10, Dennis via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Friday, 2 May 2025 at 07:44:52 UTC, Manu wrote:

 On a slight tangent; why do I need to attribute the function at
 all for it check `scope` args don't escape?


 You don't need  live for that, only  safe and -preview=dip1000.
  live / -preview=dip1021 checks for mutable aliasing, which is
 something added on top of `scope` escape analysis.


Okay, so then why should `scope` need ` safe`? It's an additional attribute
added in its own right; there's no apparent value to requiring a SECOND
attribute's presence in order to make the first attribute take effect.
`scope` should work when you write `scope`. If you don't want scope
checking, don't write `scope`...?
May 10
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/10/2025 5:10 AM, Manu wrote:

 Okay, so then why should `scope` need ` safe`?


It doesn't. It needs -dip1000, though.

```
int* foo(scope int* q) { return q; }
```
```
./cc -c x.d -dip1000
x.d(1): Error: scope parameter `q` may not be returned
int* foo(scope int* q) { return q; }
                                 ^
```
May 10
parent reply Manu <turkeyman gmail.com> writes:
On Sun, 11 May 2025, 1:46=E2=80=AFam Walter Bright via Digitalmars-d, <
digitalmars-d puremagic.com> wrote:


 On 5/10/2025 5:10 AM, Manu wrote:

 Okay, so then why should `scope` need ` safe`?


 It doesn't. It needs -dip1000, though.

 ```
 int* foo(scope int* q) { return q; }
 ```
 ```
 ./cc -c x.d -dip1000
 x.d(1): Error: scope parameter `q` may not be returned
 int* foo(scope int* q) { return q; }
                                  ^
 ```


I tested that and it didn't work for me. Maybe my test was faulty somehow
since I was working through numerous combinations, I'll try it again...






 May 10



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/10/2025 9:07 PM, Manu wrote:

 I tested that and it didn't work for me. Maybe my test was faulty somehow
since 
 I was working through numerous combinations, I'll try it again...



I just tried it again, with the same result.



 May 11



  parent reply Manu <turkeyman gmail.com>  writes:


On Mon, 12 May 2025 at 03:55, Walter Bright via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On 5/10/2025 9:07 PM, Manu wrote:

 I tested that and it didn't work for me. Maybe my test was faulty


 somehow since

 I was working through numerous combinations, I'll try it again...



 I just tried it again, with the same result.


__gshared int* g;
void test(scope int* x)
{
    g = x;
}

I just compiled this with -dip1000, and it compiles... This looks like an
escape to me! What have I misunderstood here?



 May 12



 next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz>  writes:


On 12/05/2025 9:24 PM, Manu wrote:

 On Mon, 12 May 2025 at 03:55, Walter Bright via Digitalmars-d 
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:
 
     On 5/10/2025 9:07 PM, Manu wrote:
      > I tested that and it didn't work for me. Maybe my test was faulty
     somehow since
      > I was working through numerous combinations, I'll try it again...
 
 
     I just tried it again, with the same result.
 
 
 __gshared int* g;
 void test(scope int* x)
 {
      g = x;
 }
 
 I just compiled this with -dip1000, and it compiles... This looks like 
 an escape to me! What have I misunderstood here?


Both ``__gshared`` and ``shared`` is disallowed by `` safe`` (direct 
assign/access), and `` safe`` has to be in use.

``-preview=dip1000``

```d
int* g;
void test(scope int* x)  safe {
     g = x; // Error: scope variable `x` assigned to global variable `g`
}
```



 May 12



  parent reply Manu <turkeyman gmail.com>  writes:


On Mon, 12 May 2025 at 19:41, Richard (Rikki) Andrew Cattermole via
Digitalmars-d <digitalmars-d puremagic.com> wrote:


 On 12/05/2025 9:24 PM, Manu wrote:

 On Mon, 12 May 2025 at 03:55, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>>


 wrote:

     On 5/10/2025 9:07 PM, Manu wrote:
      > I tested that and it didn't work for me. Maybe my test was faulty
     somehow since
      > I was working through numerous combinations, I'll try it again...


     I just tried it again, with the same result.


 __gshared int* g;
 void test(scope int* x)
 {
      g = x;
 }

 I just compiled this with -dip1000, and it compiles... This looks like
 an escape to me! What have I misunderstood here?


 Both ``__gshared`` and ``shared`` is disallowed by `` safe`` (direct
 assign/access), and `` safe`` has to be in use.

 ``-preview=dip1000``

 ```d
 int* g;
 void test(scope int* x)  safe {
      g = x; // Error: scope variable `x` assigned to global variable `g`
 }
 ```


`scope` shouldn't depend on  safe, and Walter above said it depended only
on -dip1000.
So, if it only works with  safe, I repeat again; WHY? It's not only
redundant annotation, but it's also a ticking bomb waiting to explode... I
already commented on this.



 May 12



 next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz>  writes:


On 12/05/2025 10:55 PM, Manu wrote:

 `scope` shouldn't depend on  safe, and Walter above said it depended 
 only on -dip1000.
 So, if it only works with  safe, I repeat again; WHY? It's not only 
 redundant annotation, but it's also a ticking bomb waiting to explode... 
 I already commented on this.


 safe is the static analysis on a function that the compiler guarantees 
is memory safe.

 trusted is the escape hatch where you have to do something that the 
compiler cannot make guarantees for.

 system has no guarantees associated with it.

All scope says is that this variable will not escape this function.

It does not have any kind of ownership properties.

It is not redundant information, it does serve an important purpose in 
escape analysis.

When you see no attributes, think unknown escape behavior, it is not 
modeled. Not, this doesn't escape, or it escapes into an unknown 
location. These are all different things.



 May 12



  parent reply Manu <turkeyman gmail.com>  writes:


On Mon, 12 May 2025 at 21:10, Richard (Rikki) Andrew Cattermole via
Digitalmars-d <digitalmars-d puremagic.com> wrote:


 On 12/05/2025 10:55 PM, Manu wrote:

 `scope` shouldn't depend on  safe, and Walter above said it depended
 only on -dip1000.
 So, if it only works with  safe, I repeat again; WHY? It's not only
 redundant annotation, but it's also a ticking bomb waiting to explode...
 I already commented on this.


  safe is the static analysis on a function that the compiler guarantees
 is memory safe.


`scope` has nothing to do with  safe. I want to prohibit the callee
retaining the argument; simple as that.

 trusted is the escape hatch where you have to do something that the

 compiler cannot make guarantees for.


This has nothing to do with  safe.

 system has no guarantees associated with it.

 All scope says is that this variable will not escape this function.



Exactly. Nothing to do with  safe.

It does not have any kind of ownership properties.


Correct.

It is not redundant information, it does serve an important purpose in

 escape analysis.


Incorrect.

When you see no attributes, think unknown escape behavior, it is not

 modeled. Not, this doesn't escape, or it escapes into an unknown
 location. These are all different things.


Okay, I'll repeat for the 3rd time or so; when `scope` is annotated to an
argument, it MUST have effect, otherwise an escape related crash is
inevitable that the user intended to prevent, and when they start debugging
and trying to track down the issue; they will see the `scope` annotation,
assume it's not that, then move on to continue looking in the wrong place
for their issue.

`scope` has nothing to do with  safe, and it MUST work in its own right,
otherwise the code is lying to the author, and it's an effective liability.



 May 12



 next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz>  writes:


On 12/05/2025 11:16 PM, Manu wrote:

 On Mon, 12 May 2025 at 21:10, Richard (Rikki) Andrew Cattermole via 
 Digitalmars-d <digitalmars-d puremagic.com <mailto:digitalmars- 
 d puremagic.com>> wrote:
 
     On 12/05/2025 10:55 PM, Manu wrote:
      > `scope` shouldn't depend on  safe, and Walter above said it depended
      > only on -dip1000.
      > So, if it only works with  safe, I repeat again; WHY? It's not only
      > redundant annotation, but it's also a ticking bomb waiting to
     explode...
      > I already commented on this.
 
      safe is the static analysis on a function that the compiler guarantees
     is memory safe.
 
 
 `scope` has nothing to do with  safe. I want to prohibit the callee 
 retaining the argument; simple as that.


Use  safe.

 system and  trusted simply do not, and will not have these guarantees.

There must be an escape hatch, and those are what D has it defined as.


      trusted is the escape hatch where you have to do something that the
     compiler cannot make guarantees for.
 
 
 This has nothing to do with  safe.
 
      system has no guarantees associated with it.
 
     All scope says is that this variable will not escape this function.
 
 
 Exactly. Nothing to do with  safe.


It has everything to do with  safe, that is where the mechanical checks 
exist.

You do not need the escape set (``scope``), if you do not have the 
mechanical checks, and where you have an escape hatch (`` trusted`` and 
`` system``), these are intentionally not turned on.


     It is not redundant information, it does serve an important purpose in
     escape analysis.
 
 
 Incorrect.


I have spent the last couple of years studying escape analysis, along 
with what attributes mean when they are missing specifically due to 
DIP1000's ``return scope`` and ``scope return`` mess.

Not only this, but I know enough of the DFA theory including what 
clang-analyzer does for frontend analysis to have been implementing that 
in dmd.

It is not redundant, this is a defining trait of how this stuff works.


     When you see no attributes, think unknown escape behavior, it is not
     modeled. Not, this doesn't escape, or it escapes into an unknown
     location. These are all different things.
 
 
 Okay, I'll repeat for the 3rd time or so; when `scope` is annotated to 
 an argument, it MUST have effect, otherwise an escape related crash is 
 inevitable that the user intended to prevent, and when they start 
 debugging and trying to track down the issue; they will see the `scope` 
 annotation, assume it's not that, then move on to continue looking in 
 the wrong place for their issue.


Okay agreed.

We should disallow it for  system functions, they do not interact with 
any static analysis in any way.


 `scope` has nothing to do with  safe, and it MUST work in its own right, 
 otherwise the code is lying to the author, and it's an effective liability.


No, it is auxiliary lint level annotation on variables.

It is not a type qualifier and is not checked in the type system itself.

Data flow analysis in a frontend is lint level, it works after the rest 
of compilation has been completed. It adds information, but it doesn't 
mess with the type system itself, if it adds information it wants to 
check within the type system, DFA is responsible for doing the checks, 
not the type system. The distinction matters.



 May 12



 next sibling parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 5/12/25 13:52, Richard (Rikki) Andrew Cattermole wrote:

 
 It has everything to do with  safe, that is where the mechanical checks 
 exist.
 
 You do not need the escape set (``scope``), if you do not have the 
 mechanical checks, and where you have an escape hatch (`` trusted`` and 
 `` system``), these are intentionally not turned on.


Even in ` system` code you cannot assign an `int` to an `int*`, you have 
to use an explicit cast. I don't think it is true that just because 
` system` does not give you memory safety guarantees it will therefore 
be expected that features do not work at all.

This is a design decision, and I think Manu's and Walter's expectations 
are more reasonable than what the compiler actually does.



 May 12



 next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz>  writes:


On 13/05/2025 12:34 AM, Timon Gehr wrote:

 On 5/12/25 13:52, Richard (Rikki) Andrew Cattermole wrote:

 It has everything to do with  safe, that is where the mechanical 
 checks exist.

 You do not need the escape set (``scope``), if you do not have the 
 mechanical checks, and where you have an escape hatch (`` trusted`` 
 and `` system``), these are intentionally not turned on.


 
 Even in ` system` code you cannot assign an `int` to an `int*`, you have 
 to use an explicit cast. I don't think it is true that just because 
 ` system` does not give you memory safety guarantees it will therefore 
 be expected that features do not work at all.
 
 This is a design decision, and I think Manu's and Walter's expectations 
 are more reasonable than what the compiler actually does.


Right, perhaps if it was a different design, say unsafe blocks I'd agree 
with what Manu is saying.

Unfortunately the current design has a valid set of tradeoffs.

And from experience you must have some sort of escape hatch, and that is 
`` trusted`` and `` system`` currently, enforcing ``scope`` there would 
make it unusable.



 May 12



 next sibling parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 5/12/25 14:41, Richard (Rikki) Andrew Cattermole wrote:

 On 13/05/2025 12:34 AM, Timon Gehr wrote:

 On 5/12/25 13:52, Richard (Rikki) Andrew Cattermole wrote:

 It has everything to do with  safe, that is where the mechanical 
 checks exist.

 You do not need the escape set (``scope``), if you do not have the 
 mechanical checks, and where you have an escape hatch (`` trusted`` 
 and `` system``), these are intentionally not turned on.


 Even in ` system` code you cannot assign an `int` to an `int*`, you 
 have to use an explicit cast. I don't think it is true that just 
 because ` system` does not give you memory safety guarantees it will 
 therefore be expected that features do not work at all.

 This is a design decision, and I think Manu's and Walter's 
 expectations are more reasonable than what the compiler actually does.


 
 Right, perhaps if it was a different design, say unsafe blocks I'd agree 
 with what Manu is saying.
 
 Unfortunately the current design has a valid set of tradeoffs.
 
 And from experience you must have some sort of escape hatch, and that is 
 `` trusted`` and `` system`` currently, enforcing ``scope`` there would 
 make it unusable.
 


There can be another way to circumvent `scope` checks that is itself 
considered ` system`. As Walter and Manu have demonstrated, it is not 
even consistent. Some checks are still active, others are not.



 May 12



prev sibling  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2025 5:41 AM, Richard (Rikki) Andrew Cattermole wrote:

 Right, perhaps if it was a different design, say unsafe blocks I'd agree with 
 what Manu is saying.


Unsafe blocks were not chosen for D because we wanted  system code to be
clearly 
delineated with thought-out interface to them, not just slapped in to shut up 
compiler complaints about unsafe code.

For a while people would use  trusted lambdas to work like unsafe blocks would. 
The result of that confirmed that we made the right decision to make the 
boundary between safe/unsafe code at the function level. One can still use 
 trusted lambdas, but that use is considered bad practice.



 May 12



prev sibling  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2025 5:34 AM, Timon Gehr wrote:

 Even in ` system` code you cannot assign an `int` to an `int*`, you have to
use 
 an explicit cast. I don't think it is true that just because ` system` does
not 
 give you memory safety guarantees it will therefore be expected that features
do 
 not work at all.


 system code is intended to be rare in a properly written D program. 
Unfortunately, D code is not  safe by default. We are moving to  safe by 
default, but that requires the editions feature.

Where the line is drawn for what is allowed in  system code is a judgement
call, 
not an obvious thing. Requiring `scope` to be transitive across existing
default 
code would discourage people from using it.

Again, I'll point out all the complaints about  nogc being transitive and 
thereby making it difficult to use in existing code.



 May 12



prev sibling  parent reply Nick Treleaven <nick geany.org>  writes:


On Monday, 12 May 2025 at 11:52:47 UTC, Richard (Rikki) Andrew 
Cattermole wrote:

 On 12/05/2025 11:16 PM, Manu wrote:
 We should disallow it for  system functions, they do not 
 interact with any static analysis in any way.


`scope` on a ` system` function parameter is not only useful as a 
form of documentation, it actually also affects the caller:

```d
void f(scope int[])  nogc;

void g()  nogc
{
     f([1, 2]);
}

```
Remove `scope`, it no longer compiles.



 May 12



  parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz>  writes:


On 13/05/2025 2:50 AM, Nick Treleaven wrote:

 On Monday, 12 May 2025 at 11:52:47 UTC, Richard (Rikki) Andrew 
 Cattermole wrote:

 On 12/05/2025 11:16 PM, Manu wrote:
 We should disallow it for  system functions, they do not interact with 
 any static analysis in any way.


 
 `scope` on a ` system` function parameter is not only useful as a form 
 of documentation, it actually also affects the caller:
 
 ```d
 void f(scope int[])  nogc;
 
 void g()  nogc
 {
      f([1, 2]);
 }
 
 ```
 Remove `scope`, it no longer compiles.


I'm aware.

In this scenario that function would need to be made `` trusted``.



 May 12



  parent reply Nick Treleaven <nick geany.org>  writes:


On Monday, 12 May 2025 at 14:58:08 UTC, Richard (Rikki) Andrew 
Cattermole wrote:

 On 13/05/2025 2:50 AM, Nick Treleaven wrote:

 ```d
 void f(scope int[])  nogc;
 
 void g()  nogc
 {
      f([1, 2]);
 }
 
 ```
 Remove `scope`, it no longer compiles.


 I'm aware.

 In this scenario that function would need to be made 
 `` trusted``.



But what if it doesn't have a safe interface - e.g. it writes an 
unsafe value to a global? It cannot be ` trusted`.



 May 12



  parent  "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz>  writes:


On 13/05/2025 3:03 AM, Nick Treleaven wrote:

 On Monday, 12 May 2025 at 14:58:08 UTC, Richard (Rikki) Andrew 
 Cattermole wrote:

 On 13/05/2025 2:50 AM, Nick Treleaven wrote:

 ```d
 void f(scope int[])  nogc;

 void g()  nogc
 {
      f([1, 2]);
 }

 ```
 Remove `scope`, it no longer compiles.


 I'm aware.

 In this scenario that function would need to be made `` trusted``.


 
 
 But what if it doesn't have a safe interface - e.g. it writes an unsafe 
 value to a global? It cannot be ` trusted`.


Then strictly speaking it shouldn't be ``scope``.

Otherwise for codegen only changes, a new attribute in core.attributes 
would be needed.

Of course we could always leave it as is, and accept that its pragmatic.



 May 12



prev sibling  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2025 4:16 AM, Manu wrote:

 `scope` has nothing to do with  safe, and it MUST work in its own right, 
 otherwise the code is lying to the author, and it's an effective liability.


An escape hatch is always needed.

For example, you might want to interface with a 3rd party library that is 
scope-correct, but is not annotated with `scope`.



 May 12



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2025 3:55 AM, Manu wrote:

 So, if it only works with  safe, I repeat again; WHY?


So you can break the rules in  system code.

We are trying to move towards  safe by default, but the new editions feature is 
needed first.



 May 12



  parent reply jmh530 <john.michael.hall gmail.com>  writes:


On Monday, 12 May 2025 at 18:40:02 UTC, Walter Bright wrote:

 On 5/12/2025 3:55 AM, Manu wrote:

 So, if it only works with  safe, I repeat again; WHY?


 So you can break the rules in  system code.

 We are trying to move towards  safe by default, but the new 
 editions feature is needed first.


Wasn't Manu's whole point that scope only really works in  safe 
code and you disagreed and said it didn't depend on that?



 May 12



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2025 12:12 PM, jmh530 wrote:

 Wasn't Manu's whole point that scope only really works in  safe code and you 
 disagreed and said it didn't depend on that?


D is complicated. Sometimes I make mistakes.



 May 12



  parent  jmh530 <john.michael.hall gmail.com>  writes:


On Monday, 12 May 2025 at 21:18:20 UTC, Walter Bright wrote:

 On 5/12/2025 12:12 PM, jmh530 wrote:

 Wasn't Manu's whole point that scope only really works in 
  safe code and you disagreed and said it didn't depend on that?


 D is complicated. Sometimes I make mistakes.


Everyone makes mistakes sometimes.



 May 12



prev sibling next sibling parent  Paolo Invernizzi <paolo.invernizzi gmail.com>  writes:


On Monday, 12 May 2025 at 09:24:13 UTC, Manu wrote:

 On Mon, 12 May 2025 at 03:55, Walter Bright via Digitalmars-d < 
 digitalmars-d puremagic.com> wrote:


 On 5/10/2025 9:07 PM, Manu wrote:

 I tested that and it didn't work for me. Maybe my test was 
 faulty


 somehow since

 I was working through numerous combinations, I'll try it 
 again...



 I just tried it again, with the same result.


 __gshared int* g;
 void test(scope int* x)
 {
     g = x;
 }

 I just compiled this with -dip1000, and it compiles... This 
 looks like an escape to me! What have I misunderstood here?


Have you  safe: on the top?



 May 12



prev sibling  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2025 2:24 AM, Manu wrote:

 __gshared int* g;
 void test(scope int* x)
 {
      g = x;
 }


Need to add  safe. Unadorned functions are  system.



 May 12



prev sibling next sibling parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/10/2025 5:10 AM, Manu wrote:

 Okay, so then why should `scope` need ` safe`? It's an additional attribute 
 added in its own right; there's no apparent value to requiring a SECOND 
 attribute's presence in order to make the first attribute take effect.
 `scope` should work when you write `scope`. If you don't want scope checking, 
 don't write `scope`...?


Because in  system code, you can do whatever you want. But to signal to the 
caller that it is `scope`, add the attribute to the parameter list. It's up to 
the programmer to live up to that promise.

 system could should really be minimized in a properly designed program. That's 
why we're moving to safe by default.



 May 10



prev sibling  parent reply Dennis <dkorpel gmail.com>  writes:


On Saturday, 10 May 2025 at 12:10:48 UTC, Manu wrote:

 Okay, so then why should `scope` need ` safe`?


The escape checker can only maintain its invariants in ` safe` 
code, because in ` system` code, anything goes. scope pointers 
can be laundered by casting to size_t and back for example. Now 
you could argue that the same holds for `const` and `immutable`, 
but we still do the checks in ` system` code for linting 
purposes, requiring an explicit `cast()` to remove `const`. I'm 
not against scope checking in ` system` code per se, but there 
are some differences that make it more complicated. There is 
currently no `cast(notscope)` for example.


 It's an additional attribute added in its own right; there's no 
 apparent value to requiring a SECOND attribute's presence in 
 order to make the first attribute take effect. `scope` should 
 work when you write `scope`. If you don't want scope checking, 
 don't write `scope`...?


While there's in theory a single meaning of `scope`, in practice, 
the attribute is used for different things:

1. Document the lifetime of a variable
2. Let the compiler enforce that lifetime
3. Turn GC allocations of classes / arrays into stack allocations
4. Run class destructors deterministically

While those usually align, different people put a different 
aspect first. For example, some consider `scope` to be basically 

to overflow the stack here:

```D
void main()
{
     scope int[] arr = new int[32_000_000];
}
```

https://github.com/dlang/dmd/issues/20770#issuecomment-2615115810

Others may just care about running a class destructor. Now what 
happens if we enable scope checking everywhere?

```D
void main()  system
{
     scope Object o = new Object();
     const h = o.toHash();
     // Error: scope variable `o` calling non-scope member 
function `Object.toHash()`
}
```

Oops, that method is not marked `scope`. class methods rarely 
escape their `this` pointer, but since it is possible to override 
methods, scope can't be inferred, requiring annoying and ugly 
scope annotations everywhere.

A similar issue can happen with ImportC:

```D
import glfw;

void copy(scope const(char)[] str)  trusted
{
     assert(str.length > 0 && str[$ - 1] == '\0');
     glfwSetClipboardString(window, str.ptr);
}
```

I've read the documentation of glfw which says 
`glfwSetClipboardString` copies the input string, but I can't 
annotate the parameter in glfw.h with `scope` because it doesn't 
exist in C.

Finally, mechanical scope checking has false positives:
```D
void f(ref scope S ctx)
{
     const old = ctx;
     g(ctx);
     ctx = old; // Error: scope variable `old` assigned to `ref` 
variable `ctx` with longer lifetime

}
```
Some of these are bugs / limitations which can be improved, but 
in general it's impossible to do perfect scope checking (Rice's 
theorem). Even your example of assigning a scope pointer to a 
global variable can be valid if you manually restrict read access 
to that global to the current scope using ` system` variables and 
the right ` trusted` code.

Now like I said, I'm not against the idea per se. I have recently 
debugged a use-after-free bug from returning a scope array in 
code that I didn't mark ` safe` yet, which would have been caught 
immediately otherwise. But the considerations are:

- Code breakage
- Different expectations of `scope` from different people
- Nuisance with scope classes
- Passing scope pointers to C functions
- False positive errors
- "Trust me, this parameter `scope`"



 May 12



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2025 6:25 AM, Dennis wrote:

 Finally, mechanical scope checking has false positives:
 ```D
 void f(ref scope S ctx)
 {
      const old = ctx;
      g(ctx);
      ctx = old; // Error: scope variable `old` assigned to `ref` variable
`ctx` 
 with longer lifetime
 
 }
 ```


This comes about because `old`, being declared after `ctx`, has a shorter 
lifetime than `ctx`. If `old` has a destructor, that destructor will run before 
`ctx`s destructor, and `ctx` will be referring to an invalid instance of `old`.

If S does not have a destructor, does that make the error a false positive or 
not? I decided that the existence of a destructor should not change the scoping 
rules. The increase in language complexity is not worth it, as such to often 
leads to unexpected, quirky side effects.

BTW, thank you for your post. It's informative and excellent.



 May 12



  parent  Dennis <dkorpel gmail.com>  writes:


On Monday, 12 May 2025 at 19:28:10 UTC, Walter Bright wrote:

 This comes about because `old`, being declared after `ctx`, has 
 a shorter lifetime than `ctx`. If `old` has a destructor, that 
 destructor will run before `ctx`s destructor, and `ctx` will be 
 referring to an invalid instance of `old`.


You can't free a pointer that has aliases in ` safe` code. 
Therefore you can't simply have a destructor that performs a 
free(), the struct that has it needs to prevent aliasing by 
disabling copies with ` disable this(this)`, or by providing a 
copy constructor that duplicates the pointer. Otherwise you allow 
memory corruption:

```D
import std.stdio;
 safe:

struct S
{
     int* x;
     ~this() scope  trusted { writefln("~this(): free(%s)", 
cast(void*) x); }
}

void f(ref scope S ctx)
{
     {
     	auto old = ctx;
     }
     writefln("use %s", cast(void*) ctx.x);
}

void main()
{
     scope S s = S(new int(38));
     f(s);
}
```

This prints:

```
~this(): free(7F8D32557000)
use 7F8D32557000
~this(): free(7F8D32557000)
```

Both a 'use after free' and a 'double free'! But if you add 
`this(this) scope { x = new int(*x); }` to S, it works out.


 If S does not have a destructor, does that make the error a 
 false positive or not? I decided that the existence of a 
 destructor should not change the scoping rules.


I agree with that, but in light of the above, I don't think a 
copy of a scope pointer should have its lifetime shortened to the 
destination variable.

It may be by design, but it's a false positive in the sense that 
it leads to unnecessary refactoring (like 
https://github.com/dlang/phobos/pull/8125), and to unnecessary 
` trusted` code. The `ctx = old` example I gave is reduced from 
my actual code:
https://github.com/dkorpel/ctod/blob/1651fb4e31095e0cf5e0c4524149bdc2bcc7539e/source/ctod/cdeclaration.d#L231

If you have an alternate way to write that in a ` safe` way, I'd 
love to hear it.


 BTW, thank you for your post. It's informative and excellent.


:)



 May 13



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/2/2025 12:44 AM, Manu wrote:

 On a slight tangent; why do I need to attribute the function at all for it to 
 check `scope` args don't escape?


Because otherwise the compiler has to examine the function implementation to 
verify this. Doing such is called attribute inference, and D does quite a bit
of 
that. Problems are:

1. Function declarations don't have a body, so there's no way to inspect the
body.

2. Inspecting all the bodies means compilation gets slowed down quite a bit.

3. Chicken-and-egg problems when inferring attributes when there are loops in 
the function call flow graph.


 I do want to add `scope` to pointers and ref arguments, but I don't see any 
 reason that I should have to attribute `live`... why isn't `scope` enough to 
 enable escape analysis?


The DFA (Data Flow Analysis) required to determine whether a pointer is the 
"owner" or not at each point in the function is considerably more complex than 
just checking for scope-ness. DFA equations are generated and then solved 
iteratively.

The code required to do it is:

https://github.com/dlang/dmd/blob/master/compiler/src/dmd/ob.d

which is very similar to the DFA performed by the optimizer:

https://github.com/dlang/dmd/blob/master/compiler/src/dmd/backend/gflow.d



 May 03



  parent reply Manu <turkeyman gmail.com>  writes:


On Sun, 4 May 2025 at 05:15, Walter Bright via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On 5/2/2025 12:44 AM, Manu wrote:

 On a slight tangent; why do I need to attribute the function at all for


 it to

 check `scope` args don't escape?


 Because otherwise the compiler has to examine the function implementation
 to
 verify this. Doing such is called attribute inference, and D does quite a
 bit of
 that. Problems are:

 1. Function declarations don't have a body, so there's no way to inspect
 the body.


A prototype without a body must always trust the declaration. You must
simply assume that presence of `scope` on an arg asserts that the parameter
won't be retained by the call, and the caller can also bank on that fact;
ie, using temp/stack memory to allocate an argument.

2. Inspecting all the bodies means compilation gets slowed down quite a bit.


If you don't want escape analysis, don't write `scope`; point is, why do
you need to write TWO attributes together for the first one (which is
itself strictly optional) to take effect?

3. Chicken-and-egg problems when inferring attributes when there are loops

 in
 the function call flow graph.


I don't follow.
Is it something like; a template which receives an arg but doesn't
explicitly state `scope` just spend time to try and infer `scope` because a
caller of that template may rely on that attribute inference?


 I do want to add `scope` to pointers and ref arguments, but I don't see
 any

 reason that I should have to attribute `live`... why isn't `scope`


 enough to

 enable escape analysis?


 The DFA (Data Flow Analysis) required to determine whether a pointer is
 the
 "owner" or not at each point in the function is considerably more complex
 than
 just checking for scope-ness. DFA equations are generated and then solved
 iteratively.

 The code required to do it is:

 https://github.com/dlang/dmd/blob/master/compiler/src/dmd/ob.d

 which is very similar to the DFA performed by the optimizer:

 https://github.com/dlang/dmd/blob/master/compiler/src/dmd/backend/gflow.d


Yeah but I still don't get it; if you don't want escape analysis, don't
write `scope`. If you do write it, you have clearly and unambiguously
indicated that you DO want escape analysis, so it should happen.
I can't see any reason a second unrelated attribute should be required in
conjunction to make the first one work... you write `scope` thinking it
will be effective, and then by not writing (or forgetting to write) the
other thing, your code is now lying to you. You won't know this until you
are bitten by the escape bug that you were trying to prevent; making the
sutuation actually *worse* than useless, because when you're trying to
track down your bug, you will see the attribute, and continue looking
elsewhere.
`scope` should work when you write it; period. If that is to say that
writing `scope` infers the other attribute, whatever... but it's not a
reasonable situation where `scope` silently does nothing and confuses or
misleads the author.



 May 10



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/10/2025 5:25 AM, Manu wrote:

     1. Function declarations don't have a body, so there's no way to inspect
the
     body.


 A prototype without a body must always trust the declaration. You must simply 
 assume that presence of `scope` on an arg asserts that the parameter won't be 
 retained by the call, and the caller can also bank on that fact; ie, using 
 temp/stack memory to allocate an argument.


Yes, of course, that's how dip1000 works.


     2. Inspecting all the bodies means compilation gets slowed down quite a
bit.


 If you don't want escape analysis, don't write `scope`; point is, why do you 
 need to write TWO attributes together for the first one (which is itself 
 strictly optional) to take effect?


`scope` is needed to say that a pointer being passed does not escape. `return` 
is needed to specify that an escape via the function's return is allowed.



     3. Chicken-and-egg problems when inferring attributes when there are loops
in
     the function call flow graph.


 I don't follow.


f calls g calls h calls f

You can't infer attributes of f because it depends on f. You can't infer 
attributes of g because it depends on f which depends  on g. And so on. To deal 
with this problem requires constructing Data Flow Equations and then solving 
those equations by iterating and converging on a unique solution. This is what 
the code in the link to gflow.d does. It's in the comments.

Unfortunately, building and solving the DFE's is slow. This is why the
optimized 
builds are much slower than non-optimized builds. This is why the simplistic
DFA 
is used to infer attributes rather than DFA, because we want fast
non-optimizing 
builds.



 Is it something like; a template which receives an arg but doesn't explicitly 
 state `scope` just spend time to try and infer `scope` because a caller of
that 
 template may rely on that attribute inference?


Of course.


 
      > I do want to add `scope` to pointers and ref arguments, but I don't see
any
      > reason that I should have to attribute `live`... why isn't `scope`
enough to
      > enable escape analysis?
 
     The DFA (Data Flow Analysis) required to determine whether a pointer is the
     "owner" or not at each point in the function is considerably more complex
than
     just checking for scope-ness. DFA equations are generated and then solved
     iteratively.
 
     The code required to do it is:
 
     https://github.com/dlang/dmd/blob/master/compiler/src/dmd/ob.d
     <https://github.com/dlang/dmd/blob/master/compiler/src/dmd/ob.d>
 
     which is very similar to the DFA performed by the optimizer:
 
     https://github.com/dlang/dmd/blob/master/compiler/src/dmd/backend/gflow.d
     <https://github.com/dlang/dmd/blob/master/compiler/src/dmd/backend/gflow.d>
 
 
 Yeah but I still don't get it; if you don't want escape analysis, don't write 
 `scope`.


Yes, and then you cannot call that function with a scope pointer.



 If you do write it, you have clearly and unambiguously indicated that 
 you DO want escape analysis, so it should happen.
 I can't see any reason a second unrelated attribute should be required in 
 conjunction to make the first one work... you write `scope` thinking it will
be 
 effective, and then by not writing (or forgetting to write) the other thing, 
 your code is now lying to you. You won't know this until you are bitten by the 
 escape bug that you were trying to prevent; making the sutuation actually 
 *worse* than useless, because when you're trying to track down your bug, you 
 will see the attribute, and continue looking elsewhere.
 `scope` should work when you write it; period. If that is to say that writing 
 `scope` infers the other attribute, whatever... but it's not a reasonable 
 situation where `scope` silently does nothing and confuses or misleads the
author.


It does work when you write it. The "second unrelated attribute" (which I
assume 
is the "return" attribute) is necessary to indicate if the scope'd pointer 
returns or not. Not having the `return` attribute means the following code is 
impossible to write:

```
int* = foo(scope int* a) { return a; } // error: `a` is escaping via return!
scope int* p = ...;
scope int* q = foo(p);
```



 May 10



  parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz>  writes:


On 11/05/2025 3:34 AM, Walter Bright wrote:

     If you do write it, you have clearly and unambiguously indicated
     that you DO want escape analysis, so it should happen. I can't see
     any reason a second unrelated attribute should be required in
     conjunction to make the first one work... you write |scope| thinking
     it will be effective, and then by not writing (or forgetting to
     write) the other thing, your code is now lying to you. You won't
     know this until you are bitten by the escape bug that you were
     trying to prevent; making the sutuation actually /worse/ than
     useless, because when you're trying to track down your bug, you will
     see the attribute, and continue looking elsewhere. |scope| should
     work when you write it; period. If that is to say that writing |
     scope| infers the other attribute, whatever... but it's not a
     reasonable situation where |scope| silently does nothing and
     confuses or misleads the author.
 
 It does work when you write it. The "second unrelated attribute" (which 
 I assume is the "return" attribute) is necessary to indicate if the 
 scope'd pointer returns or not. Not having the |return| attribute means 
 the following code is impossible to write:
 
 |int* = foo(scope int* a) { return a; } // error: `a` is escaping via 
 return! scope int* p = ...; scope int* q = foo(p);|


There is also the mess with ``return ref`` vs ``return``, that can be 
written as ``return scope`` and ``scope return``.

These both have the same escape set, just a different relationship strength.

This is of course one of the worst design decisions the D community has 
ever implemented with full regret by all parties involved! :)

This is also a good time to remember that DIP1000 only understands no 
escape, escape into return value OR this pointer. No multiple output 
support, which severely limits its capabilities.



 May 10



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/10/2025 1:17 PM, Richard (Rikki) Andrew Cattermole wrote:

 There is also the mess with ``return ref`` vs ``return``, that can be written
as 
 ``return scope`` and ``scope return``.


That came about because of the existence of `ref`. Consider `ref int* p`. How
do 
you model the scope-ness of the reference and the scope-ness of the pointer?



 These both have the same escape set, just a different relationship strength.


Strength??



 This is of course one of the worst design decisions the D community has ever 
 implemented with full regret by all parties involved! :)


The concept behind it works fine. The trouble comes with the complexity of all 
the pointer constructions, like the invisible use of `this` references.



 This is also a good time to remember that DIP1000 only understands no escape, 
 escape into return value OR this pointer.


That's sufficient to prevent an escaping pointer to the stack. It's quite
solid. 
If you disagree, post a code snippet demonstrating it.



 No multiple output support, which severely limits its capabilities.


It does not severely limit it. Even the Rust manual says it's a rare case. It's 
a minor inconvenience that can be refactored away.



 May 10



 next sibling parent  "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz>  writes:


On 11/05/2025 11:40 AM, Walter Bright wrote:

 On 5/10/2025 1:17 PM, Richard (Rikki) Andrew Cattermole wrote:

 There is also the mess with ``return ref`` vs ``return``, that can be 
 written as ``return scope`` and ``scope return``.


 
 That came about because of the existence of `ref`. Consider `ref int* 
 p`. How do you model the scope-ness of the reference and the scope-ness 
 of the pointer?


I know what I want my DFA to do, but my understanding is taking a 
pointer is rewritten out of the AST by the time semantic 3 has finished 
and that makes things more difficult.


 These both have the same escape set, just a different relationship 
 strength.


 
 Strength??


I.e. taking a pointer to a variable has a stronger relationship than 
just copying the value stored in said variable.

So the difference between ``return scope`` and ``scope return``.


 This is also a good time to remember that DIP1000 only understands no 
 escape, escape into return value OR this pointer.


 
 That's sufficient to prevent an escaping pointer to the stack. It's 
 quite solid. If you disagree, post a code snippet demonstrating it.


We've covered this before, stack is one thing, but stack becomes heap 
the moment it crosses the function boundary and the things people 
actually want to use it for? Heap.


 No multiple output support, which severely limits its capabilities.


 
 It does not severely limit it. Even the Rust manual says it's a rare 
 case. It's a minor inconvenience that can be refactored away.


Its rare enough that it doesn't need short syntax that could interfere 
with existing syntax.

Its not rare enough for me to drop it.



 May 10



prev sibling  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 5/11/25 01:40, Walter Bright wrote:

 

 No multiple output support, which severely limits its capabilities.


 
 It does not severely limit it. Even the Rust manual says it's a rare 
 case.


Well, it's common enough to need type system support. Support for 
multiple indirections is also important. Rust actually provides both of 
these.


 It's a minor inconvenience that can be refactored away.


In general you'll have to resort to ` system` code, and it will stand in 
the way of other refactorings you may want to do.

Anyway, I don't think this is a theoretical issue anymore, even though 
my perspective was the same when we first discussed this. AFAIU, real 
industry users have already found DIP1000 to be lacking because of its 
lack of compositionality.



 May 11



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/11/2025 4:32 AM, Timon Gehr wrote:

 Well, it's common enough to need type system support.


That's not necessarily true. You can do everything in C that D does, it's just 
less convenient.



 Support for multiple 
 indirections is also important. Rust actually provides both of these.


There are other ways to do that. Multiple indirections can be handled by using 
structs and then only allowing access to structs through member function
interfaces.



 It's a minor inconvenience that can be refactored away.


 In general you'll have to resort to ` system` code,


While you can always do it with  system code, I'm not convinced some
refactoring 
won't work. It's not a disaster to use  system code now and then.



 and it will stand in the way of other refactorings you may want to do.


You've got to do a lot of refactoring to make Rust code work anyway.



 Anyway, I don't think this is a theoretical issue anymore, even though my 
 perspective was the same when we first discussed this. AFAIU, real industry 
 users have already found DIP1000 to be lacking because of its lack of 
 compositionality.


I'm not aware of any particular instance of this, so cannot help with it.



 May 11



  parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 5/11/25 20:03, Walter Bright wrote:

 On 5/11/2025 4:32 AM, Timon Gehr wrote:

 Well, it's common enough to need type system support.


 
 That's not necessarily true.


You had claimed it is not a severe limitation. It's common enough to 
need type system support to not be severely limited.


 You can do everything in C that D does, it's just less convenient.
 ...


Severely so.


 

 Support for multiple indirections is also important. Rust actually 
 provides both of these.


 
 There are other ways to do that. Multiple indirections can be handled by 
 using structs and then only allowing access to structs through member 
 function interfaces.
 ...


Well, safe dynamic arrays can be handled by creating a custom struct and 
accessing it with API functions that perform bounds checks. Yet C 
programmers rarely do this.


 

 It's a minor inconvenience that can be refactored away.


 In general you'll have to resort to ` system` code,


 
 While you can always do it with  system code, I'm not convinced some 
 refactoring won't work. It's not a disaster to use  system code now and 
 then.
 ...


The best you can usually do is to manually move safety checks to 
runtime. The point of a type system in this case is to avoid that 
overhead and to give assurances before the program actually runs or even 
ships to users.


 

 and it will stand in the way of other refactorings you may want to do.


 
 You've got to do a lot of refactoring to make Rust code work anyway.
 ...


The inflexibility of Rust is indeed one of the main points of critique 
that is directed against it.



 May 12



prev sibling next sibling parent reply Dukc <ajieskola gmail.com>  writes:


On Tuesday, 29 April 2025 at 17:12:41 UTC, Walter Bright wrote:

 I was quite intrigued with the borrow checker, and set about 
 learning about it. While D cannot be retrofitted with a borrow 
 checker, it can be enhanced with it. A borrow checker has 
 nothing tying it to the Rust syntax, so it should work.


That's right.


 So I implemented a borrow checker for D, and it is enabled by 
 adding the ` live` annotation for a function, which turns on 
 the borrow checker for that function. There are no syntax or 
 semantic changes to the language, other than laying on a borrow 
 checker.


There's a difference.

In Rust, as I understand it, if you have a function like

```Rust
fn free(ptr: MyCustomPointer)
{   // ...
}
```

it is 100% safe to use. The compiler will not let you double-free 
or use after free, unless you use the `unsafe` block to do so.

But you can't have

```D
 trusted void free(MyCustomPointer ptr)
{   // ...
}
```

in D because it would be unsafe to use from a non-` live` 
function.

If we had a way to say that "this function can be called from 
` safe`, if and only if it's usage is guarded with ` live`" then 
it would be equal to the Rust borrow checker.



 May 02



 next sibling parent  jmh530 <john.michael.hall gmail.com>  writes:


On Friday, 2 May 2025 at 10:14:34 UTC, Dukc wrote:

 There's a difference.

 In Rust, as I understand it, if you have a function like

 ```Rust
 fn free(ptr: MyCustomPointer)
 {   // ...
 }
 ```

 it is 100% safe to use. The compiler will not let you 
 double-free or use after free, unless you use the `unsafe` 
 block to do so.

 But you can't have

 ```D
  trusted void free(MyCustomPointer ptr)
 {   // ...
 }
 ```

 in D because it would be unsafe to use from a non-` live` 
 function.

 If we had a way to say that "this function can be called from 
 ` safe`, if and only if it's usage is guarded with ` live`" 
 then it would be equal to the Rust borrow checker.


I was trying to make this point before, but think you do it 
better here.



 May 02



prev sibling  parent reply Paul Backus <snarwin gmail.com>  writes:


On Friday, 2 May 2025 at 10:14:34 UTC, Dukc wrote:

 There's a difference.

 In Rust, as I understand it, if you have a function like

 ```Rust
 fn free(ptr: MyCustomPointer)
 {   // ...
 }
 ```

 it is 100% safe to use. The compiler will not let you 
 double-free or use after free, unless you use the `unsafe` 
 block to do so.

 But you can't have

 ```D
  trusted void free(MyCustomPointer ptr)
 {   // ...
 }
 ```

 in D because it would be unsafe to use from a non-` live` 
 function.


+100

Rust programmers don't put up with restrictions of the borrow 
checker just for fun. They do it because *it allows the compiler 
to verify the safety of their code.*

A borrow checker that can't be used for safety verification is 
like an oven mitt that can't be used to hold a hot pan. All it 
does is make things awkward and uncomfortable, and if you 
actually try to rely on it, you'll end up getting hurt.



 May 02



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


You add:

```
 live:
```

at the beginning of your modules.

Because it is opt-in with D rather than opt-out in Rust, does not alter the
fact 
that D has a borrow checker.

In Rust you can use "unsafe" to avoid the borrow checker.



 May 03



 next sibling parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 5/3/25 21:22, Walter Bright wrote:

 You add:
 
 ```
  live:
 ```
 
 at the beginning of your modules.
 
 Because it is opt-in with D rather than opt-out in Rust, does not alter 
 the fact that D has a borrow checker.
 
 In Rust you can use "unsafe" to avoid the borrow checker.


These are disconnected and cherry-picked operational points without 
taking into account the semantics. Even if you added ` live  safe` on 
top of all your modules, you still would not get any aliasing guarantees.

```d
 live  safe:
class C{ int* x; }

void main(){
     auto x=new int;
     auto c=new C;
     c.x=x;
     auto y=c.x;
     assert(x is y);
     imported!"std.stdio".writeln(x,y);
}
```



 May 03



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


I did not bother to support class dereferencing in the implementation due to
the 
lack of interest in it.

D has a lot of higher level constructs (like classes) that require careful 
implementation in the borrow checker. That doesn't mean the approach is unsound.



 May 03



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 5/4/25 03:26, Walter Bright wrote:

 I did not bother to support class dereferencing in the implementation 
 due to the lack of interest in it.
 
 D has a lot of higher level constructs (like classes) that require 
 careful implementation in the borrow checker. That doesn't mean the 
 approach is unsound.


No classes are needed, I just did that for readability.

```d
 live  safe:

void main(){
     auto x=new int;
     auto c=new int*;
     *c=x;
     auto y=*c;
     assert(x is y);
     imported!"std.stdio".writeln(x,y);
}
```



 May 03



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


You're correct in that it doesn't check indirect indirections.

This is equivalent to the problem of storing a pointer in an aggregate, which I 
did not work on implementing.



 May 04



prev sibling next sibling parent reply Paul Backus <snarwin gmail.com>  writes:


On Saturday, 3 May 2025 at 19:22:20 UTC, Walter Bright wrote:

 You add:

 ```
  live:
 ```

 at the beginning of your modules.


Even if you do this, you still cannot manually free memory in 
 safe code, like you can in Rust.

Borrow checking, in itself, is not a useful feature--it's just a 
bunch of arbitrary hoops the programmer has to jump through. It 
is useful in Rust ONLY because of the additional safety analysis 
it enables. In D, the borrow checker does not enable any 
additional safety analysis, so why would the programmer ever want 
to jump through its hoops?



 May 04



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/4/2025 3:57 AM, Paul Backus wrote:

 Even if you do this, you still cannot manually free memory in  safe code, like 
 you can in Rust.


Actually, you can in D.

The borrow checker doesn't know anything about free'ing memory. What it does 
know is the ownership of the pointer transferred to the function, or not.
That's 
all it needs to know.



 Borrow checking, in itself, is not a useful feature--it's just a bunch of 
 arbitrary hoops the programmer has to jump through. It is useful in Rust ONLY 
 because of the additional safety analysis it enables. In D, the borrow checker 
 does not enable any additional safety analysis, so why would the programmer
ever 
 want to jump through its hoops?


The check it does is enforce the style of only one mutable pointer being live
at 
a time, which prevents double frees. It enforces that a mutable pointer is not 
left dangling, which prevents omitted frees.



 May 04



prev sibling next sibling parent reply Paul Backus <snarwin gmail.com>  writes:


On Saturday, 3 May 2025 at 19:22:20 UTC, Walter Bright wrote:

 In Rust you can use "unsafe" to avoid the borrow checker.


Also, this is not true, which you would know if you had read 
literally anything about unsafe Rust:


 You can take five actions in unsafe Rust that you can’t in safe 
 Rust, which we call unsafe superpowers. Those superpowers 
 include the ability to:

 * Dereference a raw pointer
 * Call an unsafe function or method
 * Access or modify a mutable static variable
 * Implement an unsafe trait
 * Access fields of a union

 It’s important to understand that unsafe doesn’t turn off the 
 borrow checker or disable any other of Rust’s safety checks


Source: https://doc.rust-lang.org/book/ch20-01-unsafe-rust.html

This page is the first result on Google for "unsafe rust". The 
fact that you got this wrong shows very clearly that you have not 
done your homework on this topic.



 May 04



 next sibling parent reply Dukc <ajieskola gmail.com>  writes:


On Sunday, 4 May 2025 at 11:02:27 UTC, Paul Backus wrote:

 This page is the first result on Google for "unsafe rust". The 
 fact that you got this wrong shows very clearly that you have 
 not done your homework on this topic.


I think you're a bit too fast with your judgement here.

While the unsafe keyword doesn't directly turn off the borrow 
checker, it does let you to work your way around it. I don't know 
exactly how, but I'm sure it's possible; it's a systems 
programming language after all. Most likely it involves calling 
some sort of unsafe typecasting function. I think this is what 
Walter means.



 May 04



 next sibling parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 5/4/25 18:35, Dukc wrote:

 On Sunday, 4 May 2025 at 11:02:27 UTC, Paul Backus wrote:

 This page is the first result on Google for "unsafe rust". The fact 
 that you got this wrong shows very clearly that you have not done your 
 homework on this topic.


 
 I think you're a bit too fast with your judgement here.
 
 While the unsafe keyword doesn't directly turn off the borrow checker, 
 it does let you to work your way around it. I don't know exactly how, 
 but I'm sure it's possible; it's a systems programming language after 
 all. Most likely it involves calling some sort of unsafe typecasting 
 function. I think this is what Walter means.


Well, in unsafe Rust you can use raw pointers that are not subject to 
lifetime checks.



 May 04



prev sibling  parent reply Paul Backus <snarwin gmail.com>  writes:


On Sunday, 4 May 2025 at 16:35:23 UTC, Dukc wrote:

 On Sunday, 4 May 2025 at 11:02:27 UTC, Paul Backus wrote:

 This page is the first result on Google for "unsafe rust". The 
 fact that you got this wrong shows very clearly that you have 
 not done your homework on this topic.


 I think you're a bit too fast with your judgement here.

 While the unsafe keyword doesn't directly turn off the borrow 
 checker, it does let you to work your way around it. I don't 
 know exactly how, but I'm sure it's possible; it's a systems 
 programming language after all. Most likely it involves calling 
 some sort of unsafe typecasting function. I think this is what 
 Walter means.


No, you're giving Walter too much credit here.

In context, he is clearly trying to claim that the 
 live/non- live distinction in D is analogous to the safe/unsafe 
distinction in Rust w.r.t. borrow checking--i.e., that both 
languages have a mode with borrow checking, and a mode without 
it. The difference--he claims--is that in Rust, the mode with 
borrow checking is opt-out, and in D, it's opt-in.

The *actual* difference between D and Rust's approach is that 
Rust has two separate kinds of pointers: references, which are 
borrow-checked (in both safe and unsafe code), and raw pointers, 
which aren't. In other words, whether borrow checking is done or 
not is decided on a *type-by-type* basis, not a 
*function-by-function* basis.

The fact that Walter is (apparently) unaware of these facts shows 
that he has clearly not "studied the Rust specification," as he 
claims.



 May 04



  parent  Dukc <ajieskola gmail.com>  writes:


On Sunday, 4 May 2025 at 16:53:55 UTC, Paul Backus wrote:

 No, you're giving Walter too much credit here.

 [snip]

 The fact that Walter is (apparently) unaware of these facts 
 shows that he has clearly not "studied the Rust specification," 
 as he claims.


I'm not trying to claim he understands the Rust borrow checker 
and how it differs from ours. It's clear he's missing something, 
or at least failing to see (or admit) how inpractical our 
solution is compared to the Rust one.

I'm only saying he didn't write anything that would prove he 
didn't even attempt to study it.



 May 04



prev sibling next sibling parent  Walter Bright <newshound2 digitalmars.com>  writes:


Unsafe in Rust can be applied to functions, and Rust references can be
converted 
to raw pointers in unsafe code.

The comparison of opt-in vs opt-out is valid.



 May 04



prev sibling  parent  Don Allen <donaldcallen gmail.com>  writes:


On Sunday, 4 May 2025 at 11:02:27 UTC, Paul Backus wrote:

 On Saturday, 3 May 2025 at 19:22:20 UTC, Walter Bright wrote:

 In Rust you can use "unsafe" to avoid the borrow checker.


 Also, this is not true, which you would know if you had read 
 literally anything about unsafe Rust:


 You can take five actions in unsafe Rust that you can’t in 
 safe Rust, which we call unsafe superpowers. Those superpowers 
 include the ability to:

 * Dereference a raw pointer
 * Call an unsafe function or method
 * Access or modify a mutable static variable
 * Implement an unsafe trait
 * Access fields of a union

 It’s important to understand that unsafe doesn’t turn off the 
 borrow checker or disable any other of Rust’s safety checks


 Source: https://doc.rust-lang.org/book/ch20-01-unsafe-rust.html

 This page is the first result on Google for "unsafe rust". The 
 fact that you got this wrong shows very clearly that you have 
 not done your homework on this topic.


Walter's admission that he has not written even a single line of 
Rust code raises a similar concern. Reading Rust documentation is 
necessary but not sufficient for acquiring a real understanding 
of the borrow checker (would you get into a self-driving car 
having software written by someone who doesn't drive?). I've 
written about 10000 lines of working Rust code, so while I would 
not claim to be the world's preeminent expert on the language, 
I've been in the trenches with it. You cannot learn from reading 
a book what this taught me.

This also strikes me as an instance of gluing yet another wart 
onto this language.



 May 06



prev sibling  parent reply Dukc <ajieskola gmail.com>  writes:


On Saturday, 3 May 2025 at 19:22:20 UTC, Walter Bright wrote:

 You add:

 ```
  live:
 ```

 at the beginning of your modules.


I would never seriously consider this.

It would mean I have to make _all_ my ` safe` code ` live` before 
I can begin using the ` trusted` version of `free` without it 
being safewashing.

And thereafter I could never use any third party 
` safe`/` trusted` code that isn't verified to be ` live` 
correct. This includes Phobos.

Just dropping to ` system`/` trusted` when doing manual memory 
management is far more practical regardless of the use case.


 Because it is opt-in with D rather than opt-out in Rust, does 
 not alter the fact that D has a borrow checker.


Most certainly we don't want to force existing ` safe` D code to 
adapt ` live` rules. Not even over an edition switch. In that 
you're right.

But ironically, our opt-out borrow checker does exactly this in a 
practical sense. Like I wrote above, you can only depend on 
` live` for ` safe`ty if you use it everywhere. This is the worst 
of both worlds: not only do I have to make all my ` safe` code to 
use the borrow checker (as in Rust), I must _manually_ make sure 
I don't have any non-` live` ` safe` code around, unless I'm 
willing to manually review them like a ` trusted` function.

If the borrow checker is to be of any real use, we need some way 
to check where it is needed. So that code that does manual memory 
management will have to use ` live` (or ditch ` safe`), but your 
regular garbage-collected stuff won't.



 May 04



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/4/2025 11:03 AM, Dukc wrote:

 Like I wrote above, you can only depend on ` live` for ` safe`ty if you 
 use it everywhere.


It doesn't have to be all or nothing in order to be useful. Just like we use 
 safe on some functions, and  system on others.



 May 04



  parent reply Dukc <ajieskola gmail.com>  writes:


On Sunday, 4 May 2025 at 22:12:25 UTC, Walter Bright wrote:

 On 5/4/2025 11:03 AM, Dukc wrote:

 Like I wrote above, you can only depend on ` live` for 
 ` safe`ty if you use it everywhere.





 It doesn't have to be all or nothing in order to be useful. 
 Just like we use  safe on some functions, and  system on others.


If you mean useful as a linting tool, maybe.

But Rust's borrow checker lets you to write safe code that would 
have to be unsafe without the checker.

The D borrow checker doesn't. Everything that can be ` safe` with 
the borrow checker can be ` safe` without, _unless_ you commit to 
never use ` safe` without ` live`. If you think this isn't the 
case, try writing a code counterexample. I don't think you can do 
it.

You have to admit this makes our checker, as it currently stands, 
unusable for what many of us consider the primary purpose of a 
borrow checker.



 May 04



 next sibling parent reply jmh530 <john.michael.hall gmail.com>  writes:


On Monday, 5 May 2025 at 06:52:09 UTC, Dukc wrote:

 [snip]

 You have to admit this makes our checker, as it currently 
 stands, unusable for what many of us consider the primary 
 purpose of a borrow checker.


It might be helpful to example where Rust's version of some code 
let's you prove something is safe with the borrow checker, but 
D's version doesn't.



 May 06



  parent reply Dukc <ajieskola gmail.com>  writes:


On Tuesday, 6 May 2025 at 14:48:27 UTC, jmh530 wrote:

 It might be helpful to example where Rust's version of some 
 code let's you prove something is safe with the borrow checker, 
 but D's version doesn't.


You're right. Have a look at my post earlier on this thread:

 On Tuesday, 29 April 2025 at 17:12:41 UTC, Walter Bright wrote:

 So I implemented a borrow checker for D, and it is enabled by 
 adding the ` live` annotation for a function, which turns on 
 the borrow checker for that function. There are no syntax or 
 semantic changes to the language, other than laying on a 
 borrow checker.


 There's a difference.

 In Rust, as I understand it, if you have a function like

 ```Rust
 fn free(ptr: MyCustomPointer)
 {   // ...
 }
 ```

 it is 100% safe to use. The compiler will not let you 
 double-free or use after free, unless you use the `unsafe` 
 block to do so.

 But you can't have

 ```D
  trusted void free(MyCustomPointer ptr)
 {   // ...
 }
 ```

 in D because it would be unsafe to use from a non-` live` 
 function.

 If we had a way to say that "this function can be called from 
 ` safe`, if and only if it's usage is guarded with ` live`" 
 then it would be equal to the Rust borrow checker.





 May 06



  parent reply jmh530 <john.michael.hall gmail.com>  writes:


On Tuesday, 6 May 2025 at 20:12:18 UTC, Dukc wrote:

 On Tuesday, 6 May 2025 at 14:48:27 UTC, jmh530 wrote:

 It might be helpful to example where Rust's version of some 
 code let's you prove something is safe with the borrow 
 checker, but D's version doesn't.


 You're right. Have a look at my post earlier on this thread:

 On Tuesday, 29 April 2025 at 17:12:41 UTC, Walter Bright wrote:

 So I implemented a borrow checker for D, and it is enabled by 
 adding the ` live` annotation for a function, which turns on 
 the borrow checker for that function. There are no syntax or 
 semantic changes to the language, other than laying on a 
 borrow checker.


 There's a difference.

 In Rust, as I understand it, if you have a function like

 ```Rust
 fn free(ptr: MyCustomPointer)
 {   // ...
 }
 ```

 it is 100% safe to use. The compiler will not let you 
 double-free or use after free, unless you use the `unsafe` 
 block to do so.

 But you can't have

 ```D
  trusted void free(MyCustomPointer ptr)
 {   // ...
 }
 ```

 in D because it would be unsafe to use from a non-` live` 
 function.

 If we had a way to say that "this function can be called from 
 ` safe`, if and only if it's usage is guarded with ` live`" 
 then it would be equal to the Rust borrow checker.




Ah, I remember that. Thanks.

Just to be clear, the  trusted function you provide could be 
called by an  safe function, but your point is that it may 
actually be unsafe to do so. The real goal is to enable more code 
to actually be safe, not to just slap  trusted on things.

The examples in the  live spec don't reference  safe. But it does 
have an example of a non- live release function that is called by 
an  live test function that prevents use after free.

I can think of a few different ways forward (by no means limited 
to just this)

1) Do  safe inference on  live functions. I assume a body is 
needed on  live functions (Rust requires a body unless a trait or 
extern function, and extern is only allowed in unsafe code). So 
that might be good idea on its own merits.

That being said, it doesn't really help you if the  live function 
is calling an  system function. Also,  live functions can call 
non- live functions (although for an  safe  live function all the 
functions called by it need to be  safe/ trusted too), so it 
wouldn't necessarily be a transitive inference all the way down 
(i.e. it hits the first unattributed function and becomes 
 system).

2) Enhance the  safe definition so that an  safe  live function 
can call an  system function so long as the function it calls 
abides by certain restrictions. Obviously, the spec would need to 
clearly state what the restrictions are. One limitation would be 
that to verify that a  system function abides by these 
restrictions would require having the function body.

3) Add a  safeiflive attribute such that the function is  safe if 
called by a  live function and  system otherwise. As above, the 
team would need to pin down exactly what is allowed. After all, 
why couldn't it just be marked  safe? What  safe restrictions 
could be relaxed if  live is being enforced. Would require some 
thought, I imagine, but probably useful to think about. The 
downside is that it's more attributes.

4) Add  trustediflive, similar to above such that it is  trusted 
if called by a  live function and  system otherwise. Let's the 
user make the decision, more flexible. I think  trustediflive is 
kind of in line with what you were suggesting.



 May 07



 next sibling parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 5/7/25 16:47, jmh530 wrote:

 
 I can think of a few different ways forward (by no means limited to just 
 this)


Well, the big issue here is that  live is still not transitive. A 
function attribute is not really the right way to go about this anyway. 
Aliasing is a property of a pointer, not a function.



 May 07



 next sibling parent reply Meta <jared771 gmail.com>  writes:


On Wednesday, 7 May 2025 at 23:21:12 UTC, Timon Gehr wrote:

 On 5/7/25 16:47, jmh530 wrote:

 
 I can think of a few different ways forward (by no means 
 limited to just this)


 Well, the big issue here is that  live is still not transitive. 
 A function attribute is not really the right way to go about 
 this anyway. Aliasing is a property of a pointer, not a 
 function.


And this is exactly what Rust does with the borrowed/raw pointer 
distinction. Walter how can you claim to have studied the Rust 
spec yet miss and/or disregard this fundamental detail? And do 
not bring up that extremely outdated and irrelevant example of 
near/far pointers in DOS.



 May 07



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/7/2025 6:27 PM, Meta wrote:

 And this is exactly what Rust does with the borrowed/raw pointer distinction. 
 Walter how can you claim to have studied the Rust spec yet miss and/or
disregard 
 this fundamental detail?


I did not miss it. See my reply to Timon.



 And do not bring up that extremely outdated and 
 irrelevant example of near/far pointers in DOS.


Yes it is dated, as modern languages abandoned multiple pointer types based on 
that experience.

The only one I even know about is Microsoft Managed C++, where they have gc and 
nogc pointers. What a disaster. Nobody wants to write code in Managed C++. That 
scheme never made it into Standard C++.

A feature needs to be useful and easy to use in order to get people to use it.



 May 09



prev sibling next sibling parent reply jmh530 <john.michael.hall gmail.com>  writes:


On Wednesday, 7 May 2025 at 23:21:12 UTC, Timon Gehr wrote:

 On 5/7/25 16:47, jmh530 wrote:

 
 I can think of a few different ways forward (by no means 
 limited to just this)


 Well, the big issue here is that  live is still not transitive. 
 A function attribute is not really the right way to go about 
 this anyway. Aliasing is a property of a pointer, not a 
 function.


I think the difficulty is figuring a way to convince Walter of 
this point. It's one thing to assert that it is true, it's 
another to prove it.

That's why in my last post, I was kind of like, "ok what's the 
goal, what would we have to do to achieve it." If it can't be 
achieved with the function attribute approach, then it should be 
abandoned. But I would hope we can salvage this effort into 
something useful.

To your point about being a property of a pointer, the 
alternative approach is to introduce a reference that is borrow 
checked. Yes, it results in additional pointer types, but if 
that's the approach that Rust ultimately relies on and it's the 
only way to achieve what we want, then it's what should be done.



 May 09



  parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz>  writes:


On 10/05/2025 5:17 AM, jmh530 wrote:

 To your point about being a property of a pointer, the alternative 
 approach is to introduce a reference that is borrow checked. Yes, it 
 results in additional pointer types, but if that's the approach that 
 Rust ultimately relies on and it's the only way to achieve what we want, 
 then it's what should be done.


I want to touch upon this because its misunderstood quite widely 
regarding the difference between Rust's borrow checker and ownership 
transfer system.

The borrow checker loosens restrictions placed upon you by the ownership 
transfer system to give strong guarantees of aliasing and lifetimes. If 
you did not have the borrow checker, the ownership transfer systems 
restrictions would make things absolutely impossible to work with.

I'll translate this into D-speak and I'm sure I haven't got it 100%.

For the purposes of this discussion I'll use a non-transitive type 
qualifier called ``unique`` to indicate ownership transfer, but it can 
be done other ways like reference counting or isolated from Midori.

When ownership transfer has a variable copied into another, the old 
variable resets (back to null). You can only have a modelable number of 
instances of this pointer within a function (could be just one).

By default, any variable (that is a pointer) that is unattributed is 
equivalent to writing ``unique(int*)``, and ``unique(unique(int*)*)``.

If you add escape analysis annotations it becomes ``int*`` or 
``unique(int*)*``.

The former can be passed to the latter implicitly.

```d
void goingDown(unique(unique(int*)*) input) {
	goingUp(input);
}

void goingUp(scope unique(int*)* input) {
	unique(int*) v = *input;
}
```

Now the escape analysis, first you have the going up the stack 
guarantees, this can be seen in ``goingUp``.

It is where the compiler is establishing the relationships between the 
variables as seeable by a function prototype.

This is what our DIP1000 attempts to do poorly.

The second kind is in ``goingDown``, and goes by the name borrow checker 
or as I prefer it "owner escape analysis" and this is what " live" 
supposedly fills the gap of (but doesn't).

It adds some extra logic to escape analysis that ``goingUp`` uses, 
specifically relating to the number of mutable copies you have of it, 
and requires that object will outlive its borrow.

```d
unique(int*) owner;

{
    int* borrow1 = owner; // ok
    int* borrow2 = owner; // Error
    const(int)* borrow3 = owner; // ok
    owner= null; // Error
}
```

Its important to note here that something has to trigger owner escape 
analysis. Either the type qualifier or the rest of the escape analysis 
(via say a function call).



 May 09



 next sibling parent reply jmh530 <john.michael.hall gmail.com>  writes:


On Friday, 9 May 2025 at 18:28:11 UTC, Richard (Rikki) Andrew 
Cattermole wrote:

 On 10/05/2025 5:17 AM, jmh530 wrote:

 To your point about being a property of a pointer, the 
 alternative approach is to introduce a reference that is 
 borrow checked. Yes, it results in additional pointer types, 
 but if that's the approach that Rust ultimately relies on and 
 it's the only way to achieve what we want, then it's what 
 should be done.


 I want to touch upon this because its misunderstood quite 
 widely regarding the difference between Rust's borrow checker 
 and ownership transfer system.

 The borrow checker loosens restrictions placed upon you by the 
 ownership transfer system to give strong guarantees of aliasing 
 and lifetimes. If you did not have the borrow checker, the 
 ownership transfer systems restrictions would make things 
 absolutely impossible to work with.
 [snip]


Not sure I completely follow everything here, but is your point 
that to actually do what Rust is doing we need the 
ownership/borrowing system plus borrow checking?



 May 09



  parent  "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz>  writes:


On 10/05/2025 7:15 AM, jmh530 wrote:

 On Friday, 9 May 2025 at 18:28:11 UTC, Richard (Rikki) Andrew Cattermole 
 wrote:

 On 10/05/2025 5:17 AM, jmh530 wrote:

 To your point about being a property of a pointer, the alternative 
 approach is to introduce a reference that is borrow checked. Yes, it 
 results in additional pointer types, but if that's the approach that 
 Rust ultimately relies on and it's the only way to achieve what we 
 want, then it's what should be done.


 I want to touch upon this because its misunderstood quite widely 
 regarding the difference between Rust's borrow checker and ownership 
 transfer system.

 The borrow checker loosens restrictions placed upon you by the 
 ownership transfer system to give strong guarantees of aliasing and 
 lifetimes. If you did not have the borrow checker, the ownership 
 transfer systems restrictions would make things absolutely impossible 
 to work with.
 [snip]


 
 Not sure I completely follow everything here, but is your point that to 
 actually do what Rust is doing we need the ownership/borrowing system 
 plus borrow checking?


Strictly speaking no.

What you need is escape analysis with relationship strengths.

```d
struct Owner {
     int* borrow()  escape(return^) { ... }
}

Owner owner = ...;
int* ptr = owner.borrow;
owner = Owner.init; // Error
```

The escape attribute there, upgrades the escape to protecting the this 
pointer for as long as the borrow exists.

DIP1000 has two relationship strengths, for this we need a third.

This triggers the borrow checking aspect to escape analysis.
Without it, it can't trigger per variable.

The main problem is how costly the data flow analysis of doing this is, 
to an extent that models enough code to both be useful and fast.

I think I've solved this part of the problem, and am currently 
implementing the first portion of it up to a proof of concept stage.



 May 09



prev sibling next sibling parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/9/2025 11:28 AM, Richard (Rikki) Andrew Cattermole wrote:

 This is what our DIP1000 attempts to do poorly.


The concept of DIP1000 is solid. It does an excellent job (sans bugs) of 
preventing escaping pointers to the stack.

The problem is fitting it into the various high level constructs that D has,
and 
the confusing attributes needed to inform the function interface of the status 
of its arguments.

DIP1000 does not need to do flow analysis, and does not attempt it, but the 
borrow checker does need it.



 May 09



prev sibling  parent reply Derek Fawcus <dfawcus+dlang employees.org>  writes:


On Friday, 9 May 2025 at 18:28:11 UTC, Richard (Rikki) Andrew 
Cattermole wrote:

 On 10/05/2025 5:17 AM, jmh530 wrote:

 To your point about being a property of a pointer, the 
 alternative approach is to introduce a reference that is 
 borrow checked. Yes, it results in additional pointer types, 
 but if that's the approach that Rust ultimately relies on and 
 it's the only way to achieve what we want, then it's what 
 should be done.


 I want to touch upon this because its misunderstood quite 
 widely regarding the difference between Rust's borrow checker 
 and ownership transfer system.

 The borrow checker loosens restrictions placed upon you by the 
 ownership transfer system to give strong guarantees of aliasing 
 and lifetimes. If you did not have the borrow checker, the 
 ownership transfer systems restrictions would make things 
 absolutely impossible to work with.


I can't say I'm 100% comfortable with these things, but anyway:

A characteristic of affine and linear types is that they can be 
'used'/'consumed' at most one, with linear also being 'used' at 
least once.  Hence linear types must be 'used'.

This has some awkwardness in how that behaviour is then 
experienced by the programmer.

So Rust has something which is a form of affine types, but the 
borrowing allows one to make use of the type without consuming 
the value.  i.e. a use is not deemed a 'use'/'consumption'.

This 'borrowing' creates a restricted form of pointer, possibly 
mutable, and the checker is then validating / proving a set of 
constraints for how such borrowed references may be handled.

I view this sort of thing (along with the many different forms of 
pointer which Cyclone had) as completely different to the forms 
of near/far/huge/segment pointer which were available in various 
DOS compilers, and that (IMO) doesn't make for a sensible 
comparison.

Certain modern languages already have multiple forms of pointer 
(nullable, non-null, fat, thing, with and without arithmetic) and 
those do not create the same issues for the programmer as the DOS 
style pointers did.

D itself already has 3 different types of pointer: references, 
pointers, arrays/slices.

I don't see adding additional types as an issue, other then 
syntactical issues.  Obviously there will be implementation 
issues for the compiler, but depending upon the presented form, I 
do not see a reason why this should result in user issues.



 May 10



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/10/2025 6:07 AM, Derek Fawcus wrote:

 D itself already has 3 different types of pointer: references, pointers, 
 arrays/slices.


Plus `this`, lazy, delegates, associative arrays and class references.

They were part of the initial design of D, the language grew up around them.

They've also made for making dip1000 very complex to implement.

Trying to fold in a new pointer type is a huge change.

(If I was doing a do-over for D, I'd look hard at dumping about half of those.
I 
tried to get rid of lazy a few years ago, but got a lot of pushback on it.)

Microsoft's Managed C++ has regular pointers and gc pointers, distinguished
with 
different syntax. It was a failure in the marketplace.



 May 10



 next sibling parent  Araq <rumpf_a web.de>  writes:


On Saturday, 10 May 2025 at 23:25:13 UTC, Walter Bright wrote:


 Microsoft's Managed C++ has regular pointers and gc pointers, 
 distinguished with different syntax. It was a failure in the 
 marketplace.


C++ has unique_ptr, shared_ptr, weak_ptr ... in addition to its 
raw pointers and C++ is doing fine.

The good news for you is that indeed you don't need to build 
these into your compiler and that D can likewise have these as a 
library. (Though for unique_ptr you might need better support for 
move operations.) The bad news is that your  live cannot be 
attached to a pointer so that library implementations of xxx_ptr 
cannot make use of it to enable more safety.



 May 10



prev sibling next sibling parent  claptrap <clap trap.com>  writes:


On Saturday, 10 May 2025 at 23:25:13 UTC, Walter Bright wrote:

 On 5/10/2025 6:07 AM, Derek Fawcus wrote:

 Microsoft's Managed C++ has regular pointers and gc pointers, 
 distinguished with different syntax. It was a failure in the 
 marketplace.


That a logical fallacy. It's

X has property Y
X failed
therefor property Y causes failure

Eg..

DeLoreans had rear engines.
DeLoreans were a commercial failure.
Therefore rear engines cause commercial failure.

IE. If you want to use the failure of managed C++ as an argument 
against "managed pointers" or "multiple pointer types" you would 
have to show that it failed because of "managed pointers".



 May 11



prev sibling  parent reply Paulo <pjmlp progtools.org>  writes:


On Saturday, 10 May 2025 at 23:25:13 UTC, Walter Bright wrote:

 On 5/10/2025 6:07 AM, Derek Fawcus wrote:

 D itself already has 3 different types of pointer: references, 
 pointers, arrays/slices.


 Plus `this`, lazy, delegates, associative arrays and class 
 references.

 They were part of the initial design of D, the language grew up 
 around them.

 They've also made for making dip1000 very complex to implement.

 Trying to fold in a new pointer type is a huge change.

 (If I was doing a do-over for D, I'd look hard at dumping about 
 half of those. I tried to get rid of lazy a few years ago, but 
 got a lot of pushback on it.)

 Microsoft's Managed C++ has regular pointers and gc pointers, 
 distinguished with different syntax. It was a failure in the 
 marketplace.


You keep repeating this, sorry but it only goes to show how 
little you know about the modern Microsoft ecosystem.

Not only did Managed C++ evolve into C++/CLI, did C++/CLI get 
updated to C++20 back in 2023, and has several bug fixes coming 
up on Visual Studio 2022 17.4.

https://devblogs.microsoft.com/cppblog/cpp20-support-comes-to-cpp-cli


https://devblogs.microsoft.com/cppblog/c-language-updates-in-msvc-in-visual-studio-2022-17-14/#c++/cli

C++/CLI is the to go tool for many developers that rather not 
spend all day writing P/Invoke attributes, when it is a big C++ 
library being added into a .NET project.



 May 11



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


It never made it into the C++ Standard, and its only use is CLI.



 May 12



  parent  Paulo Pinto <pjmlp progtools.org>  writes:


On Monday, 12 May 2025 at 21:38:28 UTC, Walter Bright wrote:

 It never made it into the C++ Standard, and its only use is CLI.


So what.

It is a commercial success in the Windows developer community, to 
the point Microsoft even diverted resources from C++23 
development, to improve C++20 compliance on C++/CLI, as 
communicated by Microsoft employees, as some of the reasons 
regarding the C++23 support delays.

GCC and clang also have lots of compiler extensions that never 
made into the C or C++ standard, and only exist in one of them. 
To the point Google invested lots of money to make some projects, 
like the Linux kernel, be able to compile with both compilers.

Unreal C++ extensions are also not into the C++ standard, they 
are only available on the most used games engine in the industry.

Are they also a failure?

If anything, this is a good example of the market fit that has 
been discussed in these threads, C++/CLI does one thing, and it 
is great at it, being a productive way to do C++ interop in .NET.



 May 12



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/7/2025 4:21 PM, Timon Gehr wrote:

 On 5/7/25 16:47, jmh530 wrote:

 I can think of a few different ways forward (by no means limited to just this)


 
 Well, the big issue here is that  live is still not transitive.


We are all well aware of the problems with transitive attributes. Making  live 
transitive means that nobody will ever use it, no matter how good it is. People 
keep asking for a non-transitive  nogc.


 A function 
 attribute is not really the right way to go about this anyway. Aliasing is a 
 property of a pointer, not a function.


Attaching it to a pointer means nobody will use it, because they'll have to
redo 
their entire program.



 May 09



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 5/10/25 06:02, Walter Bright wrote:

 On 5/7/2025 4:21 PM, Timon Gehr wrote:

 On 5/7/25 16:47, jmh530 wrote:

 I can think of a few different ways forward (by no means limited to 
 just this)


 Well, the big issue here is that  live is still not transitive.


 
 We are all well aware of the problems with transitive attributes. Making 
  live transitive means that nobody will ever use it, no matter how good 
 it is. People keep asking for a non-transitive  nogc.
 ...


My point is not that ` live` should be a transitive function attribute. 
It should not be a function attribute in the first place.

For ` live` functions, not only is it a problem when they call 
non-` live` functions, it is also a problem you call _them_ from 
non-` live` functions.

And even then, you have not implemented checks that are even sufficient 
to guarantee anything if your entire program is ` live` with a custom 
` live` druntime.



 A function attribute is not really the right way to go about this 
 anyway. Aliasing is a property of a pointer, not a function.


 
 Attaching it to a pointer means nobody will use it, because they'll have 
 to redo their entire program.
 


If this is really your perspective, it is better to just drop borrow 
checking as a direction. It seems your design constraints are 
self-contradictory assuming that utility is one of them. The current 
design does not incentivize people to use it, it just makes it not very 
useful.

The entire point is allowing gradual adoption with gradual benefits. 
However, this has to be done one datastructure at as time, not one 
function at a time.

In any case, what you say is simply not true, you can e.g. pass borrowed 
things as `scope` parameters, as long as you are able to control 
accesses to the owner for the duration of that function call.

But sure, if your entire ` safe` program is based around passing raw, 
non-scope pointers around, there will _never_ be a way to use these 
functions from ` safe` code using manually allocated pointers.

Note that `int*` and `scope int*` are _already_ de facto different 
pointer types (even though `scope` is implemented as an attribute).
You cannot pass a `scope int*` to an `int*` parameter. Yet for DIP1000, 
having to "redo the entire program" to use `scope` and other annotations 
was not considered a deal-breaker.

Though neither DIP1000 nor ` live` have any meaningful story about 
multiple indirections.



 May 10



 next sibling parent  "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz>  writes:


On 11/05/2025 12:51 AM, Timon Gehr wrote:

 If this is really your perspective, it is better to just drop borrow 
 checking as a direction. It seems your design constraints are self- 
 contradictory assuming that utility is one of them. The current design 
 does not incentivize people to use it, it just makes it not very useful.


I'm inclined to suggest  live is still savable, tie it to  restrict and 
it'll be useful to people doing SIMD type data processing.

However that isn't borrow checking, that is something new that I don't 
think is in the literature currently.



 May 10



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


1. `scope` is not part of the type. It is a storage class, not a type modifier. 
The only type modifiers are immutable, const, shared and inout. Those are 
pervasively handled throughout the compiler's dealings with types. Making a new 
pointer type would be a huge effort *in addition* to all the other work needed 
to make a borrow checker.

2. We have a lot of experience with transitive attributes. There's a lot of 
resistance to using them because of this. Just look at all the complaints about 
`nogc` being transitive. Personally, I find `const` being transitive as a 
serious impediment to using `const` in the dmd source code. One of the big 
problems is it doesn't work with the Visitor pattern. (There are a couple cases 
where `const` is just cast away, shame on me.)

3. I'm well aware that ideally  live would be sounder if it is transitive. I 
disagree with the notion that it is useless if it is not. I am 98% sure that if 
 live was made transitive, it would be instantly useless because nobody is
going 
to be willing to make an existing program or library pass the borrow checker. 
(It took enormous effort for Rust to convince people they had to throw their 
existing code away and recode it from scratch to use the borrow checker.) Much 
of the existing code would have to be thrown out. Making a perfect borrow 
checker is the enemy of making a practical, useful one.

4. Making dip1000 transitive made it impractical to use in existing code.

5. dip1000 makes it an error to assign a scope pointer to multiple indirections 
(because scope isn't a type modifier).

6. When an  live function calls another function, a `scope` parameter is a 
borrow, and a non-scope parameter is an ownership transfer.

7. Compiling an  live function is slow because of the DFA. Making it transitive 
would make it apply to the whole program, which would then compile as slow as
Rust.



 May 10



 next sibling parent  Nick Treleaven <nick geany.org>  writes:


On Saturday, 10 May 2025 at 21:20:11 UTC, Walter Bright wrote:

 1. `scope` is not part of the type. It is a storage class, not 
 a type modifier. The only type modifiers are immutable, const, 
 shared and inout. Those are pervasively handled throughout the 
 compiler's dealings with types. Making a new pointer type would 
 be a huge effort *in addition* to all the other work needed to 
 make a borrow checker.


What if ` live` was a storage class? Then it might be easier to 
integrate with other code, and potentially help to provide some 
memory-safety guarantees. The compiler's understanding of unique 
expressions* might help too.

[*] https://dlang.org/spec/const3.html#unique-expressions



 May 11



prev sibling  parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 5/10/25 23:20, Walter Bright wrote:

 1. `scope` is not part of the type.


I stated as much, but it still influences type checking, so it is 
formally part of the type system.


 It is a storage class, not a type 
 modifier. The only type modifiers are immutable, const, shared and 
 inout. Those are pervasively handled throughout the compiler's dealings 
 with types. Making a new pointer type would be a huge effort *in 
 addition* to all the other work needed to make a borrow checker.
 ...


Sure, so don't do it if you think it is not worth it. It's necessary to 
have annotations that are part of types to make meaningful progress on 
this though. Note that I am pretty sure that your issues with type 
modifiers in DMD are largely due to the way they are implemented. I 
don't think it would be very hard to add such features to my own frontend.


 2. We have a lot of experience with transitive attributes. There's a lot 
 of resistance to using them because of this. Just look at all the 
 complaints about `nogc` being transitive.


Well, sometimes you want it to be transitive, sometimes you just want to 
avoid implicit allocations. It's different use cases and they should be 
distinct features. OpenD adopted my proposal for `pragma(explicit_gc)` 
and it helped me chase down some unwanted implicit GC allocations in my 
programs without flagging all the instances where I was deliberately 
using GC.


 Personally, I find `const` 
 being transitive as a serious impediment to using `const` in the dmd 
 source code. One of the big problems is it doesn't work with the Visitor 
 pattern. (There are a couple cases where `const` is just cast away, 
 shame on me.)
 ...


Well yes, as I have been saying for years, don't use `const` if you want 
to mutate. As you would not use features that restrict aliasing if you 
want to freely alias. There is no free lunch, if you want 
machine-checkable guarantees, you have to follow a certain discipline.

` live` however is discipline for its own sake, without the slightest 
hint of practical machine-checkable guarantees.


 3. I'm well aware that ideally  live would be sounder if it is 
 transitive.


No. Again, ideally there would be no ` live` function attribute in the 
first place. It's the wrong way to decide where to do type checks.

Anyway, there is sound and unsound. And there are directions to follow 
that can in principle lead to a sound design with enough effort, and 
there are directions to follow that will never lead to a sound design. 
` live` is in the second category, ` safe` is in the first category.


 I disagree with the notion that it is useless if it is not.


I am not going to use it. It is useless to me. Not because it is not 
transitive, but because it does not even pretend to want to achieve 
anything.


 I am 98% sure that if  live was made transitive, it would be instantly 
 useless because nobody is going to be willing to make an existing 
 program or library pass the borrow checker.


It appears that you did not read what I wrote in my previous post, you 
are still attacking the same straw man that I already disowned.

You are correct here. ` live` is useless whether it is transitive or not.


 (It took enormous effort for 
 Rust to convince people they had to throw their existing code away and 
 recode it from scratch to use the borrow checker.) Much of the existing 
 code would have to be thrown out. Making a perfect borrow checker is the 
 enemy of making a practical, useful one.
 ...


Nonsense. Being practical and useful is of course part of perfection. 
Soundness is just also needed, if the utility sought is memory safety 
guarantees.

And if the goal is not memory safety guarantees, I just don't see why 
people would bother with this. What customer or boss will be like: "You 
really need to use ` live` when you code up this program for me." I 
really don't see it.


 4. Making dip1000 transitive made it impractical to use in existing code.
 ...


No, the problem with DIP1000 you are referring to that it adds 
additional restrictions for existing syntax, so if you want to adopt it 
you have to first pay the up-front cost of updating your existing code. 
This simply does not happen if the new feature uses syntax that is not 
used in the wild.

Of course, even then you will get what you pay for in terms of 
annotation and type checking overhead, but with ` live` you get way less 
than what you pay.

Anyway, hardness of adoption is simply the nature of DIP1000, as its 
goal was to fix holes in  safe without making certain use cases 
impossible in  safe code outright. I suspect with editions, people will 
be able to adopt DIP1000 (or its more expressive future successor) 
incrementally, which will make it significantly more practical.


 5. dip1000 makes it an error to assign a scope pointer to multiple 
 indirections (because scope isn't a type modifier).
 ...


Sure. Code often has multiple indirections in practice though.


 6. When an  live function calls another function, a `scope` parameter is 
 a borrow, and a non-scope parameter is an ownership transfer.
 ...


Well, operationally it acts somewhat like these. Semantically this is 
not what happens.

As I have stated in the past, it's a bit like cargo cults in the 
pacific. Operationally, you have people that act kind of like staff 
maintaining a landing strip. In practice, no planes actually are landing 
or taking off from there.

` live` signs contracts for borrowing and/or sale with much fanfare and 
will sometimes stop you from doing what you want to do, but no lending 
takes place and no ownership is actually transferred.


 7. Compiling an  live function is slow because of the DFA. Making it 
 transitive would make it apply to the whole program, which would then 
 compile as slow as Rust.
 


Not true, you only have to enhance type checking where the features are 
actually used. This is not a ` live`-specific thing at all.



 May 11



prev sibling  parent reply Dukc <ajieskola gmail.com>  writes:


On Wednesday, 7 May 2025 at 14:47:49 UTC, jmh530 wrote:

 4) Add  trustediflive, similar to above such that it is 
  trusted if called by a  live function and  system otherwise. 
 Let's the user make the decision, more flexible. I think 
  trustediflive is kind of in line with what you were suggesting.


To be honest I didn't think the solution in detail. My intention 
was just to point out roughly what it would have to be like if 
` live`, is to be useful for ` safe`ty but I certainly didn't 
intend to say it's necessarily the best way to go about it. A 
totally different borrow checker is also a possibility.

There are many details to consider, like how do we check whether 
the parameters passed to / value returned from the ` safe  live` 
function using ` trustediflive` need to be borrow checked too? 
And how do we declare the type of such a function? It's going to 
be complicated.

I'm not sure whether a borrow checker is even worth it for D. 
Probably, no matter how we do it, it's still going to be 
complicated, and it's useful for only a pretty niche use case. 
For the vast majority of your memory safety problems you can and 
should just rely on the GC after all. Walter is writing something 
along these lines in his OP.

But if it's worth it, either ` live` needs way more language 
machinery to support it's purpose or we need an entirely new 
solution.



 May 08



 next sibling parent  jmh530 <john.michael.hall gmail.com>  writes:


On Thursday, 8 May 2025 at 08:54:16 UTC, Dukc wrote:

 [snip]
 To be honest I didn't think the solution in detail. My 
 intention was just to point out roughly what it would have to 
 be like if ` live`, is to be useful for ` safe`ty but I 
 certainly didn't intend to say it's necessarily the best way to 
 go about it. A totally different borrow checker is also a 
 possibility.
 [snip]


Fair points throughout. My reply to Timon applies a bit to what 
you say here as well.



 May 09



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


The  live design was done in a manner to not need any new syntax - it is an 
extra layer of error checking.



 May 09



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 5/10/25 06:13, Walter Bright wrote:

 The  live design was done in a manner to not need any new syntax - it is 
 an extra layer of error checking.


That would be giving too much credit. It is adding an extra layer of 
checking, but what it checks for are not errors, it just checks for 
adherence to a somewhat arbitrary discipline of how pointers may be 
assigned to one another.



 May 11



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/11/2025 5:32 AM, Timon Gehr wrote:

 On 5/10/25 06:13, Walter Bright wrote:

 The  live design was done in a manner to not need any new syntax - it is an 
 extra layer of error checking.


 
 That would be giving too much credit.


This is why I stopped working on D's borrow checker. It's all been relentlessly 
negative. But I did get it to the point where it proved that a borrow checker 
can work in D with zero changes to the syntax (except marking a function as 
 live), and I'm happy about that.

When I embarked on ImportC, I got the same response. Until suddenly it became a 
major asset for D. I was just more willing to push through with it.



 May 11



 next sibling parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 5/11/25 20:23, Walter Bright wrote:

 
 This is why I stopped working on D's borrow checker.


That was the right call.


 It's all been relentlessly negative.


Well, sorry, this is not fun for me.


 But I did get it to the point where it proved that a borrow checker can work
in D with zero changes to the syntax (except marking a function as  live), and
I'm happy about that. 


I think nobody competent doubted it is possible to do what ` live` does, 
it's just not a particularly sensible thing to do.

I also completely disagree that what ` live` does is somehow equivalent 
to "a borrow checker working in D" comparing to standard expectations by 
a competent user.


 When I embarked on ImportC, I got the same response.


This is not true, I was on board with ImportC.
https://forum.dlang.org/post/skmhil$p8n$1 digitalmars.com


 Until suddenly it became a major asset for D.


In addition to it becoming a minor liability due to the temptation of 
copying over the bitfields implementation to D proper. :o)


 I was just more willing to push through with it.


It's just not true that ` live` would be good if you prioritized it over 
ImportC. It's a fundamentally flawed design.



 May 11



  parent  "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz>  writes:


On 12/05/2025 7:58 AM, Timon Gehr wrote:

     It's all been relentlessly negative.
 
 Well, sorry, this is not fun for me.


It has long since crossed over into "just implement it" territory for 
energy reasons for me.



 May 12



prev sibling next sibling parent reply Paul Backus <snarwin gmail.com>  writes:


On Sunday, 11 May 2025 at 18:23:56 UTC, Walter Bright wrote:

 On 5/11/2025 5:32 AM, Timon Gehr wrote:

 On 5/10/25 06:13, Walter Bright wrote:

 The  live design was done in a manner to not need any new 
 syntax - it is an extra layer of error checking.


 
 That would be giving too much credit.


 This is why I stopped working on D's borrow checker. It's all 
 been relentlessly negative. But I did get it to the point where 
 it proved that a borrow checker can work in D with zero changes 
 to the syntax (except marking a function as  live), and I'm 
 happy about that.


The reason for the negativity is that people do not want "a 
borrow checker." What they want is more powerful safety analysis. 
A borrow checker is merely one possible means of achieving that 
goal.

In business terms, this is a failure of "product-market fit." You 
are supplying something that there is no demand for. Even if it 
works perfectly, nobody is going to buy it.



 May 11



  parent reply jmh530 <john.michael.hall gmail.com>  writes:


On Sunday, 11 May 2025 at 20:43:52 UTC, Paul Backus wrote:

 [snip]

 The reason for the negativity is that people do not want "a 
 borrow checker." What they want is more powerful safety 
 analysis. A borrow checker is merely one possible means of 
 achieving that goal.

 In business terms, this is a failure of "product-market fit." 
 You are supplying something that there is no demand for. Even 
 if it works perfectly, nobody is going to buy it.


Agreed.

But I think "what people want" and "what's the goal here" are two 
different kinds of questions. What people want is more powerful 
safety analysis. But the reason why Walter started working on 
this was that he saw safety as becoming a bigger issue in 
programming languages, most notably with Rust gaining traction, 
and wanted to help D compete on that basis. So I think one thing 
that has Walter frustrated is that he wants to be able to say D 
has a borrow checker when someone asks, but the actual users 
don't want something unless it actually enables the more powerful 
safety analysis.

And at the same time, people are frustrated with half-baked 
implementations of things. At least if Walter stops working on 
 live, then it wouldn't have been made a part of the language.

But at the end of the day, I think there are a lot of people who 
are on board with more powerful safety analysis, provided it is 
done right.



 May 12



  parent reply Paul Backus <snarwin gmail.com>  writes:


On Monday, 12 May 2025 at 13:45:04 UTC, jmh530 wrote:

 But I think "what people want" and "what's the goal here" are 
 two different kinds of questions. What people want is more 
 powerful safety analysis. But the reason why Walter started 
 working on this was that he saw safety as becoming a bigger 
 issue in programming languages, most notably with Rust gaining 
 traction, and wanted to help D compete on that basis.


Here, you say that the goal of  live is to compete with Rust on 
safety.


 So I think one thing that has Walter frustrated is that he 
 wants to be able to say D has a borrow checker when someone 
 asks, but the actual users don't want something unless it 
 actually enables the more powerful safety analysis.


Here, you say that the goal of  live is "to be able to say D has 
a borrow checker."

The problem is that these are not the same goal. Walter's work on 





is a mistake. If Walter disagrees, I can accept that, although I 
would like to at least see him make the positive case for why 




 May 12



  parent reply jmh530 <john.michael.hall gmail.com>  writes:


On Monday, 12 May 2025 at 16:11:18 UTC, Paul Backus wrote:

 On Monday, 12 May 2025 at 13:45:04 UTC, jmh530 wrote:

 But I think "what people want" and "what's the goal here" are 
 two different kinds of questions. What people want is more 
 powerful safety analysis. But the reason why Walter started 
 working on this was that he saw safety as becoming a bigger 
 issue in programming languages, most notably with Rust gaining 
 traction, and wanted to help D compete on that basis.


 Here, you say that the goal of  live is to compete with Rust on 
 safety.


 So I think one thing that has Walter frustrated is that he 
 wants to be able to say D has a borrow checker when someone 
 asks, but the actual users don't want something unless it 
 actually enables the more powerful safety analysis.


 Here, you say that the goal of  live is "to be able to say D 
 has a borrow checker."

 The problem is that these are not the same goal. Walter's work 





In my view, competing with Rust and being able to say that D has 
a borrow checker are the same thing. My point was that what 
people really want is more powerful safety analysis. Sorry if 
that was confusing.



 May 12



  parent  Paul Backus <snarwin gmail.com>  writes:


On Monday, 12 May 2025 at 19:15:56 UTC, jmh530 wrote:

 In my view, competing with Rust and being able to say that D 
 has a borrow checker are the same thing. My point was that what 
 people really want is more powerful safety analysis. Sorry if 
 that was confusing.


It's not confusing, it's just incorrect. I understand your view 
perfectly, and I disagree with it.



 May 12



prev sibling next sibling parent  Meta <jared771 gmail.com>  writes:


On Sunday, 11 May 2025 at 18:23:56 UTC, Walter Bright wrote:

 This is why I stopped working on D's borrow checker. It's all 
 been relentlessly negative.


At some point when someone won't listen and every other option to 
convince them has been tried, the only option is to get upset.



 May 11



prev sibling  parent reply claptrap <clap trap.com>  writes:


On Sunday, 11 May 2025 at 18:23:56 UTC, Walter Bright wrote:

 On 5/11/2025 5:32 AM, Timon Gehr wrote:

 On 5/10/25 06:13, Walter Bright wrote:

 The  live design was done in a manner to not need any new 
 syntax - it is an extra layer of error checking.


 
 That would be giving too much credit.


 This is why I stopped working on D's borrow checker. It's all 
 been relentlessly negative.


Isn't that the point of negative feedback? I mean people say 
"yeah thats crap, we dont want it" so you stop working on it.

The system works!

Might help to find a way to figure it out sooner tho, maybe some 
kind of proposal system where people could critique it before you 
waste your time doing all that work?



 May 11



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/11/2025 2:01 PM, claptrap wrote:

 Might help to find a way to figure it out sooner tho, maybe some kind of 
 proposal system where people could critique it before you waste your time
doing 
 all that work?


Presenting a working solution is more effective than a proposal.



 May 12



  parent  claptrap <clap trap.com>  writes:


On Monday, 12 May 2025 at 08:26:28 UTC, Walter Bright wrote:

 On 5/11/2025 2:01 PM, claptrap wrote:

 Might help to find a way to figure it out sooner tho, maybe 
 some kind of proposal system where people could critique it 
 before you waste your time doing all that work?


 Presenting a working solution is more effective than a proposal.


More effective at demonstrating the idea, not at all effective at 
avoiding time wasted on stuff nobody wants though, which was 
*actually* my point.



 May 12



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/4/2025 11:52 PM, Dukc wrote:

 You have to admit this makes our checker, as it currently stands, unusable for 
 what many of us consider the primary purpose of a borrow checker.


The borrow checker works on functions marked with  live. Just like safe checks 
are only on functions marked  safe. Both are usable.



 May 09



  parent reply Dukc <ajieskola gmail.com>  writes:


On Saturday, 10 May 2025 at 03:52:34 UTC, Walter Bright wrote:

 On 5/4/2025 11:52 PM, Dukc wrote:

 You have to admit this makes our checker, as it currently 
 stands, unusable for what many of us consider the primary 
 purpose of a borrow checker.


 The borrow checker works on functions marked with  live. Just 
 like safe checks are only on functions marked  safe. Both are 
 usable.


No aknowledgement of the langauge shortcoming multiple people 
have to pointed out again and again. You're still talking as it 
wasn't there.

I have hard time believing a successful programming language 
designer wouldn't just get it. Instead, I'm assuming you have a 
marketing philosophy that tells you to avoid admitting things 
like this because it'd be more important to show faith in the 
language and inspire confidence in it.

Maybe this sort of thinking has it's place when we're at 
news.ycombinator.com and the topic is about merits of the 
language in general. But this is largely a design discussion, and 
the participants are largely those who are already committed to 
the language. Discussions like this can't move forward without 
common understanding of the problem.

It is true that there could still be someone who searches for "D 
borrow checker" or something, lands on this discussion and goes 
for Rust instead when there's a confirmation our borrow checker 
doesn't do the same things. But hiding problems here runs into 
what I see as a far greater risk: The debators, who want to 
improve the language, or see it improving, will get frustated if 
they don't see the leadership treating their input seriously, 
which will lead to loss of morale and contributors if it goes too 
far. So please don't treat these issues with a "fake it till you 
make it" - attitude.



 May 10



 next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz>  writes:


On 11/05/2025 6:06 AM, Dukc wrote:

 I have hard time believing a successful programming language designer 
 wouldn't just get it. Instead, I'm assuming you have a marketing 
 philosophy that tells you to avoid admitting things like this because 
 it'd be more important to show faith in the language and inspire 
 confidence in it.


There is no need for this language.

Everything that I have seen and know about Walter strongly suggests that 
he doesn't understand where we are diverging.

It may not be helping that to solve this involves slower DFA's and the 
addition of a type qualifier.

I can prove that the former doesn't need to be the default (and I am 
trying to do so by implementing it).



 May 10



  parent  claptrap <clap trap.com>  writes:


On Saturday, 10 May 2025 at 20:27:16 UTC, Richard (Rikki) Andrew 
Cattermole wrote:

 On 11/05/2025 6:06 AM, Dukc wrote:

 I have hard time believing a successful programming language 
 designer wouldn't just get it. Instead, I'm assuming you have 
 a marketing philosophy that tells you to avoid admitting 
 things like this because it'd be more important to show faith 
 in the language and inspire confidence in it.


 There is no need for this language.

 Everything that I have seen and know about Walter strongly 
 suggests that he doesn't understand where we are diverging.


That reminds me I haven't watched Groundhog Day for a while.



 May 10



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/10/2025 11:06 AM, Dukc wrote:

 No aknowledgement of the langauge shortcoming multiple people have to pointed 
 out again and again. You're still talking as it wasn't there.


I've acknowledged it several times, and pointed out the problems with the 
proposed solutions (transitivity, new pointer types).

Rust is a one-note language. Everything revolves around the borrow checker.
That 
does not work with D, which is a polyglot language. A borrow checker has to fit 
in with the rest of the language. Of course that means compromise. Compromise 
doesn't mean it's useless.



 May 10



  parent  Dukc <ajieskola gmail.com>  writes:


On Saturday, 10 May 2025 at 23:32:07 UTC, Walter Bright wrote:

 On 5/10/2025 11:06 AM, Dukc wrote:

 No aknowledgement of the langauge shortcoming multiple people 
 have to pointed out again and again. You're still talking as 
 it wasn't there.


 I've acknowledged it several times,


Sorry, but not very clearly. You keep on insisting that D borrow 
checker is useful. It's true to some extent - it *can* function 
as a limited linting tool. But since the point you are replying 
to is that the borrow checker doesn't help writing more ` safe` 
code (unlike the Rust one) it leaves the impression that you 
refuse to accept the points made.



 May 10



prev sibling  parent reply Guillaume Piolat <first.nam_e gmail.com>  writes:


On Tuesday, 29 April 2025 at 17:12:41 UTC, Walter Bright wrote:

 It reminds me of OOP. OOP was sold as the answer to every 
 programming problem. Many OOP languages appeared. But, 
 eventually, things died down and OOP became just another tool 
 in the toolbox. D and C++ support OOP, too.


My issue with all the talk about how some features could our 
savior is that they are nowhere to be found in our direct 
competitors.

There is no surge of competitors boosted or enabled by 
memory-safety, borrow-checking, lack of data races or even 
advanced types. There is very little Rust competitors, to the 
point I'd argue people self-select out of the market by spending 
time on expensive features like those that bring no bread on the 
table. For example we have zero Haskell, Clojure, or Elm 
competitors, or even Ocaml in the audio space (that one could 
work there probably).

But there is a very real threat of competitors being faster with 
lower iteration times.



 May 02



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 5/2/25 16:53, Guillaume Piolat wrote:

 On Tuesday, 29 April 2025 at 17:12:41 UTC, Walter Bright wrote:

 It reminds me of OOP. OOP was sold as the answer to every programming 
 problem. Many OOP languages appeared. But, eventually, things died 
 down and OOP became just another tool in the toolbox. D and C++ 
 support OOP, too.


 
 My issue with all the talk about how some features could our savior is 
 that they are nowhere to be found in our direct competitors.
 
 There is no surge of competitors boosted or enabled by memory-safety, 


(` live` does not give you memory safety guarantees.)


 borrow-checking, lack of data races or even advanced types. There is 
 very little Rust competitors, to the point I'd argue people self-select 
 out of the market by spending time on expensive features like those that 
 bring no bread on the table. For example we have zero Haskell, Clojure, 
 or Elm competitors, or even Ocaml in the audio space (that one could 
 work there probably).
 
 But there is a very real threat of competitors being faster with lower 
 iteration times.
 
 


Well, beyond the competitive edge that D affords you, to some extent 
users get what they pay/wait for and I think/hope users will be 
increasingly conscious of the possibility that someone might take over 
their computer by e.g. sending them an mp3 file that is invalid in a 
peculiar way.

The point of memory safety features is not really to help you write 
faster unsafe code with lower iteration times (though they sometimes do).

The goal is to help you write faster safe code with lower iteration 
times. In some domains, safety is, or will be, a requirement on the 
functionality of the software.

Some businesses don't care about cybersecurity until they or their users 
get hacked (regularly). Of course, memory corruption is only one way 
attackers can succeed, so advanced type system features can help. (More 
advanced than what you cite though, so we'll see how quickly those can 
be transferred to industry. It seems that usually the type system tech 
in popular languages is on the order of 40 years old.)

` live` however does not give you additional hard guarantees, so I also 
think a priori it is not a feature on whose implementation I would spend 
time personally. Perhaps the work put into it can be reused for 
something more useful in the future.



 May 02



 next sibling parent reply Dukc <ajieskola gmail.com>  writes:


On Friday, 2 May 2025 at 19:25:53 UTC, Timon Gehr wrote:

 On 5/2/25 16:53, Guillaume Piolat wrote:

 On Tuesday, 29 April 2025 at 17:12:41 UTC, Walter Bright wrote:
 For example we have zero Haskell, Clojure, or Elm competitors
 


 Of course, memory corruption is only one way attackers can 
 succeed, so advanced type system features can help. (More 
 advanced than what you cite though, so we'll see how quickly 
 those can be transferred to industry.


More advanced than _Haskell_? Wat?

IMO it's absolutely mind-blowing how advanced Haskell is when you 
enable the more recent features such as GADTs. I feel it's 
already so powerful that my comprehension runs out well before 
the expressive power of the language, and I don't think I'm bad 
at comprehending programming language features.



 May 02



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 5/2/25 21:48, Dukc wrote:

 On Friday, 2 May 2025 at 19:25:53 UTC, Timon Gehr wrote:

 On 5/2/25 16:53, Guillaume Piolat wrote:

 On Tuesday, 29 April 2025 at 17:12:41 UTC, Walter Bright wrote:
 For example we have zero Haskell, Clojure, or Elm competitors


 Of course, memory corruption is only one way attackers can succeed, so 
 advanced type system features can help. (More advanced than what you 
 cite though, so we'll see how quickly those can be transferred to 
 industry.


 
 More advanced than _Haskell_? Wat?
 
 IMO it's absolutely mind-blowing how advanced Haskell is when you enable 
 the more recent features such as GADTs. I feel it's already so powerful 
 that my comprehension runs out well before the expressive power of the 
 language, and I don't think I'm bad at comprehending programming 
 language features.


See e.g., https://ghc.serokell.io/dh

In any case, comprehensibility is at most loosely related to 
expressiveness. I think Haskell with all its extensions is too complex 
for what it gives you in terms of expressiveness.



 May 02



  parent  Dukc <ajieskola gmail.com>  writes:


On Saturday, 3 May 2025 at 03:11:14 UTC, Timon Gehr wrote:

 See e.g., https://ghc.serokell.io/dh

 In any case, comprehensibility is at most loosely related to 
 expressiveness. I think Haskell with all its extensions is too 
 complex for what it gives you in terms of expressiveness.


Thanks for the explaination.



 May 04



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/2/2025 12:25 PM, Timon Gehr wrote:

 ` live` however does not give you additional hard guarantees,


It gives the guarantee of a borrow checker - one mutable owner, and many 
immutable non-owners.

The other safety checks needed (like array overflow checks and escaping stack 
pointers) are already in the language.



 May 03



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 5/3/25 23:30, Walter Bright wrote:

 On 5/2/2025 12:25 PM, Timon Gehr wrote:

 ` live` however does not give you additional hard guarantees,


 It gives the guarantee of a borrow checker - one mutable owner, and many 
 immutable non-owners.
 ...


It does not.


 The other safety checks needed (like array overflow checks and escaping 
 stack pointers) are already in the language.


Whether a safety check is "needed" depends on what invariants the type 
system is able to establish and rely on in ` safe` code.



 May 03



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/3/2025 2:47 PM, Timon Gehr wrote:

 It does not.


See my other reply regarding higher level pointer constructs.



 Whether a safety check is "needed" depends on what invariants the type system
is 
 able to establish and rely on in ` safe` code.


If you can enumerate the other safety checks D needs to be memory safe, I'm 
listening.



 May 03



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 5/4/25 03:30, Walter Bright wrote:

 On 5/3/2025 2:47 PM, Timon Gehr wrote:

 It does not.


 
 See my other reply regarding higher level pointer constructs.
 ...


Well, let's continue that discussion in the other subthread.



 

 Whether a safety check is "needed" depends on what invariants the type 
 system is able to establish and rely on in ` safe` code.


 
 If you can enumerate the other safety checks D needs to be memory safe, 
 I'm listening.


E.g., null checks. x)

My point is that this is an _inductive invariant_. You have to state 
what the invariant is, then you must prove that all execution steps of 
the program maintain this invariant and in turn you can rely on the 
invariant holding when you show memory safety.

This is a _design problem_. You don't _need_ any compile-time checks to 
have a memory safe language, you just need them if there are 
performance/expressiveness considerations.

` safe` with DIP1000 and GC is fine in terms of memory safety, ignoring 
some bugs. It's just not very expressive.

` live` does not add additional expressiveness to memory safe D, there 
is no new ` safe` code that it enables. This is because it does not 
actually expand the type system invariant in a way that could then be 
relied upon.



 May 03



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


The point of the borrow checker is to ensure there is exactly one point of 
origin for a pointer and exactly one point of termination for it.

In other words, one allocation and one free. More than one free is a memory 
safety bug, continuing to use a pointer after it is free'd is a memory safety 
bug, and never freeing a pointer is not a memory safety bug, but is still a bug.

I also like it for aesthetic purposes, as it makes code more readable.



 May 04



  parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz>  writes:


On 05/05/2025 8:50 AM, Walter Bright wrote:

 The point of the borrow checker is to ensure there is exactly one point 
 of origin for a pointer and exactly one point of termination for it.


Hang on, that isn't the point of it at all.

A borrow checker has got the _side effect_ of only having one entry and 
exit point for an object.

But the _objective_ is to loosen the restrictions that an _ownership 
transfer system_ imposes on the type system whilst keeping its guarantees.

We do not have an ownership transfer system in D. Therefore we cannnot 
have a borrow checker.

What we do have is a liveliness analysis with guaranteed modelability, 
and we do have a use for that,  restrict.

Enforcing  restrict could be an incredibly useful tool to those who do 
data processing with simd, right now they are doing this by hand.



 May 04



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/4/2025 3:49 PM, Richard (Rikki) Andrew Cattermole wrote:

 On 05/05/2025 8:50 AM, Walter Bright wrote:

 The point of the borrow checker is to ensure there is exactly one point of 
 origin for a pointer and exactly one point of termination for it.


 
 Hang on, that isn't the point of it at all.
 
 A borrow checker has got the _side effect_ of only having one entry and exit 
 point for an object.
 
 But the _objective_ is to loosen the restrictions that an _ownership transfer 
 system_ imposes on the type system whilst keeping its guarantees.


I don't see how that would be workable.



 We do not have an ownership transfer system in D.


Yes we do, in  live code. That's exactly what it does. A pointer assignment to 
another pointer transfers the ownership to the lvalue, and the rvalue becomes 
"dead".



 herefore we cannnot have a 
 borrow checker.


The implementation shows it works.



 What we do have is a liveliness analysis with guaranteed modelability, and we
do 
 have a use for that,  restrict.
 
 Enforcing  restrict could be an incredibly useful tool to those who do data 
 processing with simd, right now they are doing this by hand.


Restricted pointers in C are guaranteed only by the user, the compiler does no 
checking.

You cannot vet restricted pointers without a borrow checker.



 May 09



  parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz>  writes:


On 10/05/2025 4:21 PM, Walter Bright wrote:

 On 5/4/2025 3:49 PM, Richard (Rikki) Andrew Cattermole wrote:

 On 05/05/2025 8:50 AM, Walter Bright wrote:

 The point of the borrow checker is to ensure there is exactly one 
 point of origin for a pointer and exactly one point of termination 
 for it.


 Hang on, that isn't the point of it at all.

 A borrow checker has got the _side effect_ of only having one entry 
 and exit point for an object.

 But the _objective_ is to loosen the restrictions that an _ownership 
 transfer system_ imposes on the type system whilst keeping its 
 guarantees.


 
 I don't see how that would be workable.


Okay that is interesting.


 We do not have an ownership transfer system in D.


 
 Yes we do, in  live code. That's exactly what it does. A pointer 
 assignment to another pointer transfers the ownership to the lvalue, and 
 the rvalue becomes "dead".


That is an owner state in the DFA.

Ownership transfer system operates at the type qualifier level and 
applies to a variable regardless of what function it is in.

See isolated from Midori, which was designed with D's type system in mind.

Currently people fake having one, by using a struct with a copy 
constructor that resets the old value.


 herefore we cannnot have a borrow checker.


 
 The implementation shows it works.
 
 

 What we do have is a liveliness analysis with guaranteed modelability, 
 and we do have a use for that,  restrict.

 Enforcing  restrict could be an incredibly useful tool to those who do 
 data processing with simd, right now they are doing this by hand.


 
 Restricted pointers in C are guaranteed only by the user, the compiler 
 does no checking.
 
 You cannot vet restricted pointers without a borrow checker.


Exactly, so what I've been thinking is tying  live to  restrict, which 
would make it a very useful tool.

This is something it could shine at, given its current implementation 
details.

----------------------------------------------------------------------

I want to go back to an earlier example I did elsewhere, where I have 
this type qualifier ``unique``, that is for an ownership transfer system.

If you have an ownership transfer system without a borrow checker:

```d
void func(scope ref int*) {}

unique(int*) a = ...;
assert(a !is null);

unique(int*) b = a;
assert(a is null);
assert(b !is null);

func(b); // error
```

Very annoying, it is what we have today if we fake it with a struct, 
except people will use alias this... and no more compiler error.

When you have both an ownership transfer system with a borrow checker:

```d
void func(scope ref int*) {}

unique(int*) a = ...;
assert(a !is null);

unique(int*) b = a;
assert(a is null);
assert(b !is null);

func(b); // ok
```

You may notice that unique may implicitly be removed in the type system.
It is not the type systems job to care about it being removed.

It is the job of the borrow checker to enforce the guarantee the origin 
outliving the borrow.

This is what I mean by: "But the _objective_ is to loosen the 
restrictions that an _ownership  transfer system_ imposes on the type 
system whilst keeping its guarantees."

In practice it just means that the type system doesn't need to solve a 
problem that a DFA is going to later on.



 May 09



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/9/2025 9:40 PM, Richard (Rikki) Andrew Cattermole wrote:


 ```d
 void func(scope ref int*) {}
 
 unique(int*) a = ...;
 assert(a !is null);
 
 unique(int*) b = a;
 assert(a is null);
 assert(b !is null);
 
 func(b); // ok
 ```


With a borrow checker like in  live, such asserts serve no purpose:
```
int* a = ...;
int* b = a;   // ownership transferred to b
*a = 3; // error, ownership was transferred out of a
```



 May 09



  parent  "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz>  writes:


On 10/05/2025 5:27 PM, Walter Bright wrote:

 On 5/9/2025 9:40 PM, Richard (Rikki) Andrew Cattermole wrote:
 

 ```d
 void func(scope ref int*) {}

 unique(int*) a = ...;
 assert(a !is null);

 unique(int*) b = a;
 assert(a is null);
 assert(b !is null);

 func(b); // ok
 ```


 
 With a borrow checker like in  live, such asserts serve no purpose:
 ```
 int* a = ...;
 int* b = a;   // ownership transferred to b
 *a = 3; // error, ownership was transferred out of a
 ```


Yes, they were there for demonstration purposes to show what the state is.



 May 09