www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Future of SafeRefCounted

reply Dukc <ajieskola gmail.com> writes:
`SafeRefCounted` depends on DIP1000 to work. Because of that, it 
was requested early in the PR review adding it that it should 
only be ` safe` when DIP1000 is on, not otherwise. This sounded 
obviously a good idea back then, so the destructor of 
SafeRefCounted has a `static if` that checks for DIP1000 and does 
the deallocation inside a ` trusted` lambda if the switch is 
enabled, directly otherwise.

This makes the destructor ` system` for non-DIP1000 client code, 
` safe` for DIP1000 code. But, I've come to think it'd have been 
better to leave it ` safe` for all code.

Have you looked at 
[std.dirEntries](https://dlang.org/phobos/std_file.html#dirEntries) recently?
It has a compile-time template parameter called `useDIP1000`, that is always
set automatically and is even checked with a static assert to be set to be
default option. Isn't this crazy? But believe or not, the unit tests don't pass
without the template parameter!

The issue is, the return value of `DirEntries` is implemented 
with SafeRefCounted. This means it, like `SafeRefCounted` itself, 
is safe iff DIP1000 is on. Without the `useDIP1000` flag, the 
linker can attempt linking to non-DIP1000 `dirEntries` from 
`dip1000` client code or vice-versa, causing an ABI mismatch 
since the safety of the function affects it's name mangling.

Imagine some average user trying to define their own type with 
`SafeRefCounted` and getting linker errors like that. When I 
encountered the linker errors, I knew that I had just made 
`SafeRefCounter` safety to depend on the DIP1000 flag, but it was 
still very tricky to figure out what was happening and how to 
solve it. We certainly can't expect the same from users.

Now, maybe this would let non-DIP1000 users to write unsafe code 
in ` safe`. I'm not actually sure if that's the case, but 
assuming it is I still think it would be the lesser evil here. 
Remember, ` safe` has pointer escaping holes in non-DIP1000 code 
anyway. I don't see how the possibility of the 
`std.typecons.borrow` argument function escaping the reference to 
the payload is any different from the possibility of leaking a 
slice of a static array in the stack.

----------------------------

Which leads to another issue: language modules and Phobos 3.

My understanding is the current plan with Phobos 3 is to not use 
DIP1000 in it. `SafeRefCounted` depends on DIP1000. Without it it 
isn't any better than the old `RefCounted` type.

Surprisingly, this doesn't automatically mean it can't be done in 
Phobos 3. Within the present language it wouldn't be possible, 
but modules are going to change the calculus.

In the future when DIP1000 is on or off per module as opposed to 
compilation unit, it is going to the client code which has to be 
DIP1000 for ref counting to be safe. The reference counted type 
can be implemented with our without DIP1000 just as well 
(although non-DIP1000 implementation of `borrow` would probably 
need a [currently 
missing](https://github.com/dlang/dmd/issues/20302) ability to 
manually inspect the scopeness of the argument function - 
currently the safety is inferred by calling the argument function 
with a `scope` argument which probably doesn't work outside 
DIP1000. Also ` safe`ty unit tests of it would need DIP1000.).

Modules are also one more reason to do away with inspecting for 
DIP1000. With modules, the current definition of the inspection
```D
private auto dip1000Test(int x) {return *&x;}
// Used in the `static if`
package(std) enum dip1000Enabled
     = is(typeof(&dip1000Test) : int function(int)  safe);
```
is going to inspect Phobos, not the client code. While we're 
probably going to have a way to inspect client code instead, IMO 
it is a code smell if you're inspecting the module / completion 
flags of your client code.

What are your thoughts?
Feb 22
next sibling parent Dukc <ajieskola gmail.com> writes:
On Saturday, 22 February 2025 at 19:05:49 UTC, Dukc wrote:

 But believe or not, the unit tests don't pass without the 
 template parameter!


Correction: Unit tests do pass, but the test suite as whole gives 
a linker error in apparently unrelated place.
Feb 22
prev sibling next sibling parent Dukc <ajieskola gmail.com> writes:
On Saturday, 22 February 2025 at 19:05:49 UTC, Dukc wrote:

 Which leads to another issue: language modules and Phobos 3.


Wow, how did I write about modules all the way down without 
noticing? I meant *editions* in all but one instance. So:

Which leads to another issue: language editions and Phobos 3.

My understanding is the current plan with Phobos 3 is to not
use DIP1000 in it. `SafeRefCounted` depends on DIP1000. Without
it it isn't any better than the old `RefCounted` type.

Surprisingly, this doesn't automatically mean it can't be done
in Phobos 3. Within the present language it wouldn't be
possible, but editions are going to change the calculus.

In the future when DIP1000 is on or off per module as opposed
to compilation unit, it is going to the client code which has
to be DIP1000 for ref counting to be safe. The reference
counted type can be implemented with our without DIP1000 just
as well (although non-DIP1000 implementation of `borrow` would
probably need a [currently
missing](https://github.com/dlang/dmd/issues/20302) ability to
manually inspect the scopeness of the argument function -
currently the safety is inferred by calling the argument
function with a `scope` argument which probably doesn't work
outside DIP1000. Also ` safe`ty unit tests of it would need
DIP1000.).

Editions are also one more reason to do away with inspecting for
DIP1000. With editions, the current definition of the inspection
```D
private auto dip1000Test(int x) {return *&x;}
// Used in the `static if`
package(std) enum dip1000Enabled
     = is(typeof(&dip1000Test) : int function(int)  safe);
```
is going to inspect Phobos, not the client code. While we're
probably going to have a way to inspect client code instead,
IMO it is a code smell if you're inspecting the edition /
completion flags of your client code.

What are your thoughts?
Feb 23
prev sibling next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 22 February 2025 at 19:05:49 UTC, Dukc wrote:

 This sounded obviously a good idea back then, so the destructor 
 of
 SafeRefCounted has a `static if` that checks for DIP1000 and
 does the deallocation inside a ` trusted` lambda if the switch
 is enabled, directly otherwise.
 (...)
 What are your thoughts?


To me it always looked hacky, and I'd get rid of it as well.

On the other hand, I think reference counting sucks in general, 
and see little value in using it in Phobos. Especially in 
functions like dirEntires, as if that is going to be called a lot 
in performance critical real-time code.

If you want to get rid of the GC dependency, you can use arena 
allocation. ([I talked about this last 
DConf](https://dconf.org/2024/index.html#dennisk))
Feb 23
next sibling parent reply Dukc <ajieskola gmail.com> writes:
On Sunday, 23 February 2025 at 20:02:03 UTC, Dennis wrote:

 On the other hand, I think reference counting sucks in general, 
 and see little value in using it in Phobos. Especially in 
 functions like dirEntires, as if that is going to be called a 
 lot in performance critical real-time code.


I disagree, reference counting is a great tool for this use case. 
Not because of the memory, but because if the directory handles. 
Were the `dirEntries` implementation garbage collected, the 
directory handles would get released only when the collection 
happens, which could never happen if the program doesn't use much 
RAM.

Admittedly this kind of reference counting doesn't mandate using 
`SafeRefCounted` - the reference counter could just free the 
directory handles and leave the RAM management to the GC, and 
this would be totally ` safe`.

When I think of it, maybe `DirEntries` should have the 
`SafeRefCounted` field as an untyped `void[x]` field instead and 
have a manually written ` trusted` destructor to call the 
`SafeRefCounted` destructor, so the ABI of 
`dirEntries`/`DirEntries` isn't affected. Ironically this means 
there's no benefit anymore from using `SafeRefCounted` over the 
old `RefCounted`.
Feb 24
parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 24 February 2025 at 17:18:55 UTC, Dukc wrote:

 On Sunday, 23 February 2025 at 20:02:03 UTC, Dennis wrote:

 On the other hand, I think reference counting sucks in 
 general, and see little value in using it in Phobos. 
 Especially in functions like dirEntires, as if that is going 
 to be called a lot in performance critical real-time code.


 I disagree, reference counting is a great tool for this use 
 case. Not because of the memory, but because if the directory 
 handles. Were the `dirEntries` implementation garbage 
 collected, the directory handles would get released only when 
 the collection happens, which could never happen if the program 
 doesn't use much RAM.


For resource management with deterministic lifecycle, I usually 
prefer non-copyable structs with refs. Only in the rare case when 
they need to be accessible in multiple parts of the code would I 
resort to ref counting.
Feb 24
prev sibling parent Adam Wilson <flyboynw gmail.com> writes:
On Sunday, 23 February 2025 at 20:02:03 UTC, Dennis wrote:

 If you want to get rid of the GC dependency, you can use arena 
 allocation. ([I talked about this last 
 DConf](https://dconf.org/2024/index.html#dennisk))


In general I approve of using arena allocators. RC is useful, but 
I'd like to avoid RC as much as possible in Phobos. We have a GC, 
and we're getting a vastly improved one. For cases where the GC 
cannot be used, almost anything is better than RC.
Feb 25
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, February 22, 2025 12:05:49 PM MST Dukc via Digitalmars-d wrote:

 What are your thoughts?


Personally, I'm very much hoping that DIP 1000 gets killed off. There are
some niche cases where it's useful, but in general, it's just too
disruptive. In order to support it in general, code that doesn't care at all
about it has to be marked with scope in order for the code that does care
about it to be able to use that code. And the reason that the current plan
is for Phobos v3 to not do anything with DIP 1000 is because a number of us
feel that way. Atila is looking to see if he can come up with a way to make
it work without being so disruptive, but I question that that's going to
work, whereas others like Rikki are looking to do something else entirely
which will theoretically cover more than DIP 1000 does. So, I don't know
what we're going to end up with, but realistically, if a type requires DIP
1000 in order to be  safe, then it's likely not going to be  safe in Phobos
v3.

As for RefCounted specifically, I haven't looked at its implementation
recently, so I don't know why it can't be  safe, but off the top of my head,
I don't see why it can't be possible to create something like RefCounted
that is  safe. Maybe some part of what the current design does would have to
be disallowed, but it sure seems like it should be possible to have
reference counting work with  safe so long as you don't allow references to
the internals to escape. And if for some reason, it can't be, well, then
either the code using it will need to wrap it in a way that allows for
 safe, or it can't be  safe.

And for the vast majority of code, nothing like RefCounted is needed. The GC
largely solves the problem with regards to memory management, and the cases
where you want or need to do something else will probably want a solution
that's tied to whatever allocators are being used rather than a more general
ref-counting solution (assuming that reference counting is wanted at all).
It's mostly just system resources that need to be deterministically freed /
destroyed where reference counting becomes necessary. So, IMHO, we do need a
solution for reference counting in Phobos v3, but it's also niche enough
that I don't think that it's the end of the world if some  trusted is
required for it - much as I agree that it would ideally be  safe. But again,
I'd have to sit down and work through the problem to say for sure whether it
can be  safe without DIP 1000. Either way, as things stand, DIP 1000 is not
going to be a solution for making anything in Phobos v3  safe.

- Jonathan M Davis
Feb 24
next sibling parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 25/02/2025 1:17 PM, Jonathan M Davis wrote:

 On Saturday, February 22, 2025 12:05:49 PM MST Dukc via Digitalmars-d wrote:
 
 
     What are your thoughts?
 
 
 Personally, I'm very much hoping that DIP 1000 gets killed off. There 
 are some niche cases where it's useful, but in general, it's just too 
 disruptive. In order to support it in general, code that doesn't care at 
 all about it has to be marked with scope in order for the code that does 
 care about it to be able to use that code. And the reason that the 
 current plan is for Phobos v3 to not do anything with DIP 1000 is 
 because a number of us feel that way. Atila is looking to see if he can 
 come up with a way to make it work without being so disruptive, but I 
 question that that's going to work, whereas others like Rikki are 
 looking to do something else entirely which will theoretically cover 
 more than DIP 1000 does. So, I don't know what we're going to end up 
 with, but realistically, if a type requires DIP 1000 in order to be 
  safe, then it's likely not going to be  safe in Phobos v3.


I'm exploring it from both directions.

Trying to model more things (without doing whole program analysis), and 
have the default be "to give up" when it cannot analyze what it does or 
doesn't understand.

Basically get the best of both worlds by having a fast and a slow DFA 
engine.

It'll mean that PhobosV3 would have DFA language features, with some 
guarantees. There may even be whole modules that support the slow DFA, 
but we'll see about it as a stretch goal.
Feb 24
prev sibling next sibling parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Tuesday, 25 February 2025 at 00:17:19 UTC, Jonathan M Davis 
wrote:

 On Saturday, February 22, 2025 12:05:49 PM MST Dukc via 
 Digitalmars-d wrote:

 What are your thoughts?


 Personally, I'm very much hoping that DIP 1000 gets killed off. 
 There are some niche cases where it's useful [...]


I use dip1000 in two ways, firstly to avoid allocations on the 
heap and secondly to safely share references to objects with 
limited lifetime. Often both at the same time.

Avoiding allocations (GC or malloc) allows the program to run 
faster with fewer resources. I personally value efficient and 
safe programs, and IMO it should be a core focus of the language.

I do see challenges with dip1000's usage, but relegating it by 
saying its only useful for niche cases neglects a whole set of 
efficient programs. I understand not everyone cares about them to 
the same extent, but don't forget there is a chicken-and-egg 
problem as well: if it's relatively hard to write efficient 
programs, you'll tend to filter out the people who care about 
them.


 Either way, as things stand, DIP 1000 is not going to be a 
 solution for making anything in Phobos v3  safe.


That is a real shame. That leaves implementing them either 
 system, inefficiently or not at all.
Feb 25
prev sibling parent reply Dukc <ajieskola gmail.com> writes:
On Tuesday, 25 February 2025 at 00:17:19 UTC, Jonathan M Davis 
wrote:

 As for RefCounted specifically, I haven't looked at its 
 implementation recently, so I don't know why it can't be  safe, 
 but off the top of my head, I don't see why it can't be 
 possible to create something like RefCounted that is  safe.


I had another look at it, and my conclusion is, it actually 
doesn't require DIP1000. DIP25 is enough! If you don't recall, 
and if I understand it correctly DIP25 is essentially DIP1000 
limited only to `ref` and `return ref`. `scope`/`return scope` 
rules for pointers/slices/classes/struct/unions aren't included. 
It's already the default in the current language.

Now, it's still possible to escape the payload with the same 
tricks you could escape a pointer to stack in default D. But this 
will also be fixed if we implement Simple Safe D. DIP1000 is not 
strictly needed for this use case. Although in Simple Safe D the 
user will likely have to make temporary copies of the payload, or 
parts of it, that would be unneeded in present D or with DIP1000 
- just like when dealing with objects on the stack.
Feb 25
parent reply Derek Fawcus <dfawcus+dlang employees.org> writes:
On Tuesday, 25 February 2025 at 11:46:44 UTC, Dukc wrote:

 If you don't recall, and if I understand it correctly DIP25 is 
 essentially DIP1000 limited only to `ref` and `return ref`. 
 `scope`/`return scope` rules for 
 pointers/slices/classes/struct/unions aren't included. It's 
 already the default in the current language.



to have something akin to scoped pointers?

See the postscript in the below article, where this was added as 

checking", and preventing escapes.

https://em-tg.github.io/csborrow/

That brought to mind this discussion, and hence wondering if 
there are any answers there to be "stolen", since to my ignorant 
first glance it seems similar to what D is groping towards with 
DIP 1000.
Mar 05
next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Wednesday, 5 March 2025 at 22:04:24 UTC, Derek Fawcus wrote:

 That brought to mind this discussion, and hence wondering if 
 there are any answers there to be "stolen", since to my 
 ignorant first glance it seems similar to what D is groping 
 towards with DIP 1000.


Yes indeed. This goes to show dip1000's aim is the right one, 
just needs more convenient usage.

Thanks for sharing.
Mar 05
parent Derek Fawcus <dfawcus+dlang employees.org> writes:
On Wednesday, 5 March 2025 at 22:27:32 UTC, Sebastiaan Koppe 
wrote:

 On Wednesday, 5 March 2025 at 22:04:24 UTC, Derek Fawcus wrote:

 That brought to mind this discussion, and hence wondering if 
 there are any answers there to be "stolen", since to my 
 ignorant first glance it seems similar to what D is groping 
 towards with DIP 1000.


 Yes indeed. This goes to show dip1000's aim is the right one, 
 just needs more convenient usage.

 Thanks for sharing.


Here is another article which touches on the same areas, and may 
be of interest:

https://btmc.substack.com/p/borrow-checking-from-scratch
Mar 06
prev sibling parent Dukc <ajieskola gmail.com> writes:
On Wednesday, 5 March 2025 at 22:04:24 UTC, Derek Fawcus wrote:

 On Tuesday, 25 February 2025 at 11:46:44 UTC, Dukc wrote:

 If you don't recall, and if I understand it correctly DIP25 is 
 essentially DIP1000 limited only to `ref` and `return ref`. 
 `scope`/`return scope` rules for 
 pointers/slices/classes/struct/unions aren't included. It's 
 already the default in the current language.



 seems to have something akin to scoped pointers?

 See the postscript in the below article, where this was added 

 checking", and preventing escapes.

 https://em-tg.github.io/csborrow/


DIP25 is pretty similar to this, except it can also specify 
non-returning references.

DIP1000 is more advanced, though. It works not only with 
`ref`erences, but also with pointers, arrays, structs and so on. 
Meaning you can pass a slice of a static array on the stack to a 
function, and it will be checked for escaping just like a `ref` 

equivalent neither can.

Another thing DIP1000 enables over DIP25 is a `ref`erence to a 
scoped pointer (or slice, or struct, or class), meaning you can 
have a function that mutates a pointer to a local variable _in 
place_ . From [my post in the official 
blog](https://dlang.org/blog/2022/10/08/dip1000-memory-safety-in-a-modern-systems-programming-language-part-2/):

```D
 safe ref Exception nullify(return ref scope Exception obj)
{
     obj = null;
     return obj;
}

 safe unittest
{
     scope obj = new Exception("Error!");
     assert(obj.msg == "Error!");
     obj.nullify;
     assert(obj is null);
     // Since nullify returns by ref, we can assign
     // to it's return value.
     obj.nullify = new Exception("Fail!");
     assert(obj.msg == "Fail!");
}
```

. The exception `obj` points to in the unit test is allocated on 
the stack because of `scope`, yet `nullify` can mutate the class 
reference in place (as opposed to only the class itself). You 
can't do this safely with DIP25 alone, since you can't have a 
`ref` to another `ref`.
Mar 06