www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Object.toString, toHash, opCmp, opEquals

reply Walter Bright <newshound2 digitalmars.com> writes:
The prototypes are:

```
string toString();
size_t toHash()  trusted nothrow;
int opCmp(Object o);
bool opEquals(Object o);
```

which long predated `const`. The trouble is, they should be:

```
string toString() const;
size_t toHash() const  trusted nothrow;
int opCmp(const Object o) const;
bool opEquals(const Object o) const;
```

Without the `const` annotations, the functions are not usable by `const`
objects 
without doing an unsafe cast. This impairs anyone wanting to write
const-correct 
code, and also impedes use of ` live` functions.

I recommend that everyone who has overloads of these functions, alter them to 
have the `const` signatures. This will future-proof them against any changes to 
Object's signatures.
Apr 25
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 4/26/24 01:06, Walter Bright wrote:

 The prototypes are:
 
 ```
 string toString();
 size_t toHash()  trusted nothrow;
 int opCmp(Object o);
 bool opEquals(Object o);
 ```
 ...


Beautiful. If I could change anything, I would remove ` trusted nothrow` 
from `toHash`. Or just delete all the functions.


 which long predated `const`. The trouble is, they should be:
 
 ```
 string toString() const;
 size_t toHash() const  trusted nothrow;
 int opCmp(const Object o) const;
 bool opEquals(const Object o) const;
 ```
 ...


No, please.


 Without the `const` annotations, the functions are not usable by `const` 
 objects without doing an unsafe cast. This impairs anyone wanting to 
 write const-correct code,


"const correctness" does not work in D because const

a) provides actual guarantees
b) is transitive

It is fundamentally incompatible with many common patters of 
object-oriented and other state abstraction. It is not even compatible 
with the range API. Uses of `const` are niche. `const` is nice when it 
does work, but it's not something you can impose on all code, 
particularly object-oriented code.


 and also impedes use of ` live` functions.
 ...


Perfect. I have no intention of using ` live` functions. I do not see 
their utility. It would be good to reuse the dataflow analysis you 
implemented for ` live` in some productive way though.


 I recommend that everyone who has overloads of these functions, alter 
 them to have the `const` signatures. This will future-proof them against 
 any changes to Object's signatures.


I will not do that, because if it does not outright break my code (e.g. 
because Phobos cannot support `const` ranges), it actually limits my 
options in the future in a way that is entirely unnecessary.

This is a non-starter. We need another solution.
Apr 25
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/25/2024 4:36 PM, Timon Gehr wrote:

 Without the `const` annotations, the functions are not usable by `const` 
 objects without doing an unsafe cast. This impairs anyone wanting to write 
 const-correct code,


 
 "const correctness" does not work in D because const
 
 a) provides actual guarantees
 b) is transitive


It's not the C++ notion of const, sure. But the name still applies.


 It is fundamentally incompatible with many common patters of object-oriented
and 
 other state abstraction. It is not even compatible with the range API. Uses of 
 `const` are niche. `const` is nice when it does work, but it's not something
you 
 can impose on all code, particularly object-oriented code.


Why would anyone, for example, try to mutate a range when it is passed to one
of 
these functions?



 and also impedes use of ` live` functions.
 ...


 
 Perfect. I have no intention of using ` live` functions. I do not see their 
 utility.


The utility is being able to write borrow-checker style code, so you can avoid 
things like double frees.

As I recall, it was you that pointed out that reference counting can never be 
safe if two mutable pointers to the same ref counted object (one to the object, 
the other to its interior) were passed to a function. (Freeing the first can 
leave the second interior pointer pointing to a deleted object.) The entire ref 
counting scheme capsized because of this.


 I recommend that everyone who has overloads of these functions, alter them to 
 have the `const` signatures. This will future-proof them against any changes 
 to Object's signatures.


 
 I will not do that, because if it does not outright break my code (e.g.
because 
 Phobos cannot support `const` ranges), it actually limits my options in the 
 future in a way that is entirely unnecessary.


Why would anyone need toHash(), toString(), opEquals() or opCmp() to mutate 
their data? Wouldn't that be quite surprising behavior?
Apr 25
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 4/26/24 02:57, Walter Bright wrote:

 On 4/25/2024 4:36 PM, Timon Gehr wrote:

 Without the `const` annotations, the functions are not usable by 
 `const` objects without doing an unsafe cast. This impairs anyone 
 wanting to write const-correct code,


 "const correctness" does not work in D because const

 a) provides actual guarantees
 b) is transitive


 
 It's not the C++ notion of const, sure. But the name still applies.
 ...


Well, we could come up with a better name, one that actually reflects 
that there are some pitfalls.



 It is fundamentally incompatible with many common patters of 
 object-oriented and other state abstraction. It is not even compatible 
 with the range API. Uses of `const` are niche. `const` is nice when it 
 does work, but it's not something you can impose on all code, 
 particularly object-oriented code.


 
 Why would anyone, for example, try to mutate a range when it is passed 
 to one of these functions?
 ...


A range is useless unless it is mutable. The range interface is 
inherently mutable. To iterate a range, you have to call `popFront()` on 
it. There is no way to have a `const popFront()`.


 

 and also impedes use of ` live` functions.
 ...


 Perfect. I have no intention of using ` live` functions. I do not see 
 their utility.


 
 The utility is being able to write borrow-checker style code, so you can 
 avoid things like double frees.
 ...


` live` does not enable this. Anyway, you are trying to impose 
nonsensical restrictions on garbage-collected code. I have yet to run 
into a double-free using GC allocation and I doubt ` live` would help me 
avoid that if it were a thing.


 As I recall, it was you that pointed out that reference counting can 
 never be safe if two mutable pointers to the same ref counted object 
 (one to the object, the other to its interior) were passed to a 
 function. (Freeing the first can leave the second interior pointer 
 pointing to a deleted object.) The entire ref counting scheme capsized 
 because of this.
 ...


I provided the counterexample, but the unsound generalization is yours. 
(Technically, there would be ways to type check that code without 
banning mutation outright.)


 I recommend that everyone who has overloads of these functions, alter 
 them to have the `const` signatures. This will future-proof them 
 against any changes to Object's signatures.


 I will not do that, because if it does not outright break my code 
 (e.g. because Phobos cannot support `const` ranges), it actually 
 limits my options in the future in a way that is entirely unnecessary.


 
 Why would anyone need toHash(), toString(), opEquals() or opCmp() to 
 mutate their data? Wouldn't that be quite surprising behavior?
 


As I keep pointing out, there is a difference between mutating abstract 
data and concrete memory locations. For instance, data types with 
amortized guarantees usually have to reorganize the internal data 
representation on each query. (Think e.g. splay trees.)

Anyway, let's for the sake of argument assume that I want to write 
functions that leave memory in exactly the state they encountered it in. 
Const will _still_ unduly restrict me because it is not fine-grained enough.

```d
import std.stdio, std.range, std.conv;

struct S{
     auto r=iota(1,2);
     string toString()const{ return text(r); }
}

void main(){
     S s;
     writeln(s);
}
```

Writes:
```d
const(Result)(1, 2)
```

Sometimes there is not even a safe workaround to get a mutable version 
of a range, because of transitive `const`. A range can have indirections 
in its implementation.

This is just one example establishing that `const` is not expressive 
enough to say _ONLY_ "this will not mutate anything". It also spells: 
"This code can be a huge pain in the ass at any point in the future for 
dumb, incidental reasons."

I really do not want to deal with this. I'd much rather fork Phobos so 
it uses non-const alternatives to toHash and toString.

If you expect people to prove properties to an incomplete type system 
via annotations and to accept unnecessary restrictions, they have to get 
some value out of it. You also would not go: "Starting from tomorrow, 
you have to prove to me that you brush your teeth every day. I want 
video evidence." And then, when I refuse, you can't say: "Why would you 
not brush your teeth?" This is what this is.

I caution you to now not miss the forest for the trees and engage in a 
"tooth-brushing related" argument (e.g., proposing a different range 
design or something like that). This is an inherent issue. Even if you 
make the type system more expressive, the annotation overhead is still 
real, and often uneconomical.

I am perfectly fine with having some restricted system like Rust for 
people who want to do safe manual memory management. This would even be 
useful to me. But this has to be opt-in, based on data structures, and 
interoperate as seamlessly as possible with the full language.

One thing I absolutely agree on with Robert is that it should always be 
_possible_ to write simple  safe D code without any advanced type system 
shenanigans. I think any design that strays from that principle is bad. 
This proposed change absolutely torpedoes that.
Apr 25
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/25/2024 6:32 PM, Timon Gehr wrote:

 A range is useless unless it is mutable. The range interface is inherently 
 mutable. To iterate a range, you have to call `popFront()` on it. There is no 
 way to have a `const popFront()`.


I agree there's no reason to have a const popFront(). But opEquals() is 
inherently non-mutable. Let's posit a mutating opEquals() and:

```
o.opEquals(o);
```

and the opEquals() mutated which one, or both, or what would happen if it did?



 The utility is being able to write borrow-checker style code, so you can avoid 
 things like double frees.
 ...


 
 ` live` does not enable this.


```
auto p = q;
free(p);
free(q);
```


 Anyway, you are trying to impose nonsensical 
 restrictions on garbage-collected code. I have yet to run into a double-free 
 using GC allocation and I doubt ` live` would help me avoid that if it were a 
 thing.


D doesn't distinguish between gc pointers and non-gc pointers. It has been 
proposed, but I have very extensive experience with multiple pointer types and 
it is a cure worse than the disease.



 As I recall, it was you that pointed out that reference counting can never be 
 safe if two mutable pointers to the same ref counted object (one to the 
 object, the other to its interior) were passed to a function. (Freeing the 
 first can leave the second interior pointer pointing to a deleted object.) The 
 entire ref counting scheme capsized because of this.


 I provided the counterexample, but the unsound generalization is yours.


All it takes is one counterexample to capsize it.



 (Technically, there would be ways to type check that code without banning 
 mutation outright.)


Neither Andrei nor I nor anyone else working on it could figure out a solution 
(other than disallowing all pointers to payload). The borrow checker does solve 
it, though.



 Why would anyone need toHash(), toString(), opEquals() or opCmp() to mutate 
 their data? Wouldn't that be quite surprising behavior?


 
 As I keep pointing out, there is a difference between mutating abstract data
and 
 concrete memory locations. For instance, data types with amortized guarantees 
 usually have to reorganize the internal data representation on each query. 
 (Think e.g. splay trees.)
 
 Anyway, let's for the sake of argument assume that I want to write functions 
 that leave memory in exactly the state they encountered it in. Const will 
 _still_ unduly restrict me because it is not fine-grained enough.
 
 ```d
 import std.stdio, std.range, std.conv;
 
 struct S{
      auto r=iota(1,2);
      string toString()const{ return text(r); }


I agree that mutates the argument passed to toString(). That would consume the 
range. Calling toString() again would return an empty string.



 Sometimes there is not even a safe workaround to get a mutable version of a 
 range, because of transitive `const`. A range can have indirections in its 
 implementation.
 This is just one example establishing that `const` is not expressive enough to 
 say _ONLY_ "this will not mutate anything". It also spells: "This code can be
a 
 huge pain in the ass at any point in the future for dumb, incidental reasons."
 
 I really do not want to deal with this. I'd much rather fork Phobos so it uses 
 non-const alternatives to toHash and toString.


I suppose it wouldn't help if I suggest:

```
writeln(text(r));
```

I only proposed the const toString() for Object.toString(), not for struct, 
where indeed you are free to have struct toString() do anything you want.

Class and struct are fundamentally different in that class is a universal 
hierarchy with a common root, and hence we must define what that common root
is. 
Struct, on the other hand, is rootless, and hence the user can define it
however 
he pleases.

I agree with you that Object shouldn't have had any members, and Andrei and I 
did discuss that, but since it had members, we couldn't really take them away. 
Note that COM classes also have a common root with one member QueryInterface().



 If you expect people to prove properties to an incomplete type system via 
 annotations and to accept unnecessary restrictions, they have to get some
value 
 out of it. You also would not go: "Starting from tomorrow, you have to prove
to 
 me that you brush your teeth every day. I want video evidence." And then, when
I 
 refuse, you can't say: "Why would you not brush your teeth?" This is what this
is.
 
 I caution you to now not miss the forest for the trees and engage in a 
 "tooth-brushing related" argument (e.g., proposing a different range design or 
 something like that). This is an inherent issue. Even if you make the type 
 system more expressive, the annotation overhead is still real, and often 
 uneconomical.
 
 I am perfectly fine with having some restricted system like Rust for people
who 
 want to do safe manual memory management. This would even be useful to me. But 
 this has to be opt-in, based on data structures, and interoperate as
seamlessly 
 as possible with the full language.



I think I see your point of view. Mine is a little different. I have 
considerable experience with C. When I see:

```
int foo(T* p);
```

Is p an array? is foo() going to mutate what it points to? Is foo() going to 
free() it? How would I know without reading the implementation? (The 
documentation is always incomplete, wrong, or missing.) Annotations give me 
confidence that I understand what it does. const/ref/scope here answer my 
questions, and the compiler backs it up.



 One thing I absolutely agree on with Robert is that it should always be
 _possible_ to write simple  safe D code without any advanced type system
 shenanigans. I think any design that strays from that principle is bad. This
 proposed change absolutely torpedoes that.


I agree with Robert, too. I asked him to prepare a list of his proposals so I 
can see what can be done.

P.S. const class Objects are more or less unusable with the non-const toString, 
toHash, opCmp and opEquals.

P.P.S. all of D's annotations are subtractive. This means you can write code 
without annotations and it'll work. But safe, probably not.

P.P.P.S. I almost never write a multiple free bug these days. But that doesn't 
translate to "don't need double free protection", as I spent many years making 
that mistake and tracking them down. I even wrote my own malloc/free debugger
to 
help. Eventually, I simply internalized what not to do. But that isn't a 
transferable skill. I can't even explain what I do.

Anyhow, thanks for the food for thought!
Apr 25
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 4/26/24 05:13, Walter Bright wrote:

 On 4/25/2024 6:32 PM, Timon Gehr wrote:

 A range is useless unless it is mutable. The range interface is 
 inherently mutable. To iterate a range, you have to call `popFront()` 
 on it. There is no way to have a `const popFront()`.


 
 I agree there's no reason to have a const popFront(). But opEquals() is 
 inherently non-mutable.


That does not mean it can be D `const`. This is one of the two reasons I 
mentioned why "const correctness" is such a damaging concept for a D 
programmer.

Here, you are again conflating the logical with the physical semantics. 
It's a bit like saying "`opEquals` changes the state of the stack, hence 
obviously it cannot be `const`!", just one abstraction level higher.

Ideally, `opEquals` implements an equivalence relation. It is fine if it 
changes the representatives in the process, as long as it properly 
encapsulates the internal state such that whenever two values compare 
equal, the observable semantics of the two representatives is the same.


 Let's posit a mutating opEquals() and:
 
 ```
 o.opEquals(o);
 ```
 
 and the opEquals() mutated which one, or both, or what would happen if 
 it did?
 ...


"If you stop brushing your teeth, you might get cavities! There is hence 
no reason not to record video evidence!"

Anyway, here is a simple contrived example of a mutating `opEquals` that 
is not a logical problem:

```d
struct int31{
     private int payload;
     bool opEquals(int31 rhs){
         payload^=1;
         return (payload>>1)==(rhs.payload>>1);
     }
     void opBinary(string op:"+")(int31 rhs){
         return ((payload>>1)+(rhs>>1))<<1;
     }
}
```

If you want something that is actually useful, you will have to look 
into splay trees or something like that. Or e.g., maybe you have a ring 
buffer or something that compacts itself on iteration. As I said, 
amortized data structures. It may be incorrect to have a const opEquals. 
It can introduce a performance regression.


 

 The utility is being able to write borrow-checker style code, so you 
 can avoid things like double frees.
 ...


 ` live` does not enable this.


 
 ```
 auto p = q;
 free(p);
 free(q);
 ```
 ...


Well, I can just not use `malloc` and `free`. Anyway, to me this is not 
"borrow-checker style" code. This is C-style ` system` code.


 Anyway, you are trying to impose nonsensical restrictions on 
 garbage-collected code. I have yet to run into a double-free using GC 
 allocation and I doubt ` live` would help me avoid that if it were a 
 thing.


 
 D doesn't distinguish between gc pointers and non-gc pointers. It has 
 been proposed, but I have very extensive experience with multiple 
 pointer types and it is a cure worse than the disease.
 ...


I understand that there exist bad solutions to basically any problem. 
This very thread provides ample evidence of that fact. We have `scope` 
and non-`scope` pointers and the world has not ended yet.


 

 As I recall, it was you that pointed out that reference counting can 
 never be safe if two mutable pointers to the same ref counted object 
 (one to the object, the other to its interior) were passed to a 
 function. (Freeing the first can leave the second interior pointer 
 pointing to a deleted object.) The entire ref counting scheme 
 capsized because of this.


 I provided the counterexample, but the unsound generalization is yours.


 
 All it takes is one counterexample to capsize it.
 ...


Sure, I was just objecting to the characterization that I claimed a 
"Rust-style" mutation-restricting solution is the only possible one.


 

 (Technically, there would be ways to type check that code without 
 banning mutation outright.)


 
 Neither Andrei nor I nor anyone else working on it could figure out a 
 solution (other than disallowing all pointers to payload).


This is not true, it seems they just did not explain it to you. You 
could have some sort of more precise type-state system that only 
disallows operations that may deallocate the payload. This is the kind 
of thing that Rust initially explored. Anyway, I am not even saying that 
this is necessarily better, I just don't like technically wrong words 
being put into my mouth. ;)


 The borrow checker does solve it, though.
 ...


It does not, because it does not actually get aliasing under control. It 
adds checks that are incomplete in some programs, and unnecessary in 
other programs.


 

 Why would anyone need toHash(), toString(), opEquals() or opCmp() to 
 mutate their data? Wouldn't that be quite surprising behavior?


 As I keep pointing out, there is a difference between mutating 
 abstract data and concrete memory locations. For instance, data types 
 with amortized guarantees usually have to reorganize the internal data 
 representation on each query. (Think e.g. splay trees.)

 Anyway, let's for the sake of argument assume that I want to write 
 functions that leave memory in exactly the state they encountered it 
 in. Const will _still_ unduly restrict me because it is not 
 fine-grained enough.

 ```d
 import std.stdio, std.range, std.conv;

 struct S{
      auto r=iota(1,2);
      string toString()const{ return text(r); }


 
 I agree that mutates the argument passed to toString(). That would 
 consume the range. Calling toString() again would return an empty string.
 ...


No, this is not true. `text` does not accept its argument by `ref`. The 
range stays intact. This is similar to how in:

```
int[] a = [1,2,3];
writeln(a);
```

The array `a` is not empty after printing.


 

 Sometimes there is not even a safe workaround to get a mutable version 
 of a range, because of transitive `const`. A range can have 
 indirections in its implementation.
 This is just one example establishing that `const` is not expressive 
 enough to say _ONLY_ "this will not mutate anything". It also spells: 
 "This code can be a huge pain in the ass at any point in the future 
 for dumb, incidental reasons."

 I really do not want to deal with this. I'd much rather fork Phobos so 
 it uses non-const alternatives to toHash and toString.


 
 I suppose it wouldn't help if I suggest:
 
 ```
 writeln(text(r));
 ```
 ...


No, it does not. I do not see how this would help.


 I only proposed the const toString() for Object.toString(), not for 
 struct, where indeed you are free to have struct toString() do anything 
 you want.
 ...


I happen to be already using classes. Forking Phobos is less effort than 
moving to structs. Or I could just switch to OpenD I guess.


 Class and struct are fundamentally different in that class is a 
 universal hierarchy with a common root, and hence we must define what 
 that common root is. Struct, on the other hand, is rootless, and hence 
 the user can define it however he pleases.
 
 I agree with you that Object shouldn't have had any members, and Andrei 
 and I did discuss that, but since it had members, we couldn't really 
 take them away. Note that COM classes also have a common root with one 
 member QueryInterface().
 ...


I am amazed that you want to break most D code by imposing attributes on 
common root functions, but removing functions from the common root is a 
bridge too far even though the fix is usually simply to remove `override`.


 

 If you expect people to prove properties to an incomplete type system 
 via annotations and to accept unnecessary restrictions, they have to 
 get some value out of it. You also would not go: "Starting from 
 tomorrow, you have to prove to me that you brush your teeth every day. 
 I want video evidence." And then, when I refuse, you can't say: "Why 
 would you not brush your teeth?" This is what this is.

 I caution you to now not miss the forest for the trees and engage in a 
 "tooth-brushing related" argument (e.g., proposing a different range 
 design or something like that). This is an inherent issue. Even if you 
 make the type system more expressive, the annotation overhead is still 
 real, and often uneconomical.

 I am perfectly fine with having some restricted system like Rust for 
 people who want to do safe manual memory management. This would even 
 be useful to me. But this has to be opt-in, based on data structures, 
 and interoperate as seamlessly as possible with the full language.


 
 
 I think I see your point of view. Mine is a little different.


My point of view is D-focused, yours often enough seems to be C-focused. 
There is only so much insights about D's design that can be extracted 
from issues with C's design. Actual experience with D is increasingly 
important.

You will notice that all of the experience you mention in this thread is 
with systems that do not work well. I have considerable experience with 
D, and the only memory-management related issue that I care about is use 
after free. Yet ` live` does not solve this problem for me. (I am aware 
that you can write a snippet of code that is rejected by  live for use 
after free. Personally I care about code that is accepted and hence is 
guaranteed not to have use after free.)


 I have considerable experience with C. When I see:
 
 ```
 int foo(T* p);
 ```
 
 Is p an array? is foo() going to mutate what it points to? Is foo() 
 going to free() it?


I agree with this point of view, this is not what I am objecting to. 
This is a "tooth-brushing related" argument.

Anyway, this is the C point of view. OTOH, in  safe D, `p` cannot be 
`free`d. It may e.g. be a GC pointer.

If you want to allow an ` safe foo` to free its argument, you will have 
to encode in the type of that argument that it is a malloc'd pointer. 
There is just no way around that unless you say "in this language, every 
non-scope pointer comes from malloc". That would be a bad outcome.

The best way to do such an encoding is to have a struct wrapper around 
the pointer, have proper move semantics and a borrow checker that works 
well, and soundly, with data abstraction. In this case, the borrow 
checker actually makes a difference in ` safe` code. Otherwise it does not.


 How would I know without reading the implementation? 
 (The documentation is always incomplete, wrong, or missing.) Annotations 
 give me confidence that I understand what it does. const/ref/scope here 
 answer my questions, and the compiler backs it up.
 ...


Your considerable experience with C contradicts your extensive 
experience with "multiple pointer types" and D's actual, existing 
DIP1000 and `const` design. I implore you to refine your position, 
otherwise it is simply internally inconsistent and hence allows you to 
dismiss any argument. This is very frustrating for an interlocutor.

Anyway, I agree that `const` and `scope` can be very useful in cases 
where they work. They are just not a panacea.


 
  > One thing I absolutely agree on with Robert is that it should always be
  > _possible_ to write simple  safe D code without any advanced type system
  > shenanigans. I think any design that strays from that principle is 
 bad. This
  > proposed change absolutely torpedoes that.
 
 I agree with Robert, too. I asked him to prepare a list of his proposals 
 so I can see what can be done.
 ...


One concrete thing that can be done is to change course here. If you 
want to do a breaking change, do one that causes less pain and does not 
make D code more complicated by default.


 P.S. const class Objects are more or less unusable with the non-const 
 toString, toHash, opCmp and opEquals.
 ...


`const` class Objects are more or less unusable full stop. You can't 
even have a tail-const class reference.

Yet `const` class Objects are exactly what this proposal is trying to 
impose on unsuspecting D programmers. It just does not work.


 P.P.S. all of D's annotations are subtractive. This means you can write 
 code without annotations and it'll work.


That's great, but it will sometimes not interoperate with code that has 
annotations, as in this case. Hence if you start imposing annotations on 
code, you lose this property. This would be a significant loss for the 
approachability of D, particularly as a first language. Furthermore, it 
is also a slap in the face to experienced D developers that have come to 
understand the limitations and proper applications of D's annotations.


 But safe, probably not.
 ...


I do not understand. Do you agree with Robert or not?

A big strength of D is that you can start out prototyping stuff with the 
GC without unnecessary annotation overhead and then often it will be 
good enough. If it is not, you can then explore different memory 
management options, surgically for the parts of the program state where 
that actually makes a difference. Only at this point is it then okay to 
expect people to annotate things if they want checked safety.


 P.P.P.S. I almost never write a multiple free bug these days. But that 
 doesn't translate to "don't need double free protection", as I spent 
 many years making that mistake and tracking them down. I even wrote my 
 own malloc/free debugger to help. Eventually, I simply internalized what 
 not to do. But that isn't a transferable skill. I can't even explain 
 what I do.
 ...


As I said many times, if you want ` live` to be a linter to avoid manual 
memory management bugs in ` system/ trusted` functions that avoid proper 
data abstraction with constructors and destructors, that is fine. But 
you cannot hold this position and at the same time turn around and claim 
it does anything for ` safe` reference counting. It just does not. A 
more careful approach is needed.


 Anyhow, thanks for the food for thought!
 


My pleasure! Here is some more: Why did you not propose to add `pure` to 
the signatures? How about ` nogc`? `nothrow`? ` safe`? Why is `toHash` 
` trusted nothrow`, but not other functions?
Apr 26
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 4/26/24 15:27, Timon Gehr wrote:

 The borrow checker does solve it, though.
 ...


 
 It does not, because it does not actually get aliasing under control. It 
 adds checks that are incomplete


(And also insufficient.)


 in some programs, and unnecessary in other programs.


(I assumed you were talking about the ` live` borrow checker.)
Apr 26
parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 27/04/2024 1:33 AM, Timon Gehr wrote:

 On 4/26/24 15:27, Timon Gehr wrote:
 
         The borrow checker does solve it, though. ...
 
     It does not, because it does not actually get aliasing under
     control. It adds checks that are incomplete
 
 (And also insufficient.)


Yes, this is something I've been trying to explain (highly 
unsuccessfully I might add) from pretty much day 1 of  live.

For a borrow checker to actually be useful, it must start from the point 
of allocation and track all the way to deallocation.

But in my view there are two behaviors here:

- Ownership transfer
- Owner/borrow relationship

The owner/borrow relationship is the thing Walter has just given me 
green light on to do a DIP for that I've been wanting for years.

That relies on DIP1000 to detect relationships via the use of ``return`` 
(talked with Dennis, confirmed that this is what is *meant* to be 
happening).

The ownership transfer however is what I want to see solved with 
isolated. This solves aliasing since which sub graph of memory is in 
each variable at the point of a transfer.
Apr 26
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 4/26/24 15:27, Timon Gehr wrote:

 Ideally, `opEquals` implements an equivalence relation. It is fine if it
changes the representatives in the process, as long as it properly encapsulates
the internal state such that whenever two values compare equal, the observable
semantics of the two representatives is the same.
 ...
 If you want something that is actually useful, you will have to look 
 into splay trees or something like that. Or e.g., maybe you have a ring 
 buffer or something that compacts itself on iteration. As I said, 
 amortized data structures. It may be incorrect to have a const opEquals. 
 It can introduce a performance regression.


Jonathan's examples with concurrency are also a very good practical 
illustration of this.
Apr 26
prev sibling next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 26/04/2024 12:57 PM, Walter Bright wrote:

 As I recall, it was you that pointed out that reference counting can 
 never be safe if two mutable pointers to the same ref counted object 
 (one to the object, the other to its interior) were passed to a 
 function. (Freeing the first can leave the second interior pointer 
 pointing to a deleted object.) The entire ref counting scheme capsized 
 because of this.


This is the first time I have heard of this being a concern of yours.

Stuff like this is always solvable if we acknowledge (in other words 
write them all down) what the requirements are!
Apr 25
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/25/2024 6:39 PM, Richard (Rikki) Andrew Cattermole wrote:

 This is the first time I have heard of this being a concern of yours.


It was a working group.


 Stuff like this is always solvable if we acknowledge (in other words write
them 
 all down) what the requirements are!


We had a requirement for memory safety. Without it, RC was more of a step 
sideways than forwards.
Apr 25
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 26/04/2024 3:15 PM, Walter Bright wrote:

     Stuff like this is always solvable if we acknowledge (in other words
     write them all down) what the requirements are!
 
 We had a requirement for memory safety. Without it, RC was more of a 
 step sideways than forwards.


I have had a solution to this since before  live that I was screaming 
about in the guise of DIP1000's last big hole.

Xor mutable references with borrows.

Protects against assigns, function parameter passing, doesn't need any 
extra syntax...

I am highly annoyed by this. THIS WAS SOLVABLE WITH SOMETHING I HAVE 
BEEN SCREAMING ABOUT FOR ALMOST THIS EXACT THING.

If this is literally the *only* thing blocking RC, I can do the DIP for it.
Apr 25
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/25/2024 8:21 PM, Richard (Rikki) Andrew Cattermole wrote:

 If this is literally the *only* thing blocking RC, I can do the DIP for it.


The other problem with RC is the exception handler for every decrement. If 
there's a DIP in it, please do so!
Apr 25
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 26/04/2024 6:02 PM, Walter Bright wrote:

 On 4/25/2024 8:21 PM, Richard (Rikki) Andrew Cattermole wrote:

 If this is literally the *only* thing blocking RC, I can do the DIP 
 for it.


 
 The other problem with RC is the exception handler for every decrement. 
 If there's a DIP in it, please do so!


With struct destructors the unwinding table should already be in use.

So it is a cost we are already paying, that shouldn't be something to 
worry about as it is not a new cost.

On that note, unwinding tables need to be turned on for -betterC in dmd. 
Turning them off can only cause program corruption when calling non 
-betterC code including C/C++. https://github.com/dlang/dmd/pull/16177

But yes, I'll start working on a DIP!

Although I need to talk with Dennis. DIP1000 isn't doing what I would 
expect it to do for slices:

```d
import std;
void main()  safe {
     Context context;
     char[] str = context.acquire();
     char[] var = test(str);
     writeln(var); // Should be erroring with: scope variable `var` 
assigned to non-scope parameter `__param_0` calling `writeln`
}

struct Context {
     char[] acquire() scope return  trusted {
         return "Abc".dup;
     }
}

char[] test(return char[] input)  safe {
     return input;
}
```

It does for ``void*``.
Apr 25
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/25/2024 11:10 PM, Richard (Rikki) Andrew Cattermole wrote:

 With struct destructors the unwinding table should already be in use.
 So it is a cost we are already paying, that shouldn't be something to worry 
 about as it is not a new cost.


Structs are passed around by ref all the time to avoid this cost. With RC, that 
goes out the window.



 DIP1000 isn't doing what I would expect it 
 to do for slices:


Please file bug reports, and tag DIP1000 bugs with the "safe" keyword.

Also, this is drifting off topic. If you want to continue, please start a new 
thread.
Apr 26
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, April 25, 2024 6:57:49 PM MDT Walter Bright via Digitalmars-d 
wrote:

 On 4/25/2024 4:36 PM, Timon Gehr wrote:

 Without the `const` annotations, the functions are not usable by `const`
 objects without doing an unsafe cast. This impairs anyone wanting to
 write
 const-correct code,


 "const correctness" does not work in D because const

 a) provides actual guarantees
 b) is transitive


 It's not the C++ notion of const, sure. But the name still applies.


The name applies, but because D's const is transisitive and can't be
backdoored, it poses a serious problem for certain categories of types to
require it. As such, while in C++, it's normal to slap const on stuff all
over the place, because the type is logically const, and any type that needs
to mutate any portion of its state which is not part of that logical
constness (e.g. a mutex) is perfectly free to do so via using the mutable
keyword or casting away const. In contrast, it violates the type system for
any D code to work around const like that, so it becomes problematic to use
const all over the place like you would in C++, and making code "const
correct" like you would in C++ is typically bad practice in D. It's great to
use D's const where you can, but it's simply too restrictive to require it
in the general case.


 It is fundamentally incompatible with many common patters of
 object-oriented and other state abstraction. It is not even compatible
 with the range API. Uses of `const` are niche. `const` is nice when it
 does work, but it's not something you can impose on all code,
 particularly object-oriented code.


 Why would anyone, for example, try to mutate a range when it is passed to
 one of these functions?


If you can't mutate a range, you can't iterate through it. Your proposed DIP
to be able to have a form of tail-const for ranges will help with that, but
the fact still stands that some types will not work with const, because they
need to mutate some portion of their state in order to function, even with
functions that need to be logically const. If D's const were like C++'s
const, this wouldn't be a problem, but the strong guarantees that D's const
is supposed to provide make it completely incompatible with some code. As
such, we really can't require it anywhere without causing problems. If you
want to be able to require it, it needs to have backdoors; otherwise, a
number of common coding idioms become impossible to use.

So, either we have backdoors that allow mutating const, and we can require
const in places that need to be logically const, or we have const be strict
about mutation and can't require that it be used. As things stand with D's
const, that means that we can't require that it be used.


 I recommend that everyone who has overloads of these functions, alter
 them to have the `const` signatures. This will future-proof them against
 any changes to Object's signatures.


 I will not do that, because if it does not outright break my code (e.g.
 because Phobos cannot support `const` ranges), it actually limits my
 options in the future in a way that is entirely unnecessary.


 Why would anyone need toHash(), toString(), opEquals() or opCmp() to mutate
 their data? Wouldn't that be quite surprising behavior?


It would be surprising if the logical state of the type changed, but it
wouldn't be at all surprising if some portion of the type which was not part
of its logical state changed. A very simple case of this would be if the
type contains a member variable which is shared and a mutex to protect
access to that data (be it a mutex which is also a member variable or which
is a member of the shared member variable). Any of those four functions
would then need to lock that mutex in order to read the data so that they
can do stuff like hash it or compare it. So, while the logical state
wouldn't change, the object itself would be mutated in the process.

Similarly, if a type lazily initializes some portion of its state, and that
initialization hasn't happened yet before one of those functions is called,
then it's going to have to do that initialization as part of the call, which
means mutating the object's state. Its logical state doesn't change, so for
C++, this kind of thing would be a complete non-issue, but for D, because
const doesn't allow any kind of mutation, such a type cannot have const
functions.

And those are just two examples of cases where an object needs to be able to
mutate some portion of its state in functions like opEquals, meaning that if
we put const on opEquals, either such classes can no longer be written in D,
or they're going to cast away const and mutate even if that does technically
violate the type system's guarantees.

If you're just dealing with ints and pointers and arrays and the like, and
you aren't dealing with user-defined types at all, then const generally
doesn't cause many problems. But as soon as you're dealing with user-defined
types, you start running into issues with const depending on what your code
needs to do, and the more complex the code, the more likely it is that
issues with const are going to pop up. The same goes with pretty much all of
the attributes. They add restrictions which work in some cases but don't in
many others.

So, for instance, it's usually bad practice to put const on the parameters
for templated functions, since that means that whole categories of types
won't work with that code, whereas if you don't use const, the caller can
pass a const type, and it'll work just fine in that case so long as the type
in question was designed to work with const. But the types that don't work
with const will also work with that code, because the template doesn't have
its parameter marked as const, and so the generated code won't use const.

We have the same problem with member functions on classes, but since they're
virtual, we can't templatize that code. However, we can templatize the code
that uses those classes, making the use of Object completely unnecessary,
and then each class can define functions like opEquals with whatever set of
attributes makes sense for that class' hierarchy. Derived classes within
that hierarchy will then be stuck with the decisions made for the base
class, but programmers can choose what makes the most sense for that
particular class hierarchy, whereas we cannot possibly make that decision
for all classes and not screw over developers in the process, because it's
not one size fits all.

In general, we need to be trying to support the various attributes
(including const) with druntime and Phobos, but we should not be requiring
them, because they are all too restrictive for that to make sense. And that
includes const.

- Jonathan M Davis
Apr 25
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Perhaps I can help things work for you and Timon:

```
import std.stdio;

class A
{
     string xxx(const Object) const { return "A"; }
}

class B : A
{
     alias xxx = A.xxx;
     string xxx(Object) { return "B"; }
}

void main()
{
     const A a = new A();
     B b = new B();
     const B c = new B();
     writeln(a.xxx(a));
     writeln(b.xxx(b));
     writeln(c.xxx(c));
}
```
I'm calling this xxx instead of toString, just so I can show all the code. 
Compiling it and running it prints:

A
B
A

In other words, you can have a toString() that is mutable and it will work fine 
with writeln(), because writeln(x) does not look for Object.toString(), it
looks 
for x.toString().

Does this work for you?
Apr 25
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, April 26, 2024 12:44:03 AM MDT Walter Bright via Digitalmars-d 
wrote:

 Perhaps I can help things work for you and Timon:

 ```
 import std.stdio;

 class A
 {
      string xxx(const Object) const { return "A"; }
 }

 class B : A
 {
      alias xxx = A.xxx;
      string xxx(Object) { return "B"; }
 }

 void main()
 {
      const A a = new A();
      B b = new B();
      const B c = new B();
      writeln(a.xxx(a));
      writeln(b.xxx(b));
      writeln(c.xxx(c));
 }
 ```
 I'm calling this xxx instead of toString, just so I can show all the code.
 Compiling it and running it prints:

 A
 B
 A

 In other words, you can have a toString() that is mutable and it will work
 fine with writeln(), because writeln(x) does not look for
 Object.toString(), it looks for x.toString().

 Does this work for you?


The ideal situation here is that none of these functions are on Object at
all. They really aren't useful there, because it's not particularly useful
or necessary to operate on Object. Some of the druntime code does, because
it hasn't been templated yet, but once it has been, it won't need to operate
on Object at all. At that point, we won't need to have any of these
functions on Object, and Editions should give us the ability to remove them.

And then, yes, classes can define those functions as they see fit without
having to worry about an implementation on Object, because the code that's
using them will be templated. To an extent, that can be done now. However,
you then have a problem if the Object version ever gets called, and the
derived version does not override the Object version, because then the wrong
version gets called. And if we add something like const to these functions,
and Object gets used at all, then either the wrong overload will be called,
or the derived class will need to be casting away const to have the correct
version called (or druntime will be casting away const like it unfortunately
does with opEquals right now if you compare two const Objects, and when that
happens, it can easily violate the type system's guarantees with const).

Adding const to any of these functions on Object is just putting the code in
a position where either we're at risk of the wrong overload being called, or
const is going to end up being cast away if the Object overload ever gets
used. And it really doesn't buy us much, since code that wants to have them
work with const can overload them on const right now and have the non-const
overload call the const one. Object can't be compared as const, but that's
not generally necessary anyway, since normal code is going to use reference
which are typed as the actual classes, not Object. For the most part, the
only code that's going to have that issue is the druntime code that we need
to turn into templates anyway but currently uses Object because of how old
it is. And once that's done, then there's no need to have the Object
versions of these functions at all. So, I don't think that it makes any
sense whatsoever to add const to the functions on Object. Rather, we need to
be getting the druntime code to the point that it will work to remove them
from Object. And that will fix far more than the issue of const working with
these functions, because then it will allow user code to define these
functions with whatever set of attributes make sense for that code, thereby
fixing the problem for attributes in general.

Also, I would point out that if your motivation for trying to put const on
these functions is related to DIP 1021, then that's going to cause a whole
other set of problems anyway, because if I understand correctly, DIP 1021 is
trying to disallow stuff like

    foo(bar, bar);

where bar is not taken as const via at least one of those parameters. And
yet it's extremely common that neither parameter can be marked as const,
because the code is either written to work with a type that does not work
with const, or it's templated and therefore needs to not assume that the
type it's given works with const (and will often not be instantiated with
const types). The free function, opEquals, is precisely such a case, becase
it's not only designed to work with classes whether their opEquals is const
or not (so long as they're not compared as Object), but it's specifically
designed to be able to compare the same class object against itself (whether
it's literally the same reference or two references which happen to point to
the same object). So, the free function, opEquals, needs to be able to
accept the same object for both arguments without using const at all. Stuff
like

    auto eq = cls == cls;

or

    auto cls2 = cls;
    auto eq = cls == cls2;

need to compile (especially the second one), and that needs to work without
requiring const, because not all classes can use const.

And if DIP 1021 is trying to force code to use const, that's going to be
non-starter for a lot of code because of how restrictive D's const is. How
common it is for the same reference to be passed multiple times, I don't
know (certainly, it's going to be far more common with opEquals or opCmp
than with most functions), so the issue may be fairly restricted in
practice, but with pretty much any part of D, it's going to be a problem any
time that the language tries to require that stuff be const, because const
is simply too restrictive to work with code in general. It would be like
requiring that a feature be pure or  nogc. It will work in many cases, but
it also won't work in many cases, so requiring it makes it so that code that
really should work won't.

Of course, const will work with some types just fine (especially primitive
types), but there are lots of cases in D code where const is avoided
completely, because it's too restrictive for that code to use. And templated
code typically avoids explicitly using it at all on its parameters, because
if the parameters were explicitly marked as const, the code wouldn't work
with any types that don't work as const, whereas if you don't mark them as
const and then pass a const object, then the template is instantiated with
const and works just fine so long as the type itself works as const. So, for
most templated code, explicitly using const is not only completely
unnecessary, but it's bad practice.

So, I find it to be extremely concerning if we're trying to force const
anywhere. We need to support it where we can, but requiring it is going to
cause problems with any types that can't use it - or which can't use it for
the particular operations that are involved with the code trying to require
it. So, if a new language feature is trying to require const, we really need
to revisit that feature.

- Jonathan M Davis
Apr 26
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
D1 is an example of a language with no attributes and no const. D1 works as a 
good programming language.

But it gives the programmer no indication of whether the arguments get mutated 
or not. He'll have to read and understand the called function, as well as the 
functions it calls.

It is reasonable to use const parameters when the argument is not going to be 
mutated. I personally prefer to use that as much as possible, and I like very 
much that the compiler will enforce it. With the mutating 4 functions, I cannot 
use const class objects.

Mutating toString, toHash, opCmp, and opEquals is unusual behavior, which is
why 
const should be the default for them. After all, who expects a==b to change a
or b?

I showed how to use the toString, toHash, opCmp, and opEquals functions with 
objects that do want to use mutating implementations of those functions. It
will 
also be clear to the user which toString is mutating and which is not. It 
satisfies the use cases Timon mentioned - he'll still be able to use a mutating 
toString that will be used by writeln().
Apr 26
next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Friday, 26 April 2024 at 20:17:09 UTC, Walter Bright wrote:

 Mutating toString, toHash, opCmp, and opEquals is unusual 
 behavior, which is why const should be the default for them. 
 After all, who expects a==b to change a or b?


Timon has mentioned data structures with amortized time 
complexity several times now, but perhaps an example closer to 
home helps:

https://github.com/dlang/dmd/blob/9ffa763540e16228138b44c3731d9edc2a7728b6/compiler/src/dmd/dsymbol.d#L668

In this case, `toString` (or `toPrettyChars`, same idea) is 
*logically* const because it doesn't mutate the Dsymbol 
meaningfully, but it can't be D's *memory* const because it does 
change a class field to store a cached result.
Apr 26
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/26/2024 2:38 PM, Dennis wrote:

 Timon has mentioned data structures with amortized time complexity several
times 
 now, but perhaps an example closer to home helps:
 
 https://github.com/dlang/dmd/blob/9ffa763540e16228138b44c3731d9edc2a7728b6/compiler/src/dmd/dsymbol.d#L668
 
 In this case, `toString` (or `toPrettyChars`, same idea) is *logically* const 
 because it doesn't mutate the Dsymbol meaningfully, but it can't be D's
*memory* 
 const because it does change a class field to store a cached result.
 


I'm aware of that, and have investigated it several times looking for what can 
be made const. The compiler does a lot of lazy evaluation.

The simplest way to deal with that is to get the logical const value at the
call 
site, then pass it to a const parameter.
Apr 26
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, April 26, 2024 2:17:09 PM MDT Walter Bright via Digitalmars-d 
wrote:

 D1 is an example of a language with no attributes and no const. D1 works as
 a good programming language.

 But it gives the programmer no indication of whether the arguments get
 mutated or not. He'll have to read and understand the called function, as
 well as the functions it calls.

 It is reasonable to use const parameters when the argument is not going to
 be mutated. I personally prefer to use that as much as possible, and I like
 very much that the compiler will enforce it. With the mutating 4 functions,
 I cannot use const class objects.


You can do that today. E.G. This code compiles and runs just fine.

```
void main()
{
    static class C
    {
        int i;

        this(int i)
        {
            this.i = i;
        }

        override bool opEquals(Object rhs)
        {
            if(auto r = cast(C)rhs)
                return opEquals(r);
            return false;
        }

        bool opEquals(const C rhs) const
        {
            return this.i == rhs.i;
        }

        override int opCmp(Object rhs)
        {
            if(auto r = cast(C)rhs)
                return opCmp(r);

            throw new Exception("Cannot compare C with types that aren't C");
        }

        int opCmp(const C rhs) const
        {
            if(this.i < rhs.i)
                return -1;
            if(this.i > rhs.i)
                return 1;
            return true;
        }

        override string toString() const
        {
            import std.format : format;
            return format!"C(%s)"(i);
        }

        override size_t toHash() const  safe nothrow
        {
            return i;
        }
    }

    const c1 = new C(42);
    const c2 = new C(99);

    assert(c1 == c1);
    assert(c1 != c2);

    assert(c1 <= c1);
    assert(c1 < c2);

    assert(c1.toHash() == 42);

    import std.format : format;
    assert(format!"c1: %s"(c1) == "c1: C(42)");
}
```

All four functions worked with const references. What does not work is if
you use const Object references.

    assert(c1 == c2);

gets lowered to the free function, opEquals:

    assert(opEquals(c1, c2));

and because that function is templated, the derived class overloads get
used. This is what happens in almost all D code using classes. The exception
is code that uses Object directly, and pretty much no code should doing
that.

Java passes Object all over the place and stores it in data structures such
as containers, because Java doesn't have templates. For them, generic code
has to operate on Object, because they don't have any other way to do it. In
sharp contrast, we have templates. So, generic code has no need to operate
on Object, and as such, it has no need to call opEquals, opCmp, toHash, or
toString on Object. As it is, Object's opCmp throws, because there are
plenty of classes where opCmp doesn't even make sense, and there really
isn't a way to give Object an implementation that makes sense.

Generic code in D is templated, and as such, we can do stuff like we've done
with the free funtion, opEquals, and make it so the code that needs to
operate on classes generically operates on the exact type that it's given
instead of degrading to Object. And as such, we don't need any of these
functions to be on Object.

It's already the case that code like format and writeln operate on the
actual class type that they're given and not Object. You already saw that
when you talked about using alternate overloads for toString.

D code in general does not operate on Object. AFAIK, the main place that
anything in D operates on Object at present is in old druntime code that has
yet to be templated. And if that code is templated, the need to have these
functions on Object goes away entirely. Then the entire debate of which
attributes these functions should have goes away. Classes can define them in
whatever way the programmer sees fit so long as the parameters and return
types match what's necessary for them to be called by the code that uses
these functions - just like happens with structs. The only difference is
that the derived classes within a particular class hierarchy will have to be
built on top of whatever signatures those functions were given on the first
class in the hierarchy that had them, whereas structs don't have to worry
about inheritance. But those signatures can then be whatever is appropriate
for that particular class hierarchy instead of trying to come up with a set
of attributes that make sense for all classes (which isn't possible).

And given that Object really doesn't need to have any of these functions, we
likely would have removed them years ago if it weren't for the fact that
something like that would break code (in large part due to the override
keyword; a lot of the code would have worked just fine with those functions
being removed from Object if the derived classes didn't have to have
override, which will then become an error when the base class version of the
funtion is removed). Andrei also proposed ProtoObject as a way to change the
class hierarchy so that we could remove these functions (as well as the
monitor) from classes without breaking code built on top of Object. So,
we've known for years that we could fix this problem if we could just remove
these functions from Object.

Editions gives us a way to make breaking changes in a mangeable manner. This
should give us the opportunity to remove these four functions from Object
like we've discussed for years and couldn't do because it would break code.

And if we decide to not do that, putting const on these four functions would
actually make the situation worse. Yes, you could then call those four
functions on const Objects, but it would mean that every single class will
be forced to have these functions even if they cannot actually implement
them properly with const. And what do such types do at that point? Do they
throw an exception?

```
    static class C
    {
        Mutex mutex;
        shared int* ptr;

        this()
        {
            this.ptr = new shared int;
        }

        override bool opEquals(const Object rhs) const
        {
            throw new Exception("C does not support const")
        }

        bool opEquals(C rhs)
        {
            mutex.lock();
            immutable left = cast()*this.ptr;
            mutex.unlock();

            rhs.mutex.lock();
            immutable right = cast()*rhs.ptr;
            rhs.mutex.unlock();

            return left == right;
        }
    }
```

Do they cast away const and mutate?

```
    static class C
    {
        Mutex mutex;
        shared int* ptr;

        this(int i)
        {
            this.ptr = new shared int;
        }

        override bool opEquals(const Object rhs) const
        {
            if(auto r = cast(C)rhs)
                return (cast()this).opEquals(r);
            return false;
        }

        bool opEquals(C rhs)
        {
            mutex.lock();
            immutable left = cast()*this.ptr;
            mutex.unlock();

            rhs.mutex.lock();
            immutable right = cast()*rhs.ptr;
            rhs.mutex.unlock();

            return left == right;
        }
    }
```

Do they just have different behavior in the Object overload?

```
    static class C
    {
        Mutex mutex;
        shared int* ptr;

        this(int i)
        {
            this.ptr = new shared int;
        }

        override bool opEquals(const Object rhs) const
        {
            return this is rhs;
        }

        bool opEquals(C rhs)
        {
            mutex.lock();
            immutable left = cast()*this.ptr;
            mutex.unlock();

            rhs.mutex.lock();
            immutable right = cast()*rhs.ptr;
            rhs.mutex.unlock();

            return left == right;
        }
    }
```

You'd have a type which could technically be used as a const Object but
which would not do the correct thing if it ever is. In contrast, right now,
while you can't call any of these functions with a const Object, you _can_
call them on a const reference of the derived type if the derived type has
const on them.

So, the code right now will do the correct thing, and it will work with
const in any normal situation, whereas if we put const on these functions,
such classes will have overloads that will not - and cannot - do the correct
thing. And while those Object overloads would not normally be used, if they
ever are, you have a bug - one which could be pretty annoying to track down,
depending on what the const overload does.

I completely agree with you that _most_ classes should have const on these
functions so that they can work with const references, but not all classes
will work with const, and I don't see why there is any need to make these
functions work with const Objects - const class references, yes, but not
const Objects.

Normal D code does not use Object directly, and we should be able to
templatize what little druntime code there is left which operates on Object
and needs to use one or more of these functions. Once that's done, we can
use an Edition to remove these functions from Object, and this entire issue
goes up in smoke.

Instead, what you're proposing also causes breakage, but it puts perfectly
legitimate use cases in a situation where they have to implement functions
which they literally cannot implement properly. And it's for a use case that
normal D code shouldn't even be doing - that is operating on Object instead
of whatever derived types the code base in question is actually using. D is
not Java. It may have made sense to put these functions on Object with D1,
but with D2, we have a powerful template system which generally obviates the
need to operate on Object. We shouldn't need to have these functions on
Object, and Editions should give us what we need to remove them in a
manageable way.

- Jonathan M Davis
Apr 26
prev sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Friday, 26 April 2024 at 08:43:46 UTC, Jonathan M Davis wrote:

 The ideal situation here is that none of these functions are on 
 Object at all. They really aren't useful there, because it's 
 not particularly useful or necessary to operate on Object. Some 
 of the druntime code does, because it hasn't been templated 
 yet, but once it has been, it won't need to operate on Object 
 at all. At that point, we won't need to have any of these 
 functions on Object, and Editions should give us the ability to 
 remove them.


+1, this is the correct solution.

We've already had success templating the druntime opEquals 
lowering for classes [1]. We can and should do the same thing for 
the other Object methods.

[1] https://github.com/dlang/druntime/pull/3665
Apr 27
parent reply Atila Neves <atila.neves gmail.com> writes:
On Saturday, 27 April 2024 at 16:35:41 UTC, Paul Backus wrote:

 On Friday, 26 April 2024 at 08:43:46 UTC, Jonathan M Davis 
 wrote:

 The ideal situation here is that none of these functions are 
 on Object at all. They really aren't useful there, because 
 it's not particularly useful or necessary to operate on 
 Object. Some of the druntime code does, because it hasn't been 
 templated yet, but once it has been, it won't need to operate 
 on Object at all. At that point, we won't need to have any of 
 these functions on Object, and Editions should give us the 
 ability to remove them.


 +1, this is the correct solution.

 We've already had success templating the druntime opEquals 
 lowering for classes [1]. We can and should do the same thing 
 for the other Object methods.

 [1] https://github.com/dlang/druntime/pull/3665


I talked to Walter and we agreed that the best way forward is 
probably to deprecate these member functions and remove them in 
the next edition.
May 08
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, May 8, 2024 7:20:20 PM MDT Atila Neves via Digitalmars-d wrote:

 On Saturday, 27 April 2024 at 16:35:41 UTC, Paul Backus wrote:

 On Friday, 26 April 2024 at 08:43:46 UTC, Jonathan M Davis

 wrote:

 The ideal situation here is that none of these functions are
 on Object at all. They really aren't useful there, because
 it's not particularly useful or necessary to operate on
 Object. Some of the druntime code does, because it hasn't been
 templated yet, but once it has been, it won't need to operate
 on Object at all. At that point, we won't need to have any of
 these functions on Object, and Editions should give us the
 ability to remove them.


 +1, this is the correct solution.

 We've already had success templating the druntime opEquals
 lowering for classes [1]. We can and should do the same thing
 for the other Object methods.

 [1] https://github.com/dlang/druntime/pull/3665


 I talked to Walter and we agreed that the best way forward is
 probably to deprecate these member functions and remove them in
 the next edition.


Yay!

- Jonathan M Davis
May 09
prev sibling parent "H. S. Teoh" <hsteoh qfbox.info> writes:
On Thu, May 09, 2024 at 01:53:55AM -0600, Jonathan M Davis via Digitalmars-d
wrote:

 On Wednesday, May 8, 2024 7:20:20 PM MDT Atila Neves via Digitalmars-d wrote:


[...]

 I talked to Walter and we agreed that the best way forward is
 probably to deprecate these member functions and remove them in the
 next edition.


 
 Yay!


[...]

Finally!!  Should've done this years ago.


T

-- 
Curiosity kills the cat. Moral: don't be the cat.
May 09
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 4/26/24 08:44, Walter Bright wrote:

 Perhaps I can help things work for you and Timon:
 
 ```
 import std.stdio;
 
 class A
 {
      string xxx(const Object) const { return "A"; }
 }
 
 class B : A
 {
      alias xxx = A.xxx;
      string xxx(Object) { return "B"; }
 }
 
 void main()
 {
      const A a = new A();
      B b = new B();
      const B c = new B();
      writeln(a.xxx(a));
      writeln(b.xxx(b));
      writeln(c.xxx(c));
 }
 ```
 I'm calling this xxx instead of toString, just so I can show all the 
 code. Compiling it and running it prints:
 
 A
 B
 A
 
 In other words, you can have a toString() that is mutable and it will 
 work fine with writeln(), because writeln(x) does not look for 
 Object.toString(), it looks for x.toString().
 
 Does this work for you?


It clutters the user code with aliases. Might be preferable to forking 
Phobos though. Also, it can give wrong results at runtime.

For example, if a templated library type uses DbI to check whether it 
should make `toString` `const` by checking whether there is a `const 
toString` on the argument type, it will find `Object.toString`. Then 
those types will not properly compose with my `toString`.

This is not a theoretical problem either. This kind of introspection 
would be the proper fix for the following issue with 
std.typecons.Tuple.toString:

```d
import std.stdio, std.typecons;
class C{
     override string toString()=>"correct";
}

void main(){
     writeln(new C()); // "correct"
     writeln(tuple(new C())); // "Tuple!(C)(const(tt.C))"
}
```

This is also the same issue that prevents tuples with range members from 
being printed properly.
Apr 26
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 4/26/24 16:00, Timon Gehr wrote:

 
 
 This is not a theoretical problem either. This kind of introspection 
 would be the proper fix for the following issue with 
 std.typecons.Tuple.toString:
 
 ```d
 import std.stdio, std.typecons;
 class C{
      override string toString()=>"correct";
 }
 
 void main(){
      writeln(new C()); // "correct"
      writeln(tuple(new C())); // "Tuple!(C)(const(tt.C))"
 }
 ```
 
 This is also the same issue that prevents tuples with range members from 
 being printed properly.


For reference, this is how I deal with problems like that currently:
https://github.com/tgehr/util/blob/master/tuple.d
Apr 26
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/26/2024 7:00 AM, Timon Gehr wrote:

 This is not a theoretical problem either. This kind of introspection would be 
 the proper fix for the following issue with std.typecons.Tuple.toString:
 
 ```d
 import std.stdio, std.typecons;
 class C{
      override string toString()=>"correct";
 }
 
 void main(){
      writeln(new C()); // "correct"
      writeln(tuple(new C())); // "Tuple!(C)(const(tt.C))"
 }
 ```
 
 This is also the same issue that prevents tuples with range members from being 
 printed properly.


I would like to see "new C()" and "tuple(new C())" mean exactly the same thing. 
After all, with the builtin-tuples (not the struct library version) they are
the 
same thing.
Apr 26
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 4/26/24 22:23, Walter Bright wrote:

 On 4/26/2024 7:00 AM, Timon Gehr wrote:

 This is not a theoretical problem either. This kind of introspection 
 would be the proper fix for the following issue with 
 std.typecons.Tuple.toString:

 ```d
 import std.stdio, std.typecons;
 class C{
      override string toString()=>"correct";
 }

 void main(){
      writeln(new C()); // "correct"
      writeln(tuple(new C())); // "Tuple!(C)(const(tt.C))"
 }
 ```

 This is also the same issue that prevents tuples with range members 
 from being printed properly.


 
 I would like to see "new C()" and "tuple(new C())" mean exactly the same 
 thing. After all, with the builtin-tuples (not the struct library 
 version) they are the same thing.
 


Ok, then I will further pursue an implementation of `opArgs` I guess. It 
does have a couple of drawbacks, but if this is your preference we can 
try to make it work.
Apr 26
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/26/2024 4:30 PM, Timon Gehr wrote:

 On 4/26/24 22:23, Walter Bright wrote:

 I would like to see "new C()" and "tuple(new C())" mean exactly the same 
 thing. After all, with the builtin-tuples (not the struct library version) 
 they are the same thing.


 
 Ok, then I will further pursue an implementation of `opArgs` I guess. It does 
 have a couple of drawbacks, but if this is your preference we can try to make
it 
 work.


I also mean that given:
```
int mul(int x, int y);
```
then:
```
mul(1, 2);
```
should mean the same thing as:
```
mul(tuple(1, 2));
```

This currently works with the builtin tuples. The trouble is that:

```
struct Args { int a; int b };
Args args = { 1, 2 };
mul(args);
```

fundamentally does not work, because the binary function call API for arguments 
is not the same as the binary function call API for structs with the 
corresponding fields.

This has frustrated me for some time. Static arrays and structs are binary API 
compatible, and I've been careful not to break that in D's design. I.e. they
are 
unified. I'd like to extend that unification to tuples-as-arguments.

I.e.:

```
struct <=> static array <=> tuple <=> argument list
```

should be binary interchangeable.

Note that I was very pleased to make the discovery that pointers to:

struct member functions <=> class member functions <=> nested functions

are all interchangeable delegates! This has been a big win for D.
Apr 26
prev sibling next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, April 25, 2024 5:06:27 PM MDT Walter Bright via Digitalmars-d 
wrote:

 The prototypes are:

 ```
 string toString();
 size_t toHash()  trusted nothrow;
 int opCmp(Object o);
 bool opEquals(Object o);
 ```

 which long predated `const`. The trouble is, they should be:

 ```
 string toString() const;
 size_t toHash() const  trusted nothrow;
 int opCmp(const Object o) const;
 bool opEquals(const Object o) const;
 ```

 Without the `const` annotations, the functions are not usable by `const`
 objects without doing an unsafe cast. This impairs anyone wanting to write
 const-correct code, and also impedes use of ` live` functions.

 I recommend that everyone who has overloads of these functions, alter them
 to have the `const` signatures. This will future-proof them against any
 changes to Object's signatures.


The problem with this is that D's const is not logical const, and some
objects cannot work if these functions are fully const (e.g. beacuse they're
using a mutex or because they have to lazily calculate their state). Having
_any_ attributes on these functions is a problem, because those attributes
restrict what derived classes can do. Similarly, not having any attributes
causes problems, because then they can't be used in code that requires those
attributes.

The solution to this problem (as has been discussed plenty of times in the
past) is to outright remove these functions from Object. The only reason
that they need to be there is because of code that's written to use Object
instead of using templates, and D has templates. The main blocker then has
been two things:

1. We haven't wanted to break existing code by removing these functions from
Object. Editions hopefully give us a way to move past that problem.

2. Instead of being properly templated, some of the key druntime code (e.g.
involving hash tables) has used Object. Some work has been done to fix
various parts of druntime (like the hooks for arrays) so that they're
templated, but the work has not been completed. If the various parts of
druntime which require Object are fully fixed to be templated, then we don't
need these functions on Object any longer. The code would just be
instantiated with whatever the class type that's given is, and those
functions can then have whatever attributes are appropriate for that
particular class hierarchy without Object needing to have them any more than
Object has a foobarWilly function, because some stray library needs that for
its class hierarchy.

We currently have a partial solution in druntime in that the free function,
opEquals, is templated so that if you try to use == on class references
which are not Object, it will use the derived class' version of opEquals,
thus allowing classes to define opEquals with whatever attributes are
appropriate for that particular class hierarchy (as well as allowing them to
make their opEquals take the type of that specific class instead of Object).
The problem of course is that when you compare classes as Object, you get
the Object version, but most code doesn't use Object directly, so in
general, == works with whatever attributes we want, and if we get rid of
opEquals from Object, those comparisons should still work. You then won't be
able to use == on Object, but that's a pretty nonsensical comparison anyway.
It's only ever made sense in code where you couldn't templatize it and
therefore needed a base class to use (like Java does with its containers),
and even then, for most code, it makes far more sense to use a base class
from that particular project than to use Object, in which case, that base
class can be given whatever functions or attributes are appropriate to that
particular class hierarchy.

__cmp is similarly templated, though I'm not as familiar with how the
lowerings work with regards to opCmp. But as long as all of the comparison
operators lower to templated functions that call opEquals or opCmp on the
class references with whatever type they have instead of with Object, we can
work around Object's versions of those functions.

toHash and toString would typically be called more directly, but if the code
that's calling them is templated rather than using Object, derived classes
can currently declare versions of those functions with a different set of
attributes (though since those functions don't have parameters, they're
somewhat more restricted in which attributes can be used, since they can't
overload on most attributes) - and with regards to const, they can overload
the Object versions, meaning that they only have a problem if Object is
being used.

Adding const - or any other attributes - to the functions on Object would be
a step backwards rather than forwards and needlessly restrict code. Rather,
we need to take advantage of templates and and Editions and make it so that
none of these functions are on Object at all, allowing each individual class
hierarchy to define these functions in whatever manner makes sense for that
code. Editions gives us an opportunity here which we have not had
previously, and we should take grab it.

- Jonathan M Davis
Apr 25
prev sibling next sibling parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 26/04/2024 11:06 AM, Walter Bright wrote:

 I recommend that everyone who has overloads of these functions, alter 
 them to have the `const` signatures. This will future-proof them against 
 any changes to Object's signatures.


We don't need to do this.

The solution that covers pretty much everyone needs is custom root classes.

Attributes, monitor field, reference counting, -betterC, all handled if 
we just let people define their own root class that they explicitly 
inherit from.

Right now language is far too coupled to druntime, and this is one area 
I want to see fixed. Too many issues crop up because it is too coupled.
Apr 25
prev sibling next sibling parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Thursday, 25 April 2024 at 23:06:27 UTC, Walter Bright wrote:

 The prototypes are:

 ```
 string toString();
 size_t toHash()  trusted nothrow;
 int opCmp(Object o);
 bool opEquals(Object o);
 ```

 which long predated `const`. The trouble is, they should be:


Shouldn't some or all of them be qualified as scope aswell?
Apr 26
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/26/2024 11:28 AM, Per Nordlöw wrote:

 Shouldn't some or all of them be qualified as scope aswell?


I did think of that, but also figured if I can't get const, scope is dead in
the 
water, too.
Apr 26
prev sibling next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, May 9, 2024 9:24:12 AM MDT H. S. Teoh via Digitalmars-d wrote:

 On Thu, May 09, 2024 at 01:53:55AM -0600, Jonathan M Davis via Digitalmars-d 


wrote:

 On Wednesday, May 8, 2024 7:20:20 PM MDT Atila Neves via Digitalmars-d 




wrote:

 [...]


 I talked to Walter and we agreed that the best way forward is
 probably to deprecate these member functions and remove them in the
 next edition.


 Yay!


 [...]

 Finally!!  Should've done this years ago.


Yeah, but issues with regards to code breakage have made it difficult to do
cleanly. Editions should make it much more reasonable.

- Jonathan M Davis
May 09
prev sibling next sibling parent "H. S. Teoh" <hsteoh qfbox.info> writes:
On Thu, May 09, 2024 at 01:12:35PM -0600, Jonathan M Davis via Digitalmars-d
wrote:

 On Thursday, May 9, 2024 9:24:12 AM MDT H. S. Teoh via Digitalmars-d wrote:

 On Thu, May 09, 2024 at 01:53:55AM -0600, Jonathan M Davis via Digitalmars-d 


 wrote:

 On Wednesday, May 8, 2024 7:20:20 PM MDT Atila Neves via Digitalmars-d 




 wrote:

 [...]

 I talked to Walter and we agreed that the best way forward is
 probably to deprecate these member functions and remove them in
 the next edition.


 Yay!


 [...]

 Finally!!  Should've done this years ago.


 
 Yeah, but issues with regards to code breakage have made it difficult
 to do cleanly. Editions should make it much more reasonable.


[...]

Have editions been implementing in any shape/form yet?  Been curious
about what it looks like, and how it works in practice.


T

-- 
I am Pentium of Borg. Division is futile; you will be approximated.
May 09
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, May 9, 2024 2:15:27 PM MDT H. S. Teoh via Digitalmars-d wrote:

 On Thu, May 09, 2024 at 01:12:35PM -0600, Jonathan M Davis via Digitalmars-d 


wrote:

 On Thursday, May 9, 2024 9:24:12 AM MDT H. S. Teoh via Digitalmars-d 




wrote:

 On Thu, May 09, 2024 at 01:53:55AM -0600, Jonathan M Davis via
 Digitalmars-d>


 wrote:

 On Wednesday, May 8, 2024 7:20:20 PM MDT Atila Neves via Digitalmars-d




 wrote:

 [...]


 I talked to Walter and we agreed that the best way forward is
 probably to deprecate these member functions and remove them in
 the next edition.


 Yay!


 [...]

 Finally!!  Should've done this years ago.


 Yeah, but issues with regards to code breakage have made it difficult
 to do cleanly. Editions should make it much more reasonable.


 [...]

 Have editions been implementing in any shape/form yet?  Been curious
 about what it looks like, and how it works in practice.


If I understand correctly, some changes have been made in dmd which are
supposed to only be in effect with a new edition, but I don't think that the
actual edition mechanism has been implemented yet. There's been a fair bit
of going back and forth over the details of the proposal (at the moment,
mostly how that would interact with dub), which could affect the
implementation anyway.

At its most basic though, the core idea is that a module could be marked as
being for a specific edition, and it would then be compiled with the rules
for that edition regardless of which edition the code importing it uses (and
that includes any templated code within the module). So, in theory, the code
within that module would continue to work as it always has as new editions
are released, meaning that breaking changes in future editions wouldn't
affect it until that code was updated to be marked as being for a new
edition, at which point, the maintainer would need to update it in whatever
fashion was necessary to get it working properly with the new edition.

So, the reason that we could then make breaking changes with new editions
would be because you can control which edition your code targets, making it
so that those breaking changes don't affect you until you're ready to deal
with them.

Unfortunately, because the program can only use one version of druntime,
which would then need to be able to work with all editions, there are some
things which we likely won't ever be able to change (like it sounds like
we're probably going to be stuck with the monitor in Object), but we'll
definitely be able to make more breaking changes than we can right now.

- Jonathan M Davis
May 09