www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Ownership and Borrowing in D

reply Mike Parker <aldacron gmail.com> writes:
In the Draft Review for Walter's DIP, Argument Ownership and 
Function Calls (which in the next half hour or so will be 
starting a community review as DIP 1021), some folks asked for 
some information about the bigger picture. In response, Walter 
has put together a post outlining his current thinking on how to 
add Ownership and Borrowing to D.

This is just an overview and he's currently working on a more 
comprehensive proposal, but it should provide some context for 
those who want it when reviewing DIP 1021.

The blog:
https://dlang.org/blog/2019/07/15/ownership-and-borrowing-in-d/

Reddit:
https://www.reddit.com/r/programming/comments/cdifbu/ownership_and_borrowing_in_d/
Jul 15 2019
next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Monday, 15 July 2019 at 14:58:55 UTC, Mike Parker wrote:

 In the Draft Review for Walter's DIP, Argument Ownership and 
 Function Calls (which in the next half hour or so will be 
 starting a community review as DIP 1021), some folks asked for 
 some information about the bigger picture. In response, Walter 
 has put together a post outlining his current thinking on how 
 to add Ownership and Borrowing to D.


Thank you, Walter.  This is very exciting.  I've said that if I 
could have my ideal language, it would be Dust, and now I just 
might get it :)

Mike
Jul 15 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/15/2019 7:07 PM, Mike Franklin wrote:

 Thank you, Walter.  This is very exciting.  I've said that if I could have
my 
 ideal language, it would be Dust, and now I just might get it :)


Now I just have to deliver the goods!
Jul 15 2019
next sibling parent Newbie2019 <newbie2019 gmail.com> writes:
On Tuesday, 16 July 2019 at 06:12:42 UTC, Walter Bright wrote:

 Now I just have to deliver the goods!


Very exciting about d have the  live and borrow owner feature.

How do you plan to handle move ?  dose it depends on DIP1014 ?
Jul 16 2019
prev sibling next sibling parent reply Olivier FAURE <couteaubleu gmail.com> writes:
On Tuesday, 16 July 2019 at 06:12:42 UTC, Walter Bright wrote:

 Now I just have to deliver the goods!


Lately, I've been thinking about the possibility of an 
alternative ownership system for D, one that would be radically 
different from what you're considering, but still aim for the 
same features (memory safety, compile-time checking, zero-cost 
abstraction), based on a `unique` qualifier.

If I were to write a formal proposal for it, how interested would 
you be in comparing the two schemes (DIP 1021 and eventually Rust 
"one mutable ref" rule, vs unique qualifier)?

Like, I want to make my pitch, but I don't want to spend huge 
amount of effort on it if you're just going to go with DIP 1021 
anyway.
Jul 17 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/17/2019 12:11 PM, Olivier FAURE wrote:

 On Tuesday, 16 July 2019 at 06:12:42 UTC, Walter Bright wrote:

 Now I just have to deliver the goods!


 
 Lately, I've been thinking about the possibility of an alternative ownership 
 system for D, one that would be radically different from what you're 
 considering, but still aim for the same features (memory safety, compile-time 
 checking, zero-cost abstraction), based on a `unique` qualifier.
 
 If I were to write a formal proposal for it, how interested would you be in 
 comparing the two schemes (DIP 1021 and eventually Rust "one mutable ref"
rule, 
 vs unique qualifier)?
 
 Like, I want to make my pitch, but I don't want to spend huge amount of effort 
 on it if you're just going to go with DIP 1021 anyway.


Any competing system would need to not be 'opt-in' on a type by type basis.
I.e. 
the central feature of an  live function is the user will not be able to write 
memory unsafe code within that function.

I'm interested to see your design. But I suggest you move quickly, as I've 
suggested people who talked about this propose a design for the last 10 years, 
nothing has happened, and we can't wait any longer.
Jul 17 2019
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 17.07.19 22:59, Walter Bright wrote:

 
 Any competing system would need to not be 'opt-in' on a type by type 
 basis. I.e. the central feature of an  live function is the user will 
 not be able to write memory unsafe code within that function.


Those two things are not the least bit at odds with each other. You only 
need to do the additional checks for types that actually need it. (And 
there you need to _always_ do them in  safe code, not just in 
specially-annotated " live" functions.) A dynamic array of integers that 
is owned by the garbage collector doesn't need any O/B semantics. A 
malloc-backed array that wants to borrow out its contents needs to be 
able to restrict  safe access patterns to ensure that the memory stays 
alive for the duration of the borrow.

Furthermore, making built-in types change meaning based on function 
attributes is just not a good idea, because you will get needless 
friction at the interface between functions with the attribute and 
functions without.

Anyway, it makes no sense to have a variable of type `int*` that owns 
the `int` it points to (especially in safe code), because that type 
doesn't track how to deallocate the memory.

Ownership and borrowing should be supported for user-defined types, not 
raw pointers. Rust doesn't track ownership for built-in pointers. In 
Rust, raw pointer access is unsafe.
Jul 19 2019
prev sibling parent Olivier FAURE <couteaubleu gmail.com> writes:
On Wednesday, 17 July 2019 at 20:59:27 UTC, Walter Bright wrote:

 I'm interested to see your design. But I suggest you move 
 quickly, as I've suggested people who talked about this propose 
 a design for the last 10 years, nothing has happened, and we 
 can't wait any longer.


For the record, I've been busy with WebAssembly proposals and 
haven't had the time to write a proper DIP this week.

That said, here's a **very, very rough** draft that conveys what 
the proposed design is about: 
https://gist.github.com/PoignardAzur/9896ddb17b9f6d6f3d0fa5e6fe1a7088

I'll try to make a PR before next week-end.
Jul 21 2019
prev sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Tuesday, 16 July 2019 at 06:12:42 UTC, Walter Bright wrote:

 Now I just have to deliver the goods!


I am very excited about this, together with unique these are some 
very desired semantics.

I have a practical question. In spasm I have to solve the 
lifetime of javascript objects that are referenced from D. The 
way it works is that objects in Javascript are put in a big array 
and the index - the handle if you will - is passed to D. When D 
code wants to do anything with the js object, it passes the 
handle to a js glue function which looks up the object in the 
table and does what it needs to.

The trick is finding an efficient mechanism to free the js 
objects in the js array. At first I tried reference counting - 
which worked, but introduced a lot of code bloat.

Now I have settled on having the wrapper struct (JsHandle) of the 
handle be non-copyable, and pass it around by either moving or 
using `ref scope` with dip1000 and  safe. Then if you need to 
have the handle around for longer you can always wrap it in a RC 
wrapper.

This works remarkably well. It gives cheap, efficient and safe 
code and still allows multiple long-lived references with RC if 
that is what is wanted.

There are 2 issues I currently have.

First some code:

---
alias Handle = uint;

struct JsHandle {
   nothrow:
   package Handle handle;
   alias handle this;
   ~this() {
     if (handle != invalid)
       spasm_removeObject(handle);
   }
    disable this(this);
}

struct HTMLElement {
   spasm.bindings.dom.Element _parent;
   alias _parent this;
   this(JsHandle h) {
     _parent = .Element(h);
   }
   // ... functions
}

struct HTMLCanvasElement {
   spasm.bindings.html.HTMLElement _parent;
   alias _parent this;
   this(JsHandle h) {
     _parent = .HTMLElement(h);
   }
  // ... functions
}
---

1) Sometimes the web api's accept an Nullable/Optional, which 
means I have to wrap, for instance, the HTMLElement into an 
Optional type and pass that. Of course, I want to maintain the 
unique reference constraint and enforce that the Optional won't 
outlive the original HTMLElement. Which I can't, since they are 
all non-pointers and I can't store a scope ref inside a Optional. 
The solution I have now is to create a Nullable!(T*). This does 
solve the unique semantics, at the expense of having an api that 
accepts both Nullable!(T) and Nullable!(T*). Pretty ugly.

2) the other issue is that I sometimes want to "upcast" a 
HTMLElement to a HTMLCanvasElement. I can simply do this by 
moving the handle around, but I want the HTMLCanvasElement not to 
outlive the HTMLElement. Again, the solution is to involve 
pointers. Since all wrapper structs have the same memory layout 
(only a uint), I used this hack:

---
template as(Target) {
   auto as(Source)(scope return ref Source s) {
     return cast(Target*)&s;
   }
}
---

Anyway my question is whether the ownership and borrowing 
semantics ever gets applied to non-pointers?
Jul 20 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/20/2019 5:42 AM, Sebastiaan Koppe wrote:

 Anyway my question is whether the ownership and borrowing semantics ever gets 
 applied to non-pointers?


Currently, no. The programmer would be expected to include the correct 
copyctor/opAssign/destructors which would take care of it.

We're likely going to add a move constructor, too.
Jul 20 2019
next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Saturday, 20 July 2019 at 21:56:50 UTC, Walter Bright wrote:

 On 7/20/2019 5:42 AM, Sebastiaan Koppe wrote:

 Anyway my question is whether the ownership and borrowing 
 semantics ever gets applied to non-pointers?


 Currently, no. The programmer would be expected to include the 
 correct copyctor/opAssign/destructors which would take care of 
 it.


Do you mean to keep track of ownership/borrowedness manually? 
Sure, that will work. But how would I then get the compiler to 
complain when a borrowed one outlives the owned one?
Jul 20 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/20/2019 3:39 PM, Sebastiaan Koppe wrote:

 Do you mean to keep track of ownership/borrowedness manually?


No, that's what the copyctor/opAssign/dtor semantics so.
Jul 20 2019
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 21.07.19 02:17, Walter Bright wrote:

 On 7/20/2019 3:39 PM, Sebastiaan Koppe wrote:

 Do you mean to keep track of ownership/borrowedness manually?


 
 No, that's what the copyctor/opAssign/dtor semantics so.


This is not true.
Jul 22 2019
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Tuesday, 23 July 2019 at 04:02:50 UTC, Timon Gehr wrote:

 On 21.07.19 02:17, Walter Bright wrote:

 On 7/20/2019 3:39 PM, Sebastiaan Koppe wrote:

 Do you mean to keep track of ownership/borrowedness manually?


 
 No, that's what the copyctor/opAssign/dtor semantics so.


 This is not true.


I thought as much. Thanks for the confirmation. I am considering 
moving to pointers to benefit from the future semantics. It's 
just that I don't like pointers that much...
Jul 23 2019
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 23.07.19 10:20, Sebastiaan Koppe wrote:

 On Tuesday, 23 July 2019 at 04:02:50 UTC, Timon Gehr wrote:

 On 21.07.19 02:17, Walter Bright wrote:

 On 7/20/2019 3:39 PM, Sebastiaan Koppe wrote:

 Do you mean to keep track of ownership/borrowedness manually?


 No, that's what the copyctor/opAssign/dtor semantics so.


 This is not true.


 
 I thought as much. Thanks for the confirmation. I am considering moving 
 to pointers to benefit from the future semantics. It's just that I don't 
 like pointers that much...


I think tying ownership/borrowing semantics to pointers instead of 
structs makes no sense; it's not necessary and it's not sufficient. Your 
use case illustrates why it is not sufficient.
Jul 23 2019
parent reply Olivier FAURE <couteaubleu gmail.com> writes:
On Tuesday, 23 July 2019 at 14:18:27 UTC, Timon Gehr wrote:

 I think tying ownership/borrowing semantics to pointers instead 
 of structs makes no sense; it's not necessary and it's not 
 sufficient. Your use case illustrates why it is not sufficient.


That's as good a time as any to plug my own proposal draft:

https://gist.github.com/PoignardAzur/9896ddb17b9f6d6f3d0fa5e6fe1a7088

Any thoughts?
Jul 23 2019
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Tuesday, 23 July 2019 at 19:15:44 UTC, Olivier FAURE wrote:

 That's as good a time as any to plug my own proposal draft:

 https://gist.github.com/PoignardAzur/9896ddb17b9f6d6f3d0fa5e6fe1a7088

 Any thoughts?


So in a nutshell, a variable with the unique qualifier ensures 
that there are no other references to that data during the 
lifetime of said variable? In other words, you can only take 1 
addressOf/ref at a time?

Does it prevent more that just use-after-free?

Random idea: what about an  owned destructor? One that only runs 
on the original object, not on any of its copies. Which implies 
scope of course, so that nothing of the copies or its internals 
are allowed to escape.
Jul 23 2019
parent Olivier FAURE <couteaubleu gmail.com> writes:
On Tuesday, 23 July 2019 at 22:01:56 UTC, Sebastiaan Koppe wrote:

 So in a nutshell, a variable with the unique qualifier ensures 
 that there are no other references to that data during the 
 lifetime of said variable? In other words, you can only take 1 
 addressOf/ref at a time?

 Does it prevent more that just use-after-free?


Can you ask again on the Github thread?

I don't want to hijack this thread for 12 pages.

(also, can you expand your question? I don't get what you're 
asking)
Jul 23 2019
prev sibling next sibling parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Saturday, 20 July 2019 at 21:56:50 UTC, Walter Bright wrote:


 We're likely going to add a move constructor, too.


And deprecate opPostMove?
Jul 21 2019
parent Max Haughton <maxhaton gmail.com> writes:
On Sunday, 21 July 2019 at 20:20:10 UTC, Per Nordlöw wrote:

 On Saturday, 20 July 2019 at 21:56:50 UTC, Walter Bright wrote:


 We're likely going to add a move constructor, too.


 And deprecate opPostMove?


Seems likely, especially given that DIP1014 hasn't been 
implemented in the language or made it into the spec yet.
Jul 21 2019
prev sibling parent 12345swordy <alexanderheistermann gmail.com> writes:
On Saturday, 20 July 2019 at 21:56:50 UTC, Walter Bright wrote:

 On 7/20/2019 5:42 AM, Sebastiaan Koppe wrote:

 Anyway my question is whether the ownership and borrowing 
 semantics ever gets applied to non-pointers?


 Currently, no. The programmer would be expected to include the 
 correct copyctor/opAssign/destructors which would take care of 
 it.

 We're likely going to add a move constructor, too.


Make sure that it is a separate pull request from the dip, as c++ 
std::string binding is currently block by the lack of this.
Jul 21 2019
prev sibling next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
Wonderful quote from you Walter!


 The careful reader will notice something peculiar in what I wrote: “as
if”. What do I mean by that weasel wording? Is there some skullduggery going
on? Why yes, there is. Computer languages are full of “as if” dirty deeds
under the hood, like the money you deposit in your bank account isn’t
actually there (I apologize if this is a rude shock to anyone), and this
isn’t any different. Read on!


I copied that over to Oceania's Discord server to talk about (very well 
received).
Jul 15 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/15/2019 8:52 PM, rikki cattermole wrote:

 Wonderful quote from you Walter!
 

 The careful reader will notice something peculiar in what I wrote: “as
if”. 
 What do I mean by that weasel wording? Is there some skullduggery going on? 
 Why yes, there is. Computer languages are full of “as if” dirty deeds
under 
 the hood, like the money you deposit in your bank account isn’t actually
there 
 (I apologize if this is a rude shock to anyone), and this isn’t any
different. 
 Read on!


 
 I copied that over to Oceania's Discord server to talk about (very well
received).


Thanks for letting me know. I enjoyed writing it.
Jul 15 2019
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
Now on the front page of Hacker News at number 9!

https://news.ycombinator.com/news
Jul 15 2019
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/15/2019 7:58 AM, Mike Parker wrote:

 The blog:
 https://dlang.org/blog/2019/07/15/ownership-and-borrowing-in-d/
 
 Reddit:
 https://www.reddit.com/r/programming/comments/cdifbu/ownership_and_borrowing_in_d/


Hacker News:
https://news.ycombinator.com/item?id=20441519
Jul 16 2019
prev sibling next sibling parent Olivier FAURE <couteaubleu gmail.com> writes:
On Monday, 15 July 2019 at 14:58:55 UTC, Mike Parker wrote:

 some folks asked for some information about the bigger picture.


As one of the folks that asked for that information, thank you 
Walter for posting this :)

The system described is a little rough around the edges, but it's 
really helpful to inform the discussion around DIP 1021!
Jul 17 2019
prev sibling parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 7/15/19 10:58 AM, Mike Parker wrote:

 
 The blog:
 https://dlang.org/blog/2019/07/15/ownership-and-borrowing-in-d/
 


Very interesting!

One formatting issue though:
"D has a history of using function attributes to alter the semantics of 
a function&mdash;for example"

Note the "&mdash;".
Jul 18 2019