www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP77 - Fix unsafe RC pass by 'ref'

reply Walter Bright <newshound2 digitalmars.com> writes:
http://wiki.dlang.org/DIP77
Apr 08 2015
next sibling parent "weaselcat" <weaselcat gmail.com> writes:
On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


Deceptively simple solution for a hard problem.
Apr 08 2015
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


That is pretty much the old Rust solution called boxing. This 
sound like the right way forward to me.
Apr 08 2015
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/8/2015 5:30 PM, deadalnix wrote:

 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


 That is pretty much the old Rust solution called boxing. This sound like the
 right way forward to me.


Yes, it's equivalent to it. I had started with doing an INC/DEC pair, but that 
implied adding more logic to detect how to do an INC/DEC for a particular type. 
The logic for doing a copy is much more straightforward. The RC object should 
also be designed in such a way that the copy is not costly.

Function purity is again showing what an advantage it is.

Andrei's idea was to not do the copy for  system opAssign's, thus providing C++ 
equivalence for those folks that need it and don't care about guaranteed memory 
safety.
Apr 08 2015
prev sibling parent reply "Brad Anderson" <eco gnuk.net> writes:
On Thursday, 9 April 2015 at 00:30:46 UTC, deadalnix wrote:

 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


 That is pretty much the old Rust solution called boxing. This 
 sound like the right way forward to me.


Any idea why they abandoned it?
Apr 08 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/8/2015 6:19 PM, Brad Anderson wrote:

 On Thursday, 9 April 2015 at 00:30:46 UTC, deadalnix wrote:

 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


 That is pretty much the old Rust solution called boxing. This sound like the
 right way forward to me.


 Any idea why they abandoned it?


No, but I can guess. It's less efficient.
Apr 08 2015
parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 9 April 2015 at 01:35:14 UTC, Walter Bright wrote:

 On 4/8/2015 6:19 PM, Brad Anderson wrote:

 On Thursday, 9 April 2015 at 00:30:46 UTC, deadalnix wrote:

 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright 
 wrote:

 http://wiki.dlang.org/DIP77


 That is pretty much the old Rust solution called boxing. This 
 sound like the
 right way forward to me.


 Any idea why they abandoned it?


 No, but I can guess. It's less efficient.


This and the fact that they want to prevent multiple writable 
burrow for concurrency reasons, so that seemed like the correct 
way forward for them.
Apr 08 2015
prev sibling next sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


Can this be clarified a little?

The lifetime of tmp will be the same as that of a temporary with 
a destructor.


Does that mean the lifetime of tmp is tied to the scope of the 
function being called, or the current scope calling the function?
Apr 08 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/8/2015 5:49 PM, weaselcat wrote:

 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


 Can this be clarified a little?

 The lifetime of tmp will be the same as that of a temporary with a destructor.


 Does that mean the lifetime of tmp is tied to the scope of the function being
 called, or the current scope calling the function?


The same as that of a tmp being returned from a function - to the end of the 
expression.
Apr 08 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 9 April 2015 at 01:35:54 UTC, Walter Bright wrote:

 The same as that of a tmp being returned from a function - to 
 the end of the expression.


But how will this work with anything that makes RCArrays 
reachable through indirections...? Is the compiler going to do a 
recursive scan and create temporaries of all RC-pointers that are 
reachable?

(e.g. RCArrays of RCArrays)
Apr 09 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/9/2015 12:58 PM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:

 On Thursday, 9 April 2015 at 01:35:54 UTC, Walter Bright wrote:

 The same as that of a tmp being returned from a function - to the end of the
 expression.


 But how will this work with anything that makes RCArrays reachable through
 indirections...? Is the compiler going to do a recursive scan and create
 temporaries of all RC-pointers that are reachable?

 (e.g. RCArrays of RCArrays)



It examines the types of all mutable values available to the function.

Note the discussion of the effect of purity in the DIP.
Apr 09 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 9 April 2015 at 21:47:47 UTC, Walter Bright wrote:

 On 4/9/2015 12:58 PM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=

 indirections...? Is the compiler going to do a recursive scan 
 and create
 temporaries of all RC-pointers that are reachable?

 (e.g. RCArrays of RCArrays)



 It examines the types of all mutable values available to the 
 function.


So if I pass in a graph, linked list or a tree, then it will scan 
the entire structure at runtime and build a large set of 
temporaries before calling the function?
Apr 09 2015
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 10 April 2015 at 05:56:40 UTC, Ola Fosheim Grøstad 
wrote:

 On Thursday, 9 April 2015 at 21:47:47 UTC, Walter Bright wrote:

 On 4/9/2015 12:58 PM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=

 indirections...? Is the compiler going to do a recursive scan 
 and create
 temporaries of all RC-pointers that are reachable?

 (e.g. RCArrays of RCArrays)



 It examines the types of all mutable values available to the 
 function.


 So if I pass in a graph, linked list or a tree, then it will 
 scan the entire structure at runtime and build a large set of 
 temporaries before calling the function?


No, but indeed, passing down indirection require boxing too. The 
DIP is not clear about it.
Apr 09 2015
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 10 April 2015 at 06:02:35 UTC, deadalnix wrote:

 No, but indeed, passing down indirection require boxing too. 
 The DIP is not clear about it.


Not sure what you mean by boxing. Usually it means that you wrap 
a value type in an object to turn it into a reference type. In 
Rust I think it means the same as unique_ptr ( 
http://rustbyexample.com/box.html ).

How will DIP77 work out if you represent a directed acyclic graph 
with ref counted edges? (e.g. one RCArray  per node)

Basically the problem that needs to be solved is very close to 
determining if there is no aliasing between parameters. The 
compile time solution could be either whole program pointer 
analysis or heavy type constraints  (linear typing). A general 
runtime solution would be rather inefficient IMO.

I think memory safety in D probably could be better solved, and 
with less effort, by a separate tool if DMD could be made to 
generate high level IR (a D VM that preserves high level 
information better than say LLVM).
Apr 10 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/9/2015 10:56 PM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:

 So if I pass in a graph, linked list or a tree, then it will scan the entire
 structure at runtime and build a large set of temporaries before calling the
 function?



Nope. It'll scan at compile time the types reachable through the parameters. If 
any of those types are RCO's that have a type that matches a ref parameter, the 
ref parameter's RCO will get copied.
Apr 10 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 10 April 2015 at 07:01:47 UTC, Walter Bright wrote:

 Nope. It'll scan at compile time the types reachable through 
 the parameters. If any of those types are RCO's that have a 
 type that matches a ref parameter, the ref parameter's RCO will 
 get copied.


But how can this work? If I provide 2 ref parameters to different 
Node instances (not a RCO) in the same graph. How can I then be 
sure that it will not disappear?

E.g.

ref1 -> Node->RCRef->Edge(RCO)->Node->RCRef->Edge(RCO)->Node"X"
ref2 -> Node"X"
Apr 10 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 12:10 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:

 On Friday, 10 April 2015 at 07:01:47 UTC, Walter Bright wrote:

 Nope. It'll scan at compile time the types reachable through the parameters.
 If any of those types are RCO's that have a type that matches a ref parameter,
 the ref parameter's RCO will get copied.


 But how can this work? If I provide 2 ref parameters to different Node
instances
 (not a RCO) in the same graph. How can I then be sure that it will not
disappear?

 E.g.

 ref1 -> Node->RCRef->Edge(RCO)->Node->RCRef->Edge(RCO)->Node"X"
 ref2 -> Node"X"


Because with two parameters, if one is a ref coming from an RCO rc1, and the 
other is a ref to a type that is the root of a graph that contains an RCO type, 
then the rc1 is copied.
Apr 10 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 10 April 2015 at 08:47:09 UTC, Walter Bright wrote:

 On 4/10/2015 12:10 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=

 ref1 -> Node->RCRef->Edge(RCO)->Node->RCRef->Edge(RCO)->Node"X"
 ref2 -> Node"X"


 Because with two parameters, if one is a ref coming from an RCO 
 rc1, and the other is a ref to a type that is the root of a 
 graph that contains an RCO type, then the rc1 is copied.


ref1 -> RCO1 -> N1 -> RCRef1 -> RCO2 -> N2 -> RCRef3 -> RCO3 -> N3
ref2 -> N3

N1..N3 represent resources (like a file).

process(ref1,ref2){
     ref1.stuff.stuff.stuff.release(); // Node2 and Node3 is 
released.
     ref2.read(); // fails/garbage
}
Apr 10 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 2:14 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:

 On Friday, 10 April 2015 at 08:47:09 UTC, Walter Bright wrote:

 On 4/10/2015 12:10 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=

 ref1 -> Node->RCRef->Edge(RCO)->Node->RCRef->Edge(RCO)->Node"X"
 ref2 -> Node"X"


 Because with two parameters, if one is a ref coming from an RCO rc1, and the
 other is a ref to a type that is the root of a graph that contains an RCO
 type, then the rc1 is copied.


 ref1 -> RCO1 -> N1 -> RCRef1 -> RCO2 -> N2 -> RCRef3 -> RCO3 -> N3
 ref2 -> N3

 N1..N3 represent resources (like a file).

 process(ref1,ref2){
      ref1.stuff.stuff.stuff.release(); // Node2 and Node3 is released.
      ref2.read(); // fails/garbage
 }



ref2 is copied.
Apr 10 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 10 April 2015 at 09:15:11 UTC, Walter Bright wrote:

 ref2 is copied.


You cannot copy a resource like a memfile. It is already closed 
and the memory is gone.
Apr 10 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 2:27 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:

 On Friday, 10 April 2015 at 09:15:11 UTC, Walter Bright wrote:

 ref2 is copied.


 You cannot copy a resource like a memfile. It is already closed and the memory
 is gone.


Please read the DIP again. The copy happens earlier.
Apr 10 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 10 April 2015 at 09:28:57 UTC, Walter Bright wrote:

 Please read the DIP again. The copy happens earlier.


But one cannot keep a stateful resource alive by copying memory. 
One would have to take ownership to keep it alive. In order to do 
that you need a model for ownership.
Apr 10 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 2:54 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:

 On Friday, 10 April 2015 at 09:28:57 UTC, Walter Bright wrote:

 Please read the DIP again. The copy happens earlier.


 But one cannot keep a stateful resource alive by copying memory. One would have
 to take ownership to keep it alive. In order to do that you need a model for
 ownership.


That's what ref counting is about.
Apr 10 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 10 April 2015 at 10:28:03 UTC, Walter Bright wrote:

 That's what ref counting is about.


That would require pervasive ref-counting.
Apr 10 2015
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Fri, 10 Apr 2015 14:04:39 +0000
schrieb "Ola Fosheim Gr=C3=B8stad"
<ola.fosheim.grostad+dlang gmail.com>:


 On Friday, 10 April 2015 at 10:28:03 UTC, Walter Bright wrote:

 That's what ref counting is about.


=20
 That would require pervasive ref-counting.


You make is sound like there was an alternative to ref
counting for resources. Is there?

--=20
Marco
Apr 11 2015
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 11 April 2015 at 18:20:51 UTC, Marco Leise wrote:

 Am Fri, 10 Apr 2015 14:04:39 +0000
 schrieb "Ola Fosheim Grøstad"
 <ola.fosheim.grostad+dlang gmail.com>:


 On Friday, 10 April 2015 at 10:28:03 UTC, Walter Bright wrote:

 That's what ref counting is about.


 
 That would require pervasive ref-counting.


 You make is sound like there was an alternative to ref
 counting for resources. Is there?


Ownership, GC.
Apr 11 2015
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sat, 11 Apr 2015 20:42:52 +0000
schrieb "deadalnix" <deadalnix gmail.com>:


 On Saturday, 11 April 2015 at 18:20:51 UTC, Marco Leise wrote:

 Am Fri, 10 Apr 2015 14:04:39 +0000
 schrieb "Ola Fosheim Gr=C3=B8stad"
 <ola.fosheim.grostad+dlang gmail.com>:


 On Friday, 10 April 2015 at 10:28:03 UTC, Walter Bright wrote:

 That's what ref counting is about.


=20
 That would require pervasive ref-counting.


 You make is sound like there was an alternative to ref
 counting for resources. Is there?


=20
 Ownership, GC.


That doesn't convince me yet. GC is just incorrect to use here.
D's GC like any other that is non-deterministic will hold on
to many external resources for longer than is feasible.

(Unique) ownership on the other hand isn't really what we are
looking at here, right? I mean, the DIP talks about what
happens when you are dealing with multiple references to
the same item.

That leaves us with RC, which I think is a good choice.

--=20
Marco
May 01 2015
prev sibling next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


I've only skimmed this, but:

1. I don't think Objective-C uses autorelease pools for any other 
reason than to make manual ref counting easier.

See also John McCall's reflections on ARC:

http://forum.dlang.org/thread/mcqcor$aa$1 digitalmars.com?page=2#post-hgmhgirfervrsvcghchw:40forum.dlang.org

2. How will this work with "yield"?

Why not just implement the more generic solution (shared pointers 
with move/borrow or WPO) ?
Apr 09 2015
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 9 April 2015 at 09:05:10 UTC, Ola Fosheim Grøstad 
wrote:

 2. How will this work with "yield"?


You yield both caller and callee, so you'll get caller's boxing 
in the yield.


 Why not just implement the more generic solution (shared 
 pointers with move/borrow or WPO) ?


I don't think this is possible.
Apr 09 2015
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 9 April 2015 at 18:31:24 UTC, deadalnix wrote:

 On Thursday, 9 April 2015 at 09:05:10 UTC, Ola Fosheim Grøstad 
 wrote:

 2. How will this work with "yield"?


 You yield both caller and callee, so you'll get caller's boxing 
 in the yield.


But the coroutine stack and everything on it will be intact when 
it yields, including references to array elements...?


 Why not just implement the more generic solution (shared 
 pointers with move/borrow or WPO) ?


 I don't think this is possible.


It should be possible with pointer analysis, but the easier 
approach is just to ban non-const ref parameters (c++ style) for 
rc-pointer-objects, so maybe D should provide "head const" after 
all...
Apr 09 2015
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2015-04-08 23:10:37 +0000, Walter Bright <newshound2 digitalmars.com> said:


 http://wiki.dlang.org/DIP77


In the definition of a Reference Counted Object:

"""
An object is assumed to be reference counted if it has a postblit and a 
destructor, and does not have an opAssign marked  system.
"""

Why should it not have an opAssign marked  system?

And what happens if the struct has a postblit but it is  disabled? Will 
the compiler forbid you from passing it by ref in cases where it'd need 
to make a copy, or will it just not be a RCO?

More generally, is it right to add implicit copying just because a 
struct has a postblit and a destructor? If someone implemented a 
by-value container in D (such as those found in C++), this behaviour of 
the compiler would trash the performance by silently doing useless 
unnecessary copies. You won't even get memory-safety as a benefit: if 
the container allocates from the GC it's safe anyway, otherwise you're 
referencing deallocated memory with your ref parameter (copying the 
struct would just make a copy elsewhere, not retain the memory of the 
original).

I think you're assuming too much from the presence of a postblit and a 
destructor. This implicit copy behaviour should not be trigged by 
seemingly unrelated clues. Instead of doing that:

	auto tmp = rc;

the compiler should insert this:

	auto tmp = rc.opPin();

RCArray can implement opPin by returning a copy of itself. A by-value 
container can implement opPin by returning a dummy struct that retains 
the container's memory until the dummy struct's destructor is called. 
Alternatively someone could make a dummy "void opPin()  system {}" to 
signal it isn't safe to pass internal references around (only in system 
code would the implicit call to opPin compile). If you were writing a 
layout-compatible D version of std::vector, you'd likely have to use a 
 system opPin because there's no way you can "pin" that memory and 
guaranty memory-safety when passing references around.

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca
Apr 09 2015
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/9/2015 5:05 AM, Michel Fortin wrote:

 Why should it not have an opAssign marked  system?


"Andrei's idea was to not do the copy for  system opAssign's, thus providing
C++ 
equivalence for those folks that need it and don't care about guaranteed memory 
safety."



 And what happens if the struct has a postblit but it is  disabled? Will the
 compiler forbid you from passing it by ref in cases where it'd need to make a
 copy, or will it just not be a RCO?


It wouldn't be an RCO.



 More generally, is it right to add implicit copying just because a struct has a
 postblit and a destructor? If someone implemented a by-value container in D
 (such as those found in C++), this behaviour of the compiler would trash the
 performance by silently doing useless unnecessary copies. You won't even get
 memory-safety as a benefit: if the container allocates from the GC it's safe
 anyway, otherwise you're referencing deallocated memory with your ref parameter
 (copying the struct would just make a copy elsewhere, not retain the memory of
 the original).


The only real purpose to a postblit is to support ref counting. Why would a 
by-value container use a postblit and not ref count?



 I think you're assuming too much from the presence of a postblit and a
 destructor. This implicit copy behaviour should not be trigged by seemingly
 unrelated clues. Instead of doing that:

      auto tmp = rc;

 the compiler should insert this:

      auto tmp = rc.opPin();

 RCArray can implement opPin by returning a copy of itself. A by-value container
 can implement opPin by returning a dummy struct that retains the container's
 memory until the dummy struct's destructor is called. Alternatively someone
 could make a dummy "void opPin()  system {}" to signal it isn't safe to pass
 internal references around (only in system code would the implicit call to
opPin
 compile). If you were writing a layout-compatible D version of std::vector,
 you'd likely have to use a  system opPin because there's no way you can "pin"
 that memory and guaranty memory-safety when passing references around.


My first impression is that's too complicated for the user to get right.
Apr 09 2015
next sibling parent reply "w0rp" <devw0rp gmail.com> writes:
On Thursday, 9 April 2015 at 18:44:10 UTC, Walter Bright wrote:

 The only real purpose to a postblit is to support ref counting. 
 Why would a by-value container use a postblit and not ref count?


A struct could have a postblit defined if you are implementing 
something like std::vector, where you you copy the memory when 
the struct is copied. I'm not sure why you would want to do such 
a thing in D, though. If allocating memory is your concern, you 
probably don't want any allocation, including malloc.
Apr 09 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/9/2015 11:53 AM, w0rp wrote:

 On Thursday, 9 April 2015 at 18:44:10 UTC, Walter Bright wrote:

 The only real purpose to a postblit is to support ref counting. Why would a
 by-value container use a postblit and not ref count?


 A struct could have a postblit defined if you are implementing something like
 std::vector, where you you copy the memory when the struct is copied. I'm not
 sure why you would want to do such a thing in D, though.


I'm not sure why you'd do that, either. Just make the memory part ref counted, 
then when modifying it, make the copy then if your ref count > 1.

If you want to interface with std::vector, make opAssign  system, after all, 
you're dealing with C++ :-)
Apr 09 2015
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 9 April 2015 at 18:44:10 UTC, Walter Bright wrote:

 On 4/9/2015 5:05 AM, Michel Fortin wrote:

 Why should it not have an opAssign marked  system?


 "Andrei's idea was to not do the copy for  system opAssign's, 
 thus providing C++ equivalence for those folks that need it and 
 don't care about guaranteed memory safety."


Why not bind this behavior to extern(C++) ?


 The only real purpose to a postblit is to support ref counting. 
 Why would a by-value container use a postblit and not ref count?


In C++, the only real purpose of template were generic 
containers, and that didn't ended up well.

It is dangerous, at language level, to reason from usage instead 
of first principle (not that usage do not matter, usage should 
serve as a guideline for the principles) or thing like C++ 
templates happens.


 My first impression is that's too complicated for the user to 
 get right.


Yeah, I don't think opPin is a the right way to go. It is always 
easy and tempting to add new stuff to make X or Y work, but at 
the end, it only create language complexity explosion.

There are some inefficiencies involved here, but I trust the 
compiler to be able to optimize it away for the most part, and we 
have a backdoor for thoses who want to bypass safety.

More generally, this is why I oppose the return attribute + 
adding op on reference type in favor of a principle scope 
proposal. The first one is 2 language features for a less general 
end result.
Apr 09 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/9/2015 1:21 PM, deadalnix wrote:

 On Thursday, 9 April 2015 at 18:44:10 UTC, Walter Bright wrote:

 On 4/9/2015 5:05 AM, Michel Fortin wrote:

 Why should it not have an opAssign marked  system?


 "Andrei's idea was to not do the copy for  system opAssign's, thus providing
 C++ equivalence for those folks that need it and don't care about guaranteed
 memory safety."


 Why not bind this behavior to extern(C++) ?


Because the charter of  system is "the user supplies the memory safety", which 
fits here perfectly. extern(C++) carries a lot of other stuff with it.



 It is dangerous, at language level, to reason from usage instead of first
 principle (not that usage do not matter, usage should serve as a guideline for
 the principles) or thing like C++ templates happens.


On the other hand, throwing things in the language just because you can, with
no 
idea what they are good for or how they will be used, doesn't end well.



 My first impression is that's too complicated for the user to get right.


 Yeah, I don't think opPin is a the right way to go. It is always easy and
 tempting to add new stuff to make X or Y work, but at the end, it only create
 language complexity explosion.


Glad you agree. I'm afraid of making something that is technically correct, but 
unusable.



 There are some inefficiencies involved here, but I trust the compiler to be
able
 to optimize it away for the most part, and we have a backdoor for thoses who
 want to bypass safety.


Yes. Interestingly, this is a superset of what Rust does. Rust's has the idea
of 
only one mutable reference at a time, for efficiency. As the DIP points out, if 
the function only has const references available other than the one mutable
one, 
it doesn't do the copy, either.

I was curious how Rust handled the global variable issue. Turns out it doesn't
- 
mutable global variables are marked as "unsafe" and the checker gives up on it.



 More generally, this is why I oppose the return attribute + adding op on
 reference type in favor of a principle scope proposal. The first one is 2
 language features for a less general end result.


I understand where you're coming from on this. The design was picked for being 
as simple as possible with the least disruption.
Apr 09 2015
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 9 April 2015 at 22:06:01 UTC, Walter Bright wrote:

 I was curious how Rust handled the global variable issue. Turns 
 out it doesn't - mutable global variables are marked as 
 "unsafe" and the checker gives up on it.


I think it's safe to say that if even Rust with its linear type 
system and advanced borrow checker cannot deal with it, noone 
can, short of whole program analysis. And it's easy to understand 
why once you realize that it's all about aliasing. That's why I 
also suggest to make borrowing from mutable globals unsafe.
Apr 10 2015
prev sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2015-04-09 18:43:25 +0000, Walter Bright <newshound2 digitalmars.com> said:


 The only real purpose to a postblit is to support ref counting. Why 
 would a by-value container use a postblit and not ref count?


Because reference counting has tradoffs, or because someone translates 
an exiting C++ program and want to keep the semantics the same to make 
the translation simpler. If you want to limit postblit to the reference 
counting use case I guess that's fine, although this limitation should 
probably be clarified somewhere because to someone with a C++ 
background it looks a lot like a substitute for a copy-constructor.

But this goes beyond containers. Destructors and postblits are 
implicitly created whenever a field contained within a struct has them. 
So you could have this:

	struct Big
	{
		RCArray!T array;
		int[1000] ints;
	}

and now implicit copies of Big are made because Big implicitly gained a 
destructor and postblit from its array field. But at least you gain 
memory safety.

...or maybe not. Let's add another field to Big:

	struct Handle
	{
		// shouldn't be able to make copies of this
		 disable this(this) {}
	}

	struct Big
	{
		RCArray!T array;
		int[1000] ints;
		Handle handle;
	}

Now Big's implicit postblit becomes disabled too (because Handle can't 
handle copying), therefore it is no longer a RCO object and the 
compiler will no longer create temporary copies. Now you've trashed 
memory-safety.

To be fair, my opPin idea suffers a similar fate here: there is no 
opPin defined for Big, which creates a hole in memory-safety. To 
compensate, the compiler would have to call opPin individually for each 
field that defines it.

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca
Apr 10 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 4:49 AM, Michel Fortin wrote:

 ...or maybe not. Let's add another field to Big:

      struct Handle
      {
          // shouldn't be able to make copies of this
           disable this(this) {}
      }

      struct Big
      {
          RCArray!T array;
          int[1000] ints;
          Handle handle;
      }

 Now Big's implicit postblit becomes disabled too (because Handle can't handle
 copying), therefore it is no longer a RCO object and the compiler will no
longer
 create temporary copies. Now you've trashed memory-safety.


The copy would be made of array, not Big.
Apr 10 2015
prev sibling parent "Martin Nowak" <code dawg.eu> writes:
On Thursday, 9 April 2015 at 12:05:16 UTC, Michel Fortin wrote:

 """
 An object is assumed to be reference counted if it has a 
 postblit and a destructor, and does not have an opAssign marked 
  system.
 """


It seems to only fit to RCO, I'm not positive though.
We should formalize this, by documenting it and by adding isRCO 
traits to phobos.
Apr 10 2015
prev sibling next sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


 From the DIP:
T foo(ref Payload payload);
...
RC rc;
foo(rc.payload);

rewrite foo(rc) as:

auto tmp = rc;
foo(rc.payload);

The only question I had was whether there was an advantage to 
rewriting the statement in its own scope:
{
   auto tmp = rc;
   foo(rc.payload);
}

You can save some stack space that way, but I don't know how hard 
it is to implement.
Apr 09 2015
prev sibling next sibling parent reply "ixid" <adamsibson hotmail.com> writes:
 http://wiki.dlang.org/DIP77


We seem to be incrementally edging towards Rust's memory 
management system or something similar and retracing their steps 
along the path to get there. Hypothetically how close could we 
get to being able to opt into a Rust world from the GC world or C 
worlds of memory management?
Apr 10 2015
next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Friday, 10 April 2015 at 09:16:38 UTC, ixid wrote:

 http://wiki.dlang.org/DIP77


 We seem to be incrementally edging towards Rust's memory 
 management system or something similar and retracing their 
 steps along the path to get there. Hypothetically how close 
 could we get to being able to opt into a Rust world from the GC 
 world or C worlds of memory management?


extern(Rust) embraceExtendExtinguish();
Apr 10 2015
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 2:16 AM, ixid wrote:

 http://wiki.dlang.org/DIP77


 We seem to be incrementally edging towards Rust's memory management system or
 something similar and retracing their steps along the path to get there.
 Hypothetically how close could we get to being able to opt into a Rust world
 from the GC world or C worlds of memory management?


As I explained to deadalnix in this thread, D's semantics in this regard are a 
superset of Rust's.
Apr 10 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/10/15 2:16 AM, ixid wrote:

 http://wiki.dlang.org/DIP77


 We seem to be incrementally edging towards Rust's memory management
 system or something similar and retracing their steps along the path to
 get there. Hypothetically how close could we get to being able to opt
 into a Rust world from the GC world or C worlds of memory management?


We hope to do better than Rust. -- Andrei
Apr 10 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 10 April 2015 at 15:44:32 UTC, Andrei Alexandrescu 
wrote:

 We hope to do better than Rust. -- Andrei


Hope does not cut it.

A design that breaks down even on a linked list (e.g. you have to 
turn to ref-counted pointers everywhere) and that makes 
findByRef(container,node) break should not have made it to a DIP 
in the first place.

D needs clean semantics on what is a value. The only sensible 
definition for a copyable value is that you cannot take the 
identity in _any way_ from a value or anything that can be 
reached from it.

D needs clean semantics on what is unique ownership. Ref counting 
is not it. Pervasive ref counting requires WPO to work out ok.

C++14 with sanitizer tooling is a lot more attractive than this 
direction you are going for.
Apr 11 2015
next sibling parent "w0rp" <devw0rp gmail.com> writes:
I was thinking about this again, and how the first 'ref' would 
create a copy in order to keep the object around. At first I 
thought I didn't like that, but then I realised that it's 
actually not far from what I wanted for taking r-values by 
reference. I commonly write functions which take values either by 
reference or by value in this way.

void foo(ref T value);
void foo(T value) { foo(value); }

Which obviously results in a combinatorial explosion of 
overloads, which you can write automatically some times with 
'auto ref'.

My question is, in the greater context of the language, supposing 
we had this copying behaviour for a first ref for reference 
counted objects, will it putting something in the language for 
copying r-values into functions, or will it provide a path to 
implement such a thing?
Apr 11 2015
prev sibling parent "weaselcat" <weaselcat gmail.com> writes:
On Saturday, 11 April 2015 at 08:35:18 UTC, Ola Fosheim Grøstad 
wrote:

 On Friday, 10 April 2015 at 15:44:32 UTC, Andrei Alexandrescu 
 wrote:

 We hope to do better than Rust. -- Andrei


 Hope does not cut it.

 A design that breaks down even on a linked list (e.g. you have 
 to turn to ref-counted pointers everywhere) and that makes 
 findByRef(container,node) break should not have made it to a 
 DIP in the first place.


a bidirectional linked list cannot be implemented in safe rust 
afaik
Apr 11 2015
prev sibling next sibling parent reply "Martin Nowak" <code dawg.eu> writes:
On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


So someone passes an RCO via ref to avoid the inc/dec, and 
because that imposes safety issues we turn it into some sort of 
pass by value under the hood, defeating the purpose, and provide 
an opt-out via  system opAssign.

Wouldn't it more straightforward to make pass-by-ref unsafe 
( system) for RCOs?

Then the only thing missing to make this equally powerful, would 
be an optimization opportunity for the compiler to elide copies 
of pass-by-value RCOs, e.g. it could avoid calling the postblit 
when the function retains the refcount.
Apr 10 2015
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 3:02 AM, Martin Nowak wrote:

 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


 So someone passes an RCO via ref to avoid the inc/dec, and because that imposes
 safety issues we turn it into some sort of pass by value under the hood,
 defeating the purpose, and provide an opt-out via  system opAssign.


Or you could pass it by const ref (which is what Rust essentially does).



 Wouldn't it more straightforward to make pass-by-ref unsafe ( system) for RCOs?


That's what we have now. It's not good enough.
Apr 10 2015
parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 04/10/2015 12:29 PM, Walter Bright wrote:

 So someone passes an RCO via ref to avoid the inc/dec, and because
 that imposes
 safety issues we turn it into some sort of pass by value under the hood,
 defeating the purpose, and provide an opt-out via  system opAssign.


 
 Or you could pass it by const ref (which is what Rust essentially does).


Maybe I'm missing something, but this proposal seems to make `ref RCO`
fairly useless, because it creates a copy anyhow.


 Wouldn't it more straightforward to make pass-by-ref unsafe ( system)
 for RCOs?


 
 That's what we have now. It's not good enough.


Assigning a RefCounted is marked  system, pass-by-ref is  safe.
What's missing, you want to be able to use RefCounted in  safe code? Why
not pass it by value then? That would pin the object and you could elide
the additional inc/dec just like you propose to elide the temporary copies.
It's more efficient to pass a smart pointer like RefCounted by-value anyhow.
Apr 10 2015
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 2:20 PM, Martin Nowak wrote:

 On 04/10/2015 12:29 PM, Walter Bright wrote:

 So someone passes an RCO via ref to avoid the inc/dec, and because
 that imposes
 safety issues we turn it into some sort of pass by value under the hood,
 defeating the purpose, and provide an opt-out via  system opAssign.


 Or you could pass it by const ref (which is what Rust essentially does).


 Maybe I'm missing something, but this proposal seems to make `ref RCO`
 fairly useless, because it creates a copy anyhow.


Not if you read the copy elision section.



 Assigning a RefCounted is marked  system,


That's kind of throwing in the towel, isn't it?


 pass-by-ref is  safe.
 What's missing, you want to be able to use RefCounted in  safe code? Why
 not pass it by value then? That would pin the object and you could elide
 the additional inc/dec just like you propose to elide the temporary copies.
 It's more efficient to pass a smart pointer like RefCounted by-value anyhow.
Apr 10 2015
prev sibling parent "weaselcat" <weaselcat gmail.com> writes:
On Friday, 10 April 2015 at 21:20:32 UTC, Martin Nowak wrote:

 Assigning a RefCounted is marked  system, pass-by-ref is  safe.
 What's missing, you want to be able to use RefCounted in  safe 
 code? Why
 not pass it by value then?


This actually sounds quite logical to me - passing an RC object 
by ref doesn't sound very safe to begin with.
Apr 10 2015
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 10 April 2015 at 10:02:01 UTC, Martin Nowak wrote:

 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


 So someone passes an RCO via ref to avoid the inc/dec, and 
 because that imposes safety issues we turn it into some sort of 
 pass by value under the hood, defeating the purpose, and 
 provide an opt-out via  system opAssign.

 Wouldn't it more straightforward to make pass-by-ref unsafe 
 ( system) for RCOs?

 Then the only thing missing to make this equally powerful, 
 would be an optimization opportunity for the compiler to elide 
 copies of pass-by-value RCOs, e.g. it could avoid calling the 
 postblit when the function retains the refcount.


Only the first pass by ref create a copy. You can then pass the 
ref down all you want without copy.

That is an acceptable cost IMO.
Apr 10 2015
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 10 April 2015 at 23:12:55 UTC, deadalnix wrote:

 On Friday, 10 April 2015 at 10:02:01 UTC, Martin Nowak wrote:

 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright 
 wrote:

 http://wiki.dlang.org/DIP77


 So someone passes an RCO via ref to avoid the inc/dec, and 
 because that imposes safety issues we turn it into some sort 
 of pass by value under the hood, defeating the purpose, and 
 provide an opt-out via  system opAssign.

 Wouldn't it more straightforward to make pass-by-ref unsafe 
 ( system) for RCOs?

 Then the only thing missing to make this equally powerful, 
 would be an optimization opportunity for the compiler to elide 
 copies of pass-by-value RCOs, e.g. it could avoid calling the 
 postblit when the function retains the refcount.


 Only the first pass by ref create a copy. You can then pass the 
 ref down all you want without copy.

 That is an acceptable cost IMO.


It's not acceptable that it happens behind the user's back. 
Costly operations must be explicit.
Apr 11 2015
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/11/2015 2:28 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" wrote:

 It's not acceptable that it happens behind the user's back. Costly operations
 must be explicit.


Don't know of a better solution.
Apr 11 2015
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 11 April 2015 at 09:41:07 UTC, Walter Bright wrote:

 On 4/11/2015 2:28 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:

 It's not acceptable that it happens behind the user's back. 
 Costly operations
 must be explicit.


 Don't know of a better solution.


How about this?



Btw, I also made other changes: No implied scope for  safe 
functions, no overloading on scope (instead postblit and 
destructor are skipped), and added a terminology section (rather 
important!).
Apr 11 2015
next sibling parent reply "matovitch" <camille.brugel laposte.net> writes:
On Saturday, 11 April 2015 at 11:33:51 UTC, Marc Schütz wrote:

 On Saturday, 11 April 2015 at 09:41:07 UTC, Walter Bright wrote:

 On 4/11/2015 2:28 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:

 It's not acceptable that it happens behind the user's back. 
 Costly operations
 must be explicit.


 Don't know of a better solution.


 How about this?



 Btw, I also made other changes: No implied scope for  safe 
 functions, no overloading on scope (instead postblit and 
 destructor are skipped), and added a terminology section 
 (rather important!).


Just passing, a bit off topic and clearly not familiar enough 
with the discussed subject but in case you missed it the last 
rust blog post is an nice and motivated introduction to their 
ownership system :

http://blog.rust-lang.org/2015/04/10/Fearless-Concurrency.html

I need to read about Marc's scope proposal...I am not convinced 
by the dip77, what about raii managed classes with costly 
opAssign operator (like in w0rn example) ? The goal of passing an 
object by reference is to avoid the copy and here you guess the 
parameter is ref-counted and made a (pseudo-)copy ? Then any 
allocating opAssign should be marked  system ?
Apr 11 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/11/15 5:35 AM, matovitch wrote:

 I am not convinced by the dip77, what about raii managed classes with
 costly opAssign operator (like in w0rn example) ? The goal of passing an
 object by reference is to avoid the copy and here you guess the
 parameter is ref-counted and made a (pseudo-)copy ? Then any allocating
 opAssign should be marked  system ?


Expensive opAssign or expensive postblit? -- Andrei
Apr 11 2015
parent "matovitch" <camille.brugel laposte.net> writes:
On Sunday, 12 April 2015 at 04:18:54 UTC, Andrei Alexandrescu 
wrote:

 On 4/11/15 5:35 AM, matovitch wrote:

 I am not convinced by the dip77, what about raii managed 
 classes with
 costly opAssign operator (like in w0rn example) ? The goal of 
 passing an
 object by reference is to avoid the copy and here you guess the
 parameter is ref-counted and made a (pseudo-)copy ? Then any 
 allocating
 opAssign should be marked  system ?


 Expensive opAssign or expensive postblit? -- Andrei


Well both in the example given by w0rp. I guess D as no "rule of 
five" but still a "rule of three" (which is in fact rule of four 
with the copy-and-swap idiom) about coding posblit, opAssign and 
destructor together ?
Apr 12 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/11/2015 4:33 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" wrote:

 On Saturday, 11 April 2015 at 09:41:07 UTC, Walter Bright wrote:

 On 4/11/2015 2:28 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" wrote:

 It's not acceptable that it happens behind the user's back. Costly operations
 must be explicit.


 Don't know of a better solution.


 How about this?



 Btw, I also made other changes: No implied scope for  safe functions, no
 overloading on scope (instead postblit and destructor are skipped), and added a
 terminology section (rather important!).


A quick read of this suggests it is doing the Rust model of only one mutable 
reference at a time. Is this really viable with D?
Apr 12 2015
next sibling parent "matovitch" <camille.brugel laposte.net> writes:
On Sunday, 12 April 2015 at 07:17:07 UTC, Walter Bright wrote:

 A quick read of this suggests it is doing the Rust model of 
 only one mutable reference at a time. Is this really viable 
 with D?


I would love to see a storage class to add the rust ownership 
model to D. I surely don't know if it's feasible.
Apr 12 2015
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Sunday, 12 April 2015 at 07:17:07 UTC, Walter Bright wrote:

 On 4/11/2015 4:33 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:




 A quick read of this suggests it is doing the Rust model of 
 only one mutable reference at a time. Is this really viable 
 with D?


I think it is. You have to keep in mind that this is opt-in; it 
only applies to `scope` variables. I would agree that as a 
default, it wouldn't fit D at all. I think that, for all its 
usefulness, it'd be a tad too limiting for my taste if I were 
forced to use it everywhere. In Rust, it needs to be ubiquitous, 
because their goal is to get by without a GC. We on the other 
hand just want a way to make the GC easier to avoid. We also want 
to be able to use these different programming styles side by 
side, therefore an opt-in solution is good.

Besides, I too want to recommend the article Matovitch linked to, 
and Yehuda Katz' article mentioned therein, because they make a 
good point demonstrating that it gives more correctness 
guarantees than just memory safety:

http://forum.dlang.org/post/ttjzihuffrdlgqonqiwi forum.dlang.org

It's a great building block for lots of nice things. In fact, 
it's quite likely the only way to get these guarantees once you 
look beyond reference counting.
Apr 12 2015
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Marc Schütz:


 You have to keep in mind that this is opt-in; it only applies 
 to `scope` variables. I would agree that as a default, it 
 wouldn't fit D at all. I think that, for all its usefulness, 
 it'd be a tad too limiting for my taste if I were forced to use 
 it everywhere.


I think D Zen asks for safety on default and opt-out on request.

See also " safe by default":
https://issues.dlang.org/show_bug.cgi?id=13838

Bye,
bearophile
Apr 12 2015
parent "Martin Nowak" <code dawg.eu> writes:
On Sunday, 12 April 2015 at 12:06:50 UTC, bearophile wrote:

 I think D Zen asks for safety on default and opt-out on request.


Marc talked about unique ownership vs. shared ownership, not 
about safe vs. unsafe.
Apr 12 2015
prev sibling parent reply "Martin Nowak" <code dawg.eu> writes:
On Sunday, 12 April 2015 at 07:17:07 UTC, Walter Bright wrote:

 On 4/11/2015 4:33 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:

 On Saturday, 11 April 2015 at 09:41:07 UTC, Walter Bright 
 wrote:


 A quick read of this suggests it is doing the Rust model of 
 only one mutable reference at a time. Is this really viable 
 with D?


It's a very good ownership model IMO, because it has no runtime 
overhead and often sharing/escaping isn't needed. Would at least 
be interesting to explore this in more detail for scoped 
variables.

As a side note, I find that unique ownership is strangely missing 
from our discussions, yet it's a perfect ownership model for 
resources such as a File.
https://github.com/D-Programming-Language/phobos/pull/3171
Apr 12 2015
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Sunday, 12 April 2015 at 13:12:05 UTC, Martin Nowak wrote:

 As a side note, I find that unique ownership is strangely 
 missing from our discussions, yet it's a perfect ownership 
 model for resources such as a File.
 https://github.com/D-Programming-Language/phobos/pull/3171


I'm aware of it, but didn't want to open another can of worms. I 
have a vague idea what to do about it, based on deadalnix' 
"isolated" idea and the already existing uniqueness concept. I 
need to make up my mind about moving though, whether the benefits 
of making the compiler know about it are worth the added 
complexity.
Apr 12 2015
prev sibling next sibling parent reply "w0rp" <devw0rp gmail.com> writes:
On Saturday, 11 April 2015 at 09:28:46 UTC, Marc Schütz wrote:

 On Friday, 10 April 2015 at 23:12:55 UTC, deadalnix wrote:

 On Friday, 10 April 2015 at 10:02:01 UTC, Martin Nowak wrote:

 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright 
 wrote:

 http://wiki.dlang.org/DIP77


 So someone passes an RCO via ref to avoid the inc/dec, and 
 because that imposes safety issues we turn it into some sort 
 of pass by value under the hood, defeating the purpose, and 
 provide an opt-out via  system opAssign.

 Wouldn't it more straightforward to make pass-by-ref unsafe 
 ( system) for RCOs?

 Then the only thing missing to make this equally powerful, 
 would be an optimization opportunity for the compiler to 
 elide copies of pass-by-value RCOs, e.g. it could avoid 
 calling the postblit when the function retains the refcount.


 Only the first pass by ref create a copy. You can then pass 
 the ref down all you want without copy.

 That is an acceptable cost IMO.


 It's not acceptable that it happens behind the user's back. 
 Costly operations must be explicit.


If we somehow know for a fact that the object really is a 
reference counted object, then the cost should be acceptable. 
Running the postblit will consist of incrementing a reference 
count and nothing else. There's no other way I can think of that 
permits passing the object safely.

I think the one thing in the DIP I'm not sure of is the 
definition of what is a reference counted object. Say you write 
this.

struct Widget {
     byte* data;
     size_t length;

      trusted
     opAssign(ref const(Widget) other) {
         length = other.length;

         data = malloc(length);
         memcpy(cast(void*) data, cast(void*) other.data, length);
     }

      trusted
     this(this) {
         auto oldData = data;

         data = malloc(length);
         memcpy(cast(void*) data, cast(void*) oldData, length);
     }

      trusted
     ~this() {
         free(data);
     }
}

Why would you want to do this? Who knows. According to DIP77 the 
object above is defined to be a reference counted object, when it 
isn't. If we're rejecting the C++ answer of "don't write code 
that way," then the same rejection should apply here. We need to 
be able to say with 100% certainty that we are dealing with a 
reference counted object, or at least an object where we know the 
postblit is so trivial the cost of calling it will be small.
Apr 11 2015
parent "weaselcat" <weaselcat gmail.com> writes:
On Saturday, 11 April 2015 at 10:14:00 UTC, w0rp wrote:


 I think the one thing in the DIP I'm not sure of is the 
 definition of what is a reference counted object.


I believe it's still defined by DIP74?
http://wiki.dlang.org/DIP74
Apr 11 2015
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 11 April 2015 at 09:28:46 UTC, Marc Schütz wrote:

 It's not acceptable that it happens behind the user's back. 
 Costly operations must be explicit.


Nothing is costly here.
Apr 11 2015
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 11 April 2015 at 20:40:20 UTC, deadalnix wrote:

 On Saturday, 11 April 2015 at 09:28:46 UTC, Marc Schütz wrote:

 It's not acceptable that it happens behind the user's back. 
 Costly operations must be explicit.


 Nothing is costly here.


Of course it is:
http://forum.dlang.org/post/iehijjgdlfouplcngrjy forum.dlang.org
Apr 12 2015
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


In general, this is way too much focused on reference counting. 
The DIP does define "RCO" in more general terms, but still, it 
feels like the wrong approach.

You also seem to look at Rust's type system from the same POV 
("only one mutable reference at a time, for efficiency"), but 
that's not the point at all: it's about safety in general, not 
efficiency for RC. It encompasses so much more: memory safety, 
race-free sharing of data, avoiding iterator invalidation... 
DIP77 cannot help with general iterator invalidation at all.

Another thing that just feels wrong is the automagic copying. 
We're passing something by `ref`, and the compiler inserts a copy 
behind our back?! And this isn't even visible to the user...

Because the topic is correctness,  safe/ system seems the correct 
way to approach the problem. The compiler should just treat 
unsafe pass-by-ref as  system, as Martin suggests. Basically, 
instead of inserting a copy, pass-by-ref will be  system, when no 
copy would need to be inserted, it will be  safe.

I also think that the unsafety detection heuristic is too 
simplistic. When scope/return is extended to pointers and other 
kinds of references, there would probably be way too many false 
positives. A more sophisticated analysis is necessary that can 
more realistically determine when mutable aliasing can occur. 
E.g., it can also take uniqueness into account.

Besides, the address and slice operators also need to be taken 
into account, and these can appear outside of function calls.
Apr 10 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 7:23 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" wrote:

 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


 In general, this is way too much focused on reference counting. The DIP does
 define "RCO" in more general terms, but still, it feels like the wrong
approach.

 You also seem to look at Rust's type system from the same POV ("only one
mutable
 reference at a time, for efficiency"), but that's not the point at all: it's
 about safety in general, not efficiency for RC. It encompasses so much more:
 memory safety, race-free sharing of data, avoiding iterator invalidation...
 DIP77 cannot help with general iterator invalidation at all.


D uses ranges, not iterators.



 Another thing that just feels wrong is the automagic copying. We're passing
 something by `ref`, and the compiler inserts a copy behind our back?! And this
 isn't even visible to the user...

 Because the topic is correctness,  safe/ system seems the correct way to
 approach the problem. The compiler should just treat unsafe pass-by-ref as
  system, as Martin suggests. Basically, instead of inserting a copy,
pass-by-ref
 will be  system, when no copy would need to be inserted, it will be  safe.


A difficulty with that is the semantics will change as the compiler improves
and 
gets better at realizing no copy would be needed. Having the  safety be 
"implementation defined" would be unhappy.



 I also think that the unsafety detection heuristic is too simplistic. When
 scope/return is extended to pointers and other kinds of references, there would
 probably be way too many false positives. A more sophisticated analysis is
 necessary that can more realistically determine when mutable aliasing can
occur.
 E.g., it can also take uniqueness into account.

 Besides, the address and slice operators also need to be taken into account,
and
 these can appear outside of function calls.


I expect any use of RCO's necessarily means the implementation of the RCO is 
unsafe (after all, it calls free()), but a safe interface is provided. The safe 
interface to an RCO is tightly controlled with 'return ref'.
Apr 10 2015
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 10 April 2015 at 17:43:00 UTC, Walter Bright wrote:

 On 4/10/2015 7:23 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:

 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright 
 wrote:

 http://wiki.dlang.org/DIP77


 In general, this is way too much focused on reference 
 counting. The DIP does
 define "RCO" in more general terms, but still, it feels like 
 the wrong approach.

 You also seem to look at Rust's type system from the same POV 
 ("only one mutable
 reference at a time, for efficiency"), but that's not the 
 point at all: it's
 about safety in general, not efficiency for RC. It encompasses 
 so much more:
 memory safety, race-free sharing of data, avoiding iterator 
 invalidation...
 DIP77 cannot help with general iterator invalidation at all.


 D uses ranges, not iterators.


I used "iterator invalidation" because it is an established term. 
You get the same problems with ranges, once the contents aren't 
GC managed. And even if they are, your code is merely memory 
safe, but not necessarily correct. stdin.byLine() without idup is 
a prime example.


 Another thing that just feels wrong is the automagic copying. 
 We're passing
 something by `ref`, and the compiler inserts a copy behind our 
 back?! And this
 isn't even visible to the user...

 Because the topic is correctness,  safe/ system seems the 
 correct way to
 approach the problem. The compiler should just treat unsafe 
 pass-by-ref as
  system, as Martin suggests. Basically, instead of inserting a 
 copy, pass-by-ref
 will be  system, when no copy would need to be inserted, it 
 will be  safe.


 A difficulty with that is the semantics will change as the 
 compiler improves and gets better at realizing no copy would be 
 needed. Having the  safety be "implementation defined" would be 
 unhappy.


Agreed, it needs to be specified. I'm aware that this is 
difficult. Though, why can't we use the same approach we use in 
other cases? Be restrictive at first, later accept more, as we 
refine the specification... At some point we'll find the sweet 
spot between false negatives and complexity of specification, and 
can stabilize on it, so that other compilers have a non-moving 
target.


 I also think that the unsafety detection heuristic is too 
 simplistic. When
 scope/return is extended to pointers and other kinds of 
 references, there would
 probably be way too many false positives. A more sophisticated 
 analysis is
 necessary that can more realistically determine when mutable 
 aliasing can occur.
 E.g., it can also take uniqueness into account.

 Besides, the address and slice operators also need to be taken 
 into account, and
 these can appear outside of function calls.


 I expect any use of RCO's necessarily means the implementation 
 of the RCO is unsafe (after all, it calls free()), but a safe 
 interface is provided. The safe interface to an RCO is tightly 
 controlled with 'return ref'.


Uhm... how is this related to what I wrote?

Example:

void foo()  safe {
     RCArray!int arr = [0,1,2];
     {
         int* p = &arr[0];  // legal under new scope rules
         RCArray!int other;
         arr = other;
         writeln(*p);       // OOPS
     }
}

This shows that with the extended scope rules (i.e. scope not 
restricted to `ref`), the borrowing can happen everywhere, not 
just on function calls. By your rules, every address and slice 
operator applied to a mutable local would need to trigger the 
copying, because the resulting (borrowed) reference exists in the 
same scope as the original variable (owner).

I'd prefer instead to make writing to the owner unsafe, as long 
as borrowed references to it exist (this also includes potential 
writes, like passing a mutable ref to another function). This 
doesn't require copying behind the user's back, and allows the 
user to choose the solution best suited for the situation: manual 
copying, deferring the write to the owner,  trusted...
Apr 10 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 11:28 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" wrote:

 Example:

 void foo()  safe {
      RCArray!int arr = [0,1,2];
      {
          int* p = &arr[0];  // legal under new scope rules


This would be a bad design of an RCO. RCO's must be constructed to not allow 
pointers to the payload other than by ref.


          RCArray!int other;
          arr = other;
          writeln(*p);       // OOPS
      }
 }
Apr 10 2015
next sibling parent reply "Martin Nowak" <code dawg.eu> writes:
On Friday, 10 April 2015 at 21:26:14 UTC, Walter Bright wrote:

 This would be a bad design of an RCO. RCO's must be constructed 
 to not allow pointers to the payload other than by ref.


And taking the address of that is already unsafe.
Apr 11 2015
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 11 April 2015 at 09:15:19 UTC, Martin Nowak wrote:

 On Friday, 10 April 2015 at 21:26:14 UTC, Walter Bright wrote:

 This would be a bad design of an RCO. RCO's must be 
 constructed to not allow pointers to the payload other than by 
 ref.


 And taking the address of that is already unsafe.


It isn't under my proposal, which Walter has already accepted the 
most important part of.
Apr 11 2015
prev sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 10 April 2015 at 21:26:14 UTC, Walter Bright wrote:

 On 4/10/2015 11:28 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:

 Example:

 void foo()  safe {
     RCArray!int arr = [0,1,2];
     {
         int* p = &arr[0];  // legal under new scope rules


 This would be a bad design of an RCO. RCO's must be constructed 
 to not allow pointers to the payload other than by ref.


There is no reason for this restriction. But if this is your 
opinion, why did you agree to "implement 'scope' and 'return' for 
arrays, classes, and pointers"?

http://forum.dlang.org/post/mfhkbm$2vbk$1 digitalmars.com

It makes no sense to implement that, but not allow it to be used 
safely.
Apr 11 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/11/2015 2:18 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" wrote:

 On Friday, 10 April 2015 at 21:26:14 UTC, Walter Bright wrote:

 On 4/10/2015 11:28 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>"
wrote:

 Example:

 void foo()  safe {
     RCArray!int arr = [0,1,2];
     {
         int* p = &arr[0];  // legal under new scope rules


 This would be a bad design of an RCO. RCO's must be constructed to not allow
 pointers to the payload other than by ref.


 There is no reason for this restriction.


The reason is to prevent unsafe access of the payload.



 But if this is your opinion, why did
 you agree to "implement 'scope' and 'return' for arrays, classes, and
pointers"?

 http://forum.dlang.org/post/mfhkbm$2vbk$1 digitalmars.com

 It makes no sense to implement that, but not allow it to be used safely.


In your example posted upthread, it wasn't safe. Perhaps scoped pointers can't 
be made safe for RCOs, or perhaps the proposal needs more thinking. I don't
know 
at the moment.
Apr 11 2015
prev sibling next sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 04/09/2015 01:10 AM, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


In the first problem example:

 struct S {
     RCArray!T array;
 }
 void main() {
     auto s = S(RCArray!T([T()])); // s.array's refcount is now 1
     foo(s, s.array[0]);           // pass by ref
 }
 void foo(ref S s, ref T t) {
     s.array = RCArray!T([]);      // drop the old s.array
     t.doSomething();              // oops, t is gone
 }

What do you do to pin s.array?

auto tmp = s;

or

auto tmp = s.array;
Apr 10 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 2:11 PM, Martin Nowak wrote:

 On 04/09/2015 01:10 AM, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


 In the first problem example:

   struct S {
       RCArray!T array;
   }
   void main() {
       auto s = S(RCArray!T([T()])); // s.array's refcount is now 1
       foo(s, s.array[0]);           // pass by ref
   }
   void foo(ref S s, ref T t) {
       s.array = RCArray!T([]);      // drop the old s.array
       t.doSomething();              // oops, t is gone
   }

 What do you do to pin s.array?

 auto tmp = s;

 or

 auto tmp = s.array;


The latter.
Apr 10 2015
next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2015-04-10 21:29:19 +0000, Walter Bright <newshound2 digitalmars.com> said:


 On 4/10/2015 2:11 PM, Martin Nowak wrote:

 On 04/09/2015 01:10 AM, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


 
 In the first problem example:
 
   struct S {
       RCArray!T array;
   }
   void main() {
       auto s = S(RCArray!T([T()])); // s.array's refcount is now 1
       foo(s, s.array[0]);           // pass by ref
   }
   void foo(ref S s, ref T t) {
       s.array = RCArray!T([]);      // drop the old s.array
       t.doSomething();              // oops, t is gone
   }
 
 What do you do to pin s.array?
 
 auto tmp = s;
 
 or
 
 auto tmp = s.array;
 


 
 The latter.


And how is it pinned in this case?

  struct S {
      private RCArray!T array;
      ref T opIndex(int index) return { return array[index]; }
      void clear() { s.array = RCArray!T([]); }
  }
  void main() {
      auto s = S(RCArray!T([T()])); // s.array's refcount is now 1
      foo(s, s[0]);           // pass by ref
  }
  void foo(ref S s, ref T t) {
      s.clear();            // drop the old s.array
      t.doSomething();              // oops, t is gone
  }

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca
Apr 10 2015
next sibling parent "weaselcat" <weaselcat gmail.com> writes:
On Friday, 10 April 2015 at 21:50:13 UTC, Michel Fortin wrote:

 On 2015-04-10 21:29:19 +0000, Walter Bright 
 <newshound2 digitalmars.com> said:


 On 4/10/2015 2:11 PM, Martin Nowak wrote:

 On 04/09/2015 01:10 AM, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


 
 In the first problem example:
 
  struct S {
      RCArray!T array;
  }
  void main() {
      auto s = S(RCArray!T([T()])); // s.array's refcount is 
 now 1
      foo(s, s.array[0]);           // pass by ref
  }
  void foo(ref S s, ref T t) {
      s.array = RCArray!T([]);      // drop the old s.array
      t.doSomething();              // oops, t is gone
  }
 
 What do you do to pin s.array?
 
 auto tmp = s;
 
 or
 
 auto tmp = s.array;
 


 
 The latter.


 And how is it pinned in this case?

  struct S {
      private RCArray!T array;
      ref T opIndex(int index) return { return array[index]; }
      void clear() { s.array = RCArray!T([]); }
  }
  void main() {
      auto s = S(RCArray!T([T()])); // s.array's refcount is now 
 1
      foo(s, s[0]);           // pass by ref
  }
  void foo(ref S s, ref T t) {
      s.clear();            // drop the old s.array
      t.doSomething();              // oops, t is gone
  }


AFAICT it would create a temporary copy of s.array, which would 
keep t valid.


 Nope. It'll scan at compile time the types reachable through 
 the parameters. If any of those types are RCO's that have a 
 type that matches a ref parameter, the


ref parameter's RCO will get copied.
Apr 10 2015
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 2:50 PM, Michel Fortin wrote:

 And how is it pinned in this case?

   struct S {
       private RCArray!T array;
       ref T opIndex(int index) return { return array[index]; }
       void clear() { s.array = RCArray!T([]); }
   }
   void main() {
       auto s = S(RCArray!T([T()])); // s.array's refcount is now 1
       foo(s, s[0]);           // pass by ref


The s[0] is preceded by tmp=s; Because T is reachable from typeof(first
argument s)


   }
   void foo(ref S s, ref T t) {
       s.clear();            // drop the old s.array
       t.doSomething();              // oops, t is gone
   }
Apr 10 2015
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 04/10/2015 11:29 PM, Walter Bright wrote:

 
 The latter.


Can you update that part in the DIP, it wasn't clear that the temporary
selectively pins RCO fields of a normal struct passed by ref.
Apr 10 2015
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 10 April 2015 at 23:18:59 UTC, Martin Nowak wrote:

 On 04/10/2015 11:29 PM, Walter Bright wrote:

 
 The latter.


 Can you update that part in the DIP, it wasn't clear that the 
 temporary
 selectively pins RCO fields of a normal struct passed by ref.


If a struct has RCO fields, shouldn't it be an RCO itself, and as 
such be pinned ?

It sounds like this is implied in the DIP.
Apr 10 2015
parent Michel Fortin <michel.fortin michelf.ca> writes:
On 2015-04-10 23:22:17 +0000, "deadalnix" <deadalnix gmail.com> said:


 On Friday, 10 April 2015 at 23:18:59 UTC, Martin Nowak wrote:

 On 04/10/2015 11:29 PM, Walter Bright wrote:

 
 The latter.


 
 Can you update that part in the DIP, it wasn't clear that the temporary
 selectively pins RCO fields of a normal struct passed by ref.


 
 If a struct has RCO fields, shouldn't it be an RCO itself, and as such 
 be pinned ?


Not necessarily. A  disabled postblit could make it no longer RCO 
(including a  disabled postblit in one of the fields).


 It sounds like this is implied in the DIP.


That's what I thought too. But when confronted to a case where that 
wouldn't work Walter said in this thread that the compiler would make a 
temporary of the fields. So I'm not too sure what to think anymore. The 
DIP should clarify what happens with  disabled postblit and RCO fields 
inside non-RCO structs.

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca
Apr 10 2015
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 4:18 PM, Martin Nowak wrote:

 On 04/10/2015 11:29 PM, Walter Bright wrote:

 The latter.


 Can you update that part in the DIP, it wasn't clear that the temporary
 selectively pins RCO fields of a normal struct passed by ref.


It pins RCO objects, however they are derived.
Apr 10 2015
prev sibling parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 04/09/2015 01:10 AM, Walter Bright wrote:

 http://wiki.dlang.org/DIP77


It's a very interesting proposal to tackle this specific problem.

As with all the scope/lifetime related stuff, I find it extremely
difficult to anticipate all the needs and foresee how this will
integrate with the rest of the language, or even the other scope related
DIPs.

At this point it might be better to make a dmd branch and throw up a
prototype for all the scope/ref-counted work. The litmus test for a
successful design would be safe, efficient,  nogc containers.
Apr 11 2015