www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - An exegesis of Walter's reference counted slice

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
I modified Walter's sample code to this: 
http://dpaste.dzfl.pl/f3d854feede9. It uses malloc for both the array 
and the reference count, and also uses  trusted minimally. I inserted 
assert()s here and there to clarify the workings. Nothing big except for 
the careful use of  trusted.

I'll use this as a basis of some exegesis.

1. The code is a bit more complicated than it should. Overall this is 
not a biggie; regular D users are not supposed to write reference 
counted slices casually. But I was bummed that e.g. I found no way to 
call emplace()  safe-ly.

2. Michel's point (https://issues.dlang.org/show_bug.cgi?id=14221) 
reveals the largest issue with RC/GC integration. We need to find a fix 
for it if we want to have the GC lift cycles.

3. opIndex ("here's the magic") is the most interesting part of the 
proposal. It disallows unsafe use such as:

 safe ref int fun()
{
    auto a = RCArray!int([1, 2, 3]);
    return a[1];
}

Nice. I'd go as far as saying that barring implementation bugs, with 
DIP25 in tow, and after we fix 14221, it's impossible to get an invalid 
memory access with RCArray in  safe code. This (we offer a way to design 
 safe arrays and more generally structs that are  safe) is interesting 
and important.

That said there is a rub. The part that deallocates memory in the 
destructor must be  trusted. That is fine, but the trustworthiness of 
that code depends on the "return" attribute in opIndex. Furthermore, if 
the author of RCSlice forgets to put "return" there, the compiler won't 
help - it just allows wrong code like fun() above to compile and run 
(causing dangling pointer use).

So: does DIP25 allow safe slices? Looks that way, though a proof would 
be nice. Does it allow other safe interesting structs that own data? 
Very likely. Does it allow really sophisticated ownership schemes? We 
need to explore that. Does it protect against bugs in implementations of 
safe ownership schemes that explicitly release memory? Not too well. I 
think the prevalent idiom will be to accompany such artifacts with 
unittests that make sure unsafe uses (such as fun() above) do not compile.


Andrei
Feb 24 2015
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/24/15 2:40 PM, Andrei Alexandrescu wrote:

 I modified Walter's sample code to this:
 http://dpaste.dzfl.pl/f3d854feede9. It uses malloc for both the array
 and the reference count, and also uses  trusted minimally. I inserted
 assert()s here and there to clarify the workings. Nothing big except for
 the careful use of  trusted.

 I'll use this as a basis of some exegesis.

 1. The code is a bit more complicated than it should. Overall this is
 not a biggie; regular D users are not supposed to write reference
 counted slices casually. But I was bummed that e.g. I found no way to
 call emplace()  safe-ly.


I have no problem with underlying complexity of primitive types. Using 
Objective-C objects is just fine without understanding the implementation.

Note, you need to GC.addRange all the elements if the type has 
references, or else you cannot have GC pointers in the array. For 
example, an array of class references could potentially result in those 
references being collected before the array is gone.

Ironically, if those elements are references, but are reference counted 
references, then you wouldn't need to addRange. An interesting problem...


 2. Michel's point (https://issues.dlang.org/show_bug.cgi?id=14221)
 reveals the largest issue with RC/GC integration. We need to find a fix
 for it if we want to have the GC lift cycles.


I think a system of making sure a piece of data is always destroyed in 
the same thread it was created (unless of course, the thread is gone, in 
which case it doesn't matter), should be fine.

-Steve
Feb 24 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/24/15 11:55 AM, Steven Schveighoffer wrote:

 Note, you need to GC.addRange all the elements if the type has
 references, or else you cannot have GC pointers in the array. For
 example, an array of class references could potentially result in those
 references being collected before the array is gone.

 Ironically, if those elements are references, but are reference counted
 references, then you wouldn't need to addRange. An interesting problem...


Could you please add a detailed issue about this? Thanks. -- Andrei
Feb 24 2015
next sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Tue, 24 Feb 2015 12:03:36 -0800, Andrei Alexandrescu wrote:


 On 2/24/15 11:55 AM, Steven Schveighoffer wrote:

 Note, you need to GC.addRange all the elements if the type has
 references, or else you cannot have GC pointers in the array. For
 example, an array of class references could potentially result in those
 references being collected before the array is gone.

 Ironically, if those elements are references, but are reference counted
 references, then you wouldn't need to addRange. An interesting
 problem...


=20
 Could you please add a detailed issue about this? Thanks. -- Andrei


here's the fixed code: http://dpaste.dzfl.pl/5f7dfe237d1e

btw, you have a bug in your implementation:

  (cast(E*) malloc(a.length * uint.sizeof))[0 .. a.length])();

should be:

  (cast(E*) malloc(a.length * E.sizeof))[0 .. a.length])();

sorry, i'm bad in explanations, let the code speak for itself.=
Feb 24 2015
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/24/15 3:03 PM, Andrei Alexandrescu wrote:

 On 2/24/15 11:55 AM, Steven Schveighoffer wrote:

 Note, you need to GC.addRange all the elements if the type has
 references, or else you cannot have GC pointers in the array. For
 example, an array of class references could potentially result in those
 references being collected before the array is gone.

 Ironically, if those elements are references, but are reference counted
 references, then you wouldn't need to addRange. An interesting problem...


 Could you please add a detailed issue about this? Thanks. -- Andrei


I'm not sure what you mean by 'this'. As far as I can tell, it isn't a 
bug in GC but a bug in your dpaste code.

-Steve
Feb 26 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/26/15 8:44 AM, Steven Schveighoffer wrote:

 On 2/24/15 3:03 PM, Andrei Alexandrescu wrote:

 On 2/24/15 11:55 AM, Steven Schveighoffer wrote:

 Note, you need to GC.addRange all the elements if the type has
 references, or else you cannot have GC pointers in the array. For
 example, an array of class references could potentially result in those
 references being collected before the array is gone.

 Ironically, if those elements are references, but are reference counted
 references, then you wouldn't need to addRange. An interesting
 problem...


 Could you please add a detailed issue about this? Thanks. -- Andrei


 I'm not sure what you mean by 'this'. As far as I can tell, it isn't a
 bug in GC but a bug in your dpaste code.

 -Steve


I'm referring to the "interesting problem" of adding addRange for some 
but not all types. -- Andrei
Feb 26 2015
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/26/15 11:56 AM, Andrei Alexandrescu wrote:

 On 2/26/15 8:44 AM, Steven Schveighoffer wrote:

 On 2/24/15 3:03 PM, Andrei Alexandrescu wrote:

 On 2/24/15 11:55 AM, Steven Schveighoffer wrote:

 Note, you need to GC.addRange all the elements if the type has
 references, or else you cannot have GC pointers in the array. For
 example, an array of class references could potentially result in those
 references being collected before the array is gone.

 Ironically, if those elements are references, but are reference counted
 references, then you wouldn't need to addRange. An interesting
 problem...


 Could you please add a detailed issue about this? Thanks. -- Andrei


 I'm not sure what you mean by 'this'. As far as I can tell, it isn't a
 bug in GC but a bug in your dpaste code.


 I'm referring to the "interesting problem" of adding addRange for some
 but not all types. -- Andrei


OK, but I don't know where to add that :( RC isn't supported by D today. 
You mean add it to a DIP? Add it to the docs?

I don't think it's mechanically checkable, BTW. If you have an array of 
pointers, each pointer may point at a GC-allocated block, or a C 
malloc-allocated block. I suppose at the time of setting an element, you 
can addRange if necessary. But the cost for that I think far outweighs 
the benefit of using RC.

-Steve
Feb 26 2015
prev sibling next sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Tue, 24 Feb 2015 14:55:14 -0500, Steven Schveighoffer wrote:


 Note, you need to GC.addRange all the elements if the type has
 references, or else you cannot have GC pointers in the array. For
 example, an array of class references could potentially result in those
 references being collected before the array is gone.


heh. yes, i hit this in my `iv.steam.PartialStream` implementation. those=20
occasional crashes was nasty.


 Ironically, if those elements are references, but are reference counted
 references, then you wouldn't need to addRange. An interesting
 problem...


actually, it's safe to add GC roots and regions that contains no GC=20
references. it's a little slowdown, as GC must scan that area, but=20
nothing bad happens.

yet you are right, there should be some method to distinguish "gc-
agnostic" types. and compiler should be able to infer that...=
Feb 24 2015
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/24/2015 11:55 AM, Steven Schveighoffer wrote:

 Note, you need to GC.addRange all the elements if the type has references, or
 else you cannot have GC pointers in the array.


Yes. I deliberately left that out in order to keep attention focused on the use 
of 'return ref'.



 Ironically, if those elements are references, but are reference counted
 references, then you wouldn't need to addRange. An interesting problem...


Correct, but (aside from the performance hit) it wouldn't be a bug to scan them 
unnecessarily in the GC.

One way is to have a scheme whereby, with introspection, a type can be 
determined to be ref counted. Then, using static if, doing the addRange would 
only be necessary if:

1. the element type contained indirections
2. those indirections were not ref counted



 I think a system of making sure a piece of data is always destroyed in the same
 thread it was created (unless of course, the thread is gone, in which case it
 doesn't matter), should be fine.


Yup, and I believe this is an orthogonal issue.
Feb 24 2015
prev sibling next sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Tue, 24 Feb 2015 11:40:34 -0800, Andrei Alexandrescu wrote:


 3. opIndex ("here's the magic") is the most interesting part of the
 proposal. It disallows unsafe use such as:
=20
  safe ref int fun()
 {
     auto a =3D RCArray!int([1, 2, 3]);
     return a[1];
 }


this is the important part for those who are too lazy to read DIP25 (like=20
me). thank you.=
Feb 24 2015
prev sibling next sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei Alexandrescu 
wrote:

 ...


Off-topic, sorry
Are we still going to get a Trusted block, or just going to use 
trusted lambdas?(They're kind of ugly TBH)


 But I was bummed that e.g. I found no way to call emplace() 
  safe-ly.


I assumed emplace would infer  safe?
Feb 24 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/24/15 12:03 PM, weaselcat wrote:

 On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei Alexandrescu wrote:

 ...


 Off-topic, sorry
 Are we still going to get a Trusted block, or just going to use trusted
 lambdas?(They're kind of ugly TBH)


We're going to lambdas. Ugliness is a feature.


 But I was bummed that e.g. I found no way to call emplace()  safe-ly.


 I assumed emplace would infer  safe?


It currently doesn't properly, probably because it uses T* instead of ref T.


Andrei
Feb 24 2015
parent reply "weaselcat" <weaselcat gmail.com> writes:
On Tuesday, 24 February 2015 at 20:04:27 UTC, Andrei Alexandrescu 
wrote:

 On 2/24/15 12:03 PM, weaselcat wrote:

 On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei 
 Alexandrescu wrote:

 ...


 Off-topic, sorry
 Are we still going to get a Trusted block, or just going to 
 use trusted
 lambdas?(They're kind of ugly TBH)


 We're going to lambdas. Ugliness is a feature.


Stroustrup would be proud ;)
Feb 24 2015
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Tue, 24 Feb 2015 20:11:17 +0000, weaselcat wrote:


 On Tuesday, 24 February 2015 at 20:04:27 UTC, Andrei Alexandrescu wrote:

 On 2/24/15 12:03 PM, weaselcat wrote:

 On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei Alexandrescu
 wrote:

 ...


 Off-topic, sorry Are we still going to get a Trusted block, or just
 going to use trusted lambdas?(They're kind of ugly TBH)


 We're going to lambdas. Ugliness is a feature.


 Stroustrup would be proud ;)


yet he made the whole language ugly, instead of making only ugly things=20
ugly. ;-)=
Feb 24 2015
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Tue, Feb 24, 2015 at 08:21:53PM +0000, ketmar via Digitalmars-d wrote:

 On Tue, 24 Feb 2015 20:11:17 +0000, weaselcat wrote:
 

 On Tuesday, 24 February 2015 at 20:04:27 UTC, Andrei Alexandrescu wrote:

 On 2/24/15 12:03 PM, weaselcat wrote:

 On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei Alexandrescu
 wrote:

 ...


 Off-topic, sorry Are we still going to get a Trusted block, or
 just going to use trusted lambdas?(They're kind of ugly TBH)


 We're going to lambdas. Ugliness is a feature.


 Stroustrup would be proud ;)


 
 yet he made the whole language ugly, instead of making only ugly
 things ugly. ;-)


Actually, it's worse than that. He made things that shouldn't be ugly,
ugly, and things that *don't* look ugly are outright wrong.

For example, naked pointers have built-in, convenient syntax. They are
also extremely dangerous, and almost all C++ code containing raw
pointers have bugs one way or another.

For-loops have very nice syntax... except that unless you're writing the
most inane, trivial loops, your code probably has bugs -- off-by-1
errors, wrong loop conditions, etc..

Calling malloc/free is also easy... and your program probably has memory
leak bugs too. Not to mention type-safety bugs if you're not careful
about what you cast that void* into. But hey, writing:

	// Woohoo, bare pointer! Let's dance!
	void *p = malloc(1234);

sure looks prettier than:

	// Repeat after me: Don't repeat yourself... don't repeat
	// yourself... don't repeat yourself...
	MyType *p = (MyType *)malloc(sizeof(MyType));


Well, new/delete is prettier (as far as C++ goes anyway)... But wait,
you still have memory leaks.  Augh...

And what about cleaning up resources after you're done with them?

	void func() {
		Resource *res = acquireResource();
		doStuff(res);
		freeResource(res);
	}

Easy, right? Yeah, except that doStuff throws exceptions, so you have
resource leakage. Solution? Write a long convoluted wrapper class with a
dtor that cleans up. And a copy ctor that makes sure initializing one
resource from another works correctly. And operator=() to make sure
assignments work correctly. Oh but wait, your ctor is not
exception-safe, so better rewrite *that* to use RAII too. Which means
more copy ctors, more operator=()... oh wait, your copy ctor is not
const-correct so it doesn't get called when you have a const object.
Yeah, and your operator=() too. And ...

After it's all said and done, what used to be a 3-line function has
exploded into a monstrosity who knows how many lines long, with arcane
incantations of auto_ptr<>, shared_ptr<>, and all that fancy stuff that
no newbie ever has the hope of parsing, let alone understanding.

	http://bartoszmilewski.com/2013/09/19/edward-chands/

Favorite quote:

	C++ has become an extremely complex language. There are
	countless ways of doing the same thing — almost all of them
	either plain wrong, dangerous, unmaintainable, or all of the
	above. The problem is that most code compiles and even runs. The
	mistakes and shortcomings are discovered much later, often after
	the product has been released. 

Yep, that's exactly why D has ruined my life, I just can't go back to
that C++ garbage anymore.


T

-- 
Don't throw out the baby with the bathwater. Use your hands...
Feb 24 2015
next sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Tuesday, 24 February 2015 at 20:46:28 UTC, H. S. Teoh wrote:

 	http://bartoszmilewski.com/2013/09/19/edward-chands/


I'm not generally one to read comments on blogs, but the replies 
to that blog are jawdropping.

I have to assume most of these people are either new to C++ or 
are being forced to write these replies at gunpoint.


Yep, that's exactly why D has ruined my life, I just can't go 
back to


that C++ garbage anymore.

Working with C++ after using D does leave one feeling... 
disillusioned.
Feb 24 2015
next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Tue, Feb 24, 2015 at 09:13:42PM +0000, weaselcat via Digitalmars-d wrote:

 On Tuesday, 24 February 2015 at 20:46:28 UTC, H. S. Teoh wrote:


[...]

Yep, that's exactly why D has ruined my life, I just can't go back to


 that C++ garbage anymore.
 
 Working with C++ after using D does leave one feeling...
 disillusioned.


That's putting it rather mildly. :-P

Some time ago I revisited one of my old C++ programs, hoping to optimize
performance a bit by using a hash table to cache the results of certain
expensive computations.  I was somewhat elated that *finally* after all
these years hash tables finally made it into the C++ standard... For all
their warts, D's AA's have left me confident that this would be a small
change...  Boy was I wrong. Everything from the horrible syntax to
built-in structs *not* being supported as hash keys by default, to
needing to write my own hash function, *and* having to pass it around
all over the place, etc.., ...  after several hours of fighting with the
language for something that would be just a couple dozen lines of code
in D at the most, I threw up my hands and ditched the whole idea.

C++11 feels a lot like "too little, too late" to me. I'd rather rewrite
the whole thing in D and spare myself the pain. For all of its current
flaws, D is still way ahead of C++ in terms of usability. I've stopped
caring about C++ and have not bothered to find out what C++14 (or is it
C++15 now?) has in store -- if past experience is anything to go by, it
will just be another "too little, too late".


T

-- 
"How are you doing?" "Doing what?"
Feb 24 2015
prev sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Tue, 24 Feb 2015 21:13:42 +0000, weaselcat wrote:


 Working with C++ after using D does leave one feeling... disillusioned.


that's why i'm not even considering c++ for writing new code anymore.=20
either D, or good old C, if D is not appropriate.

actually, i haven't written a line of c++ for several years now. and i=20
wrote almost no new C code for years too -- thanks to seampless interop=20
between D and C.=
Feb 24 2015
prev sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Tuesday, 24 February 2015 at 20:46:28 UTC, H. S. Teoh wrote:

 On Tue, Feb 24, 2015 at 08:21:53PM +0000, ketmar via 
 Digitalmars-d wrote:

 On Tue, 24 Feb 2015 20:11:17 +0000, weaselcat wrote:
 

 On Tuesday, 24 February 2015 at 20:04:27 UTC, Andrei 
 Alexandrescu wrote:

 On 2/24/15 12:03 PM, weaselcat wrote:

 On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei 
 Alexandrescu
 wrote:

 ...


 Off-topic, sorry Are we still going to get a Trusted 
 block, or
 just going to use trusted lambdas?(They're kind of ugly 
 TBH)


 We're going to lambdas. Ugliness is a feature.


 Stroustrup would be proud ;)


 
 yet he made the whole language ugly, instead of making only 
 ugly
 things ugly. ;-)


 Actually, it's worse than that. He made things that shouldn't 
 be ugly,
 ugly, and things that *don't* look ugly are outright wrong.

 For example, naked pointers have built-in, convenient syntax. 
 They are
 also extremely dangerous, and almost all C++ code containing raw
 pointers have bugs one way or another.

 For-loops have very nice syntax... except that unless you're 
 writing the
 most inane, trivial loops, your code probably has bugs -- 
 off-by-1
 errors, wrong loop conditions, etc..

 Calling malloc/free is also easy... and your program probably 
 has memory
 leak bugs too. Not to mention type-safety bugs if you're not 
 careful
 about what you cast that void* into. But hey, writing:

 	// Woohoo, bare pointer! Let's dance!
 	void *p = malloc(1234);

 sure looks prettier than:

 	// Repeat after me: Don't repeat yourself... don't repeat
 	// yourself... don't repeat yourself...
 	MyType *p = (MyType *)malloc(sizeof(MyType));


 Well, new/delete is prettier (as far as C++ goes anyway)... But 
 wait,
 you still have memory leaks.  Augh...

 And what about cleaning up resources after you're done with 
 them?

 	void func() {
 		Resource *res = acquireResource();
 		doStuff(res);
 		freeResource(res);
 	}

 Easy, right? Yeah, except that doStuff throws exceptions, so 
 you have
 resource leakage. Solution? Write a long convoluted wrapper 
 class with a
 dtor that cleans up. And a copy ctor that makes sure 
 initializing one
 resource from another works correctly. And operator=() to make 
 sure
 assignments work correctly. Oh but wait, your ctor is not
 exception-safe, so better rewrite *that* to use RAII too. Which 
 means
 more copy ctors, more operator=()... oh wait, your copy ctor is 
 not
 const-correct so it doesn't get called when you have a const 
 object.
 Yeah, and your operator=() too. And ...

 After it's all said and done, what used to be a 3-line function 
 has
 exploded into a monstrosity who knows how many lines long, with 
 arcane
 incantations of auto_ptr<>, shared_ptr<>, and all that fancy 
 stuff that
 no newbie ever has the hope of parsing, let alone understanding.

 	http://bartoszmilewski.com/2013/09/19/edward-chands/

 Favorite quote:

 	C++ has become an extremely complex language. There are
 	countless ways of doing the same thing — almost all of them
 	either plain wrong, dangerous, unmaintainable, or all of the
 	above. The problem is that most code compiles and even runs. 
 The
 	mistakes and shortcomings are discovered much later, often 
 after
 	the product has been released.

 Yep, that's exactly why D has ruined my life, I just can't go 
 back to
 that C++ garbage anymore.


 T


Almost all those warts are caused by the C compatibility and 
affect also D to certain extent.

Any language that tries to achieve copy-paste compatibility with 
C will suffer from it.

--
Paulo
Feb 24 2015
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei Alexandrescu 
wrote:

 I modified Walter's sample code to this: 
 http://dpaste.dzfl.pl/f3d854feede9. It uses malloc for both the 
 array and the reference count, and also uses  trusted 
 minimally. I inserted assert()s here and there to clarify the 
 workings. Nothing big except for the careful use of  trusted.


Doesn't look like it follows the "philosophy behind"  trusted.

Conceptually, you should be able to insert exceptions everywhere 
outside the called  trusted function and retain memory safety. 
The way the code is written you have to move outside the  trusted 
function to prove memory safety.
Feb 24 2015
prev sibling next sibling parent reply Ivan Timokhin <timokhin.iv gmail.com> writes:
Andrei Alexandrescu wrote:


 I modified Walter's sample code to this:
 http://dpaste.dzfl.pl/f3d854feede9. It uses malloc for both the array
 and the reference count, and also uses  trusted minimally. I inserted
 assert()s here and there to clarify the workings. Nothing big except for
 the careful use of  trusted.
 
 I'll use this as a basis of some exegesis.
 
 1. The code is a bit more complicated than it should. Overall this is
 not a biggie; regular D users are not supposed to write reference
 counted slices casually. But I was bummed that e.g. I found no way to
 call emplace()  safe-ly.
 
 2. Michel's point (https://issues.dlang.org/show_bug.cgi?id=14221)
 reveals the largest issue with RC/GC integration. We need to find a fix
 for it if we want to have the GC lift cycles.
 
 3. opIndex ("here's the magic") is the most interesting part of the
 proposal. It disallows unsafe use such as:
 
  safe ref int fun()
 {
     auto a = RCArray!int([1, 2, 3]);
     return a[1];
 }
 
 Nice. I'd go as far as saying that barring implementation bugs, with
 DIP25 in tow, and after we fix 14221, it's impossible to get an invalid
 memory access with RCArray in  safe code. This (we offer a way to design
  safe arrays and more generally structs that are  safe) is interesting
 and important.
 
 That said there is a rub. The part that deallocates memory in the
 destructor must be  trusted. That is fine, but the trustworthiness of
 that code depends on the "return" attribute in opIndex. Furthermore, if
 the author of RCSlice forgets to put "return" there, the compiler won't
 help - it just allows wrong code like fun() above to compile and run
 (causing dangling pointer use).
 
 So: does DIP25 allow safe slices? Looks that way, though a proof would
 be nice. Does it allow other safe interesting structs that own data?
 Very likely. Does it allow really sophisticated ownership schemes? We
 need to explore that. Does it protect against bugs in implementations of
 safe ownership schemes that explicitly release memory? Not too well. I
 think the prevalent idiom will be to accompany such artifacts with
 unittests that make sure unsafe uses (such as fun() above) do not compile.
 
 
 Andrei


Is there any plan to allow safe conversions to T[] (in restricted 
circumstances, of course)?
Feb 24 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/24/15 1:33 PM, Ivan Timokhin wrote:

 Is there any plan to allow safe conversions to T[] (in restricted
 circumstances, of course)?


We'd want to avoid it because that would necessitate the whole "scope" 
paraphernalia - e.g you can convert a reference counted slice to 
something like scope(T[]).

A possible option would be to allow the user to go "I want to use GC 
with this slice from now on" which gives back the T[] and lets the GC 
take care of it.

But generally I think we should let reference counted slices and 
built-in slices coexist. Generic range-based code is easy to write so 
factoring the nature of the slice away may be an option.


Andrei
Feb 24 2015
parent reply Ivan Timokhin <timokhin.iv gmail.com> writes:
Andrei Alexandrescu wrote:


 On 2/24/15 1:33 PM, Ivan Timokhin wrote:

 Is there any plan to allow safe conversions to T[] (in restricted
 circumstances, of course)?


 
 We'd want to avoid it because that would necessitate the whole "scope"
 paraphernalia - e.g you can convert a reference counted slice to
 something like scope(T[]).
 
 A possible option would be to allow the user to go "I want to use GC
 with this slice from now on" which gives back the T[] and lets the GC
 take care of it.
 
 But generally I think we should let reference counted slices and
 built-in slices coexist. Generic range-based code is easy to write so
 factoring the nature of the slice away may be an option.
 
 
 Andrei


Oh. So, whenever you pass a reference-counted slice around, you need to do 
it with the full inc/dec protocol, which, as Walter has mentioned several 
times already, leads to code bloat and performance hits. So... no to 
efficient reference counting? Also, no slicing of static arrays in  safe 
code?
Feb 25 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/25/15 1:58 AM, Ivan Timokhin wrote:

 Oh. So, whenever you pass a reference-counted slice around, you need to do
 it with the full inc/dec protocol, which, as Walter has mentioned several
 times already, leads to code bloat and performance hits. So... no to
 efficient reference counting? Also, no slicing of static arrays in  safe
 code?


Correct.

There's an issue of perception that I just figured, which explains a lot 
of the drama and trash talk about DIP25.

DIP25 is not a borrowing mechanism. Its charter is to make reference 
counted structs (and a variety of other structs) usable in  safe code. 
As RCSlice shows, with DIP25 a reference counted slice switches from 
unusable to usable in  safe code by adding one token to an otherwise 
unchanged implementation. That is Remarkable, and is Good Programming 
Language Design(tm).

Borrowing data in a scoped manner is the charter of DIP69. Given the 
excellent quality of DIP25, it's likely it is Here To Stay and DIP69, or 
any other proposal for borrowing, will work with it (and probably 
leverage it).


Andrei
Feb 25 2015
parent reply Ivan Timokhin <timokhin.iv gmail.com> writes:
On Wed, Feb 25, 2015 at 08:06:48AM -0800, Andrei Alexandrescu wrote:

 On 2/25/15 1:58 AM, Ivan Timokhin wrote:

 Oh. So, whenever you pass a reference-counted slice around, you need to do
 it with the full inc/dec protocol, which, as Walter has mentioned several
 times already, leads to code bloat and performance hits. So... no to
 efficient reference counting? Also, no slicing of static arrays in  safe
 code?


 
 Correct.
 
 There's an issue of perception that I just figured, which explains a lot 
 of the drama and trash talk about DIP25.
 
 DIP25 is not a borrowing mechanism. Its charter is to make reference 
 counted structs (and a variety of other structs) usable in  safe code. 
 As RCSlice shows, with DIP25 a reference counted slice switches from 
 unusable to usable in  safe code by adding one token to an otherwise 
 unchanged implementation. That is Remarkable, and is Good Programming 
 Language Design(tm).
 
 Borrowing data in a scoped manner is the charter of DIP69. Given the 
 excellent quality of DIP25, it's likely it is Here To Stay and DIP69, or 
 any other proposal for borrowing, will work with it (and probably 
 leverage it).
 
 
 Andrei
 


OK, thank you. I wasn't asking about DIP25 in particular (I think I
understand what it's about), but about your (and Walter's) plans on
the topic in general. I apologise if it sounded offensive in any way.
Feb 25 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/25/15 11:04 AM, Ivan Timokhin wrote:

 OK, thank you. I wasn't asking about DIP25 in particular (I think I
 understand what it's about), but about your (and Walter's) plans on
 the topic in general.


The medium-term plan is two-pronged: (a) immediately apply DIP25 to 
http://wiki.dlang.org/?title=DIP74, which will propose  safe reference 
counted class objects; (b) examine scoped borrowing (DIP69 and friends) 
opportunities post DIP25.

Walter and I are working on DIP74 right now.


Andrei
Feb 25 2015
parent "weaselcat" <weaselcat gmail.com> writes:
  Wednesday, 25 February 2015 at 20:01:51 UTC, Andrei Alexandrescu 
wrote:

 On 2/25/15 11:04 AM, Ivan Timokhin wrote:

 OK, thank you. I wasn't asking about DIP25 in particular (I 
 think I
 understand what it's about), but about your (and Walter's) 
 plans on
 the topic in general.


 The medium-term plan is two-pronged: (a) immediately apply 
 DIP25 to http://wiki.dlang.org/?title=DIP74, which will propose 
  safe reference counted class objects; (b) examine scoped 
 borrowing (DIP69 and friends) opportunities post DIP25.

 Walter and I are working on DIP74 right now.


 Andrei


As someone who directly complained about this, this is awesome. 
Can't wait to read DIP74.
Feb 25 2015
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei Alexandrescu 
wrote:

 So: does DIP25 allow safe slices? Looks that way


I believe it does, but at the cost of forced reference counting. 
As I pointed out, the `ref` solution is not applicable to slices, 
therefore it needs to return an RCArray. This in turn forces an 
inc/dec whenever it is sliced or copied, and - maybe worse - it 
requires all consumers to support RCArray (or whatever other 
idiosyncratic RC implementation users come up with).
Feb 24 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/24/15 1:55 PM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:

 On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei Alexandrescu wrote:

 So: does DIP25 allow safe slices? Looks that way


 I believe it does, but at the cost of forced reference counting. As I
 pointed out, the `ref` solution


What is the ref solution?


 is not applicable to slices, therefore
 it needs to return an RCArray. This in turn forces an inc/dec whenever
 it is sliced or copied, and - maybe worse - it requires all consumers to
 support RCArray (or whatever other idiosyncratic RC implementation users
 come up with).


It seems to me it's time to have a UniqueArray proof of concept.


Andrei
Feb 24 2015
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 24 February 2015 at 22:11:36 UTC, Andrei Alexandrescu 
wrote:

 On 2/24/15 1:55 PM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:

 On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei 
 Alexandrescu wrote:

 So: does DIP25 allow safe slices? Looks that way


 I believe it does, but at the cost of forced reference 
 counting. As I
 pointed out, the `ref` solution


 What is the ref solution?


DIP25, in that it applies the semantics of `scope` to `ref`. The 
safety gained is thereby restricted to `ref`, but does not apply 
to slices, pointers and class references. That's the reason - as 
you already replied to Ivan - that RCArray cannot return a real 
slice.


 is not applicable to slices, therefore
 it needs to return an RCArray. This in turn forces an inc/dec 
 whenever
 it is sliced or copied, and - maybe worse - it requires all 
 consumers to
 support RCArray (or whatever other idiosyncratic RC 
 implementation users
 come up with).


 It seems to me it's time to have a UniqueArray proof of concept.


How is that related to what I wrote above?
Feb 25 2015
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei Alexandrescu 
wrote:

 I modified Walter's sample code to this: 
 http://dpaste.dzfl.pl/f3d854feede9. It uses malloc for both the 
 array and the reference count, and also uses  trusted 
 minimally. I inserted assert()s here and there to clarify the 
 workings. Nothing big except for the careful use of  trusted.


Using malloc make the GC blind. It means you can't stored 
anything GCed in this. Which means you either need to enforce 
this at the interface level (I have no idea how) or remove all 
these  trusted in the code.

Generally using malloc is not the right way forward. I do think 
that, even for RCed resource, we want them backed by the GC. It 
will allow for cycle collection. The main problem with the GC is 
not allocation, it is collection, and collection won't fire if 
one do not leak.


 2. Michel's point 
 (https://issues.dlang.org/show_bug.cgi?id=14221) reveals the 
 largest issue with RC/GC integration. We need to find a fix for 
 it if we want to have the GC lift cycles.


There is a set of feature that create implicit sharing. GC is 
only one of them. They either need to be fixed (and here, you'll 
get my point about simple fix aggregating to something more 
complex) or make this thread safe.


 So: does DIP25 allow safe slices? Looks that way, though a 
 proof would be nice. Does it allow other safe interesting 
 structs that own data? Very likely.


As long as you don't plan to own an arbitrary sub graph.


 Does it allow really sophisticated ownership schemes? We need 
 to explore that.


Limited, unless you wrap all indirection in some sort of struct 
down the road.


 Does it protect against bugs in implementations of safe 
 ownership schemes that explicitly release memory? Not too well. 
 I think the prevalent idiom will be to accompany such artifacts 
 with unittests that make sure unsafe uses (such as fun() above) 
 do not compile.


I though we wanted to do better than C++...
Feb 24 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/24/2015 2:45 PM, deadalnix wrote:

 Using malloc make the GC blind. It means you can't stored anything GCed in
this.


GC.addRange() solves this problem.



 Generally using malloc is not the right way forward. I do think that, even for
 RCed resource, we want them backed by the GC. It will allow for cycle
 collection.


If cycles are not possible, malloc/free should be used. Cycles are not possible 
in objects without indirections, and we can assert they are not possible it the 
type can be introspected at compile time to see if cycles are not possible.


 So: does DIP25 allow safe slices? Looks that way, though a proof would be
 nice. Does it allow other safe interesting structs that own data? Very likely.


 As long as you don't plan to own an arbitrary sub graph.


To have an arbitrary sub graph be memory safe with return ref, the interface to 
it will have to be constructed to only allow access by values or return refs.



 Does it protect against bugs in implementations of safe ownership schemes that
 explicitly release memory? Not too well. I think the prevalent idiom will be
 to accompany such artifacts with unittests that make sure unsafe uses (such as
 fun() above) do not compile.


 I though we wanted to do better than C++...


No language offers a way to check system code for safety. Not Java, not Rust, 

usage 
of it. C++ doesn't have that.
Feb 24 2015
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 25 February 2015 at 00:11:28 UTC, Walter Bright 
wrote:

 Generally using malloc is not the right way forward. I do 
 think that, even for
 RCed resource, we want them backed by the GC. It will allow 
 for cycle
 collection.


 If cycles are not possible, malloc/free should be used. Cycles 
 are not possible in objects without indirections, and we can 
 assert they are not possible it the type can be introspected at 
 compile time to see if cycles are not possible.


Why ? There is very little chance that malloc + GC.addRange 
becomes any faster that GC.malloc in the first place.


 So: does DIP25 allow safe slices? Looks that way, though a 
 proof would be
 nice. Does it allow other safe interesting structs that own 
 data? Very likely.


 As long as you don't plan to own an arbitrary sub graph.


 To have an arbitrary sub graph be memory safe with return ref, 
 the interface to it will have to be constructed to only allow 
 access by values or return refs.


That means all library code must be duplicated. I could explain 
you why this is bad, but someone already made a very good point 
about it there: 
http://www.drdobbs.com/cpp/type-qualifiers-and-wild-cards/231902461


 Does it protect against bugs in implementations of safe 
 ownership schemes that
 explicitly release memory? Not too well. I think the 
 prevalent idiom will be
 to accompany such artifacts with unittests that make sure 
 unsafe uses (such as
 fun() above) do not compile.


 I though we wanted to do better than C++...


 No language offers a way to check system code for safety. Not 

 is safety for the client's usage of it. C++ doesn't have that.


A language is an API to the system you are running on. A very 
expressive and complex one, but an API nevertheless.

A good API will make correct and safe use easy and incorrect and 
unsafe use convoluted, so they are done on purpose and not by 
mistake.

That is not about proving system code correct (impossible) that 
is about making unprovable code harder to spit out than provable 
one (or getting an error).
Feb 24 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/24/2015 4:41 PM, deadalnix wrote:

 Why ? There is very little chance that malloc + GC.addRange becomes any faster
 that GC.malloc in the first place.


1. GC is designed to do GC, which is more than malloc/free is designed to do. 
Less work usually is faster.

2. There are many malloc/free implementations available, and people can and do 
swap them out to find one that works best for their application. Also, compiler 
vendors often expend a great deal of effort making malloc/free work well. Why 
not take advantage of that? It's like it's pretty hard to compete with the 
system memcpy() - people expend enormous effort on that.



 To have an arbitrary sub graph be memory safe with return ref, the interface
 to it will have to be constructed to only allow access by values or return
refs.


 That means all library code must be duplicated.


Why?



 but someone already made a very good point about it there:
 http://www.drdobbs.com/cpp/type-qualifiers-and-wild-cards/231902461


Don't see how that applies.
Feb 24 2015