www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Safe reference counting cannot be implemented as a library

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
(Title is borrowed from Hans Boehm's famous "Threads cannot be 
implemented as a library", 
http://www.hpl.hp.com/techreports/2004/HPL-2004-209.pdf.)

We want to implement safe reference counting in D, and the sentiment 
that it can be done in a library (with no or minimal changes to the 
language) is often echoed in the newsgroup. This post explains that 
reference counting can be implemented as a library _only_ if safety is 
forgone. If safety is desired, reference counting must be built into 
language semantics.

The crux of the matter is modular typechecking. Consider the following 
example:

// module widget.d
 safe class Widget {
   void fun() {
     g_widget = this;
   }
}
static Widget g_widget;
// end of module widget.d

This is a perfect valid D class, and safe too. The typechecker assumes 
infinite lifetime for all Widget objects, and allows escaping the 
reference to this from foo() into the global.

Numerous similar examples can be constructed involving more elaborate 
escaping patterns than this simple global assignment.

Now, once the typechecker OKs module widget.d, the summary that all 
other typechecking "sees" is:

 safe class Widget {
   void fun();
}

A library reference counting wrapper a la RC!Widget needs to allow calls 
to fun(). Once that happens, the global will hold an alias to the 
respective Widget object indefinitely, which means as soon as the 
RC!Widget object is released by the reference counting protocol, 
g_widget will become a dangling reference.

It follows that if we want safe reference counting, there must be 
language support for it. One possibility is to attach an attribute to 
the class definition:

 safe  rc class Widget {
   ...
}

Then the compiler is able to enforce more stringent typechecking on 
Widget (for example, address of fields are not allowed to escape) and 
also insert the appropriate reference counting logic. This will be a 
topic discussed on the lifetime list.


Andrei
Oct 27 2015
next sibling parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 28/10/15 12:41 AM, Andrei Alexandrescu wrote:

 (Title is borrowed from Hans Boehm's famous "Threads cannot be
 implemented as a library",
 http://www.hpl.hp.com/techreports/2004/HPL-2004-209.pdf.)

 We want to implement safe reference counting in D, and the sentiment
 that it can be done in a library (with no or minimal changes to the
 language) is often echoed in the newsgroup. This post explains that
 reference counting can be implemented as a library _only_ if safety is
 forgone. If safety is desired, reference counting must be built into
 language semantics.

 The crux of the matter is modular typechecking. Consider the following
 example:

 // module widget.d
  safe class Widget {
    void fun() {
      g_widget = this;
    }
 }
 static Widget g_widget;
 // end of module widget.d

 This is a perfect valid D class, and safe too. The typechecker assumes
 infinite lifetime for all Widget objects, and allows escaping the
 reference to this from foo() into the global.

 Numerous similar examples can be constructed involving more elaborate
 escaping patterns than this simple global assignment.

 Now, once the typechecker OKs module widget.d, the summary that all
 other typechecking "sees" is:

  safe class Widget {
    void fun();
 }

 A library reference counting wrapper a la RC!Widget needs to allow calls
 to fun(). Once that happens, the global will hold an alias to the
 respective Widget object indefinitely, which means as soon as the
 RC!Widget object is released by the reference counting protocol,
 g_widget will become a dangling reference.

 It follows that if we want safe reference counting, there must be
 language support for it. One possibility is to attach an attribute to
 the class definition:

  safe  rc class Widget {
    ...
 }

 Then the compiler is able to enforce more stringent typechecking on
 Widget (for example, address of fields are not allowed to escape) and
 also insert the appropriate reference counting logic. This will be a
 topic discussed on the lifetime list.


 Andrei


Thought: we have RTInfo where we can place anything we want for a type.
Perhaps we could explore this avenue a bit?

A library solution, but perhaps in druntime instead of e.g. Phobos.
Add in compiler hooks and wala. Language support for it.

Other random thoughts, a storage attribute cast( rc)myRef;
On a class (as you shown)  rc class Widget {}

I think there is no one solution here. But similar behavior with 
slightly different semantics.

Anyway, the only wrong solution involves:

RefCount!T func(IAllocator alloc=theAllocator) {
	return RefCount!T(alloc.make!T, alloc);
}

Which I'm already doing.
And now I want to join that group... great.
Oct 27 2015
prev sibling next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 27 October 2015 at 21:41, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 It follows that if we want safe reference counting, there must be language
 support for it. One possibility is to attach an attribute to the class
 definition:

  safe  rc class Widget {
   ...
 }


An attribute? Is presence of opInc()/opDec() insufficient? Would the
attribute signal fabrication of some default opInc/opDec operators
applicable for general use?
Oct 27 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/27/2015 07:57 AM, Manu via Digitalmars-d wrote:

 On 27 October 2015 at 21:41, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 It follows that if we want safe reference counting, there must be language
 support for it. One possibility is to attach an attribute to the class
 definition:

  safe  rc class Widget {
    ...
 }


 An attribute? Is presence of opInc()/opDec() insufficient? Would the
 attribute signal fabrication of some default opInc/opDec operators
 applicable for general use?


You're right, opInc/opDec detection would suffice.

Unrelated, and a foreshadowing of the discussion on the lifetime mailing 
list: the compiler has ample opportunity to fuse incs/decs together, so 
the signatures of these functions is:

void opInc(uint delta);
void opDec(uint delta);

For example, consider:

class Widget {
   void fun(Widget, Widget);
}
...
auto w = new Widget;
w.fun(w, w);

In this case the compiler may insert opInc with a value larger than 1 
prior to entering the call.


Andrei
Oct 27 2015
next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 27 October 2015 at 22:27, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 10/27/2015 07:57 AM, Manu via Digitalmars-d wrote:

 On 27 October 2015 at 21:41, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 It follows that if we want safe reference counting, there must be
 language
 support for it. One possibility is to attach an attribute to the class
 definition:

  safe  rc class Widget {
    ...
 }



 An attribute? Is presence of opInc()/opDec() insufficient? Would the
 attribute signal fabrication of some default opInc/opDec operators
 applicable for general use?



 You're right, opInc/opDec detection would suffice.

 Unrelated, and a foreshadowing of the discussion on the lifetime mailing
 list: the compiler has ample opportunity to fuse incs/decs together, so the
 signatures of these functions is:

 void opInc(uint delta);
 void opDec(uint delta);

 For example, consider:

 class Widget {
   void fun(Widget, Widget);
 }
 ...
 auto w = new Widget;
 w.fun(w, w);

 In this case the compiler may insert opInc with a value larger than 1 prior
 to entering the call.


Awesome. Hadn't thought of that. Nice catch!
Perhaps add me to the list. I don't think I have anything to add that
I haven't said before, but I have very high interest in this topic.
Oct 27 2015
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, 27 October 2015 at 12:27:29 UTC, Andrei Alexandrescu 
wrote:

 Unrelated, and a foreshadowing of the discussion on the 
 lifetime mailing list:


What's the lifetime mailing list?

- Jonathan M Davis
Oct 27 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/27/15 11:59 AM, Jonathan M Davis wrote:

 On Tuesday, 27 October 2015 at 12:27:29 UTC, Andrei Alexandrescu wrote:

 Unrelated, and a foreshadowing of the discussion on the lifetime
 mailing list:


 What's the lifetime mailing list?


To be created. -- Andrei
Oct 27 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/27/2015 9:03 AM, Andrei Alexandrescu wrote:

 On 10/27/15 11:59 AM, Jonathan M Davis wrote:

 What's the lifetime mailing list?


 To be created. -- Andrei


And you'll be in it for life, so be sure you want to join :-)
Oct 27 2015
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 10/27/2015 01:27 PM, Andrei Alexandrescu wrote:

 Unrelated, and a foreshadowing of the discussion on the lifetime mailing
 list: the compiler has ample opportunity to fuse incs/decs together, so
 the signatures of these functions is:
 
 void opInc(uint delta);
 void opDec(uint delta);


Any hint/numbers showing that this is actually useful?
Implementing such a cross statement optimization is quite some work. If
this occurs often enough (in particular for shared classes with atomic
ref counting) it might be worth the effort.
Nov 01 2015
next sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 11/01/2015 09:51 PM, Martin Nowak wrote:

 Any hint/numbers showing that this is actually useful?


Also doesn't a good backend optimizer already fuse writes?
Nov 01 2015
next sibling parent rsw0x <anonymous anonymous.com> writes:
On Sunday, 1 November 2015 at 20:55:00 UTC, Martin Nowak wrote:

 On 11/01/2015 09:51 PM, Martin Nowak wrote:

 Any hint/numbers showing that this is actually useful?


 Also doesn't a good backend optimizer already fuse writes?


AFAIK the fear of RC being too slow comes from C++'s shared_ptr's 
reference count being required to be synchronized by the C++ 
spec, which heavily limits what kinds of optimizations that can 
be done on it. Thread-local by default RC should, in theory, be 
able to be optimized much much more aggressively.
Nov 01 2015
prev sibling next sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Sunday, 1 November 2015 at 20:55:00 UTC, Martin Nowak wrote:

 On 11/01/2015 09:51 PM, Martin Nowak wrote:

 Any hint/numbers showing that this is actually useful?


 Also doesn't a good backend optimizer already fuse writes?


Yes but you have this myth flying around that it is necessary for 
good RC, because language like C++ do implicit sharing, so RC 
must be done atomically, so the optimizer can't optimize.
Nov 01 2015
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Sunday, 1 November 2015 at 22:04:51 UTC, deadalnix wrote:

 On Sunday, 1 November 2015 at 20:55:00 UTC, Martin Nowak wrote:

 On 11/01/2015 09:51 PM, Martin Nowak wrote:

 Any hint/numbers showing that this is actually useful?


 Also doesn't a good backend optimizer already fuse writes?


 Yes but you have this myth flying around that it is necessary 
 for good RC, because language like C++ do implicit sharing, so 
 RC must be done atomically, so the optimizer can't optimize.


Yes, ARC does the atomic RC too, but ARC is bound by prior focus 
on manual RC without compiler support and should not be used as 
an example to be followed.

D could go for a more generic solution with specialized 
restricted integer types for resource tracking with different 
capabilites. Same amount of work, but much more versatile.

The library could then use templated RCPointers with the 
capabilites as parameters which recast the resource tracking 
integer to the desired type which injects the appropriate 
assumptions to the optimizer before/after each operation on the 
integer.

E.g. for the non-transfer-to-other-threads/fiber integer counter 
type:

1. the optimizer needs to know that below a specific 
block/stackframe the test "counter==0" always fails.

2. semantics that ensures that  contexts (fiber/thread) only are 
allowed to subtract a value they already have added. Easy way out 
is RAII.
Nov 02 2015
parent reply sclytrack <sclytrack fake.com> writes:
On Monday, 2 November 2015 at 09:56:14 UTC, Ola Fosheim Grøstad 
wrote:

 On Sunday, 1 November 2015 at 22:04:51 UTC, deadalnix wrote:

 On Sunday, 1 November 2015 at 20:55:00 UTC, Martin Nowak wrote:

 On 11/01/2015 09:51 PM, Martin Nowak wrote:







http://lists.puremagic.com/cgi-bin/mailman/listinfo/dlang-study


The last post on the study group ended a while ago. Does that 
mean it is not going forward. It suggest to talk about lifetime. 
Is that with or without talking about
reference counting simultaneously?

What approach will be taken?


1) Will for every assignment to the object a hidden temporary be 
created by the compiler to keep the object alive.

RCObject obj = new RCObject();
Item item = obj.items[x];

_temp1 = obj;       //The compiler inserts this automatically and 
is not seen.
obj=new RCObject();


2) Will you disable the item by some magical way? Item might 
still be valid.

RCObject obj = new RCObject();
...
Item item = obj.items[x];

obj=new RCObject();
//item is no longer accessible beyond this point.
writeln(item); //error


3) Somehow disable the RCObject for the duration of the borrowing.

with(Item item = RCObject.items[x])
{
   writeln(item)
}


4) Solved by eating pizza or watching a movie ...



Maybe the discussion should be a tree.
Dec 05 2015
parent deadalnix <deadalnix gmail.com> writes:
On Saturday, 5 December 2015 at 22:06:33 UTC, sclytrack wrote:

 On Monday, 2 November 2015 at 09:56:14 UTC, Ola Fosheim Grøstad 
 wrote:

 On Sunday, 1 November 2015 at 22:04:51 UTC, deadalnix wrote:

 On Sunday, 1 November 2015 at 20:55:00 UTC, Martin Nowak 
 wrote:

 On 11/01/2015 09:51 PM, Martin Nowak wrote:







 http://lists.puremagic.com/cgi-bin/mailman/listinfo/dlang-study


 The last post on the study group ended a while ago. Does that 
 mean it is not going forward. It suggest to talk about 
 lifetime. Is that with or without talking about
 reference counting simultaneously?


It means that now that all the bikescheding is behind, people can 
work on the hard problems. And it takes time. Especially when you 
have goto int he language.
Dec 05 2015
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/01/2015 03:54 PM, Martin Nowak wrote:

 On 11/01/2015 09:51 PM, Martin Nowak wrote:

 Any hint/numbers showing that this is actually useful?


 Also doesn't a good backend optimizer already fuse writes?


My understanding is that no, that won't happen in most patterns that 
matter. -- Andrei
Nov 01 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/01/2015 03:51 PM, Martin Nowak wrote:

 On 10/27/2015 01:27 PM, Andrei Alexandrescu wrote:

 Unrelated, and a foreshadowing of the discussion on the lifetime mailing
 list: the compiler has ample opportunity to fuse incs/decs together, so
 the signatures of these functions is:

 void opInc(uint delta);
 void opDec(uint delta);


 Any hint/numbers showing that this is actually useful?


Would be great to collect some, and generally get rigorous about the 
whole approach.


 Implementing such a cross statement optimization is quite some work. If
 this occurs often enough (in particular for shared classes with atomic
 ref counting) it might be worth the effort.


Most reference counting techniques revolve around reducing mutation of 
the reference count. See e.g. 
https://users.cecs.anu.edu.au/~steveb/downloads/pdf/rc-ismm-2012.pdf.

So we need to show that many refcount updates take it from 1 to larger 
than 1 and back. According to 
https://users.cecs.anu.edu.au/~steveb/downloads/pdf/rc-ismm-2012.pdf, 
many objects have a reference count of just one; the "eclipse" benchmark 
has 31.8% objects with a refcount greater than 1.


Andrei
Nov 01 2015
parent reply rsw0x <anonymous anonymous.com> writes:
On Sunday, 1 November 2015 at 22:36:46 UTC, Andrei Alexandrescu 
wrote:

 On 11/01/2015 03:51 PM, Martin Nowak wrote:

 On 10/27/2015 01:27 PM, Andrei Alexandrescu wrote:

 Unrelated, and a foreshadowing of the discussion on the 
 lifetime mailing
 list: the compiler has ample opportunity to fuse incs/decs 
 together, so
 the signatures of these functions is:

 void opInc(uint delta);
 void opDec(uint delta);


 Any hint/numbers showing that this is actually useful?


 Would be great to collect some, and generally get rigorous 
 about the whole approach.


 Implementing such a cross statement optimization is quite some 
 work. If
 this occurs often enough (in particular for shared classes 
 with atomic
 ref counting) it might be worth the effort.


 Most reference counting techniques revolve around reducing 
 mutation of the reference count. See e.g. 
 https://users.cecs.anu.edu.au/~steveb/downloads/pdf/rc-ismm-2012.pdf.

 So we need to show that many refcount updates take it from 1 to 
 larger than 1 and back. According to 
 https://users.cecs.anu.edu.au/~steveb/downloads/pdf/rc-ismm-2012.pdf, many
objects have a reference count of just one; the "eclipse" benchmark has 31.8%
objects with a refcount greater than 1.


 Andrei


That paper is assuming that you take Java(a language that does 
*not* have allocation patterns like D such as favoring data on 
the stack, tightly packed arrays of data, and immutability) rip 
out its GC, and replace it with a RC-based GC with no concept of 
unique ownership - no?
Nov 01 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/1/15 5:52 PM, rsw0x wrote:

 On Sunday, 1 November 2015 at 22:36:46 UTC, Andrei Alexandrescu wrote:

 On 11/01/2015 03:51 PM, Martin Nowak wrote:

 On 10/27/2015 01:27 PM, Andrei Alexandrescu wrote:

 Unrelated, and a foreshadowing of the discussion on the lifetime
 mailing
 list: the compiler has ample opportunity to fuse incs/decs together, so
 the signatures of these functions is:

 void opInc(uint delta);
 void opDec(uint delta);


 Any hint/numbers showing that this is actually useful?


 Would be great to collect some, and generally get rigorous about the
 whole approach.


 Implementing such a cross statement optimization is quite some work. If
 this occurs often enough (in particular for shared classes with atomic
 ref counting) it might be worth the effort.


 Most reference counting techniques revolve around reducing mutation of
 the reference count. See e.g.
 https://users.cecs.anu.edu.au/~steveb/downloads/pdf/rc-ismm-2012.pdf.

 So we need to show that many refcount updates take it from 1 to larger
 than 1 and back. According to
 https://users.cecs.anu.edu.au/~steveb/downloads/pdf/rc-ismm-2012.pdf,
 many objects have a reference count of just one; the "eclipse"
 benchmark has 31.8% objects with a refcount greater than 1.


 Andrei


 That paper is assuming that you take Java(a language that does *not*
 have allocation patterns like D such as favoring data on the stack,
 tightly packed arrays of data, and immutability) rip out its GC, and
 replace it with a RC-based GC with no concept of unique ownership - no?


The class objects we're focusing on for RC support are supposed to be 
used much like in Java. -- Andrei
Nov 01 2015
prev sibling next sibling parent reply deadalnix <deadalnix gmail.com> writes:
I've made the claim that we should implement reference counting 
as a library many time, so I think I should explicit my position. 
Indeed, RC require some level a compiler support to be safe. That 
being said, the support does not need to be specific to RC. On 
fact, my position is that the language should provide some basic 
mechanism on top of which safe RC can be implemented, as a 
library.

The problem at hand here is escape analysis. The compiler must be 
able to ensure that a reference doesn't escape the RC mechanism 
in an uncontrolled manner. I'd like to add such mechanism to the 
language rather than bake in reference counting, as it can be 
used to solve other problem we are facing today ( nogc exception 
for instance).
Oct 27 2015
next sibling parent reply PuglyWUgly <nospamPls yahoo.com> writes:
  Why care about this?

  Even Rust doesn't try to solve this problem...because it isn't 
really a problem in practice.

  In c++/rust code you use value/unique types 99.9% of the time, 
and reference counting is only for shared/aliased objects.


  Reference counting == rare and unimportant
Oct 27 2015
next sibling parent Namespace <rswhite4 gmail.com> writes:
On Tuesday, 27 October 2015 at 20:19:42 UTC, PuglyWUgly wrote:

  Why care about this?

  Even Rust doesn't try to solve this problem...because it isn't 
 really a problem in practice.

  In c++/rust code you use value/unique types 99.9% of the time, 
 and reference counting is only for shared/aliased objects.


  Reference counting == rare and unimportant


+1
Oct 27 2015
prev sibling next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Tuesday, 27 October 2015 at 20:19:42 UTC, PuglyWUgly wrote:

  Why care about this?

  Even Rust doesn't try to solve this problem...because it isn't 
 really a problem in practice.

  In c++/rust code you use value/unique types 99.9% of the time, 
 and reference counting is only for shared/aliased objects.


I agree with your sentiment that local reference counting is 
seldom needed, but shared reference counting is needed for shared 
cached objects. So shared (atomic) reference counting have a 
common and important use case.

Not sure if shared ref counting is addressed here at all though.
Oct 27 2015
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, 27 October 2015 at 20:19:42 UTC, PuglyWUgly wrote:

  Why care about this?

  Even Rust doesn't try to solve this problem...because it isn't 
 really a problem in practice.

  In c++/rust code you use value/unique types 99.9% of the time, 
 and reference counting is only for shared/aliased objects.


  Reference counting == rare and unimportant


Really? I've seen tons of C++ code that's written using smart 
pointers with objects living on the heap which then get passed 
around all over the place.

Sure, a lot of stuff in D should be structs on the stack, but 
there are plenty of cases where you need stuff on the heap, in 
which case, you either have to let the GC take care of it (which 
means no deterministic destruction), have something specific own 
it and destroy it when it's no longer needed, or reference count 
it so that it gets destroyed immediately after it's no longer 
needed. For cases where you don't care about deterministic 
destruction, using the GC is fine, but for those cases where 
deterministic destruction is required (e.g. because the object 
currently has ownership of an OS resource), the GC doesn't cut 
it, and ref-counting is very much what's needed.

If/when we introduce ref-counting into the language, I fully 
expect that there will be a lot of D programs written which 
specifically use it in order to avoid the GC. And while in many 
cases, that's going to be an unnecessary, in some cases, it'll be 
a lifesaver.

As it stands, we can add ref-counting via libraries just fine, 
but it requires giving up on  safe, which we could probably live 
with, but it would make  safe a lot less valuable in the long 
run. So, a solution that enables  safe ref-counting in D would 
certainly be desirable, and we definitely need a language 
improvement of some kind if we want to be able to ref-count 
something like exceptions (be it by building ref-counting into 
the language like Andrei is proposing or by adding other features 
which enable us to build it in the library like deadalnix is 
proposing).

- Jonathan M Davis
Oct 27 2015
next sibling parent reply rsw0x <anonymous anonymous.com> writes:
On Tuesday, 27 October 2015 at 20:41:49 UTC, Jonathan M Davis 
wrote:

 On Tuesday, 27 October 2015 at 20:19:42 UTC, PuglyWUgly wrote:

  Why care about this?

  Even Rust doesn't try to solve this problem...because it 
 isn't really a problem in practice.

  In c++/rust code you use value/unique types 99.9% of the 
 time, and reference counting is only for shared/aliased 
 objects.


  Reference counting == rare and unimportant


 Really? I've seen tons of C++ code that's written using smart 
 pointers with objects living on the heap which then get passed 
 around all over the place.


if they're using shared_ptr all over the place, they're doing it 
wrong.
shared_ptr is supposed to be a last resort
Oct 27 2015
next sibling parent deadalnix <deadalnix gmail.com> writes:
On Tuesday, 27 October 2015 at 20:45:34 UTC, rsw0x wrote:

 if they're using shared_ptr all over the place, they're doing 
 it wrong.
 shared_ptr is supposed to be a last resort


According to Herb Sutter, it is a zero cost abstraction, not like 
these pesky GC that makes your program slow.
Oct 27 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/27/2015 04:45 PM, rsw0x wrote:

 On Tuesday, 27 October 2015 at 20:41:49 UTC, Jonathan M Davis wrote:

 On Tuesday, 27 October 2015 at 20:19:42 UTC, PuglyWUgly wrote:

  Why care about this?

  Even Rust doesn't try to solve this problem...because it isn't
 really a problem in practice.

  In c++/rust code you use value/unique types 99.9% of the time, and
 reference counting is only for shared/aliased objects.


  Reference counting == rare and unimportant


 Really? I've seen tons of C++ code that's written using smart pointers
 with objects living on the heap which then get passed around all over
 the place.


 if they're using shared_ptr all over the place, they're doing it wrong.
 shared_ptr is supposed to be a last resort


This is awesomely Kafkian. So we have no problem after all - just like 
Go with generics :o). -- Andrei
Oct 27 2015
parent reply rsw0x <anonymous anonymous.com> writes:
On Tuesday, 27 October 2015 at 21:20:58 UTC, Andrei Alexandrescu 
wrote:

 On 10/27/2015 04:45 PM, rsw0x wrote:

 On Tuesday, 27 October 2015 at 20:41:49 UTC, Jonathan M Davis 
 wrote:

 On Tuesday, 27 October 2015 at 20:19:42 UTC, PuglyWUgly wrote:

  Why care about this?

  Even Rust doesn't try to solve this problem...because it 
 isn't
 really a problem in practice.

  In c++/rust code you use value/unique types 99.9% of the 
 time, and
 reference counting is only for shared/aliased objects.


  Reference counting == rare and unimportant


 Really? I've seen tons of C++ code that's written using smart 
 pointers
 with objects living on the heap which then get passed around 
 all over
 the place.


 if they're using shared_ptr all over the place, they're doing 
 it wrong.
 shared_ptr is supposed to be a last resort


 This is awesomely Kafkian. So we have no problem after all - 
 just like Go with generics :o). -- Andrei


It has been a great success for Rust, I rarely ever see RC used 
anywhere in Rust code thanks to borrowing. The new C++ core 
guidelines are also heavily based on this cf. *_view types in GSL.

The problem D has is that RC isn't even implemented at library 
level let alone in a state where it's unsafe. IIRC Phobos RC 
doesn't even allow classes.
Oct 27 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/27/2015 05:26 PM, rsw0x wrote:

 It has been a great success for Rust, I rarely ever see RC used anywhere
 in Rust code thanks to borrowing. The new C++ core guidelines are also
 heavily based on this cf. *_view types in GSL.


You can safely ignore the C++ part, the views are unsafe. I'd appreciate 
if you backed up your claim on Rust. -- Andrei
Oct 27 2015
next sibling parent rsw0x <anonymous anonymous.com> writes:
On Tuesday, 27 October 2015 at 21:50:15 UTC, Andrei Alexandrescu 
wrote:

 On 10/27/2015 05:26 PM, rsw0x wrote:

 It has been a great success for Rust, I rarely ever see RC 
 used anywhere
 in Rust code thanks to borrowing. The new C++ core guidelines 
 are also
 heavily based on this cf. *_view types in GSL.


 You can safely ignore the C++ part, the views are unsafe. I'd 
 appreciate if you backed up your claim on Rust. -- Andrei


rustc's source has about a 5:1 Box:Rc usage ratio after a quick 
grepping of the codebase(ignoring tests ofcourse.) This isn't 
counting all the cases where borrowed pointers were used instead 
of RC(which has about a 33.4:1 Borrowed:Rc usage after a quick 
grepping)
Oct 27 2015
prev sibling next sibling parent deadalnix <deadalnix gmail.com> writes:
On Tuesday, 27 October 2015 at 21:50:15 UTC, Andrei Alexandrescu 
wrote:

 On 10/27/2015 05:26 PM, rsw0x wrote:

 It has been a great success for Rust, I rarely ever see RC 
 used anywhere
 in Rust code thanks to borrowing. The new C++ core guidelines 
 are also
 heavily based on this cf. *_view types in GSL.


 You can safely ignore the C++ part, the views are unsafe. I'd 
 appreciate if you backed up your claim on Rust. -- Andrei


That is ultimately irrelevant. Be it unique_ptr or shared_ptr, 
the same problem arise: this must not escape.
Oct 27 2015
prev sibling next sibling parent PuglyWUgly <nospamPls yahoo.com> writes:
 You can safely ignore the C++ part, the views are unsafe. I'd 
 appreciate if you backed up your claim on Rust. -- Andrei



  I did a rough check of the Rust compiler source(from a copy I 
downloaded a couple months ago).

I think the compiler is supposedly filled with old code, not sure 
if it is the best example of Rust.

  I filtered out the rc/arc.rs files, and doc/test folders

  Some of these hits were comments/commented out, but I'm too lazy 
to bother removing them

Anyway:

Searching 7524 files for "Rc::new(" (case sensitive)

122 matches across 49 files

Searching 7524 files for "Arc::new(" (case sensitive)

60 matches across 28 files
Oct 27 2015
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, 27 October 2015 at 21:50:15 UTC, Andrei Alexandrescu 
wrote:

 On 10/27/2015 05:26 PM, rsw0x wrote:

 It has been a great success for Rust, I rarely ever see RC 
 used anywhere
 in Rust code thanks to borrowing. The new C++ core guidelines 
 are also
 heavily based on this cf. *_view types in GSL.


 You can safely ignore the C++ part, the views are unsafe. I'd 
 appreciate if you backed up your claim on Rust. -- Andrei


Well, even if the claims for Rust and C++ were both 100% true, 
the fact remains that we need to do something about ref-counting 
in D, because the alternative is the GC, which is not necessarily 
acceptable. We already put lots of stuff on the stack and 
encourage avoiding allocations (so, in that respect, we're 
already doing what it's being suggested C++ and Rust do) - but 
some code simply needs to allocate to do what it does, and while 
the GC actually works fantastically for that in many cases, there 
are cases where it doesn't - particularly when you need 
deterministic destruction. So, even if if ref-counting in D ends 
up being very rare, and even if we don't care about  safety, we 
still need a way to support ref-counting for classes (preferably 
a standard way). And since we do care about  safety, it behooves 
us to at least investigate what the possible solutions are for 
having  safe ref-counting in D (and preferably implement one of 
them) rather than simply accepting that ref-counting can't be 
 safe.

Personally, I think that one of the most important aspects of 
this is finding a way to make it possible to reasonably use 
exceptions in  nogc code. Right now,  nogc is almost synonymous 
with nothrow, which is not at all good IMHO. I do not want us to 
start doing stuff like returning error codes just so that our 
code can be  nogc.

- Jonathan M Davis
Oct 27 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-10-27 22:50, Andrei Alexandrescu wrote:


 You can safely ignore the C++ part, the views are unsafe. I'd appreciate
 if you backed up your claim on Rust. -- Andrei


Rust is unsafe as well, when you interface with unsafe code.

-- 
/Jacob Carlborg
Oct 28 2015
prev sibling next sibling parent bitwise <bitwise.pvt gmail.com> writes:
On Tuesday, 27 October 2015 at 20:41:49 UTC, Jonathan M Davis 
wrote:

 On Tuesday, 27 October 2015 at 20:19:42 UTC, PuglyWUgly wrote:

  [...]


 Really? I've seen tons of C++ code that's written using smart 
 pointers with objects living on the heap which then get passed 
 around all over the place.

 [...]


I can't believe you actually took the time to dignify this with a 
response...but on the other hand, I previously had no opinion 
about  safety...But if its a reason to include language level ref 
counting, then I guess I'm all for it :)

    Bit
Oct 27 2015
prev sibling parent reply PuglyWUgly <nospamPls yahoo.com> writes:
 Really? I've seen tons of C++ code that's written using smart 
 pointers with objects living on the heap which then get passed 
 around all over the place.


  Yes there is lots of old C++ code that does this, largely 
because unique_ptr wasn't implementable until C++11 added R value 
refs and move semantics.


 Sure, a lot of stuff in D should be structs on the stack, but 
 there are plenty of cases where you need stuff on the heap, in 
 which case, you either have to let the GC take care of it 
 (which means no deterministic destruction), have something 
 specific own it and destroy it when it's no longer needed, or 
 reference count it so that it gets destroyed immediately after 
 it's no longer needed.


  on the heap == unique_ptr
  on the heap + multiple owners = shared_ptr

  If you work in a codebase that properly uses unique types, like 
Rust does by default, you soon notice how rare multiple owners 
actually is.
Oct 27 2015
parent PuglyWUgly <nospamPls yahoo.com> writes:
  Well perhaps D prefers the Swift route, with everyone ref 
counted & some compiler help.

Nothing wrong with that I guess, I haven't been paying attention 
to what D is planning exactly.
Oct 27 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/27/2015 04:19 PM, PuglyWUgly wrote:

   Reference counting == rare and unimportant


That doesn't seem to be the case at all. -- Andrei
Oct 27 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-10-27 22:19, Andrei Alexandrescu wrote:


 That doesn't seem to be the case at all. -- Andrei


I'm not a C++ or Rust expert. But I think that in Rust and with the new 
C++ guide lines the idea is to use reference counting pointers only for 
owning resources. If you want to pass the data to some of part of the 
code, that does not need to own the resource, a raw pointer should be used.

-- 
/Jacob Carlborg
Oct 28 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/28/2015 04:13 AM, Jacob Carlborg wrote:

 On 2015-10-27 22:19, Andrei Alexandrescu wrote:


 That doesn't seem to be the case at all. -- Andrei


 I'm not a C++ or Rust expert. But I think that in Rust and with the new
 C++ guide lines the idea is to use reference counting pointers only for
 owning resources. If you want to pass the data to some of part of the
 code, that does not need to own the resource, a raw pointer should be used.


Problem with that is in C++ it's just unsafe and in Rust it's requires 
too much work from the programmer. We don't think either style is best 
for D. -- Andrei
Oct 28 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/28/2015 7:58 AM, Andrei Alexandrescu wrote:

 On 10/28/2015 04:13 AM, Jacob Carlborg wrote:

 On 2015-10-27 22:19, Andrei Alexandrescu wrote:


 That doesn't seem to be the case at all. -- Andrei


 I'm not a C++ or Rust expert. But I think that in Rust and with the new
 C++ guide lines the idea is to use reference counting pointers only for
 owning resources. If you want to pass the data to some of part of the
 code, that does not need to own the resource, a raw pointer should be used.


 Problem with that is in C++ it's just unsafe and in Rust it's requires too much
 work from the programmer. We don't think either style is best for D. -- Andrei


Let's continue this in the mailing list.
Oct 28 2015
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Tuesday, 27 October 2015 at 18:10:18 UTC, deadalnix wrote:

 The problem at hand here is escape analysis. The compiler must 
 be able to ensure that a reference doesn't escape the RC 
 mechanism in an uncontrolled manner. I'd like to add such 
 mechanism to the language rather than bake in reference 
 counting, as it can be used to solve other problem we are 
 facing today ( nogc exception for instance).


I would also have a definite interest in this for the direction 
I'm considering taking with an alternative std.random design.

Technically, I suspect my particular use-case is covered by 
DIP25, but a more comprehensive solution to escape analysis would 
definitely make it more secure.
Oct 27 2015
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/27/2015 11:10 AM, deadalnix wrote:

 I've made the claim that we should implement reference counting as a library
 many time, so I think I should explicit my position. Indeed, RC require some
 level a compiler support to be safe. That being said, the support does not need
 to be specific to RC. On fact, my position is that the language should provide
 some basic mechanism on top of which safe RC can be implemented, as a library.



It's not just safety. If the compiler knows that reference counting is going
on, 
it can potentially elide a lot of the overhead. If it is faced with an
arbitrary 
library solution, it only has a worm's eye view of it, and cannot do higher 
level optimizations.
Oct 27 2015
next sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Wednesday, 28 October 2015 at 01:13:16 UTC, Walter Bright 
wrote:

 It's not just safety. If the compiler knows that reference 
 counting is going on, it can potentially elide a lot of the 
 overhead. If it is faced with an arbitrary library solution, it 
 only has a worm's eye view of it, and cannot do higher level 
 optimizations.


I don't think the compiler can do that much more, but before I 
address that point, let me mention that intrinsic can be added 
for inc and dec, which would be much more lightweight for the 
language at large.

Now as to why I think this wouldn't give that much. First, if 
exception can be thrown, then all bets are pretty much off, as 
inc and dec do not go by pairs anymore. So we are down to the no 
exception situation. In that case, pairs are fairly visible to 
the compiler and can be optimized away or combined, that is 
already the kind of things that optimizer are good at.

But if, so, how do you explain the C++ situation, where nothing 
is elided (or ObjC's) ?

Well, there is a major difference with these languages: sharing 
by default. It means that inc and dec must be (atomic and 
ordered) or synchronized, which means that, as far as the 
compiler is concerned, all bets are off and the optimizer can't 
do its job.

This doesn't really apply to D, so I don't expect this to be a 
problem. And even if there is: intrinsic can save the day to hint 
the optimizer, no need for heavyweight language addition.

Now, let's get back to the exception case, as it is IMO the most 
interesting one. What if one is willing to accept leakage on 
exception throwing. That would get the optimizer back into the 
game and remove a lot of "dark matter" as Andrei calls it, which 
have a real cost on term of icache pressure and exception 
unwinding cost (one now doesn't have to resume each frame to 
maintain refcount).

If I had to go about this, I'd rather see the introduction a 
scope(exit/success/failure) like mechanism for destructors rather 
than something ref counting specific.
Oct 27 2015
next sibling parent rsw0x <anonymous anonymous.com> writes:
On Wednesday, 28 October 2015 at 03:55:25 UTC, deadalnix wrote:

 If I had to go about this, I'd rather see the introduction a 
 scope(exit/success/failure) like mechanism for destructors 
 rather than something ref counting specific.


can you expand upon this?
Oct 27 2015
prev sibling parent reply Paulo Pinto <pjmlp progtools.org> writes:
On Wednesday, 28 October 2015 at 03:55:25 UTC, deadalnix wrote:

 On Wednesday, 28 October 2015 at 01:13:16 UTC, Walter Bright 
 wrote:

 It's not just safety. If the compiler knows that reference 
 counting is going on, it can potentially elide a lot of the 
 overhead. If it is faced with an arbitrary library solution, 
 it only has a worm's eye view of it, and cannot do higher 
 level optimizations.


 I don't think the compiler can do that much more, but before I 
 address that point, let me mention that intrinsic can be added 
 for inc and dec, which would be much more lightweight for the 
 language at large.

 Now as to why I think this wouldn't give that much. First, if 
 exception can be thrown, then all bets are pretty much off, as 
 inc and dec do not go by pairs anymore. So we are down to the 
 no exception situation. In that case, pairs are fairly visible 
 to the compiler and can be optimized away or combined, that is 
 already the kind of things that optimizer are good at.

 But if, so, how do you explain the C++ situation, where nothing 
 is elided (or ObjC's) ?

 Well, there is a major difference with these languages: sharing 
 by default. It means that inc and dec must be (atomic and 
 ordered) or synchronized, which means that, as far as the 
 compiler is concerned, all bets are off and the optimizer can't 
 do its job.

 This doesn't really apply to D, so I don't expect this to be a 
 problem. And even if there is: intrinsic can save the day to 
 hint the optimizer, no need for heavyweight language addition.

 Now, let's get back to the exception case, as it is IMO the 
 most interesting one. What if one is willing to accept leakage 
 on exception throwing. That would get the optimizer back into 
 the game and remove a lot of "dark matter" as Andrei calls it, 
 which have a real cost on term of icache pressure and exception 
 unwinding cost (one now doesn't have to resume each frame to 
 maintain refcount).

 If I had to go about this, I'd rather see the introduction a 
 scope(exit/success/failure) like mechanism for destructors 
 rather than something ref counting specific.


Objective-C  does elide refcounting, there are a few WWDC ARC 
sessions where it is mentioned. Same applies to Swift.

However their exceptions work in a more RC friendly way.
Oct 27 2015
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-10-28 07:07, Paulo Pinto wrote:


 However their exceptions work in a more RC friendly way.


Swift doesn't support exceptions. And in Objective-C exceptions are like 
Errors in D. They should not be caught and the program should terminate.

The error handling support that was added in Swift 2.0 is syntax sugar 
for the Objective-C pattern to use NSError out parameters for error 
handling.

-- 
/Jacob Carlborg
Oct 28 2015
parent reply Paulo Pinto <pjmlp progtools.org> writes:
On Wednesday, 28 October 2015 at 08:07:40 UTC, Jacob Carlborg 
wrote:

 On 2015-10-28 07:07, Paulo Pinto wrote:


 However their exceptions work in a more RC friendly way.


 Swift doesn't support exceptions. And in Objective-C exceptions 
 are like Errors in D. They should not be caught and the program 
 should terminate.

 The error handling support that was added in Swift 2.0 is 
 syntax sugar for the Objective-C pattern to use NSError out 
 parameters for error handling.


Hence why I mentioned they are more RC friendly.

Swift, because it doesn't have them.

Objective-C, because termination is the only option so no need to 
worry about preserving counters.

I was typing on the phone, so didn't want to provide the full 
explanation.

--
Paulo
Oct 28 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-10-28 09:50, Paulo Pinto wrote:


 Hence why I mentioned they are more RC friendly.

 Swift, because it doesn't have them.

 Objective-C, because termination is the only option so no need to worry
 about preserving counters.

 I was typing on the phone, so didn't want to provide the full explanation.


Fair enough :)

-- 
/Jacob Carlborg
Oct 28 2015
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Wednesday, 28 October 2015 at 06:07:12 UTC, Paulo Pinto wrote:

 Objective-C  does elide refcounting, there are a few WWDC ARC 
 sessions where it is mentioned. Same applies to Swift.


Indeed, John McCall from Apple has already described how ARC 
works in these forums (astonishingly nobody felt like thanking 
him for the input... :-/):

http://forum.dlang.org/post/hgmhgirfervrsvcghchw forum.dlang.org

To what extent you can elide inc/dec depends on how you define 
and track ownership and whether you do whole program analysis, of 
course.
Oct 28 2015
prev sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 28 October 2015 at 11:13, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 10/27/2015 11:10 AM, deadalnix wrote:

 I've made the claim that we should implement reference counting as a
 library
 many time, so I think I should explicit my position. Indeed, RC require
 some
 level a compiler support to be safe. That being said, the support does not
 need
 to be specific to RC. On fact, my position is that the language should
 provide
 some basic mechanism on top of which safe RC can be implemented, as a
 library.




 It's not just safety. If the compiler knows that reference counting is going
 on, it can potentially elide a lot of the overhead. If it is faced with an
 arbitrary library solution, it only has a worm's eye view of it, and cannot
 do higher level optimizations.


I just want to drop in that I strongly feel both points here, they are
not at odds. I've been arguing for years now that D needs effective
escape analysis, this will allow all sorts of safe allocation and
lifetime patterns; and while it may enable some improved library
solutions to refcounting, I think the key advantage is actually
related to making better and safe use of stack allocation. I think
that is a much better focus when considering the need for
comprehensive escape analysis tools.
That has little to do with the language also benefiting from RC
primitives such that the compiler is able to do a quality job of
optimising ref-counting, which is a spectacularly prevalent pattern,
particularly so when used in conjunction with libraries such that the
inc/dec functions are opaque indirect calls into some foreign lib and
can't be optimised (this is the majority case in my experience). If
they can't be wrapped by a language primitive that it knows can
optimise this particular calling pattern, then the compiler has no
power to optimise such opaque calls at all.

As an anecdote, since I operate almost exclusively via practical
experience; my current project would heavily benefit from both, and
they would each contribute to a strong case for migration to D. These
2 issues alone represent, by far, the greatest trouble we face with
C++ currently.
RC is okay-ish in C++11 (with rval references), although it could be
much better, for instance, the type mangling/wrapping induced by this
sort of library solution always leads to awkward situations, ie,
'this' pointer in a method is not an RC object anymore! Methods can't
give out pointers to themselves (ie, signaling events where it's
conventional to pass a 'sender' to the subscribers). Pretty massive
fail!
But what we completely fail at is making good use of stack allocation;
requiring conservative fallback to heap allocations because we have no
proof mechanism for containing temporary ownership. We need expressive
escape analysis.

This is a very heavily object orientated codebase, rife with shared
pointers, with a strong focus on the external API and user
extensibility. Focus on the public API implies conservative allocation
habits; ie, RC is prevalent because we don't want to place complex
restrictions on users, and we must also be safe. If we has an
effective escape analysis mechanism, we would gain a lot of
opportunities to revert RC to stack allocations because we can
statically prove via the API that the user won't escape pointers.

The program consists of a typical hierarchical ownership structure, an
arbitrarily shared generalised resource pool, a highly interactive
scene graph with runtime datasets scaling to 10s of gigabytes, and a
ridiculously abstract API's (lots of painful C++ meta-magic). It is
also realtime. GC was considered and rejected on the premise that it
is realtime, and operates on truly gargantuan working datasets.
It's the most ambitious thing I've ever written, and I am dying inside
a little bit more every single day that I remain stuck with C++. I
want to start writing front-end plugin code in D as soon as possible,
which means, at very least, comprehensive RC interaction.
Oct 28 2015
parent reply David Nadlinger <code klickverbot.at> writes:
On Wednesday, 28 October 2015 at 11:21:17 UTC, Manu wrote:

 RC is okay-ish in C++11 (with rval references), although it 
 could be
 much better, for instance, the type mangling/wrapping induced 
 by this
 sort of library solution always leads to awkward situations, ie,
 'this' pointer in a method is not an RC object anymore! Methods 
 can't
 give out pointers to themselves (ie, signaling events where it's
 conventional to pass a 'sender' to the subscribers). Pretty 
 massive
 fail!


Did you look into doing something like 
std::enable_shared_from_this? I use it pretty routinely in 
networking code (boost.asio), and while it is not as pretty as it 
could be, it does the trick.

  – David
Oct 28 2015
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 28 October 2015 at 21:29, David Nadlinger via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Wednesday, 28 October 2015 at 11:21:17 UTC, Manu wrote:

 RC is okay-ish in C++11 (with rval references), although it could be
 much better, for instance, the type mangling/wrapping induced by this
 sort of library solution always leads to awkward situations, ie,
 'this' pointer in a method is not an RC object anymore! Methods can't
 give out pointers to themselves (ie, signaling events where it's
 conventional to pass a 'sender' to the subscribers). Pretty massive
 fail!



 Did you look into doing something like std::enable_shared_from_this? I use
 it pretty routinely in networking code (boost.asio), and while it is not as
 pretty as it could be, it does the trick.


I did. We ended up with custom solutions, tailored to take advantage
of our specific and fairly unconventional environment. Our RC is
invasive in most cases, using a duck-typed approach to the general
API.
I'm not really proud that we re-invented that wheel (among many,
because C++ offerings are generally insufficient), but it had to be
done.
Oct 28 2015
prev sibling parent reply Zach the Mystic <reachzach gggmail.com> writes:
On Tuesday, 27 October 2015 at 18:10:18 UTC, deadalnix wrote:

 I've made the claim that we should implement reference counting 
 as a library many time, so I think I should explicit my 
 position. Indeed, RC require some level a compiler support to 
 be safe. That being said, the support does not need to be 
 specific to RC. On fact, my position is that the language 
 should provide some basic mechanism on top of which safe RC can 
 be implemented, as a library.

 The problem at hand here is escape analysis. The compiler must 
 be able to ensure that a reference doesn't escape the RC 
 mechanism in an uncontrolled manner. I'd like to add such 
 mechanism to the language rather than bake in reference 
 counting, as it can be used to solve other problem we are 
 facing today ( nogc exception for instance).


Here's a link to the reference safety system I proposed some 
months ago:

http://forum.dlang.org/post/offurllmuxjewizxedab forum.dlang.org

I'm very far from having the expertise needed to know whether it 
would be worth its weight in practice, but it was better to write 
it out than to keep it bottled up in my head. I hope it will be 
of some use.
Oct 29 2015
parent sclytrack <sclytrack fake.com> writes:
On Thursday, 29 October 2015 at 20:31:49 UTC, Zach the Mystic 
wrote:

 On Tuesday, 27 October 2015 at 18:10:18 UTC, deadalnix wrote:

 I've made the claim that we should implement reference 
 counting as a library many time, so I think I should explicit 
 my position. Indeed, RC require some level a compiler support 
 to be safe. That being said, the support does not need to be 
 specific to RC. On fact, my position is that the language 
 should provide some basic mechanism on top of which safe RC 
 can be implemented, as a library.

 The problem at hand here is escape analysis. The compiler must 
 be able to ensure that a reference doesn't escape the RC 
 mechanism in an uncontrolled manner. I'd like to add such 
 mechanism to the language rather than bake in reference 
 counting, as it can be used to solve other problem we are 
 facing today ( nogc exception for instance).


 Here's a link to the reference safety system I proposed some 
 months ago:

 http://forum.dlang.org/post/offurllmuxjewizxedab forum.dlang.org

 I'm very far from having the expertise needed to know whether 
 it would be worth its weight in practice, but it was better to 
 write it out than to keep it bottled up in my head. I hope it 
 will be of some use.



1) Assignment to RCObject
-------------------------


{
	RCObject obj = new RCObject()
	Item item1 = obj.items[x];
	
	_temp1 = obj;
	obj = new RCObject();


	_temp2 = obj;
	obj = new RCObject();
}



2) Entering a subscope
----------------------

With this I mean entering a subscope that assigns to the RCObject.


{
	RCObject obj2 = new RCObject()
	RCObject obj = new RCObject()
	

	_temp1 = obj;				//(2) Entering a subscope
	{
		_temp2 = obj;			//(1) Assignment to RCObject
		obj = new RCObject();
	}
}


3) Leaving a scope.
-------------------

The Item is not reference counted.

{
	Item item1;
	{
		RCObject obj = new RCObject();
		//item1 = obj.items[x];		//(3) Leaving subscope
	}
}



{
	RCObject obj = new RCObject();
	Item item1;
	
	_temp1 = obj;				//(2) Entering subscope
	{
		_temp2 = obj;			//(1) Assignement to RCObject
		obj = new RCObject();
		//item1 = obj.items[x];		//(3) Leaving subscope

		_temp3 = obj;			//(1) Assignement to RCObject
		obj = new RCObject();
	}
}




4) RCObject joins a scope
-------------------------


{
	_temp1 = obj.field.rcobject;		//(4) RCObject joins current scope.
	Item item1 = obj.field.rcobject.a.items[0];


	//_temp1;				//(2) Entering subscope
	{
		_temp3 = obj.field.rcobject;	//(1) Assignment to RCObject
		obj.field.rcobject = new RCObject();
	}

	_temp4 = obj.field.rcobject;		//(4) RCObject joins current scope.
	item1 = obj.field.rcobject.a.items[2];
}
Oct 31 2015
prev sibling next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Tuesday, 27 October 2015 at 11:41:52 UTC, Andrei Alexandrescu 
wrote:

 The crux of the matter is modular typechecking. Consider the 
 following example:

 // module widget.d
  safe class Widget {
   void fun() {
     g_widget = this;
   }
 }
 static Widget g_widget;
 // end of module widget.d

 Now, once the typechecker OKs module widget.d, the summary that 
 all other typechecking "sees" is:

  safe class Widget {
   void fun();
 }


Isn't it a shame that that kind of information gets tossed aside? 
Seems to be very valuable and the loss of it the cause of several 
issues.
Oct 27 2015
parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, 28 October 2015 at 02:08:18 UTC, Sebastiaan Koppe 
wrote:

 On Tuesday, 27 October 2015 at 11:41:52 UTC, Andrei 
 Alexandrescu wrote:

 The crux of the matter is modular typechecking. Consider the 
 following example:

 // module widget.d
  safe class Widget {
   void fun() {
     g_widget = this;
   }
 }
 static Widget g_widget;
 // end of module widget.d

 Now, once the typechecker OKs module widget.d, the summary 
 that all other typechecking "sees" is:

  safe class Widget {
   void fun();
 }


 Isn't it a shame that that kind of information gets tossed 
 aside? Seems to be very valuable and the loss of it the cause 
 of several issues.


Perhaps, but that assignment is perfectly  safe. What isn't  safe 
is when you then free the Widget later because a ref-count hit 0. 
And that code already has to be  system or  trusted given what 
it's doing. So, really, the problem is that some  safe operations 
can lead to problems when  system code does stuff that assumes 
that those operations did not occur. So, unfortunately, whether 
that code can be  trusted depends on what some  safe code did 
rather than  system code like you'd normally expect. But the 
compiler simply doesn't understand enough about why something is 
 system to be able to figure out where  safe code could foul it 
up even if it has all of the source code to look at.

- Jonathan M Davis
Oct 27 2015
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 10/27/2015 12:41 PM, Andrei Alexandrescu wrote:

 It follows that if we want safe reference counting, there must be
 language support for it. One possibility is to attach an attribute to
 the class definition:
 
  safe  rc class Widget {
   ...
 }


Let's think about this more clearly before we bake a monolithic feature
for a single problem into the language.

A few thoughts:

-  rc and  noescape are orthogonal

  while  rc requires  noescape the latter could
  be useful for other purposes

- If the compiler knows a reference has a limited lifetime
  it could check for  noescape making most of RC implementable
  in a library.

  struct RC
  {
    Object get() return; // lifetime of Object is bound to RC, compiler
could check any function called on Object for  noescape
  }

- I'm not a fan of adding yet another attribute but as inference support
  is currently limited it seems we'd need an explicit attribute for
  public APIs.
Nov 01 2015
parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 01.11.2015 um 21:47 schrieb Martin Nowak:

 On 10/27/2015 12:41 PM, Andrei Alexandrescu wrote:
 - I'm not a fan of adding yet another attribute but as inference support
    is currently limited it seems we'd need an explicit attribute for
    public APIs.


I've very likely missed that part of the discussion - what were the 
reasons to not use "scope" for this?
Dec 06 2015
parent Martin Nowak <code dawg.eu> writes:
On Monday, 7 December 2015 at 07:10:42 UTC, Sönke Ludwig wrote:

 I've very likely missed that part of the discussion - what were 
 the reasons to not use "scope" for this?


Yeah good point, it should be possible to reuse scope as method 
attribute. I just used  noescape as a placeholder for the 
discussion.
Dec 13 2015