www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - borrowed pointers vs ref

reply Walter Bright <newshound2 digitalmars.com> writes:
It's been brought up more than once that the 'scope' storage class is an 
unimplemented borrowed pointer. But thinking a bit more along those lines, 
actually 'ref' fills the role of a borrowed pointer.

One particularly apropos behavior is that struct member functions pass 'this'
by 
ref, meaning that members can be called without the inc/dec millstone.

ref is still incomplete as far as this goes, but we can go the extra distance 
with it, and then it will be of great help in supporting any ref counting
solution.

What it doesn't work very well with are class references. But Andrei suggested 
that we can focus the use of 'scope' to deal with that in an analogous way.

What do you think?

Anyone want to enumerate a list of the current deficiencies of 'ref' in regards 
to this, so we can think about solving it?
May 12 2014
next sibling parent reply "Kagamin" <spam here.lot> writes:
How would you assign a borrowed pointer?
May 12 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 1:49 PM, Kagamin wrote:

 How would you assign a borrowed pointer?


A ref could only be assigned to another ref.
May 12 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 2:13 PM, Walter Bright wrote:

 On 5/12/2014 1:49 PM, Kagamin wrote:

 How would you assign a borrowed pointer?


 A ref could only be assigned to another ref.


I mean to a ref of the same or smaller scope.
May 12 2014
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 12 May 2014 16:36:12 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:


 It's been brought up more than once that the 'scope' storage class is an  
 unimplemented borrowed pointer. But thinking a bit more along those  
 lines, actually 'ref' fills the role of a borrowed pointer.

 One particularly apropos behavior is that struct member functions pass  
 'this' by ref, meaning that members can be called without the inc/dec  
 millstone.

 ref is still incomplete as far as this goes, but we can go the extra  
 distance with it, and then it will be of great help in supporting any  
 ref counting solution.


Hm... the one piece that I think would be damaging is to not be able to  
take an address of the 'this' reference. It's probably OK to just use  
pointers and static functions in some cases, but member functions do not  
have that luxury. In other words, operators.

Big example would be a doubly linked list used with ~=.

-Steve
May 12 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 2:15 PM, Steven Schveighoffer wrote:

 Hm... the one piece that I think would be damaging is to not be able to take an
 address of the 'this' reference. It's probably OK to just use pointers and
 static functions in some cases, but member functions do not have that luxury.
In
 other words, operators.

 Big example would be a doubly linked list used with ~=.


 trusted/ system code will be able to take the address of a ref.
May 12 2014
prev sibling parent reply "logicchains" <jonathan.t.barnard gmail.com> writes:
On Monday, 12 May 2014 at 21:15:38 UTC, Steven Schveighoffer 
wrote:

 Hm... the one piece that I think would be damaging is to not be 
 able to take an address of the 'this' reference. It's probably 
 OK to just use pointers and static functions in some cases, but 
 member functions do not have that luxury. In other words, 
 operators.

 Big example would be a doubly linked list used with ~=.

 -Steve


This sounds a bit like an 'issue' of sorts that Rust has with 
borrowed pointers, where certain types of datastructures cannot 
be written without resorting to the 'unsafe' parts of the 
language. The solution they've adopted is having such code 
written in libraries so that the user doesn't have to mess around 
with 'unsafe'.
May 12 2014
parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2014-05-13 at 04:07 +0000, logicchains via Digitalmars-d wrote:
[…]

 This sounds a bit like an 'issue' of sorts that Rust has with 
 borrowed pointers, where certain types of datastructures cannot 
 be written without resorting to the 'unsafe' parts of the 
 language. The solution they've adopted is having such code 
 written in libraries so that the user doesn't have to mess around 
 with 'unsafe'.


Probably re-finding many of the things people have to use
sun.misc.Unsafe for on the JVM. 

-- 
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.net
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
May 12 2014
parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Tuesday, 13 May 2014 at 04:46:41 UTC, Russel Winder via
Digitalmars-d wrote:

 On Tue, 2014-05-13 at 04:07 +0000, logicchains via 
 Digitalmars-d wrote:
 […]

 This sounds a bit like an 'issue' of sorts that Rust has with 
 borrowed pointers, where certain types of datastructures 
 cannot be written without resorting to the 'unsafe' parts of 
 the language. The solution they've adopted is having such code 
 written in libraries so that the user doesn't have to mess 
 around with 'unsafe'.


 Probably re-finding many of the things people have to use
 sun.misc.Unsafe for on the JVM.


Which is why the Java designers are looking on how to make Unsafe
an official package as of Java 9.

And did the survey a few months ago, about how Unsafe was being
used in major Java projects.

--
Paulo
May 12 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
The first thing that comes to my mind is applying this somehow to 
the (T) vs (ref T) function problem. (const ref, scope ref, 
references to r-values, you know the problem.) At the moment I 
just follow this pattern.

void foo(ref const T bar) {
     /* ... */
}

// Second overload to make r-values just work.
void foo(const T bar) {
     foo(bar);
}

auto ref sometimes works, sometimes it's more trouble than its 
worth.
May 12 2014
prev sibling next sibling parent "Daniel N" <ufo orbiting.us> writes:
On Monday, 12 May 2014 at 20:36:10 UTC, Walter Bright wrote:

 It's been brought up more than once that the 'scope' storage 
 class is an unimplemented borrowed pointer. But thinking a bit 
 more along those lines, actually 'ref' fills the role of a 
 borrowed pointer.

 One particularly apropos behavior is that struct member 
 functions pass 'this' by ref, meaning that members can be 
 called without the inc/dec millstone.

 ref is still incomplete as far as this goes, but we can go the 
 extra distance with it, and then it will be of great help in 
 supporting any ref counting solution.

 What it doesn't work very well with are class references. But 
 Andrei suggested that we can focus the use of 'scope' to deal 
 with that in an analogous way.

 What do you think?

 Anyone want to enumerate a list of the current deficiencies of 
 'ref' in regards to this, so we can think about solving it?


I would prefer 'scope ref' that would allow the solution for 
classes and everything else to be unified, i.e. everything uses 
scope.

When it comes to the implicit 'this' by ref, it could be 
redefined to pass by scope ref.

Another reason is: I know this doesn't(and might never) work in 
D, but based on the intuitive meaning of 'ref' I fully expected 
the below example to work when I first started learning the 
language.

struct A
{
   ref int a_m;
   this(ref int a)
   {
     a_m = a;
   }
}

Whereas 'scope' on the other hand is self documenting imho.
May 12 2014
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 05/12/2014 10:36 PM, Walter Bright wrote:

 It's been brought up more than once that the 'scope' storage class is an
unimplemented borrowed pointer. But thinking a bit more along those lines,
actually 'ref' fills the role of a borrowed pointer.

 One particularly apropos behavior is that struct member functions pass 'this'
by ref, meaning that members can be called without the inc/dec millstone.

 ref is still incomplete as far as this goes, but we can go the extra distance
with it, and then it will be of great help in supporting any ref counting
solution.

 What it doesn't work very well with are class references. But Andrei suggested
that we can focus the use of 'scope' to deal with that in an analogous way.

 What do you think?


I think everything should be treated uniformly. But a storage class is 
not sufficient.


 Anyone want to enumerate a list of the current deficiencies of 'ref' in
 regards to this, so we can think about solving it?


Eg:

- Cannot make tail const. / Cannot be reassigned.
- Cannot store in data structures.
- Cannot borrow slices of memory.
- Closures?
- (Probably more)
May 12 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 May 2014 06:36, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 It's been brought up more than once that the 'scope' storage class is an
 unimplemented borrowed pointer. But thinking a bit more along those lines,
 actually 'ref' fills the role of a borrowed pointer.

 One particularly apropos behavior is that struct member functions pass
 'this' by ref, meaning that members can be called without the inc/dec
 millstone.

 ref is still incomplete as far as this goes, but we can go the extra
 distance with it, and then it will be of great help in supporting any ref
 counting solution.

 What it doesn't work very well with are class references. But Andrei
 suggested that we can focus the use of 'scope' to deal with that in an
 analogous way.

 What do you think?

 Anyone want to enumerate a list of the current deficiencies of 'ref' in
 regards to this, so we can think about solving it?


I agree, I think finishing scope appears to deserve a priority boost,
it would be enabling to a lot of developments in D to have reliable
escape analysis.

It seems more problematic to repurpose ref than to finish scope though.
ref would change meaning quite significantly.
ref would probably have to become part of the type (I can imagine
needs for overloads arising?).
You would need to be able to make ref locals, and ref members of
structs so you can do useful work with them.
You'd need to be able to create an array of 'ref's

I think by-value scope still has some value too. A small struct that's
passed by value (like slices) may contain a pointer. You shouldn't
need to handle that small struct by reference when you really just
wanted to attribute it with scope.

I never saw any problems with the scope idea as it stood, and I think
ref is still useful in it's existing incarnation; the same way that
it's useful in C++, ie, a pointer that must be initialised, hides
reassignment and offset/indexing semantics (which can often interfere
with generic code).
extern(C++) would gain a new problem if ref were repurposed.
May 12 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 12 May 2014 at 20:36:10 UTC, Walter Bright wrote:

 It's been brought up more than once that the 'scope' storage 
 class is an unimplemented borrowed pointer. But thinking a bit 
 more along those lines, actually 'ref' fills the role of a 
 borrowed pointer.

 One particularly apropos behavior is that struct member 
 functions pass 'this' by ref, meaning that members can be 
 called without the inc/dec millstone.

 ref is still incomplete as far as this goes, but we can go the 
 extra distance with it, and then it will be of great help in 
 supporting any ref counting solution.

 What it doesn't work very well with are class references. But 
 Andrei suggested that we can focus the use of 'scope' to deal 
 with that in an analogous way.

 What do you think?

 Anyone want to enumerate a list of the current deficiencies of 
 'ref' in regards to this, so we can think about solving it?


There are 2 `scope` uses to think about. One is storage class and 
in that context `scope` is more of owned / unique pointer. Other 
is parameter qualifier and that one is closer to ref / borrowed 
pointer.

Main problem about making `ref` borrowed pointer is that you will 
need to prohibit storing it in function transitively. This will 
need to become invalid code:

struct A
{
     int* ptr;
}

int* gptr;

void foo(ref A a)
{
     gptr = a.ptr; // error, can't leak borrowed a.ptr into global 
context
}

This feels like too much of a breakage, this is why `scope` (or 
`scope ref`) feels more appropriate.
May 13 2014
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 13/05/14 15:36, Dicebot wrote:


 There are 2 `scope` uses to think about. One is storage class and in
 that context `scope` is more of owned / unique pointer. Other is
 parameter qualifier and that one is closer to ref / borrowed pointer.

 Main problem about making `ref` borrowed pointer is that you will need
 to prohibit storing it in function transitively. This will need to
 become invalid code:

 struct A
 {
      int* ptr;
 }

 int* gptr;

 void foo(ref A a)
 {
      gptr = a.ptr; // error, can't leak borrowed a.ptr into global context
 }

 This feels like too much of a breakage, this is why `scope` (or `scope
 ref`) feels more appropriate.


I always though "scope" would behave like that.

-- 
/Jacob Carlborg
May 13 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 13 May 2014 at 13:40:42 UTC, Jacob Carlborg wrote:

 On 13/05/14 15:36, Dicebot wrote:


 There are 2 `scope` uses to think about. One is storage class 
 and in
 that context `scope` is more of owned / unique pointer. Other 
 is
 parameter qualifier and that one is closer to ref / borrowed 
 pointer.

 Main problem about making `ref` borrowed pointer is that you 
 will need
 to prohibit storing it in function transitively. This will 
 need to
 become invalid code:

 struct A
 {
     int* ptr;
 }

 int* gptr;

 void foo(ref A a)
 {
     gptr = a.ptr; // error, can't leak borrowed a.ptr into 
 global context
 }

 This feels like too much of a breakage, this is why `scope` 
 (or `scope
 ref`) feels more appropriate.


 I always though "scope" would behave like that.


Walter's initial post implies that he wanted to re-used `ref` for 
borrowed pointer (which would mean same semantics as `scope` 
parameter qualifier)
May 13 2014
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 13 May 2014 09:50:12 -0400, Dicebot <public dicebot.lv> wrote:


 On Tuesday, 13 May 2014 at 13:40:42 UTC, Jacob Carlborg wrote:

 On 13/05/14 15:36, Dicebot wrote:


 There are 2 `scope` uses to think about. One is storage class and in
 that context `scope` is more of owned / unique pointer. Other is
 parameter qualifier and that one is closer to ref / borrowed pointer.

 Main problem about making `ref` borrowed pointer is that you will need
 to prohibit storing it in function transitively. This will need to
 become invalid code:

 struct A
 {
     int* ptr;
 }

 int* gptr;

 void foo(ref A a)
 {
     gptr = a.ptr; // error, can't leak borrowed a.ptr into global  
 context
 }

 This feels like too much of a breakage, this is why `scope` (or `scope
 ref`) feels more appropriate.


 I always though "scope" would behave like that.


 Walter's initial post implies that he wanted to re-used `ref` for  
 borrowed pointer (which would mean same semantics as `scope` parameter  
 qualifier)


Yes, the difference here is that scope is a storage class, and only  
affects the "head", whereas borrowed would have to be transitive.

-Steve
May 13 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 13 May 2014 at 17:09:17 UTC, Steven Schveighoffer 
wrote:

 On Tue, 13 May 2014 09:50:12 -0400, Dicebot <public dicebot.lv> 
 wrote:


 On Tuesday, 13 May 2014 at 13:40:42 UTC, Jacob Carlborg wrote:

 On 13/05/14 15:36, Dicebot wrote:


 There are 2 `scope` uses to think about. One is storage 
 class and in
 that context `scope` is more of owned / unique pointer. 
 Other is
 parameter qualifier and that one is closer to ref / borrowed 
 pointer.

 Main problem about making `ref` borrowed pointer is that you 
 will need
 to prohibit storing it in function transitively. This will 
 need to
 become invalid code:

 struct A
 {
    int* ptr;
 }

 int* gptr;

 void foo(ref A a)
 {
    gptr = a.ptr; // error, can't leak borrowed a.ptr into 
 global context
 }

 This feels like too much of a breakage, this is why `scope` 
 (or `scope
 ref`) feels more appropriate.


 I always though "scope" would behave like that.


 Walter's initial post implies that he wanted to re-used `ref` 
 for borrowed pointer (which would mean same semantics as 
 `scope` parameter qualifier)


 Yes, the difference here is that scope is a storage class, and 
 only affects the "head", whereas borrowed would have to be 
 transitive.

 -Steve


`scope` has to be both storage class and qualifier to work
May 13 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/13/2014 10:09 AM, Steven Schveighoffer wrote:

 Yes, the difference here is that scope is a storage class, and only affects the
 "head", whereas borrowed would have to be transitive.


Are you sure it would have to be transitive?
May 13 2014
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 13 May 2014 13:51:21 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:


 On 5/13/2014 10:09 AM, Steven Schveighoffer wrote:

 Yes, the difference here is that scope is a storage class, and only  
 affects the
 "head", whereas borrowed would have to be transitive.


 Are you sure it would have to be transitive?


I'm not certain I understand the concept correctly, but from Dicebot's  
code example, it is transitive.

I defer to the experts. Just trying to help explain the point he was  
making (and clearly doing a poor job).

-Steve
May 13 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/13/2014 11:21 AM, Steven Schveighoffer wrote:

 I'm not certain I understand the concept correctly, but from Dicebot's code
 example, it is transitive.


I know Dicebot's example assumes a transitivity property, but I question that 
assumption.

In my cursory reading of Rust documentation, it wasn't clear to me if it was 
transitive or not.
May 13 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/13/2014 6:50 AM, Dicebot wrote:

 Walter's initial post implies that he wanted to re-used `ref` for borrowed
 pointer (which would mean same semantics as `scope` parameter qualifier)


'ref' already is to much extent, for example:

    safe int foo(ref int x) {
     auto a = &x;
     return 3;
   }

dmd foo -c
foo.d(4): Error: cannot take address of parameter x in  safe function foo
May 13 2014
parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Tuesday, 13 May 2014 at 17:44:02 UTC, Walter Bright wrote:

 On 5/13/2014 6:50 AM, Dicebot wrote:

 Walter's initial post implies that he wanted to re-used `ref` 
 for borrowed
 pointer (which would mean same semantics as `scope` parameter 
 qualifier)


 'ref' already is to much extent, for example:

    safe int foo(ref int x) {
     auto a = &x;
     return 3;
   }

 dmd foo -c
 foo.d(4): Error: cannot take address of parameter x in  safe 
 function foo


This has nothing to do with `ref`. If you remove the `ref` you 
get the exact same error.
May 14 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/14/2014 12:29 AM, Idan Arye wrote:

 On Tuesday, 13 May 2014 at 17:44:02 UTC, Walter Bright wrote:

 On 5/13/2014 6:50 AM, Dicebot wrote:

 Walter's initial post implies that he wanted to re-used `ref` for borrowed
 pointer (which would mean same semantics as `scope` parameter qualifier)


 'ref' already is to much extent, for example:

    safe int foo(ref int x) {
     auto a = &x;
     return 3;
   }

 dmd foo -c
 foo.d(4): Error: cannot take address of parameter x in  safe function foo


 This has nothing to do with `ref`. If you remove the `ref` you get the exact
 same error.


Right, but the thing is, the only address of a local/parameter you can take is 
by ref, not &.
May 14 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/13/2014 6:36 AM, Dicebot wrote:

 Main problem about making `ref` borrowed pointer is that you will need to
 prohibit storing it in function transitively. This will need to become invalid
 code:

 struct A
 {
      int* ptr;
 }

 int* gptr;

 void foo(ref A a)
 {
      gptr = a.ptr; // error, can't leak borrowed a.ptr into global context
 }


The lifetime of &a is not at all the same as the lifetime of a.ptr, those are 
independent pointers. I.e. ref is not transitive (unlike const which is
transitive).
May 13 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 13 May 2014 at 17:41:23 UTC, Walter Bright wrote:

 On 5/13/2014 6:36 AM, Dicebot wrote:

 Main problem about making `ref` borrowed pointer is that you 
 will need to
 prohibit storing it in function transitively. This will need 
 to become invalid
 code:

 struct A
 {
     int* ptr;
 }

 int* gptr;

 void foo(ref A a)
 {
     gptr = a.ptr; // error, can't leak borrowed a.ptr into 
 global context
 }


 The lifetime of &a is not at all the same as the lifetime of 
 a.ptr, those are independent pointers. I.e. ref is not 
 transitive (unlike const which is transitive).


It has to be transitive to be useful as borrowed pointer. 
Consider this example:

{
     scope A a; // has some internally managed resources
     foo(a);
}

It is not safe to destruct a in the end of the scope here because 
foo may have stored references to a owned resources. But if foo 
signature is `foo(scope ref A a)` then compiler can statically 
verify that it is safe which is the very point of borrowing 
guarantees. It must be transitive to guarantee anything of course.
May 13 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/13/2014 10:52 AM, Dicebot wrote:

 It has to be transitive to be useful as borrowed pointer. Consider this
example:

 {
      scope A a; // has some internally managed resources
      foo(a);
 }

 It is not safe to destruct a in the end of the scope here because foo may have
 stored references to a owned resources. But if foo signature is `foo(scope ref
A
 a)` then compiler can statically verify that it is safe which is the very point
 of borrowing guarantees. It must be transitive to guarantee anything of course.


If those internal resources of A are marked as refcounted, then transitivity is 
not necessary. Consider also that a struct A can completely control any
escaping 
references - transitive borrowing is not necessary.
May 13 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 13 May 2014 at 18:48:14 UTC, Walter Bright wrote:

 On 5/13/2014 10:52 AM, Dicebot wrote:

 It has to be transitive to be useful as borrowed pointer. 
 Consider this example:

 {
     scope A a; // has some internally managed resources
     foo(a);
 }

 It is not safe to destruct a in the end of the scope here 
 because foo may have
 stored references to a owned resources. But if foo signature 
 is `foo(scope ref A
 a)` then compiler can statically verify that it is safe which 
 is the very point
 of borrowing guarantees. It must be transitive to guarantee 
 anything of course.


 If those internal resources of A are marked as refcounted, then 
 transitivity is not necessary. Consider also that a struct A 
 can completely control any escaping references - transitive 
 borrowing is not necessary.


No, it still can be necessary. `scope` can greatly help not only 
with resource releasing, it is also missing tool to safely cast 
from shared. Locking shared variable can return same variable 
casted to scope qualifier which will guarantee  that no reference 
has been stored to shared object by the time lock is released.

And "if those are marked as refcounted" as assumption is no 
better than "if those are owned by GC" ;)

Also A can only control escaping of any internal references only 
by completely prohibiting access to it which is not good. You 
have no means to say "feel free to use this reference as long as 
you don't keep it outside of current scope". And you effectively 
say "make all your array members private to keep borrowing 
guarantees".

Rust situation is quite different here because all their safe 
pointers have ownership/lifetime annotation. D doesn't and thus 
imaginary scope/borrowed rules need to assume worst case 
scenarios (which is still good enough for many cases).
May 13 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/13/2014 12:06 PM, Dicebot wrote:

 No, it still can be necessary. `scope` can greatly help not only with resource
 releasing, it is also missing tool to safely cast from shared. Locking shared
 variable can return same variable casted to scope qualifier which will
 guarantee  that no reference has been stored to shared object by the time lock
 is released.


I believe that is the role of `unique`. DIP69 addresses making unique pointers 
in D, and there have been several PR's implementing aspects of it.



 And "if those are marked as refcounted" as assumption is no better than "if
 those are owned by GC" ;)


I think that an object that wants to completely own its resources must properly 
encapsulate and restrict unsafe access to them itself.



 Also A can only control escaping of any internal references only by completely
 prohibiting access to it which is not good. You have no means to say "feel free
 to use this reference as long as you don't keep it outside of current scope".
 And you effectively say "make all your array members private to keep borrowing
 guarantees".


You can by only returning ref's.
May 13 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/13/2014 1:46 PM, Walter Bright wrote:

 I believe that is the role of `unique`. DIP69 addresses making unique pointers
 in D, and there have been several PR's implementing aspects of it.


Argh, DIP29:

http://wiki.dlang.org/DIP29
May 13 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 13 May 2014 at 20:46:19 UTC, Walter Bright wrote:

 On 5/13/2014 12:06 PM, Dicebot wrote:

 No, it still can be necessary. `scope` can greatly help not 
 only with resource
 releasing, it is also missing tool to safely cast from shared. 
 Locking shared
 variable can return same variable casted to scope qualifier 
 which will
 guarantee  that no reference has been stored to shared object 
 by the time lock
 is released.


 I believe that is the role of `unique`. DIP69 addresses making 
 unique pointers in D, and there have been several PR's 
 implementing aspects of it.


I don't really understand how you see this working. Sure, unique 
concept from that DIP can somewhat replace `scope` storage class. 
But you can't prove that some function does not leak unique 
variable internals if this is not annotated as such.


 And "if those are marked as refcounted" as assumption is no 
 better than "if
 those are owned by GC" ;)


 I think that an object that wants to completely own its 
 resources must properly encapsulate and restrict unsafe access 
 to them itself.


This statement is not much different from "any programmer who 
cares about memory should manage it manually" or "instead of 
const qualifier you can simply use convention". Doing this 
without compiler help is tedious. Mistakes result in bugs that 
are insanely hard to debug (storing non-shared reference to 
shared object by an accident).


 Also A can only control escaping of any internal references 
 only by completely
 prohibiting access to it which is not good. You have no means 
 to say "feel free
 to use this reference as long as you don't keep it outside of 
 current scope".
 And you effectively say "make all your array members private 
 to keep borrowing
 guarantees".


 You can by only returning ref's.


Also slices and pointers (or structs with pointers inside).
May 14 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/14/2014 6:09 AM, Dicebot wrote:

 On Tuesday, 13 May 2014 at 20:46:19 UTC, Walter Bright wrote:

 On 5/13/2014 12:06 PM, Dicebot wrote:

 No, it still can be necessary. `scope` can greatly help not only with resource
 releasing, it is also missing tool to safely cast from shared. Locking shared
 variable can return same variable casted to scope qualifier which will
 guarantee  that no reference has been stored to shared object by the time lock
 is released.


 I believe that is the role of `unique`. DIP69 addresses making unique pointers
 in D, and there have been several PR's implementing aspects of it.


 I don't really understand how you see this working. Sure, unique concept from
 that DIP can somewhat replace `scope` storage class. But you can't prove that
 some function does not leak unique variable internals if this is not annotated
 as such.


I'm not suggesting unique can replace scope, I am suggesting they are separate 
concepts.



 And "if those are marked as refcounted" as assumption is no better than "if
 those are owned by GC" ;)


 I think that an object that wants to completely own its resources must
 properly encapsulate and restrict unsafe access to them itself.


 This statement is not much different from "any programmer who cares about
memory
 should manage it manually" or "instead of const qualifier you can simply use
 convention". Doing this without compiler help is tedious. Mistakes result in
 bugs that are insanely hard to debug (storing non-shared reference to shared
 object by an accident).


My comment applies when the destructor behaves as if it owns resources other 
than the object instance itself, in which case the class designer is already 
manually managing them.



 Also A can only control escaping of any internal references only by completely
 prohibiting access to it which is not good. You have no means to say "feel free
 to use this reference as long as you don't keep it outside of current scope".
 And you effectively say "make all your array members private to keep borrowing
 guarantees".


 You can by only returning ref's.


 Also slices and pointers (or structs with pointers inside).


The idea is that 'ref' are borrowed pointers, so if you're returning pointers, 
the borrowed semantics do not apply.
May 14 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 14 May 2014 at 19:03:20 UTC, Walter Bright wrote:

 Also A can only control escaping of any internal references 
 only by completely
 prohibiting access to it which is not good. You have no 
 means to say "feel free
 to use this reference as long as you don't keep it outside 
 of current scope".
 And you effectively say "make all your array members private 
 to keep borrowing
 guarantees".


 You can by only returning ref's.


 Also slices and pointers (or structs with pointers inside).


 The idea is that 'ref' are borrowed pointers, so if you're 
 returning pointers, the borrowed semantics do not apply.


Somewhat more extended example:

struct Buffer
{
     private byte[] data;

     this() { this.data = (cast(byte*)malloc(42))[0..42]; }
     ~this() { free(this.data.ptr); }

     byte[] get(size_t a, size_t b) { return this.data[a..b]; }
}

void foo(ref Buffer buff)
{
     // here be trouble
     static int[] slice = buff.get(10, 20);
}

void foo2()
{
     Buffer buff;
     foo(buff);
     // destructor gets called, foo now has pointer to freed memory
}

Transitivity of borrowing ensures that you can use any object as 
an argument for function that takes a borrowed pointer and no 
reference to its internals will persist. Whatever memory 
management model of object type is.

With such borrowing implementation this example code is also 
totally  safe in spirit (assignment to static var will result in 
compile-time error).
May 15 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/15/2014 5:18 AM, Dicebot wrote:

 On Wednesday, 14 May 2014 at 19:03:20 UTC, Walter Bright wrote:

 The idea is that 'ref' are borrowed pointers, so if you're returning pointers,
 the borrowed semantics do not apply.


 Somewhat more extended example:

 struct Buffer
 {
      private byte[] data;

      this() { this.data = (cast(byte*)malloc(42))[0..42]; }
      ~this() { free(this.data.ptr); }

      byte[] get(size_t a, size_t b) { return this.data[a..b]; }
 }

 void foo(ref Buffer buff)
 {
      // here be trouble
      static int[] slice = buff.get(10, 20);
 }

 void foo2()
 {
      Buffer buff;
      foo(buff);
      // destructor gets called, foo now has pointer to freed memory
 }

 Transitivity of borrowing ensures that you can use any object as an argument
for
 function that takes a borrowed pointer and no reference to its internals will
 persist. Whatever memory management model of object type is.

 With such borrowing implementation this example code is also totally  safe in
 spirit (assignment to static var will result in compile-time error).


get() is returning a pointer to its internally managed data (in the form of
[]). 
You're right that transitivity of borrowing would support this safely, but I am 
not proposing that for ref. To make slicing Buffer safe, one would have to 
overload opSlice and then manage access to the slice.
May 15 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 15 May 2014 at 18:08:06 UTC, Walter Bright wrote:

 get() is returning a pointer to its internally managed data (in 
 the form of []). You're right that transitivity of borrowing 
 would support this safely, but I am not proposing that for ref. 
 To make slicing Buffer safe, one would have to overload opSlice 
 and then manage access to the slice.


Sure, I simply question its practical applicability in that case. 
It is not like I want borrowed pointer semantics only because it 
looks cool :) Transitive borrowing solves certain class of issues 
that currently rely on convention, enabling whole new type of 
verified safe code (both memory safe and concurrency safe). 
Head-only? Doesn't look so.
May 16 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/16/2014 9:43 AM, Dicebot wrote:

 Transitive
 borrowing solves certain class of issues that currently rely on convention,
 enabling whole new type of verified safe code (both memory safe and concurrency
 safe). Head-only? Doesn't look so.


I'm concerned that transitive borrowing will *preclude* a number of useful
cases.
May 16 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Friday, 16 May 2014 at 17:22:21 UTC, Walter Bright wrote:

 On 5/16/2014 9:43 AM, Dicebot wrote:

 Transitive
 borrowing solves certain class of issues that currently rely 
 on convention,
 enabling whole new type of verified safe code (both memory 
 safe and concurrency
 safe). Head-only? Doesn't look so.


 I'm concerned that transitive borrowing will *preclude* a 
 number of useful cases.


Which is why `ref` itself can't be used for that and usage of 
`scope` as qualifier is necessary to enable transitive behavior :)
May 16 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/16/2014 10:33 AM, Dicebot wrote:

 On Friday, 16 May 2014 at 17:22:21 UTC, Walter Bright wrote:

 On 5/16/2014 9:43 AM, Dicebot wrote:

 Transitive
 borrowing solves certain class of issues that currently rely on convention,
 enabling whole new type of verified safe code (both memory safe and concurrency
 safe). Head-only? Doesn't look so.


 I'm concerned that transitive borrowing will *preclude* a number of useful
cases.


 Which is why `ref` itself can't be used for that and usage of `scope` as
 qualifier is necessary to enable transitive behavior :)


True, but there comes a point where something gets complicated enough that 
nobody understands it and the implementation gets full of bugs.
May 16 2014
next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, May 16, 2014 at 11:57:36AM -0700, Walter Bright via Digitalmars-d wrote:

 On 5/16/2014 10:33 AM, Dicebot wrote:

On Friday, 16 May 2014 at 17:22:21 UTC, Walter Bright wrote:

On 5/16/2014 9:43 AM, Dicebot wrote:

Transitive borrowing solves certain class of issues that currently
rely on convention, enabling whole new type of verified safe code
(both memory safe and concurrency safe). Head-only? Doesn't look
so.


I'm concerned that transitive borrowing will *preclude* a number of
useful cases.


Which is why `ref` itself can't be used for that and usage of `scope`
as qualifier is necessary to enable transitive behavior :)


 
 True, but there comes a point where something gets complicated enough
 that nobody understands it and the implementation gets full of bugs.


*snicker*

	What I am going to tell you about is what we teach our
	programming students in the third or fourth year of graduate
	school... It is my task to convince you not to turn away because
	you don't understand it. You see my programming students don't
	understand it... That is because I don't understand it. Nobody
	does. 
	Richard Deeman

Source: dlang.org. :-P


T

-- 
It is impossible to make anything foolproof because fools are so ingenious. --
Sammy
May 16 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Friday, 16 May 2014 at 18:57:34 UTC, Walter Bright wrote:

 On 5/16/2014 10:33 AM, Dicebot wrote:

 On Friday, 16 May 2014 at 17:22:21 UTC, Walter Bright wrote:

 On 5/16/2014 9:43 AM, Dicebot wrote:

 Transitive
 borrowing solves certain class of issues that currently rely 
 on convention,
 enabling whole new type of verified safe code (both memory 
 safe and concurrency
 safe). Head-only? Doesn't look so.


 I'm concerned that transitive borrowing will *preclude* a 
 number of useful cases.


 Which is why `ref` itself can't be used for that and usage of 
 `scope` as
 qualifier is necessary to enable transitive behavior :)


 True, but there comes a point where something gets complicated 
 enough that nobody understands it and the implementation gets 
 full of bugs.


Then we are back to square one "`scope` is needed but difficult 
to implement so lets not touch it" :) Which is understandable but 
does not warrant adding mostly useless concept simply because it 
is easier.

Also I believe `scope` is one of concepts that are hard to define 
but incredibly easy to grasp intuitively.
May 16 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/16/2014 1:18 PM, Dicebot wrote:

 Also I believe `scope` is one of concepts that are hard to define but
incredibly
 easy to grasp intuitively.


How and where to put the annotations, and the notion of transitivity, are not
so 
incredibly easy to grasp. The mix of ref and scope is also a bit of a wtf.
May 16 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-05-13 19:52, Dicebot wrote:


 It has to be transitive to be useful as borrowed pointer. Consider this
 example:

 {
      scope A a; // has some internally managed resources
      foo(a);
 }

 It is not safe to destruct a in the end of the scope here because foo
 may have stored references to a owned resources. But if foo signature is
 `foo(scope ref A a)` then compiler can statically verify that it is safe
 which is the very point of borrowing guarantees. It must be transitive
 to guarantee anything of course.


What is "scope ref" supposed to do in this example, compared to just 
"scope"?

-- 
/Jacob Carlborg
May 13 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 13 May 2014 at 19:17:16 UTC, Jacob Carlborg wrote:

 What is "scope ref" supposed to do in this example, compared to 
 just "scope"?


To be a reference ;) But yeah, it is not important in this 
example, plain scope should behave the same if transitive.
May 14 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-05-14 15:00, Dicebot wrote:


 To be a reference ;) But yeah, it is not important in this example,
 plain scope should behave the same if transitive.


I though that "A" was a class in the previous example, but now I see 
that it was a struct.

-- 
/Jacob Carlborg
May 14 2014