www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - The next iteration of scope

reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
Here's the new version of my scope proposal:
http://wiki.dlang.org/User:Schuetzm/scope2

It's still missing real-life examples, a section on the 
implementation, and a more formal specification, as well as a 
discussion of backwards compatibility. But I thought I'd show 
what I have, so that it can be discussed early on.

I hope it will be more digestible for Walter & Andrei. It's more 
or less an extended version of DIP25, and avoids the need for 
most explicit annotations.
Mar 15 2015
next sibling parent reply Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 15/03/2015 14:10, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:

 Here's the new version of my scope proposal:
 http://wiki.dlang.org/User:Schuetzm/scope2

 It's still missing real-life examples, a section on the implementation,
 and a more formal specification, as well as a discussion of backwards
 compatibility. But I thought I'd show what I have, so that it can be
 discussed early on.

 I hope it will be more digestible for Walter & Andrei. It's more or less
 an extended version of DIP25, and avoids the need for most explicit
 annotations.


I too want a scope attribute e.g. for safe slicing of static arrays, 
etc. I'm not sure if it's too late for scope by default though, perhaps.

I like postblit overloading on whether 'this' can be scope or not, 
allowing efficient ref-counting.

     scope T payload;

^ This is a nice way to help enforce the correctness of  safe wrapper types.
Mar 15 2015
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Sunday, 15 March 2015 at 17:31:17 UTC, Nick Treleaven wrote:

 On 15/03/2015 14:10, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:

 Here's the new version of my scope proposal:
 http://wiki.dlang.org/User:Schuetzm/scope2

 It's still missing real-life examples, a section on the 
 implementation,
 and a more formal specification, as well as a discussion of 
 backwards
 compatibility. But I thought I'd show what I have, so that it 
 can be
 discussed early on.

 I hope it will be more digestible for Walter & Andrei. It's 
 more or less
 an extended version of DIP25, and avoids the need for most 
 explicit
 annotations.


 I too want a scope attribute e.g. for safe slicing of static 
 arrays, etc. I'm not sure if it's too late for scope by default 
 though, perhaps.


If we get  safe by default, we automatically get scope by 
default, too.


 I like postblit overloading on whether 'this' can be scope or 
 not, allowing efficient ref-counting.

     scope T payload;

 ^ This is a nice way to help enforce the correctness of  safe 
 wrapper types.


Yes, it's an exception to the general rule of "scope only in 
function signatures", but it's so useful I think it's worth it.
Mar 15 2015
next sibling parent reply Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 15/03/2015 19:11, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:

 On Sunday, 15 March 2015 at 17:31:17 UTC, Nick Treleaven wrote:

 I too want a scope attribute e.g. for safe slicing of static arrays,
 etc. I'm not sure if it's too late for scope by default though, perhaps.


 If we get  safe by default, we automatically get scope by default, too.


I don't follow that. In  safe code parameters could still default to 
escaping, causing compiler errors when passing things that can't be 
allowed to escape.

I do see that scope parameters by default would probably cause less 
churn than having to put `scope` on non-template function parameters in 
order to be usable by non-GC-heap memory.


     scope T payload;

 ^ This is a nice way to help enforce the correctness of  safe wrapper
 types.


 Yes, it's an exception to the general rule of "scope only in function
 signatures", but it's so useful I think it's worth it.


We already have scope on local delegates (at least it's accepted by dmd):

Object obj;
scope del = ()=>obj;

Also, what would be the 'type' of a static array slice without `scope` 
applying to locals?

int[2] arr = [1, 2];
scope int[] s = arr;

I suppose `scope` can be inferred for s, I'm just pointing out that it 
can apply to locals.
Mar 16 2015
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 16 March 2015 at 12:21:17 UTC, Nick Treleaven wrote:

 On 15/03/2015 19:11, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:

 On Sunday, 15 March 2015 at 17:31:17 UTC, Nick Treleaven wrote:

 I too want a scope attribute e.g. for safe slicing of static 
 arrays,
 etc. I'm not sure if it's too late for scope by default 
 though, perhaps.


 If we get  safe by default, we automatically get scope by 
 default, too.


 I don't follow that. In  safe code parameters could still 
 default to escaping, causing compiler errors when passing 
 things that can't be allowed to escape.

 I do see that scope parameters by default would probably cause 
 less churn than having to put `scope` on non-template function 
 parameters in order to be usable by non-GC-heap memory.


It's part of the proposal, under "Implicit scope":  safe implies 
scope. I think this would cause the least compatibility problems, 
because what can be done with references in  safe code is already 
restricted. And as  safe is going to be used more and more, scope 
will spread with it.


 We already have scope on local delegates (at least it's 
 accepted by dmd):

 Object obj;
 scope del = ()=>obj;


Oh, I thought this was deprecated, together with scope classes. I 
was aware that `scope` already works (at least partially) for 
delegate params, but not for locals. Anyway, I think it's 
unnecessary for locals, because:


 Also, what would be the 'type' of a static array slice without 
 `scope` applying to locals?

 int[2] arr = [1, 2];
 scope int[] s = arr;

 I suppose `scope` can be inferred for s, I'm just pointing out 
 that it can apply to locals.


Yes, it will be inferred. I've removed this part of the proposal 
from the article, because I thought it is too technical, and I 
wanted to describe only what would be visible to the end-user. 
But I moved my notes about that to the talk page [1]. The 
examples further up on this page are a bit chaotic; they were 
experiments for me to try out the algorithm. While the algorithm 
as described there probably works, I want to change it a bit to 
allow detection for the RCArray problem.

The important thing is that the end-user doesn't have to annotate 
local variables as scope. As it's unnecessary, we can even 
consider disallowing it.

[1] 
http://wiki.dlang.org/User_talk:Schuetzm/scope2#Scope_inference
Mar 16 2015
prev sibling parent reply "Oren Tirosh" <orent hishome.net> writes:
On Sunday, 15 March 2015 at 19:11:36 UTC, Marc Schütz wrote:

 On Sunday, 15 March 2015 at 17:31:17 UTC, Nick Treleaven wrote:

 On 15/03/2015 14:10, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:

 Here's the new version of my scope proposal:
 http://wiki.dlang.org/User:Schuetzm/scope2

 It's still missing real-life examples, a section on the 
 implementation,
 and a more formal specification, as well as a discussion of 
 backwards
 compatibility. But I thought I'd show what I have, so that it 
 can be
 discussed early on.

 I hope it will be more digestible for Walter & Andrei. It's 
 more or less
 an extended version of DIP25, and avoids the need for most 
 explicit
 annotations.


 I too want a scope attribute e.g. for safe slicing of static 
 arrays, etc. I'm not sure if it's too late for scope by 
 default though, perhaps.


 If we get  safe by default, we automatically get scope by 
 default, too.


The scope storage class is a two way contract. The function 
promises not to escape the reference. The caller promises to 
ensure the storage that the reference is pointing to will remain 
valid for the duration of the function call. In some cases, the 
caller code may need to take active steps to ensure that, like 
keeping an otherwise temporary reference alive to prevent it from 
being deallocated.

But what if the pointer is null? Can this be considered to 
fulfill the caller's part of the deal?

Yes, the old  notnull debate again. For me,  safe by default and 
scope by default also suggests  notnull by default for scope 
references. Sorry if this opens up directions you don't want to 
think about at the moment...
Mar 18 2015
next sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Wednesday, 18 March 2015 at 13:01:50 UTC, Oren Tirosh wrote:

 On Sunday, 15 March 2015 at 19:11:36 UTC, Marc Schütz wrote:

 On Sunday, 15 March 2015 at 17:31:17 UTC, Nick Treleaven wrote:

 On 15/03/2015 14:10, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:

 Here's the new version of my scope proposal:
 http://wiki.dlang.org/User:Schuetzm/scope2

 It's still missing real-life examples, a section on the 
 implementation,
 and a more formal specification, as well as a discussion of 
 backwards
 compatibility. But I thought I'd show what I have, so that 
 it can be
 discussed early on.

 I hope it will be more digestible for Walter & Andrei. It's 
 more or less
 an extended version of DIP25, and avoids the need for most 
 explicit
 annotations.


 I too want a scope attribute e.g. for safe slicing of static 
 arrays, etc. I'm not sure if it's too late for scope by 
 default though, perhaps.


 If we get  safe by default, we automatically get scope by 
 default, too.


 The scope storage class is a two way contract. The function 
 promises not to escape the reference. The caller promises to 
 ensure the storage that the reference is pointing to will 
 remain valid for the duration of the function call. In some 
 cases, the caller code may need to take active steps to ensure 
 that, like keeping an otherwise temporary reference alive to 
 prevent it from being deallocated.

 But what if the pointer is null? Can this be considered to 
 fulfill the caller's part of the deal?

 Yes, the old  notnull debate again. For me,  safe by default 
 and scope by default also suggests  notnull by default for 
 scope references. Sorry if this opens up directions you don't 
 want to think about at the moment...


So far, null pointers haven't been a big part of the discussion. 
By the existing definition, a null pointer is memory safe, 
because it doesn't point to anything. But they are obviously a 
problem in their own right.
Mar 18 2015
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 18 March 2015 at 13:01:50 UTC, Oren Tirosh wrote:

 The scope storage class is a two way contract. The function 
 promises not to escape the reference. The caller promises to 
 ensure the storage that the reference is pointing to will 
 remain valid for the duration of the function call. In some 
 cases, the caller code may need to take active steps to ensure 
 that, like keeping an otherwise temporary reference alive to 
 prevent it from being deallocated.

 But what if the pointer is null? Can this be considered to 
 fulfill the caller's part of the deal?

 Yes, the old  notnull debate again. For me,  safe by default 
 and scope by default also suggests  notnull by default for 
 scope references. Sorry if this opens up directions you don't 
 want to think about at the moment...


Don't be sorry, I agree with you 100%, and you stated it more 
clearly than i could have.
Mar 19 2015
prev sibling next sibling parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Sunday, 15 March 2015 at 14:10:02 UTC, Marc Schütz wrote:

 Here's the new version of my scope proposal:
 http://wiki.dlang.org/User:Schuetzm/scope2

 It's still missing real-life examples, a section on the 
 implementation, and a more formal specification, as well as a 
 discussion of backwards compatibility. But I thought I'd show 
 what I have, so that it can be discussed early on.

 I hope it will be more digestible for Walter & Andrei. It's 
 more or less an extended version of DIP25, and avoids the need 
 for most explicit annotations.


It's great to see your design evolving like this. BIG plus for 
`scope` by default in  safe code -- this makes the proposal much 
more attractive than the alternative.

"Functions and methods can be overloaded on scope. This allows 
efficient passing of RC wrappers for instance..."

How does the compiler figure out which of the variables it's 
passing to the parameters are `scope` or not? Does the caller try 
the scoped overloads first by default, and only if there's an 
error tries the non-scoped overloads? If so, what causes the 
error?

"To specify that the value is returned through another parameter, 
the return!ident syntax can be used...

struct RC(T) if(is(T == class)) {
     scope T payload;
     T borrow() return {    // `return` applies to `this`
         return payload;
     }
}"

The example contains no use of `return!ident`.

Also, what exactly does the `scope` on T payload get you? Is it 
just a more specific version of `return` on the this parameter, 
i.e. `return this.payload`? Why would you need that specificity? 
What is the dangerous operation it is intended to prevent?
Mar 15 2015
next sibling parent Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 16/03/2015 04:00, Zach the Mystic wrote:

 struct RC(T) if(is(T == class)) {
      scope T payload;
      T borrow() return {    // `return` applies to `this`
          return payload;
      }
 }


...

 Also, what exactly does the `scope` on T payload get you?


It means if you forget the `return` attribute the compiler would issue 
an error about escaping payload. payload only has to be annotated once 
and the whole body of RC is checked to prevent implicit escaping of it.
Mar 16 2015
prev sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 16 March 2015 at 04:00:51 UTC, Zach the Mystic wrote:

 "Functions and methods can be overloaded on scope. This allows 
 efficient passing of RC wrappers for instance..."

 How does the compiler figure out which of the variables it's 
 passing to the parameters are `scope` or not? Does the caller 
 try the scoped overloads first by default, and only if there's 
 an error tries the non-scoped overloads? If so, what causes the 
 error?


Hmm... I guess it only makes sense for postblits and destructors. 
I'm not sure about constructors and opAssign, so I'll leave these 
out for now. I've changed the wiki page accordingly.


 "To specify that the value is returned through another 
 parameter, the return!ident syntax can be used...

 struct RC(T) if(is(T == class)) {
     scope T payload;
     T borrow() return {    // `return` applies to `this`
         return payload;
     }
 }"

 The example contains no use of `return!ident`.


I added an example.


 Also, what exactly does the `scope` on T payload get you? Is it 
 just a more specific version of `return` on the this parameter, 
 i.e. `return this.payload`? Why would you need that 
 specificity? What is the dangerous operation it is intended to 
 prevent?


Nick already answered that. I'll expand on his explanation:

Let's take the RC struct as an example. Instances of RC can 
appear with and without scope. Because structs members inherit 
the scope-ness from the struct, `payload` could therefore be an 
unscoped pointer. It could therefore be escaped unintentionally. 
By adding `scope` to its declaration, we force it to be scoped to 
the structs lifetime, no matter how it's accessed.
Mar 16 2015
next sibling parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Monday, 16 March 2015 at 13:55:43 UTC, Marc Schütz wrote:

 Also, what exactly does the `scope` on T payload get you? Is 
 it just a more specific version of `return` on the this 
 parameter, i.e. `return this.payload`? Why would you need that 
 specificity? What is the dangerous operation it is intended to 
 prevent?


 Nick already answered that. I'll expand on his explanation:

 Let's take the RC struct as an example. Instances of RC can 
 appear with and without scope. Because structs members inherit 
 the scope-ness from the struct, `payload` could therefore be an 
 unscoped pointer. It could therefore be escaped 
 unintentionally. By adding `scope` to its declaration, we force 
 it to be scoped to the structs lifetime, no matter how it's 
 accessed.


If an RC'd struct is heap-allocated, but one of its members 
points to the stack, how is it ever safe to escape it? Why 
shouldn't the heap variable be treated as scoped too, inheriting 
the most restricted scope of any of its members? To me, the 
question is not how you can know that a member is scoped, so much 
as how you could know that it *isn't* scoped, i.e. that a 
sub-pointer was global while the parent was local. I think it 
would require a very complicated type system:

struct S {
   T* p;
}

// note the elaborate new return signature
T* fun(return!(S.p) S x) {
   return x.p;
}

T* run() {
   S s;
   s.p = new T; // s local, s.p global
   return fun(s);
}

The above is technically safe, but the question is whether it's 
too complicated for the benefit. In the absence of such a 
complicated system, the safe default is to assume a struct is 
always as scoped as its most scoped member (i.e. transitive 
scoping). Your idea of `scope` members would only be valid in the 
absence of this safe default. But even then it would be of 
limited usefulness, because it would prevent all uses of global 
references in those members, even if the parent was global. For 
me, it comes down to that you can't know if anything is global or 
local until you define an instance of it, which you can't do in 
the struct definition.
Mar 16 2015
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 16 March 2015 at 15:25:45 UTC, Zach the Mystic wrote:

 If an RC'd struct is heap-allocated, but one of its members 
 points to the stack, how is it ever safe to escape it?


It isn't.


 Why shouldn't the heap variable be treated as scoped too, 
 inheriting the most restricted scope of any of its members?


See below.


 To me, the question is not how you can know that a member is 
 scoped, so much as how you could know that it *isn't* scoped, 
 i.e. that a sub-pointer was global while the parent was local. 
 I think it would require a very complicated type system:

 struct S {
   T* p;
 }

 // note the elaborate new return signature
 T* fun(return!(S.p) S x) {
   return x.p;
 }

 T* run() {
   S s;
   s.p = new T; // s local, s.p global
   return fun(s);
 }

 The above is technically safe, but the question is whether it's 
 too complicated for the benefit. In the absence of such a 
 complicated system, the safe default is to assume a struct is 
 always as scoped as its most scoped member (i.e. transitive 
 scoping).


My old proposal treated scope as a type modifier. It would have 
made something like your example possible. But it's now a storage 
class, which means that we lose information when we go through an 
indirection.

The solution is indeed a kind of transitivity. I describe this in 
the section on multiple indirections:
http://wiki.dlang.org/User:Schuetzm/scope2#Multiple_indirections


 Your idea of `scope` members would only be valid in the absence 
 of this safe default. But even then it would be of limited 
 usefulness, because it would prevent all uses of global 
 references in those members, even if the parent was global. For 
 me, it comes down to that you can't know if anything is global 
 or local until you define an instance of it, which you can't do 
 in the struct definition.


I think there is a misunderstanding. scope members don't really 
introduce anything new. They are exactly equivalent to an 
accessor method:

struct S {
     scope int* payload;
     scope int** indirect;
}

behaves exactly the same as:

struct S {
     private int* payload_;
     ref int* payload() return {
         return payload_;
     }
}

By the rules for multiple indirections, it's not possible to do 
unsafe things with them:

void foo() {
     // reading (right hand side):

     S s;
     int* p = s.payload;  // OK, `payload` lives at least as long 
as `s`
     p = *s.indirect;     // ditto

     {
         S s2;
         p = s2.payload;  // NOT OK, we don't know how long 
`payload`
                          // really lives, so we assume it is
                          // destroyed when `s2` goes out of scope
         p = *s2.indirect;// ditto
     }

     // assignment (left hand side):

     s.payload = new int; // OK, `new int` is on the heap

     int x;
     s.payload = &x;      // NOT OK, `payload` is scope by its 
`this`
                          // which is `s`, which outlives `x`
     *s.indirect = new int*;
                          // OK: `new int*` is on the heap
     *s.indirect = &p;    // NOT OK: accessed through indirection
                          // must assume destination exists forever
                          // => cannot contain pointer to local
}

The only difference it makes whether a member is annotated as 
scope or not is when it's accessed from inside a method, because 
in this case, the compiler doesn't know anything about the 
context in which the method is called.

BUt there is indeed still some confusion on my side. It's about 
the question whether `this` should implicitly be passed as 
`scope` or not. Because if it is, scope members are probably 
useless, because they are already implied. I think I should 
remove this suggestion, because it would break too much code (in 
 system).
Mar 16 2015
parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Monday, 16 March 2015 at 17:00:12 UTC, Marc Schütz wrote:

 BUt there is indeed still some confusion on my side. It's about 
 the question whether `this` should implicitly be passed as 
 `scope` or not. Because if it is, scope members are probably 
 useless, because they are already implied. I think I should 
 remove this suggestion, because it would break too much code 
 (in  system).


I always tend to think of member functions as if they weren't:

struct S {
   T t;
   ref T fun() return {
     return t;
   }
}

In my head, I just translate fun() above to:

ref T fun(return S* __this) {
   return __this.t;
}

Therefore whatever the scope of `__this`, that's the scope of the 
return, just like it would be for any other parameter. Then:

S s;
s.fun();

... is really just `fun(s);` in disguise. That's why it's hard 
for me to grasp `scope` members, because they seem to me to be 
just as scope as their parent, whether global or local. How a 
member could be scope when the parent is global is hard for me to 
imagine.
Mar 16 2015
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 16 March 2015 at 19:43:01 UTC, Zach the Mystic wrote:

 On Monday, 16 March 2015 at 17:00:12 UTC, Marc Schütz wrote:

 BUt there is indeed still some confusion on my side. It's 
 about the question whether `this` should implicitly be passed 
 as `scope` or not. Because if it is, scope members are 
 probably useless, because they are already implied. I think I 
 should remove this suggestion, because it would break too much 
 code (in  system).


 I always tend to think of member functions as if they weren't:

 struct S {
   T t;
   ref T fun() return {
     return t;
   }
 }

 In my head, I just translate fun() above to:

 ref T fun(return S* __this) {
   return __this.t;
 }

 Therefore whatever the scope of `__this`, that's the scope of 
 the return, just like it would be for any other parameter. Then:

 S s;
 s.fun();

 ... is really just `fun(s);` in disguise. That's why it's hard 
 for me to grasp `scope` members, because they seem to me to be 
 just as scope as their parent, whether global or local.


It works just the same:

struct S {
     private int* payload_;
     ref int* payload() return {
         return payload_;
     }
}

ref int* payload(scope ref S __this) return {
     return __this.payload_;    // well, imagine it's not private
}

Both the S.payload() and the free-standing payload() do the same 
thing.

 From inside the functions, `return` tells us that we're allowed 
to a reference to our payload. From the caller's point of view, 
it signifies that the return value is scoped to the first 
argument, or `this` respectively.

To reiterate, `scope` members are just syntactical sugar for the 
kinds of accessor methods/functions in the example code. There's 
nothing special about them.


 How a member could be scope when the parent is global is hard 
 for me to imagine.


The following is clear, right?

int* p;
scope int* borrowed = p;

That's clearly allowed, we're storing a reference to a global or 
GC object into a scope variable. Now let's use `S`, which 
contains an `int*` member:

S s;
scope S borrowed_s = s;

That's also ok. Doesn't matter whether it's the pointer itself, 
or something containing the pointer. And now the final step:

scope int* p2;
p2 = s.payload;          // OK
p2 = borrowed_s.payload; // also OK
static int* p3;
p3 = s.payload;          // NOT OK!

However, if `payload` were not the accessor method/function, but 
instead a simple (non-scope) member of `S`, that last line would 
be allowed, because there is nothing restricting its use. For 
members that the struct owns and want's to manage itself, this is 
not good. Therefore, we make it private and allow access to it 
only through accessor methods/functions that are annotated with 
`return`. But we could accidentally forget an annotation, and the 
pointer could escape. That's what the `scope` annotation on the 
member is fore.

Does that clear it up?
Mar 16 2015
parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Monday, 16 March 2015 at 20:50:46 UTC, Marc Schütz wrote:

 On Monday, 16 March 2015 at 19:43:01 UTC, Zach the Mystic wrote:

 I always tend to think of member functions as if they weren't:

 struct S {
  T t;
  ref T fun() return {
    return t;
  }
 }

 In my head, I just translate fun() above to:

 ref T fun(return S* __this) {
  return __this.t;
 }

 Therefore whatever the scope of `__this`, that's the scope of 
 the return, just like it would be for any other parameter. 
 Then:

 S s;
 s.fun();

 ... is really just `fun(s);` in disguise. That's why it's hard 
 for me to grasp `scope` members, because they seem to me to be 
 just as scope as their parent, whether global or local.


 It works just the same:

 struct S {
     private int* payload_;
     ref int* payload() return {
         return payload_;
     }
 }

 ref int* payload(scope ref S __this) return {
     return __this.payload_;    // well, imagine it's not private
 }


More accurately,

// `return` is moved
ref int* payload(return scope ref S __this) {
    return __this.payload_;
}

I think that if you need `return` to make it safe, there's much 
less need for `scope`.


 Both the S.payload() and the free-standing payload() do the 
 same thing.

 From inside the functions, `return` tells us that we're allowed 
 to a reference to our payload. From the caller's point of view, 
 it signifies that the return value is scoped to the first 
 argument, or `this` respectively.

 To reiterate, `scope` members are just syntactical sugar for 
 the kinds of accessor methods/functions in the example code. 
 There's nothing special about them.


That's fine, but then there's the argument that syntax sugar is 
different from "real" functionality. To add it would require a 
compelling use case.

My fundamental issue with `scope` in general is that it should be 
the safe default, which means it doesn't really need to appear 
that often. If  safe is default, the compiler would force you to 
mark any parameter `return` when it detected such a return.


 How a member could be scope when the parent is global is hard 
 for me to imagine.


 The following is clear, right?

 int* p;
 scope int* borrowed = p;

 That's clearly allowed, we're storing a reference to a global 
 or GC object into a scope variable. Now let's use `S`, which 
 contains an `int*` member:

 S s;
 scope S borrowed_s = s;

 That's also ok. Doesn't matter whether it's the pointer itself, 
 or something containing the pointer. And now the final step:

 scope int* p2;
 p2 = s.payload;          // OK
 p2 = borrowed_s.payload; // also OK
 static int* p3;
 p3 = s.payload;          // NOT OK!

 However, if `payload` were not the accessor method/function, 
 but instead a simple (non-scope) member of `S`, that last line 
 would be allowed, because there is nothing restricting its use.


See above. With `return` being forced on the implicit this 
parameter:

ref int* payload(return /*scope*/ ref S __this) { ... }

`return` covers the need for safety, unless I'm still missing 
something.


 For members that the struct owns and want's to manage itself, 
 this is not good. Therefore, we make it private and allow 
 access to it only through accessor methods/functions that are 
 annotated with `return`. But we could accidentally forget an 
 annotation, and the pointer could escape.


Same argument. Forgetting `return` in safe code == compiler 
error. I think DIP25 already does this.
Mar 16 2015
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 17 March 2015 at 01:13:41 UTC, Zach the Mystic wrote:

 On Monday, 16 March 2015 at 20:50:46 UTC, Marc Schütz wrote:

 It works just the same:

 struct S {
    private int* payload_;
    ref int* payload() return {
        return payload_;
    }
 }

 ref int* payload(scope ref S __this) return {
    return __this.payload_;    // well, imagine it's not private
 }


 More accurately,

 // `return` is moved
 ref int* payload(return scope ref S __this) {
    return __this.payload_;
 }


Right, copy&paste mistake.


 I think that if you need `return` to make it safe, there's much 
 less need for `scope`.


 Both the S.payload() and the free-standing payload() do the 
 same thing.

 From inside the functions, `return` tells us that we're 
 allowed to a reference to our payload. From the caller's point 
 of view, it signifies that the return value is scoped to the 
 first argument, or `this` respectively.

 To reiterate, `scope` members are just syntactical sugar for 
 the kinds of accessor methods/functions in the example code. 
 There's nothing special about them.


 That's fine, but then there's the argument that syntax sugar is 
 different from "real" functionality. To add it would require a 
 compelling use case.

 My fundamental issue with `scope` in general is that it should 
 be the safe default, which means it doesn't really need to 
 appear that often. If  safe is default, the compiler would 
 force you to mark any parameter `return` when it detected such 
 a return.


Hmmm... you have a point there. On the other hand, if  safe is 
merely inferred (for templates), then you wouldn't get an error, 
but it would get inferred  system instead.
Mar 17 2015
prev sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 16 March 2015 at 13:55:43 UTC, Marc Schütz wrote:

 On Monday, 16 March 2015 at 04:00:51 UTC, Zach the Mystic wrote:

 "Functions and methods can be overloaded on scope. This allows 
 efficient passing of RC wrappers for instance..."

 How does the compiler figure out which of the variables it's 
 passing to the parameters are `scope` or not? Does the caller 
 try the scoped overloads first by default, and only if there's 
 an error tries the non-scoped overloads? If so, what causes 
 the error?


 Hmm... I guess it only makes sense for postblits and 
 destructors. I'm not sure about constructors and opAssign, so 
 I'll leave these out for now. I've changed the wiki page 
 accordingly.


To clarify: When a scoped variable is assigned to a location with 
infinite lifetime (`static`), this assignment will fail unless 
the type provides an assignment operator/postblit that is 
correctly annotated. Depending on whether `this` is implicitly 
passed as `scope` or not, this annotation needs to be `scope`, 
otherwise, conversely, the non-scope overload needs to be marked 
as `static`. Likewise, a scope variable without a destructor 
accepting scope cannot be destroyed.

I added the rules for overloading in the wiki. Now that I'm 
seeing clearer, I've restored the part where it says that all 
functions can be overloaded. No need to restrict it artificially.
Mar 16 2015
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Marc Schütz:


 Here's the new version of my scope proposal:
 http://wiki.dlang.org/User:Schuetzm/scope2


Let's see what Andrei and Walter think about this all :-)

Bye,
bearophile
Mar 16 2015
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Sunday, 15 March 2015 at 14:10:02 UTC, Marc Schütz wrote:

 Here's the new version of my scope proposal:
 http://wiki.dlang.org/User:Schuetzm/scope2

 It's still missing real-life examples, a section on the 
 implementation, and a more formal specification, as well as a 
 discussion of backwards compatibility. But I thought I'd show 
 what I have, so that it can be discussed early on.

 I hope it will be more digestible for Walter & Andrei. It's 
 more or less an extended version of DIP25, and avoids the need 
 for most explicit annotations.


BUMP

 Walter & Andrei: What is your opinion? Is it better than the 
last proposal? Worse? Good enough?
Mar 22 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/22/15 10:20 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:

 On Sunday, 15 March 2015 at 14:10:02 UTC, Marc Schütz wrote:

 Here's the new version of my scope proposal:
 http://wiki.dlang.org/User:Schuetzm/scope2

 It's still missing real-life examples, a section on the
 implementation, and a more formal specification, as well as a
 discussion of backwards compatibility. But I thought I'd show what I
 have, so that it can be discussed early on.

 I hope it will be more digestible for Walter & Andrei. It's more or
 less an extended version of DIP25, and avoids the need for most
 explicit annotations.


 BUMP

  Walter & Andrei: What is your opinion? Is it better than the last
 proposal? Worse? Good enough?


Speaking for myself (though I suspect Walter is in a similar position) 
I'm not sure when I'll have time to get to it. I'm to my head in stuff 
that's important and urgent.

I need to get DConf submissions reviewed and published. (That should 
happen later today - stay tuned!) What seems to be a trivial process for 
anyone who hasn't done it turns out to be a ton of grind.

At work I'm working on reviewing a bunch of non-idiomatic D (rather 
Java/Python/Go written in D) and refactor it into much shorter and more 
efficient idiomatic D. That is also important and urgent; with badly 
written D there's little visible advantage to D at large and a lot of 
incentive to fall back on the more familiar languages. That spills into 
documentation work, and recent discussions between Walter and me led to 
him (and thankfully others) being preoccupied with improving the 
documentation.

As odd as it sounds, better documentation for what we have is more 
"important * urgent" cross-product than exploration of language 
improvements.

Then I also need to get diverted into PR work like improving byLine. 
Something that has not really been internalized by folks here is that 
statistically EVERYBODY is on StackOverflow and NOBODY is on 
forum.dlang.org. Clearly the few folks in this forum are quite 
influential in the technical direction of the language, but the 
long-term pull (upwards or downward) of people influenced by articles 
like 
http://stackoverflow.com/questions/28922323/improving-line-wise-i-o-oper
tions-in-d/29153508 
can turn things radically better (or respectively worse). On that page 
there is one relevant piece of information - the link to the PR I 
created. It has just one vote, which pretty much means that next to 
nobody in this forum bothered to muster even that level of participation.

I can't express how much relief there is that Adam's weekly newsletter 
and Martin's work on the release mean that essentially I can take such 
matters off my mind.

Above all, my main preoccupation is to convert the energy going in 
incessant conversations in this forum on a variety of topics, from 
Brownian motion into directed propelling force.

===

A good trait of a graduate student is being able to act creatively on 
little advisor input. Academic advisors at top universities are almost 
always exceptionally good in their field; incredibly troves of insight 
and information. However, they are also immensely - and proverbially - 
busy which means they have little time to dedicate to any particular 
tidbit of information. For graduate students, that means if a particular 
matter reached the consciousness level of the adviser, it should 
definitely be looked into without further prodding. I sometimes wish D's 
leadership were in that position - if there is agreement some matter is 
important, it would be naturally looked into without a need to insist on 
it or debate it at nauseam.

Your proposal? I want to look into it. I must look into it, and it's a 
requirement that I look into it in order for it to make it into D. But I 
cannot promise when for as long as I am unable to delegate even the most 
trivial matters.


Andrei
Mar 22 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/22/15 12:29 PM, Andrei Alexandrescu wrote:

 Then I also need to get diverted into PR work like improving byLine.


Here "PR" stands for "Public Relations". -- Andrei
Mar 22 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/15/2015 7:10 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" wrote:

 Here's the new version of my scope proposal:
 http://wiki.dlang.org/User:Schuetzm/scope2

 It's still missing real-life examples, a section on the implementation, and a
 more formal specification, as well as a discussion of backwards compatibility.
 But I thought I'd show what I have, so that it can be discussed early on.

 I hope it will be more digestible for Walter & Andrei. It's more or less an
 extended version of DIP25, and avoids the need for most explicit annotations.


Thanks for doing this. It is a good step forward.

Consider the code:

   struct Foo { C obj; }
    safe void bar(C c, Foo* f) { f.obj = c; }

Under the proposal, for  safe code, this would have to be written as:

    safe void bar(static C c, Foo* f) ...

I'm concerned this may break an astonishing amount of code.
Mar 31 2015
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 31 March 2015 at 20:07:19 UTC, Walter Bright wrote:

 Consider the code:

   struct Foo { C obj; }
    safe void bar(C c, Foo* f) { f.obj = c; }

 Under the proposal, for  safe code, this would have to be 
 written as:

    safe void bar(static C c, Foo* f) ...


Either that, or:

      safe void bar(C c return!f, Foo* f) ...

or:

      safe void bar()(C c, Foo* f) ...


 I'm concerned this may break an astonishing amount of code.


We could go a step further and infer scope for all  safe 
functions, except those with at least one explicit annotation. 
This would be almost backward compatible, except for the fact 
that some arguments then suddenly become scope and therefore the 
mangling changes.
Apr 01 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/1/2015 10:00 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" wrote:

 On Tuesday, 31 March 2015 at 20:07:19 UTC, Walter Bright wrote:

 Consider the code:

   struct Foo { C obj; }
    safe void bar(C c, Foo* f) { f.obj = c; }

 Under the proposal, for  safe code, this would have to be written as:

    safe void bar(static C c, Foo* f) ...


 Either that, or:

       safe void bar(C c return!f, Foo* f) ...

 or:

       safe void bar()(C c, Foo* f) ...


 I'm concerned this may break an astonishing amount of code.


 We could go a step further and infer scope for all  safe functions, except
those
 with at least one explicit annotation. This would be almost backward
compatible,
 except for the fact that some arguments then suddenly become scope and
therefore
 the mangling changes.


The problem with inferring attributes on non-template functions is the source 
must be available to the compiler. That's obviously true for templates, but not 
so true for non-templates, where only the signature is available.

I'm thinking of a modest step which would be a subset of your proposal:

1. implement 'scope' and 'return' for arrays, classes, and pointers
2. implement inference for templates and lambdas
3. enable it with the -dip25 switch

and see how far that takes us.
Apr 01 2015
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:


 I'm thinking of a modest step which would be a subset of your 
 proposal:

 1. implement 'scope' and 'return' for arrays, classes, and 
 pointers
 2. implement inference for templates and lambdas
 3. enable it with the -dip25 switch

 and see how far that takes us.


This is interesting. For the final D programmer what's the 
practical difference between your proposed subset compared to the 
full proposal?

Bye,
bearophile
Apr 01 2015
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Wednesday, 1 April 2015 at 23:03:02 UTC, bearophile wrote:

 Walter Bright:


 I'm thinking of a modest step which would be a subset of your 
 proposal:

 1. implement 'scope' and 'return' for arrays, classes, and 
 pointers
 2. implement inference for templates and lambdas
 3. enable it with the -dip25 switch

 and see how far that takes us.


 This is interesting. For the final D programmer what's the 
 practical difference between your proposed subset compared to 
 the full proposal?


The main difference is that ` safe` doesn't imply `scope` 
anymore, which means a bit more typing. It also makes `static` 
annotations unnecessary.

I'm fine with that, because I only suggested it to reduce the 
need for explicit annotations, as Walter said that's important 
for him. Of course, it would also raise the risk of breaking 
existing code considerably, so that the potential benefits may 
not pull their weight overall.

But I hope overloading is included under point 1. It's very 
important for efficiency.
Apr 02 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/2/2015 4:44 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" wrote:

 But I hope overloading is included under point 1. It's very important for
 efficiency.


I'm not convinced of the need for overloading on 'scope'.
Apr 04 2015
parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:


 I'm not convinced of the need for overloading on 'scope'.


Do you want to explain some of the advantages and disadvantages 
of that? It will help understand your reasons.

Bye,
bearophile
Apr 04 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-04-01 22:26, Walter Bright wrote:


 The problem with inferring attributes on non-template functions is the
 source must be available to the compiler. That's obviously true for
 templates, but not so true for non-templates, where only the signature
 is available.


In my experience with D, the source code for most the functions are 
available. Everyone is doing open source and no one bother with 
writing/generating .di files. Not implementing attribute inference for 
non-template functions just because the source code of _some_ functions 
might not be present is not a good reason. Why should the non-template 
functions that _have_ the source code available suffer?

-- 
/Jacob Carlborg
Apr 01 2015
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/1/15 11:45 PM, Jacob Carlborg wrote:

 On 2015-04-01 22:26, Walter Bright wrote:


 The problem with inferring attributes on non-template functions is the
 source must be available to the compiler. That's obviously true for
 templates, but not so true for non-templates, where only the signature
 is available.


 In my experience with D, the source code for most the functions are
 available. Everyone is doing open source and no one bother with
 writing/generating .di files. Not implementing attribute inference for
 non-template functions just because the source code of _some_ functions
 might not be present is not a good reason. Why should the non-template
 functions that _have_ the source code available suffer?


Because large projects. -- Andrei
Apr 01 2015
prev sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Thu, 02 Apr 2015 08:45:10 +0200, Jacob Carlborg wrote:


 Why should the non-template
 functions that _have_ the source code available suffer?


'cause inferring attributes requires running full semantic analysis on=20
the code, so each imported function *must* be fully processed. and with=20
explicit attributes it's enough to parse it and setup signature types.=
Apr 01 2015
next sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Thu, 02 Apr 2015 06:59:40 +0000, ketmar wrote:


 On Thu, 02 Apr 2015 08:45:10 +0200, Jacob Carlborg wrote:
=20

 Why should the non-template functions that _have_ the source code
 available suffer?


=20
 'cause inferring attributes requires running full semantic analysis on
 the code, so each imported function *must* be fully processed. and with
 explicit attributes it's enough to parse it and setup signature types.


p.s. you can always fix your code instead, turning your functions to=20
argless templates.=
Apr 02 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-04-02 08:59, ketmar wrote:


 'cause inferring attributes requires running full semantic analysis on
 the code, so each imported function *must* be fully processed.


Isn't this required anyway?

-- 
/Jacob Carlborg
Apr 02 2015
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Thu, 02 Apr 2015 13:39:01 +0200, Jacob Carlborg wrote:


 On 2015-04-02 08:59, ketmar wrote:
=20

 'cause inferring attributes requires running full semantic analysis on
 the code, so each imported function *must* be fully processed.


=20
 Isn't this required anyway?


.di files has no function bodies, yet they still works. so compiler *can*=20
do full processing, but it is not required, and if compiler does that,=20
this restriction can be removed in the future. but with automatic=20
inference for functions we are stuck with full semantic analysis.=
Apr 02 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-04-02 14:02, ketmar wrote:


 .di files has no function bodies, yet they still works. so compiler *can*
 do full processing, but it is not required, and if compiler does that,
 this restriction can be removed in the future. but with automatic
 inference for functions we are stuck with full semantic analysis.


I talked about when the body is available. In my experience most 
projects do not use .di files.

-- 
/Jacob Carlborg
Apr 02 2015
parent ketmar <ketmar ketmar.no-ip.org> writes:
On Thu, 02 Apr 2015 16:04:59 +0200, Jacob Carlborg wrote:


 On 2015-04-02 14:02, ketmar wrote:
=20

 .di files has no function bodies, yet they still works. so compiler
 *can*
 do full processing, but it is not required, and if compiler does that,
 this restriction can be removed in the future. but with automatic
 inference for functions we are stuck with full semantic analysis.


=20
 I talked about when the body is available. In my experience most
 projects do not use .di files.


i just used .di an example that demonstrates that there is no need to do=20
full-blown semantic analysis even for functions with bodies. this process=20
(semantic analysis) can be both slow and memory consuming, so building=20
big application with many modules will be unnecessary slow and will=20
require more resources. and we already has a kind of "mark" that told=20
compiler to do attribute inference: empty template arg specifier.=
Apr 02 2015