www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP69 - Implement scope for escape proof references

reply Walter Bright <newshound2 digitalmars.com> writes:
http://wiki.dlang.org/DIP69

Despite its length, this is a fairly simple proposal. It adds the missing 
semantics for the 'scope' storage class in order to make it possible to pass a 
reference to a function without it being possible for it to escape.

This, among other things, makes a ref counting type practical. It also makes it 
more practical to use other storage allocation schemes than garbage collection.

It does not make scope into a type constructor, nor a general type-annotation 
system.

It does not provide an ownership system, though it would complement one.
Dec 04 2014
next sibling parent reply "eles" <eles eles.com> writes:
On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds 
 the missing semantics for the 'scope' storage class in order to 
 make it possible to pass a reference to a function without it 
 being possible for it to escape.


Making it implicit and requiring an explicit "escape" for 
un-scoped variables?
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 1:51 AM, eles wrote:

 On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds the missing
 semantics for the 'scope' storage class in order to make it possible to pass a
 reference to a function without it being possible for it to escape.


 Making it implicit and requiring an explicit "escape" for un-scoped variables?


Was afraid that would break too much code.
Dec 04 2014
next sibling parent reply "eles" <eles eles.com> writes:
On Thursday, 4 December 2014 at 10:00:37 UTC, Walter Bright wrote:

 On 12/4/2014 1:51 AM, eles wrote:

 On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright 
 wrote:

 http://wiki.dlang.org/DIP69







 Was afraid that would break too much code.


An annotation for functions could make all variables "scope"?

Similar to:

int foo(int x)  scopedvars {
  //all declarations are implicit "scope"; use "escape" for the 
others

}
Dec 04 2014
parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thu, 04 Dec 2014 10:04:07 +0000
eles via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 On Thursday, 4 December 2014 at 10:00:37 UTC, Walter Bright wrote:

 On 12/4/2014 1:51 AM, eles wrote:

 On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright=20
 wrote:

 http://wiki.dlang.org/DIP69






=20

 Was afraid that would break too much code.


=20
 An annotation for functions could make all variables "scope"?
=20
 Similar to:
=20
 int foo(int x)  scopedvars {
   //all declarations are implicit "scope"; use "escape" for the=20
 others
=20
 }


please, don't steal this nice identifier! i love variables with name
"escape"! stolen "body" still hurts me.
Dec 04 2014
parent "eles" <eles eles.com> writes:
On Thursday, 4 December 2014 at 10:11:25 UTC, ketmar via 
Digitalmars-d wrote:

 On Thu, 04 Dec 2014 10:04:07 +0000
 eles via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 On Thursday, 4 December 2014 at 10:00:37 UTC, Walter Bright 
 wrote:

 On 12/4/2014 1:51 AM, eles wrote:

 On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright 
 wrote:









 please, don't steal this nice identifier! i love variables with 
 name
 "escape"! stolen "body" still hurts me.


It was just an idea, I do not push for it.
Dec 04 2014
prev sibling next sibling parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2014-12-04 10:00:28 +0000, Walter Bright said:


 On 12/4/2014 1:51 AM, eles wrote:

 On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP69
 
 Despite its length, this is a fairly simple proposal. It adds the missing
 semantics for the 'scope' storage class in order to make it possible to pass a
 reference to a function without it being possible for it to escape.


 
 Making it implicit and requiring an explicit "escape" for un-scoped variables?
 


 
 Was afraid that would break too much code.


No, this is super important.  Break it all!  This kind of change will 
significantly increase the performance of D code by almost eliminating 
GC thrashing.

Having to type scope EVERYWHERE will make it unused and therefore 
unusable in other people's libraries.

-Shammah.
Dec 10 2014
parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 10 Dec 2014 23:06:13 -0800
Shammah Chancellor via Digitalmars-d <digitalmars-d puremagic.com>
wrote:


 Was afraid that would break too much code.


 No, this is super important.  Break it all!


alas, not in this life. each "break our code" just making resistance
harder. ;-)
Dec 11 2014
prev sibling parent "zeljkog" <zeljkog home.com> writes:
On Thursday, 4 December 2014 at 10:00:37 UTC, Walter Bright wrote:

 On 12/4/2014 1:51 AM, eles wrote:

 Making it implicit and requiring an explicit "escape" for 
 un-scoped variables?


 Was afraid that would break too much code.


Compiler switch: escape=I(gnore)|W(arning)|E(rror).
For transition default Ignore.
Dec 11 2014
prev sibling next sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thu, 04 Dec 2014 01:24:13 -0800
Walter Bright via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 http://wiki.dlang.org/DIP69
=20
 Despite its length, this is a fairly simple proposal. It adds the missing=


=20

 semantics for the 'scope' storage class in order to make it possible to p=


ass a=20

 reference to a function without it being possible for it to escape.
=20
 This, among other things, makes a ref counting type practical. It also ma=


kes it=20

 more practical to use other storage allocation schemes than garbage colle=


ction.

=20
 It does not make scope into a type constructor, nor a general type-annota=


tion=20

 system.
=20
 It does not provide an ownership system, though it would complement one.


cosmetic issue: some comments are referring to rules by number ("Error,
rule 5"), yet the rules aren't explicitly numbered. not a big deal, but
still somewhat hard to follow.
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 1:53 AM, ketmar via Digitalmars-d wrote:

 cosmetic issue: some comments are referring to rules by number ("Error,
 rule 5"), yet the rules aren't explicitly numbered. not a big deal, but
 still somewhat hard to follow.


Yeah, still learning wiki markup!
Dec 04 2014
next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 01:57:43AM -0800, Walter Bright via Digitalmars-d wrote:

 On 12/4/2014 1:53 AM, ketmar via Digitalmars-d wrote:

cosmetic issue: some comments are referring to rules by number
("Error, rule 5"), yet the rules aren't explicitly numbered. not a
big deal, but still somewhat hard to follow.


 
 Yeah, still learning wiki markup!


I don't understand the line where rule 5 was invoked:

	scope int* a;
	...
	scope int** f = &a;    // Error, rule 5

Why is it an error, since f has a shorter lifetime than a? And what has
it gotta do with rule 5, which currently reads:

	A scope ref variable can be initialized with another scope ref
	variable - scope ref is idempotent.

?


T

-- 
Real men don't take backups. They put their source on a public
FTP-server and let the world mirror it. -- Linus Torvalds
Dec 04 2014
prev sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 10:31:07AM -0800, H. S. Teoh via Digitalmars-d wrote:

 On Thu, Dec 04, 2014 at 01:57:43AM -0800, Walter Bright via Digitalmars-d
wrote:

 On 12/4/2014 1:53 AM, ketmar via Digitalmars-d wrote:

cosmetic issue: some comments are referring to rules by number
("Error, rule 5"), yet the rules aren't explicitly numbered. not a
big deal, but still somewhat hard to follow.


 
 Yeah, still learning wiki markup!


 
 I don't understand the line where rule 5 was invoked:
 
 	scope int* a;
 	...
 	scope int** f = &a;    // Error, rule 5
 
 Why is it an error, since f has a shorter lifetime than a? And what
 has it gotta do with rule 5, which currently reads:
 
 	A scope ref variable can be initialized with another scope ref
 	variable - scope ref is idempotent.
 
 ?


[...]

Ah, nevermind, it should be rule 4, not rule 5. Rule 4 states that the
address of scope variables cannot be assigned to another scope variable.
Please fix the comment. ;-)


T

-- 
I've been around long enough to have seen an endless parade of magic new
techniques du jour, most of which purport to remove the necessity of
thought about your programming problem.  In the end they wind up
contributing one or two pieces to the collective wisdom, and fade away
in the rearview mirror. -- Walter Bright
Dec 04 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 10:34 AM, H. S. Teoh via Digitalmars-d wrote:

 Please fix the comment. ;-)


done
Dec 04 2014
prev sibling next sibling parent "Robert burner Schadek" <rburners gmail.com> writes:
thank you for pushing on this.

Lifetime last bullet point: ", but lower than any variables in 
higher scopes."
isn't that redundant to the first bullet point? Or am I missing 
something?

Scope affects variables according to these rules:
Could you enumerate the list instead of bullet points, I can't 
count.

Base operation:
That one I don't get. I would assume this is legal as the 
function is not  safe.
e = &c; // Error, lifetime(e's view) is &infin; and is greater 
than lifetime(c)
Dec 04 2014
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds 
 the missing semantics for the 'scope' storage class in order to 
 make it possible to pass a reference to a function without it 
 being possible for it to escape.

 This, among other things, makes a ref counting type practical. 
 It also makes it more practical to use other storage allocation 
 schemes than garbage collection.

 It does not make scope into a type constructor, nor a general 
 type-annotation system.

 It does not provide an ownership system, though it would 
 complement one.


Haven't got the time to look at this in detail right now, but if 
it's as good as it looks on the surface, I'm very excited to use 
it. Thanks for all the hard work, all involved.
Dec 04 2014
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
 Errors for scope violations are only reported in  safe code.


Why? If I've explicitly designated a reference as scope, why 
should it be ignored in un- safe code?
Dec 04 2014
next sibling parent "Martin Nowak" <code dawg.eu> writes:
On Thursday, 4 December 2014 at 11:21:27 UTC, Marc Schütz wrote:

 Errors for scope violations are only reported in  safe code.


 Why? If I've explicitly designated a reference as scope, why 
 should it be ignored in un- safe code?


Agreed, it should also work for any other code with some function 
to cast away scope.

ref T unscope(scope ref T t)  system
{
     auto p = &t;
     return *p;
}
Dec 04 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 3:21 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:

 Errors for scope violations are only reported in  safe code.


 Why? If I've explicitly designated a reference as scope, why should it be
 ignored in un- safe code?


To interface to code that presents a safe interface, but does things under the 
hood that can't be verified.
Dec 04 2014
prev sibling next sibling parent reply "Martin Nowak" <code dawg.eu> writes:
On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP69


Great stuff.

Will this be possible, it's a fairly important use-case.

scope ref T setVal(scope ref T t)
{
     t.val = 12;
     return t;
}

Another question, how would a reference counted pointer take 
advantage of scope, i.e. avoid the increment/decrement when being 
passed to a function?
One solution would be to add a function that returns a scoped 
reference to the underlying value.
     struct RefCounted(T)
     {
         scope ref T borrow() { return *p; }
     }
Will it be possible to deduce, that the lifetime of that scoped 
value is tied to the smart pointer?
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 4:03 AM, Martin Nowak wrote:

 On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP69


 Great stuff.

 Will this be possible, it's a fairly important use-case.

 scope ref T setVal(scope ref T t)
 {
      t.val = 12;
      return t;
 }


Yes, it would be written:

   scope ref T setVal(ref T t)
   {
      t.val = 12;
      return t;
   }



 Another question, how would a reference counted pointer take advantage of
scope,
 i.e. avoid the increment/decrement when being passed to a function?
 One solution would be to add a function that returns a scoped reference to the
 underlying value.
      struct RefCounted(T)
      {
          scope ref T borrow() { return *p; }
      }
 Will it be possible to deduce, that the lifetime of that scoped value is tied
to
 the smart pointer?


   struct RefCounted(T)
   {
     T t;
     scope ref T borrow() { return t; }
     alias this t;
   }

This enables RefCounted!T to be implicitly converted to a T, but with a scoped 
result. This is a critical feature, one I spent a lot of time thinking about, 
and hope it's right :-)
Dec 04 2014
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/4/14 3:41 PM, Walter Bright wrote:

 On 12/4/2014 4:03 AM, Martin Nowak wrote:

 Will it be possible to deduce, that the lifetime of that scoped value
 is tied to
 the smart pointer?


    struct RefCounted(T)
    {
      T t;
      scope ref T borrow() { return t; }
      alias this t;
    }

 This enables RefCounted!T to be implicitly converted to a T, but with a
 scoped result. This is a critical feature, one I spent a lot of time
 thinking about, and hope it's right :-)


Hm... did you mean `alias this borrow`?

-Steve
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 1:24 PM, Steven Schveighoffer wrote:

 Hm... did you mean `alias this borrow`?



yes (hangs head in shame)
Dec 04 2014
parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:


 On 12/4/2014 1:24 PM, Steven Schveighoffer wrote:

 Hm... did you mean `alias this borrow`?



 yes (hangs head in shame)


No need to be ashamed for similar things, you are doing your best.

Bye,
bearophile
Dec 04 2014
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 12:41:58PM -0800, Walter Bright via Digitalmars-d wrote:
[...]

   struct RefCounted(T)
   {
     T t;
     scope ref T borrow() { return t; }
     alias this t;
   }
 
 This enables RefCounted!T to be implicitly converted to a T, but with
 a scoped result. This is a critical feature, one I spent a lot of time
 thinking about, and hope it's right :-)


Hold on, how does this convert to T with a scoped result? Or did you
mean:

	...
	 property scope ref T borrow() { ... }
	alias borrow this;

?


T

-- 
Let's call it an accidental feature. -- Larry Wall
Dec 04 2014
parent Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 04/12/2014 21:23, H. S. Teoh via Digitalmars-d wrote:

 	 property scope ref T borrow() { return t; }
 	alias borrow this;


While this DIP enabling the above to be memory-safe is awesome, a later 
tweak to AliasThis grammar to allow storage classes could make 'borrow' 
redundant. Plain alias now allows storage classes:

AliasDeclaration:
     alias StorageClassesopt BasicType Declarator ;

So:

TweakedAliasThis:
     alias StorageClassesopt Identifier this ;

Given ref and now scope will be storage classes, that would then allow just:

alias scope ref t this;
Dec 06 2014
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 12/04/2014 09:41 PM, Walter Bright wrote:

 Yes, it would be written:

    scope ref T setVal(ref T t)
    {
       t.val = 12;
       return t;
    }


But when there is no scope on the argument, I could not call setVal with 
a local T variable.
Dec 04 2014
next sibling parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 12/04/2014 10:54 PM, Martin Nowak wrote:

 On 12/04/2014 09:41 PM, Walter Bright wrote:

 Yes, it would be written:

    scope ref T setVal(ref T t)
    {
       t.val = 12;
       return t;
    }


 But when there is no scope on the argument, I could not call setVal with
 a local T variable.


Ah, it's inferred. Makes sense now.
Dec 04 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 1:54 PM, Martin Nowak wrote:

 On 12/04/2014 09:41 PM, Walter Bright wrote:

 Yes, it would be written:

    scope ref T setVal(ref T t)
    {
       t.val = 12;
       return t;
    }


 But when there is no scope on the argument, I could not call setVal with a
local
 T variable.


Actually, you can. The difference between 'ref' and 'scope ref' parameters is 
the latter cannot be returned by ref.
Dec 04 2014
prev sibling next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
I am not expert in such things, but here are few comments and 
questions.


 Errors for scope violations are only reported in  safe code.


This seems acceptable only if the compiler switch "-scope" 
implies functions to be  safe by default and  system on request, 
because currently lot of D programmers don't apply annotations 
like  safe to their D code. Opt-in safety doesn't work well (and 
in D we still have the problems caused by null pointers and 
references).

- - - - - - - - - -

Also consider an annotation like "!scope" or " escape" for the 
opposite purpose.

- - - - - - - - - -


Delegates currently defensively allocate closures with the GC. 
Few actually escape, and with scope only those that actually 
escape need to have the closures allocated.<


So there's no need for extra annotations to make delegates  nogc 
in most cases?

- - - - - - - - - -

Regarding array literals, some people proposed a syntax for 
fixed-size arrays to avoid heap-allocations (the "s" after the 
array literal):

void foo(int[2]) {}
void bar(int[]) {}
void main()  nogc {
     foo([1, 2]s);
     bar([1, 2]s);
}


Is DIP69 able to infer those arrays can be both stack-allocated? 
Is the "s" annotations still useful?

- - - - - - - - - -

Regarding the benefits, is escape analysis going to be used to 
allocate _automatically_ some small dynamic arrays and 
classes/structs on the stack instead of heap?

And is escape analysis going to automatically give hints to the 
GC to deallocate faster GC-allocated memory that doesn't escape?

void foo() {
     // Both dynamic arrays don't escape foo

     // Automatically stack allocated.
     auto a = new int[3];

     // Heap-allocated but deterministically
     // deleted at the end of foo scope.
     auto b = new int[10_000];
}

- - - - - - - - - -

Bye,
bearophile
Dec 04 2014
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
 void foo(int[2]) {}
 void bar(int[]) {}
 void main()  nogc {
     foo([1, 2]s);
     bar([1, 2]s);
 }


Perhaps better:

void foo(int[2]) {}
void bar(scope int[]) {}
void main()  nogc {
     foo([1, 2]s);
     bar([1, 2]s);
}

Bye,
bearophile
Dec 04 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 4:55 AM, bearophile wrote:

 I am not expert in such things, but here are few comments and questions.


 Errors for scope violations are only reported in  safe code.


 This seems acceptable only if the compiler switch "-scope" implies functions to
 be  safe by default and  system on request, because currently lot of D
 programmers don't apply annotations like  safe to their D code. Opt-in safety
 doesn't work well (and in D we still have the problems caused by null pointers
 and references).


Safety by default is outside of the sco[pe (!) of this discussion. Also, you
can 
merely put:

      safe:

at the beginning of a module and it is now all safe.


 Also consider an annotation like "!scope" or " escape" for the opposite
purpose.


That's part of a more general issue of negation of attributes.



 Delegates currently defensively allocate closures with the GC. Few actually
 escape, and with scope only those that actually escape need to have the
 closures allocated.<


 So there's no need for extra annotations to make delegates  nogc in most cases?


Marking delegates as  nogc does not affect the closure, as the closure is 
allocated when the enclosing function is entered.




 Regarding array literals, some people proposed a syntax for fixed-size arrays
to
 avoid heap-allocations (the "s" after the array literal):

 void foo(int[2]) {}
 void bar(int[]) {}
 void main()  nogc {
      foo([1, 2]s);
      bar([1, 2]s);
 }


 Is DIP69 able to infer those arrays can be both stack-allocated? Is the "s"
 annotations still useful?


It's an idea worth exploring.



 Regarding the benefits, is escape analysis going to be used to allocate
 _automatically_ some small dynamic arrays and classes/structs on the stack
 instead of heap?


This is an optimization opportunity that scope enables.



 And is escape analysis going to automatically give hints to the GC to
deallocate
 faster GC-allocated memory that doesn't escape?

 void foo() {
      // Both dynamic arrays don't escape foo

      // Automatically stack allocated.
      auto a = new int[3];

      // Heap-allocated but deterministically
      // deleted at the end of foo scope.
      auto b = new int[10_000];
 }


Another optimization opportunity.
Dec 04 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:


 This seems acceptable only if the compiler switch "-scope" 
 implies functions to
 be  safe by default and  system on request, because currently 
 lot of D
 programmers don't apply annotations like  safe to their D 
 code. Opt-in safety
 doesn't work well (and in D we still have the problems caused 
 by null pointers and references).


 Safety by default is outside of the sco[pe (!) of this 
 discussion. Also, you can merely put:

      safe:

 at the beginning of a module and it is now all safe.


I agree that  system code should allow to circumvent the part of 
the type system that enforces the scoping (this happens in Rust 
too) (but perhaps this has to happen only on scopes that gets 
inferred and not the ones specified explicitly).

Perhaps you can't leave the " safe on default" issue out of the 
DIP69. Currently most D code I see around is not tagged with 
 safe. If people don't bother applying that annotation, all the 
work you will do to implement DIP69 will be wasted or partially 
wasted.

------------------

 From your answer to H. S. Teoh:


 Are there bug reports on this?


I and other people have opened several bug reports on  safe, some 
of them are (but others are present, this is a subset):

https://issues.dlang.org/show_bug.cgi?id=13615
https://issues.dlang.org/show_bug.cgi?id=13681
https://issues.dlang.org/show_bug.cgi?id=12948
https://issues.dlang.org/show_bug.cgi?id=13607
https://issues.dlang.org/show_bug.cgi?id=12845
https://issues.dlang.org/show_bug.cgi?id=6646
https://issues.dlang.org/show_bug.cgi?id=11176
https://issues.dlang.org/show_bug.cgi?id=6333
https://issues.dlang.org/show_bug.cgi?id=13054
https://issues.dlang.org/show_bug.cgi?id=13506
https://issues.dlang.org/show_bug.cgi?id=13188


Some examples of the problems (but there are also opposite 
problems, like in Issue 6646):

void main()  safe {
     import std.stdio, std.algorithm, std.bigint, std.typecons, 
std.array;
     [1, 2].sort!("a < b", SwapStrategy.stable);
     auto r = [1, 2].sort().release;
     writeln;
     BigInt a;
     a = a + a;
     alias Foo = Tuple!int;
     Foo[] data;
     data.remove!(x => x == Foo());
     int[] b;
     auto c = b.capacity;
     b.schwartzSort!(x => x);
     const r2 = cartesianProduct([1], [1]).array;
     [Typedef!int(1)].array;
}


Output, dmd 2.067alpha:

test.d(3,11): Error: safe function 'D main' cannot call system 
function 'std.algorithm.sort!("a < b", cast(SwapStrategy)2, 
int[]).sort'
test.d(4,25): Error: safe function 'D main' cannot call system 
function 'std.range.SortedRange!(int[], "a < 
b").SortedRange.release'
test.d(5,5): Error: safe function 'D main' cannot call system 
function 'std.stdio.writeln!().writeln'
test.d(7,9): Error: safe function 'D main' cannot call system 
function 'std.bigint.BigInt.opBinary!("+", BigInt).opBinary'
test.d(10,9): Error: safe function 'D main' cannot call system 
function 'test.main.remove!((x) => x == Foo(), 
cast(SwapStrategy)2, Tuple!int[]).remove'
test.d(12,15): Error: safe function 'D main' cannot call system 
function 'object.capacity!int.capacity'
test.d(13,6): Error: safe function 'D main' cannot call system 
function 'test.main.schwartzSort!((x) => x, "a < b", 
cast(SwapStrategy)0, int[]).schwartzSort'
test.d(14,42): Error: safe function 'D main' cannot call system 
function 'std.array.array!(Result).array'
test.d(15,21): Error: safe function 'D main' cannot call system 
function 'std.array.array!(Typedef!(int, 0, null)[]).array'

Bye,
bearophile
Dec 04 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 2:08 PM, bearophile wrote:

 Perhaps you can't leave the " safe on default" issue out of the DIP69.
Currently
 most D code I see around is not tagged with  safe. If people don't bother
 applying that annotation, all the work you will do to implement DIP69 will be
 wasted or partially wasted.


The attributes don't have much use for small programs. When a program is small 
enough that one person can keep it all in their head, there is not much use for 
it. I'd like to keep D usable for quick small program writing.

Attributes become more and more useful the larger the program gets, and the
more 
people are working on it. In such cases, I don't think it's near as much of a 
burden to put in a few annotations, and the payoff is significantly larger.



 I and other people have opened several bug reports on  safe, some of them are
 (but others are present, this is a subset):

 https://issues.dlang.org/show_bug.cgi?id=13615
 https://issues.dlang.org/show_bug.cgi?id=13681
 https://issues.dlang.org/show_bug.cgi?id=12948
 https://issues.dlang.org/show_bug.cgi?id=13607
 https://issues.dlang.org/show_bug.cgi?id=12845
 https://issues.dlang.org/show_bug.cgi?id=6646
 https://issues.dlang.org/show_bug.cgi?id=11176
 https://issues.dlang.org/show_bug.cgi?id=6333
 https://issues.dlang.org/show_bug.cgi?id=13054
 https://issues.dlang.org/show_bug.cgi?id=13506
 https://issues.dlang.org/show_bug.cgi?id=13188


Thanks you.
Dec 04 2014
prev sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:


 Safety by default is outside of the sco[pe (!) of this 
 discussion. Also, you can merely put:

      safe:

 at the beginning of a module and it is now all safe.


If that topic is outside the scope of this discussion, then I 
have opened an ER on it:
https://issues.dlang.org/show_bug.cgi?id=13838

Bye,
bearophile
Dec 08 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2014 3:34 PM, bearophile wrote:

 If that topic is outside the scope of this discussion, then I have opened an ER
 on it:
 https://issues.dlang.org/show_bug.cgi?id=13838


1. You can always start a new thread here.
2. Thanks for filing the E.R.
3. When filing such, please check the [Hardware] settings. You have it set to 
[x86][Windows]. I reset them to [All][All].
Dec 09 2014
prev sibling next sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thu, 04 Dec 2014 12:55:34 +0000
bearophile via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 This seems acceptable only if the compiler switch "-scope"=20
 implies functions to be  safe by default and  system on request,=20
 because currently lot of D programmers don't apply annotations=20
 like  safe to their D code. Opt-in safety doesn't work well (and=20
 in D we still have the problems caused by null pointers and=20
 references).


actually, for me explicit attributes doesn't work well at all. and D
compiler can't infer atributes for functions (ok, it can, but it can't
apply that without changing function signature). that's why alot of my
code compiles awfully slow with separate compilation: i'm used to write
argument-less templates everywhere, so compiler can infer and apply
attributes by itself. i even stopped commenting that practice, 'cause
it means that much of the code will carry comments like this:

  // template to allow attribute inference
  void foo() (A a, int c) { ... }


i think that something is very wrong with the function attributes. but
i still can't think out how to improve that. but it still feels wrong.

that's not about "D designers made a mistake", that's about "i want to
invent a better thing!" ;-)
Dec 04 2014
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Dec 05, 2014 at 03:16:36AM +0200, ketmar via Digitalmars-d wrote:

 On Thu, 04 Dec 2014 12:55:34 +0000
 bearophile via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 

 This seems acceptable only if the compiler switch "-scope" 
 implies functions to be  safe by default and  system on request, 
 because currently lot of D programmers don't apply annotations 
 like  safe to their D code. Opt-in safety doesn't work well (and 
 in D we still have the problems caused by null pointers and 
 references).


 actually, for me explicit attributes doesn't work well at all. and D
 compiler can't infer atributes for functions (ok, it can, but it can't
 apply that without changing function signature). that's why alot of my
 code compiles awfully slow with separate compilation: i'm used to write
 argument-less templates everywhere, so compiler can infer and apply
 attributes by itself. i even stopped commenting that practice, 'cause
 it means that much of the code will carry comments like this:
 
   // template to allow attribute inference
   void foo() (A a, int c) { ... }


I've been considering to start adopting that practice too.



 i think that something is very wrong with the function attributes. but
 i still can't think out how to improve that. but it still feels wrong.
 
 that's not about "D designers made a mistake", that's about "i want to
 invent a better thing!" ;-)


In an ideal world, programmers would never have to know attributes even
existed, the compiler would infer everything and optimize based on that.

However, we're not quite there yet. :-) (And I don't know when we'll
ever get there. Maybe D3. But we probably won't see that in the next 10
years... probably more.)

I've often pondered about the possibility of a language where the
compiler will analyze each module and infer any number of attributes and
optimization opportunities for each symbol exported by that module, and
this information will be saved in the object file (or some other kind of
interfacing file). This includes any half-compiled template bodies and
whatever else that can't be fully codegen'd until actual use.  The
attributes will include all sorts of stuff that programmers normally
wouldn't want to deal with -- there could be 10+ or 50+ attributes
representing various optimization / static checking opportunities.  Then
every time a module is imported by another module, the compiler never
goes to the source code of the imported module anymore, but it will read
the object (interface) file, which is fully attributed, and the saved
attributes will be used internally for static checking, optimization,
and inferring attributes for the current module.

This way, the programmer not only gets the benefit of attributes without
actually having to explicitly use them, but the language designers can
also add many more attributes than you'd ever want to manually type out,
or even remove old attributes that are no longer useful. The interface
files will also serve as a convenient (and superior) substitute for .di
files (uuugly) or .h files (even worse).


T

-- 
Right now I'm having amnesia and deja vu at the same time. I think I've
forgotten this before.
Dec 04 2014
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"H. S. Teoh via Digitalmars-d"  wrote in message 
news:mailman.2709.1417745546.9932.digitalmars-d puremagic.com...


 I've often pondered about the possibility of a language where the
 compiler will analyze each module and infer any number of attributes and
 optimization opportunities for each symbol exported by that module, and
 this information will be saved in the object file (or some other kind of
 interfacing file). This includes any half-compiled template bodies and
 whatever else that can't be fully codegen'd until actual use.  The
 attributes will include all sorts of stuff that programmers normally
 wouldn't want to deal with -- there could be 10+ or 50+ attributes
 representing various optimization / static checking opportunities.  Then
 every time a module is imported by another module, the compiler never
 goes to the source code of the imported module anymore, but it will read
 the object (interface) file, which is fully attributed, and the saved
 attributes will be used internally for static checking, optimization,
 and inferring attributes for the current module.


This can't be used to infer attributes that can produce errors - those 
attributes have to be user-visible or the errors don't make any sense.  If 
it's purely for optimization, then that's basically what LTO does.
Dec 04 2014
parent "Tobias Pankrath" <tobias pankrath.net> writes:
On Friday, 5 December 2014 at 02:38:48 UTC, Daniel Murphy wrote:

 "H. S. Teoh via Digitalmars-d"  wrote in message 
 news:mailman.2709.1417745546.9932.digitalmars-d puremagic.com...


 I've often pondered about the possibility of a language where 
 the
 compiler will analyze each module and infer any number of 
 attributes and
 optimization opportunities for each symbol exported by that 
 module, and
 this information will be saved in the object file (or some 
 other kind of
 interfacing file). This includes any half-compiled template 
 bodies and
 whatever else that can't be fully codegen'd until actual use.  
 The
 attributes will include all sorts of stuff that programmers 
 normally
 wouldn't want to deal with -- there could be 10+ or 50+ 
 attributes
 representing various optimization / static checking 
 opportunities.  Then
 every time a module is imported by another module, the 
 compiler never
 goes to the source code of the imported module anymore, but it 
 will read
 the object (interface) file, which is fully attributed, and 
 the saved
 attributes will be used internally for static checking, 
 optimization,
 and inferring attributes for the current module.


 This can't be used to infer attributes that can produce errors 
 - those attributes have to be user-visible or the errors don't 
 make any sense.  If it's purely for optimization, then that's 
 basically what LTO does.


We could inter attributes if not specified instead of assuming a
default, for example  nogc and a possible  gc.

---

int[] foo1(int[] bar)  nogc // function is  nogc, error if gc is
used
int[] foo2(int[] bar)  gc // function is  gc, functions that call
foo2 cannot be  nogc
int[] foo3(int[] bar) // neither  nogc nor  gc, compiler infers
attribute
---
Dec 05 2014
prev sibling next sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thu, 4 Dec 2014 18:10:18 -0800
"H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> wrote:


 This way, the programmer not only gets the benefit of attributes without
 actually having to explicitly use them, but the language designers can
 also add many more attributes than you'd ever want to manually type out,
 or even remove old attributes that are no longer useful. The interface
 files will also serve as a convenient (and superior) substitute for .di
 files (uuugly) or .h files (even worse).


and those interface files will ease creation of my pet project: dynamic
component framework a-la BlackBox Component Builder. i'm already
thinking about exporting semi-processed ASTs as ".mda" files, so i can
write tools that bypasses parsing and checking phases. such files can
be used as module interfaces, as syntax completion databases and so on.
they can even be processed with some bytecode generator to improve CTFE.

yet my project plan is to release pre-alpha version not sooner than in
2030, so i'm not really working on it with full speed. ;-)
Dec 04 2014
prev sibling next sibling parent reply "ixid" <nuaccount gmail.com> writes:
 void foo(int[2]) {}
 void bar(int[]) {}
 void main()  nogc {
     foo([1, 2]s);
     bar([1, 2]s);
 }


That is a rather unfriendly syntax, it is the kind that 
degenerates into noise with other structures.
Dec 05 2014
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
ixid:


 void foo(int[2]) {}
 void bar(int[]) {}
 void main()  nogc {
    foo([1, 2]s);
    bar([1, 2]s);
 }


 That is a rather unfriendly syntax, it is the kind that 
 degenerates into noise with other structures.


Can you show an example of the noisy code it causes?

And are you able to invent something succint that is better?

Bye,
bearophile
Dec 05 2014
parent "ixid" <nuaccount gmail.com> writes:
On Friday, 5 December 2014 at 14:10:44 UTC, bearophile wrote:

 ixid:


 void foo(int[2]) {}
 void bar(int[]) {}
 void main()  nogc {
   foo([1, 2]s);
   bar([1, 2]s);
 }


 That is a rather unfriendly syntax, it is the kind that 
 degenerates into noise with other structures.


 Can you show an example of the noisy code it causes?

 And are you able to invent something succint that is better?

 Bye,
 bearophile


[1,2].stack
stack [1,2]
 stack [1,2]
[1,2]stack
Dec 05 2014
prev sibling parent reply Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 04/12/2014 12:55, bearophile wrote:

 Regarding array literals, some people proposed a syntax for fixed-size
 arrays to avoid heap-allocations (the "s" after the array literal):

 void foo(int[2]) {}
 void bar(scope int[]) {}
 void main()  nogc {
      foo([1, 2]s);
      bar([1, 2]s);
 }


I think even if the compiler could infer them as static arrays, it may 
still be useful to be explicit sometimes. We can already use a library 
template:

template staticArray(items...)
{
     import std.traits;
     alias T = CommonType!items[items.length];
     enum T staticArray = [items];
}

auto s = staticArray!(1, 2);
static assert(is(typeof(s) == int[2]));

bar(staticArray!(1, 2));

This might also make the proposed 'int[$] = [...];' syntax unnecessary.

I think jmdavis once wrote something similar - although I've used enum 
here in case it helps avoid function template bloat.
Dec 07 2014
parent "bearophile" <bearophileHUGS lycos.com> writes:
Nick Treleaven:


 This might also make the proposed 'int[$] = [...];' syntax 
 unnecessary.


Or might not. The [$] proposal is very refined.

Bye,
bearophile
Dec 07 2014
prev sibling next sibling parent reply "Kagamin" <spam here.lot> writes:
On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP69


How inout fits the picture? It has some scope semantics too.
Section about expressions suggests you will tack scoping 
carefully, but then why would you need return by ref tricks, 
which don't rely on scope tracking and hence look hackable?
Dec 04 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 4:56 AM, Kagamin wrote:

 On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP69


 How inout fits the picture? It has some scope semantics too.


s/inout/ref/



 Section about expressions suggests you will tack scoping carefully, but then
why
 would you need return by ref tricks, which don't rely on scope tracking and
 hence look hackable?


Couldn't think of another way.
Dec 04 2014
prev sibling next sibling parent reply "Daniel N" <ufo orbiting.us> writes:
On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds 
 the missing semantics for the 'scope' storage class in order to 
 make it possible to pass a reference to a function without it 
 being possible for it to escape.


Looks pretty solid, but I have one question based on the 
following two statements:

1) "Scope is inferred for function parameters if not specified, 
under the same circumstances as pure, nothrow,  nogc and safety 
are inferred."
2) "Scope is covariant, meaning it can be added to overriding 
functions."

How would this be handled in the current proposal?

class C
{
   int bar(ref T); // <-- inferred to be scope
}

class D : C
{
   override int bar(ref T); // <-- inferred to be NOT scope (but 
cannot remove scope when overriding)
}
Dec 04 2014
parent reply "Tobias Pankrath" <tobias pankrath.net> writes:
 How would this be handled in the current proposal?

 class C
 {
   int bar(ref T); // <-- inferred to be scope
 }

 class D : C
 {
   override int bar(ref T); // <-- inferred to be NOT scope (but 
 cannot remove scope when overriding)
 }


Attribute inference only works for function literals and template 
functions. So no inference is done for both methods. Template 
methods can not be overwritten.
Dec 04 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 01:49:06PM +0000, Tobias Pankrath via Digitalmars-d
wrote:

How would this be handled in the current proposal?

class C
{
  int bar(ref T); // <-- inferred to be scope
}

class D : C
{
  override int bar(ref T); // <-- inferred to be NOT scope (but cannot
remove scope when overriding)
}


 
 Attribute inference only works for function literals and template
 functions.  So no inference is done for both methods. Template methods
 can not be overwritten.


However, AFAIK, template *classes* trigger attribute inference on its
(non-template) member functions, so this would be problematic:

	class Base(T) {
		T data;
		void method(ref T); // inferred to be scope
	}

	class Derived : Base!int {
		override void method(ref T); // oops, cannot override
	}


T

-- 
I think Debian's doing something wrong, `apt-get install pesticide', doesn't
seem to remove the bugs on my system! -- Mike Dresser
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 10:45 AM, H. S. Teoh via Digitalmars-d wrote:

 However, AFAIK, template *classes* trigger attribute inference on its
 (non-template) member functions, so this would be problematic:

 	class Base(T) {
 		T data;
 		void method(ref T); // inferred to be scope
 	}

 	class Derived : Base!int {
 		override void method(ref T); // oops, cannot override
 	}


I agree, it's a good point. Scope inference cannot be done for virtual 
functions. I amended the DIP.
Dec 04 2014
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 5 December 2014 at 07:14, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 12/4/2014 10:45 AM, H. S. Teoh via Digitalmars-d wrote:

 However, AFAIK, template *classes* trigger attribute inference on its
 (non-template) member functions, so this would be problematic:

         class Base(T) {
                 T data;
                 void method(ref T); // inferred to be scope
         }

         class Derived : Base!int {
                 override void method(ref T); // oops, cannot override
         }



 I agree, it's a good point. Scope inference cannot be done for virtual
 functions. I amended the DIP.


No comment...
Dec 05 2014
prev sibling next sibling parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Walter Bright"  wrote in message news:m5p99m$luk$1 digitalmars.com...


 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds the missing 
 semantics for the 'scope' storage class in order to make it possible to 
 pass a reference to a function without it being possible for it to escape.

 This, among other things, makes a ref counting type practical. It also 
 makes it more practical to use other storage allocation schemes than 
 garbage collection.

 It does not make scope into a type constructor, nor a general 
 type-annotation system.

 It does not provide an ownership system, though it would complement one.


This looks really good.  Nice work.
Dec 04 2014
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/4/14 4:24 AM, Walter Bright wrote:

 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds the
 missing semantics for the 'scope' storage class in order to make it
 possible to pass a reference to a function without it being possible for
 it to escape.

 This, among other things, makes a ref counting type practical. It also
 makes it more practical to use other storage allocation schemes than
 garbage collection.

 It does not make scope into a type constructor, nor a general
 type-annotation system.

 It does not provide an ownership system, though it would complement one.


"There can be at most one owner for any piece of data."

This doesn't seem right. For GC data, the GC owns the data, that is 
true. But for Ref-counted data, there is more than one owner, and only 
when all the owners disown the data can it be destroyed.

I think there is a disconnect here, you can't say *nobody* owns the 
data, and if you say one variable owns the data, which one is it?

Continuing to read...

-Steve
Dec 04 2014
next sibling parent reply "Tobias Pankrath" <tobias pankrath.net> writes:
 "There can be at most one owner for any piece of data."

 This doesn't seem right. For GC data, the GC owns the data, 
 that is true. But for Ref-counted data, there is more than one 
 owner, and only when all the owners disown the data can it be 
 destroyed.

 I think there is a disconnect here, you can't say *nobody* owns 
 the data, and if you say one variable owns the data, which one 
 is it?

 Continuing to read...

 -Steve


It is in line with this definition


 A variable owns the data it contains if, when the lifetime of 
 the variable is ended, the data can be destroyed.


and the definiton of the lifetime of a variable


 The lifetime of variables is based purely on their lexical 
 scope and order of declaration. The following rules define a 
 hierarchy of lifetimes:


A variable's lifetime starts at the point of its declaration, and 
ends with the lexical scope it is defined in.
An (rvalue) expression's lifetime is temporary; it lives till the 
end of the statement that it appears in.
The lifetime of A is higher than that of B, if A appears in a 
higher scope than B, or if both appear in the same scope, but A 
comes lexically before B. This matches the order of destruction 
of local variables.
The lifetime of a function parameter is higher than that of that 
function's local variables, but lower than any variables in 
higher scopes.

Because the lifetimes of any two variables are different by this 
definition and the definition of ownership links to variable 
lifetime, only one variable can own the data.

So if you have multiple ref-counted slices to an array, no one 
owns the array until the ref-count goes down to 1. Question 
remains, if this is a definition of ownership we want to employ.






 There can be at most one owner for any piece of data.
Dec 04 2014
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/4/14 10:06 AM, Tobias Pankrath wrote:


 So if you have multiple ref-counted slices to an array, no one owns the
 array until the ref-count goes down to 1. Question remains, if this is a
 definition of ownership we want to employ.


This is like saying you have multiple owners :) I don't see the reason 
to avoid that. It's not like you can't define the owners when there are 
multiple references. Only one of them will end up destroying the 
variable, but it's not some other entity that will do it, you have a 
clear definition of who the owners are.

-Steve
Dec 04 2014
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 6:58 AM, Steven Schveighoffer wrote:

 This doesn't seem right. For GC data, the GC owns the data, that is true. But
 for Ref-counted data, there is more than one owner, and only when all the
owners
 disown the data can it be destroyed.

 I think there is a disconnect here, you can't say *nobody* owns the data, and
if
 you say one variable owns the data, which one is it?


The owner of the refcounted data is the refcounting wrapper - and the wrapper 
decides when to destroy the payload.

(For GC, the "wrapper" is the GC system, which decides to destroy the payload 
when there are no longer references to it. Just like a ref counting system.)
Dec 04 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 4 December 2014 at 14:58:47 UTC, Steven 
Schveighoffer wrote:

 "There can be at most one owner for any piece of data."

 This doesn't seem right. For GC data, the GC owns the data, 
 that is true. But for Ref-counted data, there is more than one 
 owner, and only when all the owners disown the data can it be 
 destroyed.


The RC mechanism is the owner. Ownership is loosly defined in 
this DIp so that it do not close any door for future language 
evolution.
Dec 04 2014
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/4/14 5:48 PM, deadalnix wrote:

 On Thursday, 4 December 2014 at 14:58:47 UTC, Steven Schveighoffer wrote:

 "There can be at most one owner for any piece of data."

 This doesn't seem right. For GC data, the GC owns the data, that is
 true. But for Ref-counted data, there is more than one owner, and only
 when all the owners disown the data can it be destroyed.


 The RC mechanism is the owner. Ownership is loosly defined in this DIp
 so that it do not close any door for future language evolution.


Well, actually the DIP is pretty rigid, it speaks only of ownership in 
terms of variables -- which variable owns a piece of data. It doesn't 
allow this kind of ownership via a concept or condition.

I would change the DIP to reflect this clarification, if that is what is 
intended.

-Steve
Dec 05 2014
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/4/14 4:24 AM, Walter Bright wrote:

 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds the
 missing semantics for the 'scope' storage class in order to make it
 possible to pass a reference to a function without it being possible for
 it to escape.

 This, among other things, makes a ref counting type practical. It also
 makes it more practical to use other storage allocation schemes than
 garbage collection.

 It does not make scope into a type constructor, nor a general
 type-annotation system.

 It does not provide an ownership system, though it would complement one.


"Scope affects:

     local variables allocated on the stack"

...

"scope int i;       // scope is ignored because integers are not 
references and so are not views"

I think I understand what you are trying to say -- the (big S) Scope of 
a local variable cannot escape, but it serves no purpose to declare a 
local int as (keyword) scope, since it's not going to be assigned any 
references.

But it reads contradictory.

-Steve
Dec 04 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 7:04 AM, Steven Schveighoffer wrote:

 "Scope affects:

      local variables allocated on the stack"

 ...

 "scope int i;       // scope is ignored because integers are not references and
 so are not views"

 I think I understand what you are trying to say -- the (big S) Scope of a local
 variable cannot escape, but it serves no purpose to declare a local int as
 (keyword) scope, since it's not going to be assigned any references.

 But it reads contradictory.


If you think that is contradictory, you should have read the earlier drafts :-)

Basically, I had to get straight in my head the difference between the scope of 
the variable and the scope of its payload.

The keyword 'scope' only affects the payload.
Dec 04 2014
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
int* bar(scope int*);
scope int* foo();

bar(foo());           // Ok, lifetime(foo()) > lifetime(bar())

I'm trying to understand how foo can be implemented in any case. It has 
no scope ints to return, so where does it get the int from?

I don't see where the proposal defines what exactly can be returned via 
scope.

Another thing I saw early on:

void abc() {
     scope int* a;
     int* b;
     scope ref int* c = a;  // Error, rule 5
     scope ref int* d = b;  // Ok
     int* i = a;            // Ok, scope is inferred for i
     global_ptr = d;        // Error, lifetime(d) < lifetime(global_ptr)
     global_ptr = i;        // Error, lifetime(i) < lifetime(global_ptr)
     int* j;                // Ok, scope is inferred for i
     global_ptr = j;        // Ok, j is not scope
}

Does this mean ref can now be applied to a variable?

I'm not sure what the difference between scope ref and scope is. is d 
defined as a reference to b, or is d defined as a new variable that is 
initialized to what b points at (a la C++ &) ?

-Steve
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 7:25 AM, Steven Schveighoffer wrote:

 int* bar(scope int*);
 scope int* foo();

 bar(foo());           // Ok, lifetime(foo()) > lifetime(bar())

 I'm trying to understand how foo can be implemented in any case. It has no
scope
 ints to return, so where does it get the int from?


Could be from a global variable. Or a new'd value.



 I don't see where the proposal defines what exactly can be returned via scope.


The scope return value does not affect what can be returned. It affects how
that 
return value can be used. I.e. the return value cannot be used in such a way 
that it escapes the lifetime of the expression.



 Another thing I saw early on:

 void abc() {
      scope int* a;
      int* b;
      scope ref int* c = a;  // Error, rule 5
      scope ref int* d = b;  // Ok
      int* i = a;            // Ok, scope is inferred for i
      global_ptr = d;        // Error, lifetime(d) < lifetime(global_ptr)
      global_ptr = i;        // Error, lifetime(i) < lifetime(global_ptr)
      int* j;                // Ok, scope is inferred for i
      global_ptr = j;        // Ok, j is not scope
 }

 Does this mean ref can now be applied to a variable?


     scope ref int p = x;

is the same meaning as:

     foo(scope ref int p);
     foo(x);

as far as what scope ref means. I found it convenient to use scope ref local 
variables to describe semantics. Whether we want to actually enable their usage 
hinges on if there's a point to it. Technically, it should work.



 I'm not sure what the difference between scope ref and scope is. is d defined
as
 a reference to b, or is d defined as a new variable that is initialized to what
 b points at (a la C++ &) ?


Scope refers to the payload. A scope ref is applying scope to the implied ref 
pointer. ref works like C++ &.
Dec 04 2014
next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 12:58:56PM -0800, Walter Bright via Digitalmars-d wrote:

 On 12/4/2014 7:25 AM, Steven Schveighoffer wrote:


[...]

I don't see where the proposal defines what exactly can be returned
via scope.


 
 The scope return value does not affect what can be returned. It
 affects how that return value can be used. I.e. the return value
 cannot be used in such a way that it escapes the lifetime of the
 expression.


[...]

Ahhh, this certainly clears things up a lot. You should add this
explanation to the DIP.


T

-- 
Famous last words: I *think* this will work...
Dec 04 2014
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/4/14 3:58 PM, Walter Bright wrote:

 On 12/4/2014 7:25 AM, Steven Schveighoffer wrote:

 int* bar(scope int*);
 scope int* foo();

 bar(foo());           // Ok, lifetime(foo()) > lifetime(bar())

 I'm trying to understand how foo can be implemented in any case. It
 has no scope
 ints to return, so where does it get the int from?


 Could be from a global variable. Or a new'd value.


Well, OK, but why do that?


 I don't see where the proposal defines what exactly can be returned
 via scope.


 The scope return value does not affect what can be returned. It affects
 how that return value can be used. I.e. the return value cannot be used
 in such a way that it escapes the lifetime of the expression.


I assumed the scope return was so you could do things like:

scope int *foo(scope int *x)
{
    return x;
}

which would be fine, I assume, right?


 Another thing I saw early on:

 void abc() {
      scope int* a;
      int* b;
      scope ref int* c = a;  // Error, rule 5
      scope ref int* d = b;  // Ok
      int* i = a;            // Ok, scope is inferred for i
      global_ptr = d;        // Error, lifetime(d) < lifetime(global_ptr)
      global_ptr = i;        // Error, lifetime(i) < lifetime(global_ptr)
      int* j;                // Ok, scope is inferred for i
      global_ptr = j;        // Ok, j is not scope
 }

 Does this mean ref can now be applied to a variable?


      scope ref int p = x;

 is the same meaning as:

      foo(scope ref int p);
      foo(x);

 as far as what scope ref means. I found it convenient to use scope ref
 local variables to describe semantics. Whether we want to actually
 enable their usage hinges on if there's a point to it. Technically, it
 should work.


My question was about how this kind of allows declaring a ref variable 
in the middle of a function, which was never allowed before.

-Steve
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 1:32 PM, Steven Schveighoffer wrote:

 On 12/4/14 3:58 PM, Walter Bright wrote:

 On 12/4/2014 7:25 AM, Steven Schveighoffer wrote:

 int* bar(scope int*);
 scope int* foo();

 bar(foo());           // Ok, lifetime(foo()) > lifetime(bar())

 I'm trying to understand how foo can be implemented in any case. It
 has no scope
 ints to return, so where does it get the int from?


 Could be from a global variable. Or a new'd value.


 Well, OK, but why do that?


Why would a programmer do that? I often ask that question! But the language 
allows it, therefore we must support it.



 I don't see where the proposal defines what exactly can be returned
 via scope.


 The scope return value does not affect what can be returned. It affects
 how that return value can be used. I.e. the return value cannot be used
 in such a way that it escapes the lifetime of the expression.


 I assumed the scope return was so you could do things like:

 scope int *foo(scope int *x)
 {
     return x;
 }

 which would be fine, I assume, right?


No. A scope parameter means the value does not escape the function. That means 
you can't return it.



 My question was about how this kind of allows declaring a ref variable in the
 middle of a function, which was never allowed before.


There's no technical reason it is disallowed - it's just that I didn't see a 
point to it.
Dec 05 2014
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/5/14 6:09 PM, Walter Bright wrote:

 On 12/4/2014 1:32 PM, Steven Schveighoffer wrote:

 On 12/4/14 3:58 PM, Walter Bright wrote:

 On 12/4/2014 7:25 AM, Steven Schveighoffer wrote:

 int* bar(scope int*);
 scope int* foo();

 bar(foo());           // Ok, lifetime(foo()) > lifetime(bar())

 I'm trying to understand how foo can be implemented in any case. It
 has no scope
 ints to return, so where does it get the int from?


 Could be from a global variable. Or a new'd value.


 Well, OK, but why do that?


 Why would a programmer do that? I often ask that question! But the
 language allows it, therefore we must support it.


But you're the programmer that did it, it's YOUR example! :)

However, it's not just that, this is the ONLY example given as to why we 
support scope returns. The language allows it, therefore we must support 
it? But it's not allowed now, right? Why add it?


 The scope return value does not affect what can be returned. It affects
 how that return value can be used. I.e. the return value cannot be used
 in such a way that it escapes the lifetime of the expression.


 I assumed the scope return was so you could do things like:

 scope int *foo(scope int *x)
 {
     return x;
 }

 which would be fine, I assume, right?


 No. A scope parameter means the value does not escape the function. That
 means you can't return it.


I think you should eliminate scope returns then. They are not useful. I 
can't think of a single reason why a newly allocated via GC or global 
reference return should have to be restricted to exist only within the 
statement. Both have infinite lifetimes.

-Steve
Dec 08 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 8 December 2014 at 15:12:51 UTC, Steven Schveighoffer 
wrote:

 I think you should eliminate scope returns then. They are not 
 useful. I can't think of a single reason why a newly allocated 
 via GC or global reference return should have to be restricted 
 to exist only within the statement. Both have infinite 
 lifetimes.


It's for references to objects that are owned by the function (or 
object of which the function is a method). These don't have 
infinite lifetimes.
Dec 08 2014
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/8/14 10:45 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:

 On Monday, 8 December 2014 at 15:12:51 UTC, Steven Schveighoffer wrote:

 I think you should eliminate scope returns then. They are not useful.
 I can't think of a single reason why a newly allocated via GC or
 global reference return should have to be restricted to exist only
 within the statement. Both have infinite lifetimes.


 It's for references to objects that are owned by the function (or object
 of which the function is a method). These don't have infinite lifetimes.


Why not? An object is allocated on the heap, and has infinite lifetime.

-Steve
Dec 08 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 8 December 2014 at 16:04:42 UTC, Steven Schveighoffer 
wrote:

 On 12/8/14 10:45 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:

 On Monday, 8 December 2014 at 15:12:51 UTC, Steven 
 Schveighoffer wrote:

 I think you should eliminate scope returns then. They are not 
 useful.
 I can't think of a single reason why a newly allocated via GC 
 or
 global reference return should have to be restricted to exist 
 only
 within the statement. Both have infinite lifetimes.


 It's for references to objects that are owned by the function 
 (or object
 of which the function is a method). These don't have infinite 
 lifetimes.


 Why not? An object is allocated on the heap, and has infinite 
 lifetime.


"object" as in "instance of struct" ;-)

And I was referring to objects owned by something with finite 
lifetime, e.g. a container.

     struct Array(T) {
         scope ref T opIndex(size_t);
     }

     Array!int container;
     ref x = container[42];  // no
     container[42]++;        // yes
     writeln(container[42]); // yes
Dec 08 2014
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/8/14 11:27 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:

 On Monday, 8 December 2014 at 16:04:42 UTC, Steven Schveighoffer wrote:

 On 12/8/14 10:45 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?=
 <schuetzm gmx.net>" wrote:

 On Monday, 8 December 2014 at 15:12:51 UTC, Steven Schveighoffer wrote:

 I think you should eliminate scope returns then. They are not useful.
 I can't think of a single reason why a newly allocated via GC or
 global reference return should have to be restricted to exist only
 within the statement. Both have infinite lifetimes.


 It's for references to objects that are owned by the function (or object
 of which the function is a method). These don't have infinite lifetimes.


 Why not? An object is allocated on the heap, and has infinite lifetime.


 "object" as in "instance of struct" ;-)

 And I was referring to objects owned by something with finite lifetime,
 e.g. a container.

      struct Array(T) {
          scope ref T opIndex(size_t);
      }

      Array!int container;
      ref x = container[42];  // no


Why not? x has the same lifetime as container!

This is an incorrect arbitrary decision:

module foo;

Array!int arr;

int *x;

void bar()
{
    x = &arr[0]; // should be ok.
}

A struct's member function has no idea of its lifetime, except what it's 
told.

Now, it would make more sense for this:

ref T opIndex(size_t);
scope ref T opIndex(size_t) scope;

But it's sounding to me (from Walter's other comments) like the latter 
function can't exist, because you can't return scope from scope? And 
furthermore, there's no way to overload based on scope? I'm really not 
liking this whole proposal, it either needs to be explained better, or 
rewritten.

-Steve
Dec 08 2014
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 01:24:13AM -0800, Walter Bright via Digitalmars-d wrote:

 http://wiki.dlang.org/DIP69
 
 Despite its length, this is a fairly simple proposal. It adds the
 missing semantics for the 'scope' storage class in order to make it
 possible to pass a reference to a function without it being possible
 for it to escape.


Finally! Thanks for the hard work, looking forward for this to be
implemented (in some shape of form -- see comments below).



 This, among other things, makes a ref counting type practical. It also
 makes it more practical to use other storage allocation schemes than
 garbage collection.
 
 It does not make scope into a type constructor, nor a general
 type-annotation system.


[...]

Can we pretty please use the term "type qualifier" instead of "type
constructor"? ;-)

Anyway, here are a few comments:

1) Why are scope violations only reported for  safe code?? IMO this
greatly limits the usefulness of this DIP.  If I mark something as
scope, I'd expect it should be enforced by the compiler regardless of
 safe annotations, otherwise what's the point??

Currently, due to the incompleteness of  safe, it's difficult to use
 safe annotations everywhere I'd like to (e.g. I need to use some
un- safe Phobos functions that really ought to be  safe, but aren't due
to various reasons, like compiler limitations, etc.). This greatly
limits the usefulness of this DIP, if scope is only enforced in  safe
code!


2) Is there a way to detect if something is marked as scope? If not, how
does this proposal actually enable ref-counted types? (I'm assuming a
library ref-counting type here; or are we expecting further compiler
enhancements for ref counting?)


3) What does scope mean for delegate parameters? To what does the scope
apply, the delegate itself, its body, or its return value, or ...?

	struct S {
		void opApply(scope void delegate(ref int) loopBody) {
			...
			// what restrictions (might) apply here w.r.t.
			// how loopBody can be called?
		}
	}

And what would be the effect on the caller's side?

	S s;
	foreach (i; s) {
		// what restrictions (might) apply here?
	}


4) Under "Expressions", how does scope interact with overloaded
operators? Do the same rules apply to expressions that use overloaded
operators as they apply to built-in operators, or do they apply as
though the overloaded operators were written out in function-call
syntax?  What happens if some overloaded operators take a mix of scope
and non-scope parameters?


Finally, the following isn't directly related to this DIP, since scope
is intended to solve this problem, but I thought I should bring it up.
:-) In the section "escaping via return", 5 points are listed as
sufficient for detecting the "return func(t);" case. The following
section "scope ref" states that these 5 points are "correct" (in
implementing escaping reference detection). However, isn't the following
a loophole?

	struct S {
		int x;
	}
	ref int func(ref S s) {
		return s.x;
	}
	ref T foo() {
		S s;
		return func(s); // escaping reference
	}

Since S cannot implicitly convert to int, it would appear that this
circumvents escaping reference detection. (In fact, dmd happily accepts
this code, even if everything is annotated with  safe.) Note that the
escaping reference to x can be arbitrarily deeply nested inside S, so
it's non-trivial to decide whether there's no possibility for references
to (parts of) S to leak from func().


T

-- 
If I were two-faced, would I be wearing this one? -- Abraham Lincoln
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 11:41 AM, H. S. Teoh via Digitalmars-d wrote:

 Can we pretty please use the term "type qualifier" instead of "type
 constructor"? ;-)


I think "type constructor" is the more pedantically correct term, but you'll 
have to ask Andrei :-)



 1) Why are scope violations only reported for  safe code?? IMO this
 greatly limits the usefulness of this DIP.  If I mark something as
 scope, I'd expect it should be enforced by the compiler regardless of
  safe annotations, otherwise what's the point??


Because there are inevitably cases where you'll need to wrap unsafe code with a 
safe interface. By screwing this down too tightly for  system code, you'll
force 
people to use really ugly workarounds.



 Currently, due to the incompleteness of  safe, it's difficult to use
  safe annotations everywhere I'd like to (e.g. I need to use some
 un- safe Phobos functions that really ought to be  safe, but aren't due
 to various reasons, like compiler limitations, etc.). This greatly
 limits the usefulness of this DIP, if scope is only enforced in  safe
 code!


Are there bug reports on this?



 2) Is there a way to detect if something is marked as scope?


Using __traits(compiles,...) ought to work.


 If not, how
 does this proposal actually enable ref-counted types? (I'm assuming a
 library ref-counting type here; or are we expecting further compiler
 enhancements for ref counting?)


The ref counting would be done by a wrapper, which implicitly converts to the 
wrappee by exposing a scope ref.



 3) What does scope mean for delegate parameters? To what does the scope
 apply, the delegate itself, its body, or its return value, or ...?

 	struct S {
 		void opApply(scope void delegate(ref int) loopBody) {
 			...
 			// what restrictions (might) apply here w.r.t.
 			// how loopBody can be called?
 		}
 	}


Hmmm, looks like a problem I didn't think of. darnit!



 And what would be the effect on the caller's side?

 	S s;
 	foreach (i; s) {
 		// what restrictions (might) apply here?
 	}


i would get its scope inferred as necessary, so restrictions would apply as
they 
would to a 'scope i'.



 4) Under "Expressions", how does scope interact with overloaded
 operators? Do the same rules apply to expressions that use overloaded
 operators as they apply to built-in operators, or do they apply as
 though the overloaded operators were written out in function-call
 syntax?  What happens if some overloaded operators take a mix of scope
 and non-scope parameters?


Overloaded operators are treated like calls to the functions that back them.



 Finally, the following isn't directly related to this DIP, since scope
 is intended to solve this problem, but I thought I should bring it up.
 :-) In the section "escaping via return", 5 points are listed as
 sufficient for detecting the "return func(t);" case. The following
 section "scope ref" states that these 5 points are "correct" (in
 implementing escaping reference detection). However, isn't the following
 a loophole?

 	struct S {
 		int x;
 	}
 	ref int func(ref S s) {
 		return s.x;
 	}
 	ref T foo() {
 		S s;
 		return func(s); // escaping reference
 	}

 Since S cannot implicitly convert to int, it would appear that this
 circumvents escaping reference detection.


No, because under this proposal, s.x is treated as if it were just s as far a 
scope rules are concerned.


 (In fact, dmd happily accepts
 this code, even if everything is annotated with  safe.)


I know, the current escape detection is quite inadequate - hence this proposal.


 Note that the
 escaping reference to x can be arbitrarily deeply nested inside S, so
 it's non-trivial to decide whether there's no possibility for references
 to (parts of) S to leak from func().


This proposal should lock that down.
Dec 04 2014
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 01:10:31PM -0800, Walter Bright via Digitalmars-d wrote:

 On 12/4/2014 11:41 AM, H. S. Teoh via Digitalmars-d wrote:

Can we pretty please use the term "type qualifier" instead of "type
constructor"? ;-)


 
 I think "type constructor" is the more pedantically correct term, but
 you'll have to ask Andrei :-)


Recently there was a bug filed by Andrei himself, and a bunch of merged
PRs, that renamed "type constructor" to "type qualifier". The general
sense I got was that we're deprecating "type constructor" in preference
for "type qualifier".



1) Why are scope violations only reported for  safe code?? IMO this
greatly limits the usefulness of this DIP.  If I mark something as
scope, I'd expect it should be enforced by the compiler regardless of
 safe annotations, otherwise what's the point??


 
 Because there are inevitably cases where you'll need to wrap unsafe
 code with a safe interface. By screwing this down too tightly for
  system code, you'll force people to use really ugly workarounds.


Are there any use cases for bypassing scope semantics in  system code?
How common are they? Maybe such code *should* look ugly -- to draw
attention to the fact that something un- safe is going on.

Or perhaps we can make cast() strip away scope? I much rather require
explicit annotation for potentially unsafe operations, than to have
subtle bugs creep into the code just because I forgot to mark something
as  safe.



Currently, due to the incompleteness of  safe, it's difficult to use
 safe annotations everywhere I'd like to (e.g. I need to use some
un- safe Phobos functions that really ought to be  safe, but aren't
due to various reasons, like compiler limitations, etc.). This
greatly limits the usefulness of this DIP, if scope is only enforced
in  safe code!


 
 Are there bug reports on this?


There are, though probably incomplete, and there's also an ongoing
stream of Phobos PR's for marking things  safe that ought not to be
 system.  There's a *lot* of Phobos code that needs to be cleaned up in
this way before this can be workable. (Unless we abuse  trusted as a
temporary workaround -- but that's not an approach I'd like to
recommend!)



2) Is there a way to detect if something is marked as scope?


 
 Using __traits(compiles,...) ought to work.


I see.



If not, how does this proposal actually enable ref-counted types?
(I'm assuming a library ref-counting type here; or are we expecting
further compiler enhancements for ref counting?)


 
 The ref counting would be done by a wrapper, which implicitly converts
 to the wrappee by exposing a scope ref.


Just saw your other post on this topic, makes sense.

Still not sure how it would address the RC folks' request for compiler
support, but I suppose that's a different topic.



3) What does scope mean for delegate parameters? To what does the
scope apply, the delegate itself, its body, or its return value, or
...?

	struct S {
		void opApply(scope void delegate(ref int) loopBody) {
			...
			// what restrictions (might) apply here w.r.t.
			// how loopBody can be called?
		}
	}


 
 Hmmm, looks like a problem I didn't think of. darnit!


I think it's a fairly important issue, since making this work correctly
will also open up optimization opportunities for a good chunk of
delegate-based code.



And what would be the effect on the caller's side?

	S s;
	foreach (i; s) {
		// what restrictions (might) apply here?
	}


 
 i would get its scope inferred as necessary, so restrictions would
 apply as they would to a 'scope i'.


I was thinking more of what implications may hold in the loop body for
references to (local) variables outside the loop, since that presents
another potential optimization opportunity if we do it right.



4) Under "Expressions", how does scope interact with overloaded
operators? Do the same rules apply to expressions that use overloaded
operators as they apply to built-in operators, or do they apply as
though the overloaded operators were written out in function-call
syntax?  What happens if some overloaded operators take a mix of
scope and non-scope parameters?


 
 Overloaded operators are treated like calls to the functions that back
 them.


OK. You might want to state this explicitly in the DIP, just so it isn't
left up to interpretation. :-)



Finally, the following isn't directly related to this DIP, since
scope is intended to solve this problem, but I thought I should bring
it up.  :-) In the section "escaping via return", 5 points are listed
as sufficient for detecting the "return func(t);" case. The following
section "scope ref" states that these 5 points are "correct" (in
implementing escaping reference detection). However, isn't the
following a loophole?

	struct S {
		int x;
	}
	ref int func(ref S s) {
		return s.x;
	}
	ref T foo() {
		S s;
		return func(s); // escaping reference
	}

Since S cannot implicitly convert to int, it would appear that this
circumvents escaping reference detection.


 
 No, because under this proposal, s.x is treated as if it were just s
 as far a scope rules are concerned.


[...]

I'm not quite sure I see how this could be implemented. When the
compiler is compiling func(), say it's in a different module, there's
nothing about the function that says it's illegal to return a reference
to the member of an incoming argument. First of all, this isn't a
template function so scope inference isn't in effect; secondly, even of
it were in effect (say we write it as `func()(ref S s)` instead), it
wouldn't be able to infer the return value as scope, since it's
obviously escaping a reference to a function argument. So, unless I'm
missing something obvious, func() will be accepted by the compiler (as
it should, since we didn't mark anything as scope here).

Now when the compiler compiles foo(), it can no longer see what's inside
func(), so there's no way for it to know that the ref int being returned
actually comes from the ref S parameter. Furthermore, func()'s signature
is not annotated with scope, so how would the compiler know where to
apply scoping rules?

Unless I'm missing something, this still looks like a loophole.


T

-- 
If you compete with slaves, you become a slave. -- Norbert Wiener
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 1:52 PM, H. S. Teoh via Digitalmars-d wrote:

 On Thu, Dec 04, 2014 at 01:10:31PM -0800, Walter Bright via Digitalmars-d
wrote:

 On 12/4/2014 11:41 AM, H. S. Teoh via Digitalmars-d wrote:

 Can we pretty please use the term "type qualifier" instead of "type
 constructor"? ;-)


 I think "type constructor" is the more pedantically correct term, but
 you'll have to ask Andrei :-)


 Recently there was a bug filed by Andrei himself, and a bunch of merged
 PRs, that renamed "type constructor" to "type qualifier". The general
 sense I got was that we're deprecating "type constructor" in preference
 for "type qualifier".


Looks like Andrei Hath Spaketh!



 Because there are inevitably cases where you'll need to wrap unsafe
 code with a safe interface. By screwing this down too tightly for
  system code, you'll force people to use really ugly workarounds.


 Are there any use cases for bypassing scope semantics in  system code?


One is to interface with legacy code that does not use 'scope' properly in its 
declarations. There are probably more.


 How common are they?


I don't know.


 Maybe such code *should* look ugly -- to draw
 attention to the fact that something un- safe is going on.


 system is ugly enough.


 Or perhaps we can make cast() strip away scope? I much rather require
 explicit annotation for potentially unsafe operations, than to have
 subtle bugs creep into the code just because I forgot to mark something
 as  safe.


Start your modules with:

    safe:



 There are, though probably incomplete, and there's also an ongoing
 stream of Phobos PR's for marking things  safe that ought not to be
  system.  There's a *lot* of Phobos code that needs to be cleaned up in
 this way before this can be workable. (Unless we abuse  trusted as a
 temporary workaround -- but that's not an approach I'd like to
 recommend!)


As long as there's ongoing progress!



 Still not sure how it would address the RC folks' request for compiler
 support, but I suppose that's a different topic.


Yup, different issue.



 I think it's a fairly important issue, since making this work correctly
 will also open up optimization opportunities for a good chunk of
 delegate-based code.


See my other reply with resolution.



 I was thinking more of what implications may hold in the loop body for
 references to (local) variables outside the loop, since that presents
 another potential optimization opportunity if we do it right.


Yes. I think there are a number of optimization properties we can exploit over
time.



 Overloaded operators are treated like calls to the functions that back
 them.


 OK. You might want to state this explicitly in the DIP, just so it isn't
 left up to interpretation. :-)


Ok.


 Finally, the following isn't directly related to this DIP, since
 scope is intended to solve this problem, but I thought I should bring
 it up.  :-) In the section "escaping via return", 5 points are listed
 as sufficient for detecting the "return func(t);" case. The following
 section "scope ref" states that these 5 points are "correct" (in
 implementing escaping reference detection). However, isn't the
 following a loophole?

 	struct S {
 		int x;
 	}
 	ref int func(ref S s) {
 		return s.x;
 	}
 	ref T foo() {
 		S s;
 		return func(s); // escaping reference
 	}

 Since S cannot implicitly convert to int, it would appear that this
 circumvents escaping reference detection.


 No, because under this proposal, s.x is treated as if it were just s
 as far a scope rules are concerned.


 [...]

 I'm not quite sure I see how this could be implemented. When the
 compiler is compiling func(), say it's in a different module, there's
 nothing about the function that says it's illegal to return a reference
 to the member of an incoming argument.


That's right, so the compiler assumes it does, unless the parameter is marked
as 
scope.


 First of all, this isn't a
 template function so scope inference isn't in effect; secondly, even of
 it were in effect (say we write it as `func()(ref S s)` instead), it
 wouldn't be able to infer the return value as scope, since it's
 obviously escaping a reference to a function argument.


'scope ref' parameters cannot be returned by 'ref' or by 'scope ref'. The only 
distinction between 'ref' and 'scope ref' parameters is the latter cannot be 
returned by 'ref' or 'scope ref'.


 So, unless I'm
 missing something obvious, func() will be accepted by the compiler (as
 it should, since we didn't mark anything as scope here).


The compiler assumes func(s) returns a ref to s, and so it won't allow func(s) 
to be returned from foo().



 Now when the compiler compiles foo(), it can no longer see what's inside
 func(), so there's no way for it to know that the ref int being returned
 actually comes from the ref S parameter.


It assumes it does, unless 'ref S' is marked 'scope ref S'. It's the whole
point 
of 'scope ref'.
Dec 04 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 03:26:32PM -0800, Walter Bright via Digitalmars-d wrote:

 On 12/4/2014 1:52 PM, H. S. Teoh via Digitalmars-d wrote:


[...]

Are there any use cases for bypassing scope semantics in  system code?


 
 One is to interface with legacy code that does not use 'scope'
 properly in its declarations. There are probably more.
 

How common are they?


 
 I don't know.
 

Maybe such code *should* look ugly -- to draw attention to the fact
that something un- safe is going on.


 
  system is ugly enough.


Not when all code is  system by default!



Or perhaps we can make cast() strip away scope? I much rather require
explicit annotation for potentially unsafe operations, than to have
subtle bugs creep into the code just because I forgot to mark
something as  safe.


 
 Start your modules with:
 
    safe:


I would, except that Phobos limitations preclude that. Someone else has
already posted the list of bugs, so I won't repeat that here. Needless
to say, it's still a ways off before putting  safe: at the top of your
modules will solve the problem.


[...]

	struct S {
		int x;
	}
	ref int func(ref S s) {
		return s.x;
	}
	ref T foo() {
		S s;
		return func(s); // escaping reference
	}

Since S cannot implicitly convert to int, it would appear that this
circumvents escaping reference detection.


No, because under this proposal, s.x is treated as if it were just s
as far a scope rules are concerned.


[...]

I'm not quite sure I see how this could be implemented. When the
compiler is compiling func(), say it's in a different module, there's
nothing about the function that says it's illegal to return a reference
to the member of an incoming argument.


 
 That's right, so the compiler assumes it does, unless the parameter is
 marked as scope.
 

First of all, this isn't a template function so scope inference isn't
in effect; secondly, even of it were in effect (say we write it as
`func()(ref S s)` instead), it wouldn't be able to infer the return
value as scope, since it's obviously escaping a reference to a
function argument.


 
 'scope ref' parameters cannot be returned by 'ref' or by 'scope ref'.
 The only distinction between 'ref' and 'scope ref' parameters is the
 latter cannot be returned by 'ref' or 'scope ref'.
 

So, unless I'm missing something obvious, func() will be accepted by
the compiler (as it should, since we didn't mark anything as scope
here).


 
 The compiler assumes func(s) returns a ref to s, and so it won't allow
 func(s) to be returned from foo().


Ah, I see. So you're saying that under this DIP, `return func(...);`
from a ref function will be prohibited unless the arguments to func()
were all either scope or non-reference?

That sounds good, though it *is* pretty invasive, as any existing ref
functions that do this will need to be revised, which potentially
spreads 'scope' to many other places, depending on how deeply the chain
of tail recursion / tail func calls go.



Now when the compiler compiles foo(), it can no longer see what's
inside func(), so there's no way for it to know that the ref int
being returned actually comes from the ref S parameter.


 
 It assumes it does, unless 'ref S' is marked 'scope ref S'. It's the
 whole point of 'scope ref'.


Got it, thanks.


T

-- 
Shin: (n.) A device for finding furniture in the dark.
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 3:54 PM, H. S. Teoh via Digitalmars-d wrote:

 The compiler assumes func(s) returns a ref to s, and so it won't allow
 func(s) to be returned from foo().


 Ah, I see. So you're saying that under this DIP, `return func(...);`
 from a ref function will be prohibited unless the arguments to func()
 were all either scope or non-reference?


Right.


 That sounds good, though it *is* pretty invasive, as any existing ref
 functions that do this will need to be revised, which potentially
 spreads 'scope' to many other places, depending on how deeply the chain
 of tail recursion / tail func calls go.


I know:

1. that's why the extensive support for scope inference
2. I couldn't see any other way
Dec 04 2014
parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 04:33:22PM -0800, Walter Bright via Digitalmars-d wrote:

 On 12/4/2014 3:54 PM, H. S. Teoh via Digitalmars-d wrote:

The compiler assumes func(s) returns a ref to s, and so it won't allow
func(s) to be returned from foo().


Ah, I see. So you're saying that under this DIP, `return func(...);`
from a ref function will be prohibited unless the arguments to func()
were all either scope or non-reference?


 
 Right.
 

That sounds good, though it *is* pretty invasive, as any existing ref
functions that do this will need to be revised, which potentially
spreads 'scope' to many other places, depending on how deeply the
chain of tail recursion / tail func calls go.


 
 I know:
 
 1. that's why the extensive support for scope inference
 2. I couldn't see any other way


I'm not saying it's necessary a bad thing... just that it will be an
invasive change that we should be prepared to deal with.


T

-- 
English is useful because it is a mess. Since English is a mess, it maps
well onto the problem space, which is also a mess, which we call
reality. Similarly, Perl was designed to be a mess, though in the
nicest of all possible ways. -- Larry Wall
Dec 04 2014
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 01:52:09PM -0800, H. S. Teoh via Digitalmars-d wrote:

 On Thu, Dec 04, 2014 at 01:10:31PM -0800, Walter Bright via Digitalmars-d
wrote:

 On 12/4/2014 11:41 AM, H. S. Teoh via Digitalmars-d wrote:

Can we pretty please use the term "type qualifier" instead of "type
constructor"? ;-)


 
 I think "type constructor" is the more pedantically correct term,
 but you'll have to ask Andrei :-)


 
 Recently there was a bug filed by Andrei himself, and a bunch of
 merged PRs, that renamed "type constructor" to "type qualifier". The
 general sense I got was that we're deprecating "type constructor" in
 preference for "type qualifier".


[...]

FYI: https://issues.dlang.org/show_bug.cgi?id=13671


T

-- 
"A man's wife has more power over him than the state has." -- Ralph Emerson
Dec 04 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 1:56 PM, H. S. Teoh via Digitalmars-d wrote:

 FYI: https://issues.dlang.org/show_bug.cgi?id=13671


Ok, ok, I fixed the DIP :-)
Dec 04 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 1:10 PM, Walter Bright wrote:

 On 12/4/2014 11:41 AM, H. S. Teoh via Digitalmars-d wrote:

 3) What does scope mean for delegate parameters? To what does the scope
 apply, the delegate itself, its body, or its return value, or ...?

     struct S {
         void opApply(scope void delegate(ref int) loopBody) {
             ...
             // what restrictions (might) apply here w.r.t.
             // how loopBody can be called?
         }
     }


 Hmmm, looks like a problem I didn't think of. darnit!


Turns out, 'ref' has exactly the same issue. The resolution is the same:

   alias ref int delegate() dg_t; // ref applies to return type
   void foo(dg_t dg) {
     static int i;
     ref int bar() { return i; } // ref applies to return type
     dg = &bar;
     dg() = 3;
   }

versus:

   void foo(ref int deletate() dg) { // ref applies to declaration dg
     int bar() { return 3; }
     dg = &bar;
   }

Replace 'ref' with 'scope'.
Dec 04 2014
parent reply "Basile Burg" <basile.burg. gmx.com> writes:
 Hmmm, looks like a problem I didn't think of. darnit!


 Turns out, 'ref' has exactly the same issue. The resolution is 
 the same:

   alias ref int delegate() dg_t; // ref applies to return type


Was the goal of the post to represent the new alias syntax ?
If so, big fail, Walter,...
Dec 04 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 3:58 PM, Basile Burg wrote:

 Was the goal of the post to represent the new alias syntax ?


??
Dec 04 2014
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds 
 the missing semantics for the 'scope' storage class in order to 
 make it possible to pass a reference to a function without it 
 being possible for it to escape.

 This, among other things, makes a ref counting type practical. 
 It also makes it more practical to use other storage allocation 
 schemes than garbage collection.

 It does not make scope into a type constructor, nor a general 
 type-annotation system.

 It does not provide an ownership system, though it would 
 complement one.


So as mentioned, there are various problem with this DIP :
  - rvalue are defined as having a scope that goes to the end of 
the statement. That mean they can never be assigned to anything 
as per spec.
  - It add more special casing with & (as if it wasn't enough of a 
mess with  property, optional () and the fact the function aren't 
first class). For instance *e has infinite lifetime when &(*e) is 
lifetime(e).

Also, considering *e has infinite lifetime, (but not e[i] ???) 
what is the point of scope at all ? If all indirection goes to 
infinite lifetime (ie GC) then lifetime only apply to local 
variable and it should not be able to escape these, scope or not.

During discussion, I proposed to differentiate lifetime 
calculation between lvalues and rvalues (which are inherently 
different beasts with different lifetime) and carry (or not) the 
scope flag with each expression.
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 3:04 PM, deadalnix wrote:

 So as mentioned, there are various problem with this DIP :
   - rvalue are defined as having a scope that goes to the end of the statement.
 That mean they can never be assigned to anything as per spec.


I don't believe this is correct. Rvalues can be assigned, just like:

    __gshared int x;
    { int i; x = i; }

i's scope ends at the } but it can still be assigned to x.



   - It add more special casing with & (as if it wasn't enough of a mess with
  property, optional () and the fact the function aren't first class). For
 instance *e has infinite lifetime when &(*e) is lifetime(e).


That's right. I know you're worried about that, but I still don't see it as an 
actual problem. (The optimizer makes use of this special case all the time.)



 Also, considering *e has infinite lifetime, (but not e[i] ???)


e[i] should be same as *e, unless e is a literal or tuple.


 what is the point of scope at all ? If all indirection goes to infinite
lifetime (ie GC) then
 lifetime only apply to local variable and it should not be able to escape
these,
 scope or not.


I originally had scope only apply to ref, but that made having scoped classes 
impossible.



 During discussion, I proposed to differentiate lifetime calculation between
 lvalues and rvalues (which are inherently different beasts with different
 lifetime) and carry (or not) the scope flag with each expression.


I'm not sure how that would be different from the DIP as it stands now.
Dec 04 2014
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Dec 04, 2014 at 04:31:18PM -0800, Walter Bright via Digitalmars-d wrote:

 On 12/4/2014 3:04 PM, deadalnix wrote:

So as mentioned, there are various problem with this DIP :
  - rvalue are defined as having a scope that goes to the end of the statement.
That mean they can never be assigned to anything as per spec.


 
 I don't believe this is correct. Rvalues can be assigned, just like:
 
    __gshared int x;
    { int i; x = i; }
 
 i's scope ends at the } but it can still be assigned to x.


[...]

Well, the "Expressions" section defines certain lifetimes to be
infinity, however, it contradicts the definition of an rvalue's lifetime
given under "Definitions" - "Lifetime", which states that an rvalue's
lifetime lasts only to the end of the statement. This would imply that
the following ought to be rejected:

	int x = 1 + 2; // lifetime(1 + 2) = end of statement
			// but lifetime(x) = infinity

Sounds like the Definitions section is not precise enough. :-)


T

-- 
Curiosity kills the cat. Moral: don't be the cat.
Dec 04 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 4:48 PM, H. S. Teoh via Digitalmars-d wrote:

  Well, the "Expressions" section defines certain lifetimes to be
 infinity, however, it contradicts the definition of an rvalue's lifetime
 given under "Definitions" - "Lifetime", which states that an rvalue's
 lifetime lasts only to the end of the statement. This would imply that
 the following ought to be rejected:

 	int x = 1 + 2; // lifetime(1 + 2) = end of statement
 			// but lifetime(x) = infinity

 Sounds like the Definitions section is not precise enough. :-)


An 'int' is not a view!
Dec 04 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 5 December 2014 at 00:32:32 UTC, Walter Bright wrote:

 On 12/4/2014 3:04 PM, deadalnix wrote:

 So as mentioned, there are various problem with this DIP :
  - rvalue are defined as having a scope that goes to the end 
 of the statement.
 That mean they can never be assigned to anything as per spec.


 I don't believe this is correct. Rvalues can be assigned, just 
 like:

    __gshared int x;
    { int i; x = i; }

 i's scope ends at the } but it can still be assigned to x.


It work even better when i has indirections.


  - It add more special casing with & (as if it wasn't enough 
 of a mess with
  property, optional () and the fact the function aren't first 
 class). For
 instance *e has infinite lifetime when &(*e) is lifetime(e).


 That's right. I know you're worried about that, but I still 
 don't see it as an actual problem. (The optimizer makes use of 
 this special case all the time.)


Yes, this is the job of the optimizer to do this kind of stunt.
Not the semantic analysis.


 Also, considering *e has infinite lifetime, (but not e[i] ???)


 e[i] should be same as *e, unless e is a literal or tuple.


Ho I missed the *. Sorry for that.


 what is the point of scope at all ? If all indirection goes to 
 infinite lifetime (ie GC) then
 lifetime only apply to local variable and it should not be 
 able to escape these,
 scope or not.


 I originally had scope only apply to ref, but that made having 
 scoped classes impossible.


Promoting scoped class on stack is an ownership problem, and out
of scope (!). It make sense to allow it as an optimization.

Problem is, lifetime goes to infinite after indirection, so I'm
not sure what the guarantee is.


 During discussion, I proposed to differentiate lifetime 
 calculation between
 lvalues and rvalues (which are inherently different beasts 
 with different
 lifetime) and carry (or not) the scope flag with each 
 expression.


 I'm not sure how that would be different from the DIP as it 
 stands now.


I cause everything reached through the view to be scope and
obliviate the need for things like &(*e) having special meaning.
Dec 04 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2014 6:56 PM, deadalnix wrote:

 On Friday, 5 December 2014 at 00:32:32 UTC, Walter Bright wrote:

 On 12/4/2014 3:04 PM, deadalnix wrote:

 So as mentioned, there are various problem with this DIP :
  - rvalue are defined as having a scope that goes to the end of the statement.
 That mean they can never be assigned to anything as per spec.


 I don't believe this is correct. Rvalues can be assigned, just like:

    __gshared int x;
    { int i; x = i; }

 i's scope ends at the } but it can still be assigned to x.


 It work even better when i has indirections.


I understand what you're driving at, but only a scoped rvalue would not be
copyable.



  - It add more special casing with & (as if it wasn't enough of a mess with
  property, optional () and the fact the function aren't first class). For
 instance *e has infinite lifetime when &(*e) is lifetime(e).


 That's right. I know you're worried about that, but I still don't see it as an
 actual problem. (The optimizer makes use of this special case all the time.)


 Yes, this is the job of the optimizer to do this kind of stunt.
 Not the semantic analysis.


I don't see any other way, nor do I see the practical problem.



 I originally had scope only apply to ref, but that made having scoped classes
 impossible.


 Promoting scoped class on stack is an ownership problem, and out
 of scope (!). It make sense to allow it as an optimization.

 Problem is, lifetime goes to infinite after indirection, so I'm
 not sure what the guarantee is.


The guarantee is there will be no references to the class instance after the 
scoped class goes out of scope.



 During discussion, I proposed to differentiate lifetime calculation between
 lvalues and rvalues (which are inherently different beasts with different
 lifetime) and carry (or not) the scope flag with each expression.


 I'm not sure how that would be different from the DIP as it stands now.


 I cause everything reached through the view to be scope and
 obliviate the need for things like &(*e) having special meaning.


Are you suggesting transitive scope?
Dec 05 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 5 December 2014 at 21:32:53 UTC, Walter Bright wrote:

 I don't believe this is correct. Rvalues can be assigned, 
 just like:

   __gshared int x;
   { int i; x = i; }

 i's scope ends at the } but it can still be assigned to x.


 It work even better when i has indirections.


 I understand what you're driving at, but only a scoped rvalue 
 would not be copyable.


The DIP say nothing about scoped rvalue having different behavior
than non scoped ones.


 I originally had scope only apply to ref, but that made

 having scoped classes
 impossible.


 Promoting scoped class on stack is an ownership problem, and 
 out
 of scope (!). It make sense to allow it as an optimization.

 Problem is, lifetime goes to infinite after indirection, so I'm
 not sure what the guarantee is.


 The guarantee is there will be no references to the class 
 instance after the scoped class goes out of scope.


Through use of that view. I see it as follow:
   - When unconsumed owner goes out of scope, it can (must ?) be
free automatically.
   - scope uses do not consume.
   - When the compiler see a pair of alloc/free, it can promote on
stack.

That is a much more useful definition as it allow for stack
promotion after inlining:
class FooBuilder {
      Foo build() { return new Foo(); }
}

class Foo {}

void bar() {
      auto f = new FooBuilder().build();
      // Use f and do not consume...
}

This can be reduced in such a way no allocation happen at all.


 I cause everything reached through the view to be scope and
 obliviate the need for things like &(*e) having special 
 meaning.


 Are you suggesting transitive scope?


For rvalues, yes. Not for lvalues.
Dec 05 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/5/2014 3:59 PM, deadalnix wrote:

 On Friday, 5 December 2014 at 21:32:53 UTC, Walter Bright wrote:

 I don't believe this is correct. Rvalues can be assigned, just like:

   __gshared int x;
   { int i; x = i; }

 i's scope ends at the } but it can still be assigned to x.


 It work even better when i has indirections.


 I understand what you're driving at, but only a scoped rvalue would not be
 copyable.


 The DIP say nothing about scoped rvalue having different behavior
 than non scoped ones.


Can you propose some new wording?




 I originally had scope only apply to ref, but that made

 having scoped classes
 impossible.


 Promoting scoped class on stack is an ownership problem, and out
 of scope (!). It make sense to allow it as an optimization.

 Problem is, lifetime goes to infinite after indirection, so I'm
 not sure what the guarantee is.


 The guarantee is there will be no references to the class instance after the
 scoped class goes out of scope.


 Through use of that view. I see it as follow:
    - When unconsumed owner goes out of scope, it can (must ?) be
 free automatically.
    - scope uses do not consume.
    - When the compiler see a pair of alloc/free, it can promote on
 stack.

 That is a much more useful definition as it allow for stack
 promotion after inlining:
 class FooBuilder {
       Foo build() { return new Foo(); }
 }

 class Foo {}

 void bar() {
       auto f = new FooBuilder().build();
       // Use f and do not consume...
 }

 This can be reduced in such a way no allocation happen at all.


Yes, but I think the proposal allows for that.



 I cause everything reached through the view to be scope and
 obliviate the need for things like &(*e) having special meaning.


 Are you suggesting transitive scope?


 For rvalues, yes. Not for lvalues.


I don't think that is workable.
Dec 06 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 6 December 2014 at 21:57:03 UTC, Walter Bright wrote:

 On 12/5/2014 3:59 PM, deadalnix wrote:

 The DIP say nothing about scoped rvalue having different 
 behavior
 than non scoped ones.


 Can you propose some new wording?


I did:


An infinite lifetime is a lifetime greater or equal than any
other lifetime. Expression of infinite lifetime are:
   - literals
   - GC heap allocated objects
   - statics and enums.
   - rvalues of type that do not contain indirections.
   - non scope rvalues.


Dereference share the lifetime of the dereferenced expression (ie
infinite lifetime unless the expression is scope). Address of
expression shared the lifetime of the base expression, and in
addition gain the scope flag.


 Are you suggesting transitive scope?


 For rvalues, yes. Not for lvalues.


 I don't think that is workable.


That is the only way. For rvalue, you got to take the lower
possible lifetime, but for lvalue, you want to consider the
highest possible lifetime.
Dec 08 2014
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/4/14 4:24 AM, Walter Bright wrote:

 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds the
 missing semantics for the 'scope' storage class in order to make it
 possible to pass a reference to a function without it being possible for
 it to escape.

 This, among other things, makes a ref counting type practical. It also
 makes it more practical to use other storage allocation schemes than
 garbage collection.

 It does not make scope into a type constructor, nor a general
 type-annotation system.

 It does not provide an ownership system, though it would complement one.


Can we take a step back here?

I read many people's comments and I understand only about half of them.

Can someone who knows what this new feature is supposed to do give some 
Ali Çehreli-like description on the feature? Basically, let's strip out 
the *proof* in the DIP (the how it works and why we have it), and focus 
on how it is to be used.

I still am having a hard time wrapping my head around the benefits and 
when to use scope, scope ref, why I would use it. I'm worried that we 
are adding all this complication and it will confuse the shit out of 
users, to the point where they won't use it.

-Steve
Dec 05 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/5/2014 7:27 AM, Steven Schveighoffer wrote:

 Can someone who knows what this new feature is supposed to do give some Ali
 Çehreli-like description on the feature? Basically, let's strip out the
*proof*
 in the DIP (the how it works and why we have it), and focus on how it is to be
 used.

 I still am having a hard time wrapping my head around the benefits and when to
 use scope, scope ref, why I would use it. I'm worried that we are adding all
 this complication and it will confuse the shit out of users, to the point where
 they won't use it.


The tl;dr version is when a declaration is tagged with 'scope', the contents of 
that variable will not escape the lifetime of that declaration.

It means that this code will be safe:

    void foo(scope int* p);

    p = malloc(n);
    foo(p);
    free(p);

The rest is all the nuts and bolts of making that work.
Dec 05 2014
next sibling parent reply "Sebastiaan Koppe" <mail skoppe.eu> writes:
On Friday, 5 December 2014 at 20:55:55 UTC, Walter Bright wrote:

 On 12/5/2014 7:27 AM, Steven Schveighoffer wrote:

 Can someone who knows what this new feature is supposed to do 
 give some Ali
 Çehreli-like description on the feature? Basically, let's 
 strip out the *proof*
 in the DIP (the how it works and why we have it), and focus on 
 how it is to be
 used.

 I still am having a hard time wrapping my head around the 
 benefits and when to
 use scope, scope ref, why I would use it. I'm worried that we 
 are adding all
 this complication and it will confuse the shit out of users, 
 to the point where
 they won't use it.


 The tl;dr version is when a declaration is tagged with 'scope', 
 the contents of that variable will not escape the lifetime of 
 that declaration.

 It means that this code will be safe:

    void foo(scope int* p);

    p = malloc(n);
    foo(p);
    free(p);

 The rest is all the nuts and bolts of making that work.


<brainstorm>
What about also adding the inverse of scope? Then scope can be 
inferred. As in:

```
void foo(int* p);
void free(P)(consume P* p);

p = malloc(n);
foo(p);  // here foo() gets scope, since free consumes it.
free(p);
```

So you do not need to write scope everywhere. And it would 
prevent this code:

```
{
   free(p);
   free(p);  // Error: p already consumed
}
```
</brainstorm>
Dec 05 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 6 December 2014 at 04:31:48 UTC, Sebastiaan Koppe 
wrote:

 What about also adding the inverse of scope? Then scope can be 
 inferred. As in:

 ```
 void foo(int* p);
 void free(P)(consume P* p);



Yes, this is much better. When I suggested it, it was rejected 
because D is too concerned about breaking existing code. Which is 
a not-very-good argument since this breaking change is 
concervative (you only have to add "consume" or something similar 
when the compiler complains).

The obvious solution is to do as you suggest and in addition do 
all  safe analysis on a high level IR layer using dataflow 
through and through.

Instead D continues down the rather flimsy path of partially 
addressing these issues in the type system… which will lead to a 
more complicated and less complete solution where  safe basically 
continues to be a leaky cauldron…
Dec 06 2014
next sibling parent reply "Sebastiaan Koppe" <mail skoppe.eu> writes:
On Saturday, 6 December 2014 at 12:38:24 UTC, Ola Fosheim Grøstad 
wrote:

 On Saturday, 6 December 2014 at 04:31:48 UTC, Sebastiaan Koppe 
 wrote:

 What about also adding the inverse of scope? Then scope can be 
 inferred. As in:

 ```
 void foo(int* p);
 void free(P)(consume P* p);



 Yes, this is much better. When I suggested it, it was rejected 
 because D is too concerned about breaking existing code. Which 
 is a not-very-good argument since this breaking change is 
 concervative (you only have to add "consume" or something 
 similar when the compiler complains).


Hmm, I see.

Whenever I suggest to break something at my job - which, 
admittedly is much easier there than in a language - I get faced 
with some serious reluctance. My argument is always the same: It 
is going to make you happy, eventually.

How would it break anything though? Wouldn't functions eligible 
for `consume` already have the programmer ensuring the arguments 
haven't escaped before/after the function call? In case they did 
a bad job - sure it would break - but it would have been a bug.

 The obvious solution is to do as you suggest and in addition do 
 all  safe analysis on a high level IR layer using dataflow 
 through and through.


I am a big proponent of dataflow analyses, but I got the feeling 
people think is it pretty hard. Couldn't find much detailed 
papers on it, so I don't know.
Dec 06 2014
next sibling parent "eles" <eles eles.com> writes:
On Sunday, 7 December 2014 at 06:52:38 UTC, Sebastiaan Koppe 
wrote:

 On Saturday, 6 December 2014 at 12:38:24 UTC, Ola Fosheim 
 Grøstad wrote:

 On Saturday, 6 December 2014 at 04:31:48 UTC, Sebastiaan Koppe 
 wrote:





 I am a big proponent of dataflow analyses, but I got the 
 feeling people think is it pretty hard.


I think D is in a good position to use the information available 
in assert* and in contracts for that.

Other languages have to use dedicated tools for that.

In a latter step, D could even formalize this into, as discussed 
in other threads, VRP sub-language. But, for the time being, the 
leverage offered by assert and in /out is available there to be 
used.


*yes, I know the war between assert and assume
Dec 07 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 7 December 2014 at 06:52:38 UTC, Sebastiaan Koppe 
wrote:

 How would it break anything though? Wouldn't functions eligible 
 for `consume` already have the programmer ensuring the 
 arguments haven't escaped before/after the function call? In 
 case they did a bad job - sure it would break - but it would 
 have been a bug.


I don't think the breakage is a serious problem in this case, so 
I obviously agree with you…

But, the deep-rooted problem is that you actually need different 
properties associated with  pointers to do this properly: unique, 
single-threaded shared, multi-threaded shared, gc, non-owning… 
And this could be resolved with templates and having the compiler 
recognize pointer resolutions that lead to the same lower level 
code (to avoid bloat). But the D authors don't want ownership 
related pointertypes… and I can't see that work.


 I am a big proponent of dataflow analyses, but I got the 
 feeling people think is it pretty hard. Couldn't find much 
 detailed papers on it, so I don't know.


You can do it in a simpler and conservative fashion and accept 
false positives (or negatives) so that you here-and-there have to 
inject annotations like this: "trust me I have proven that this 
property holds here", then let the compiler infer the rest.

With a good IDE this should be no biggie: just inject suggestions 
in the code and let the programmer confirm/refine them.

If you want to guarantee memory-safety you should do it with a 
single uniform mechanism, trying to do it bit-by-bit special 
casing is not a good idea. So given the current approach, I'm 
inclined to turn off  safe, make everything  nogc, write my own 
libraries with unique_ptr/shared_ptr and call it a day… The 
current approach does not address situations where I run into 
memory related bugs in my own code.
Dec 07 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 6 December 2014 at 12:38:24 UTC, Ola Fosheim Grøstad
wrote:

 On Saturday, 6 December 2014 at 04:31:48 UTC, Sebastiaan Koppe 
 wrote:

 What about also adding the inverse of scope? Then scope can be 
 inferred. As in:

 ```
 void foo(int* p);
 void free(P)(consume P* p);



 Yes, this is much better. When I suggested it, it was rejected 
 because D is too concerned about breaking existing code. Which 
 is a not-very-good argument since this breaking change is 
 concervative (you only have to add "consume" or something 
 similar when the compiler complains).

 The obvious solution is to do as you suggest and in addition do 
 all  safe analysis on a high level IR layer using dataflow 
 through and through.

 Instead D continues down the rather flimsy path of partially 
 addressing these issues in the type system… which will lead to 
 a more complicated and less complete solution where  safe 
 basically continues to be a leaky cauldron…


This is inherently about ownership. I have a proposal about this.
Scope is about using things without ownership.

Both are linked but different beast.
Dec 08 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 8 December 2014 at 23:00:05 UTC, deadalnix wrote:

 This is inherently about ownership. I have a proposal about 
 this.
 Scope is about using things without ownership.

 Both are linked but different beast.


Not really different. The activation record (stack frame) is 
conceptually an object. Scoped objects are owned by the 
activation record. You have the same problem when handing out 
references to parts of composite objects.

When propagating references down a call chain you can keep track 
of the associated call-depth, when the call-depth associated with 
the reference is greater than 0, then it safe to return the 
reference from a function. Right?
Dec 09 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 9 December 2014 at 19:39:31 UTC, Ola Fosheim Grøstad
wrote:

 On Monday, 8 December 2014 at 23:00:05 UTC, deadalnix wrote:

 This is inherently about ownership. I have a proposal about 
 this.
 Scope is about using things without ownership.

 Both are linked but different beast.


 Not really different. The activation record (stack frame) is 
 conceptually an object. Scoped objects are owned by the 
 activation record. You have the same problem when handing out 
 references to parts of composite objects.

 When propagating references down a call chain you can keep 
 track of the associated call-depth, when the call-depth 
 associated with the reference is greater than 0, then it safe 
 to return the reference from a function. Right?


That why i say they are linked. I don't think your way of stating
it contradict what I said.

scope allow for manipulation of data without owning them.
Whatever the owner is (be it the stack frame or anything else)
doesn't really matter here.
Dec 09 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 9 December 2014 at 21:58:48 UTC, deadalnix wrote:

 That why i say they are linked. I don't think your way of 
 stating
 it contradict what I said.

 scope allow for manipulation of data without owning them.
 Whatever the owner is (be it the stack frame or anything else)
 doesn't really matter here.


It does when you return parts of it, like if you pass in a binary 
tree and return a node. Which you have to be able to do for the 
concept to make sense.

When you solve issues in language design you should address the 
hard issues first, because the easy issues will then tend to 
resolve themselves if the design is good. If you start with "the 
low hanging fruit" you end up with pointless special casing… D is 
already in that landscape and should try hard not to sink deeper 
into the muddy waters.

I posit that if you have a solution for ownership/retaining 
references then the solution for scope will come from this as a 
side-effect.
Dec 10 2014
next sibling parent reply "eles" <eles eles.com> writes:
On Wednesday, 10 December 2014 at 15:15:59 UTC, Ola Fosheim 
Grøstad wrote:

 On Tuesday, 9 December 2014 at 21:58:48 UTC, deadalnix wrote:



 D is already in that landscape and should try hard not to sink 
 deeper into the muddy waters.


Funny thing it is that it started exactly as a reaction to those 
muddy waters.

I believe this is the "Stroustrup curse":

"
Much of the relative simplicity of Java is - like for most new 
languages - partly an illusion and partly a function of its 
incompleteness. As time passes, Java will grow significantly in 
size and complexity. It will double or triple in size and grow 
implementation-dependent extensions or libraries. That is the way 
every commercially successful language has developed. Just look 
at any language you consider successful on a large scale. I know 
of no exceptions, and there are good reasons for this phenomenon. 
[I wrote this before 2000; now (2012), the language part of the 
Java 7 specification is slightly longer in terms of number of 
pages than the ISO C++11 language specification.]
"
Dec 10 2014
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 10 December 2014 at 16:17:16 UTC, eles wrote:

 I believe this is the "Stroustrup curse":

 "
 Much of the relative simplicity of Java is - like for most new 
 languages - partly an illusion and partly a function of its 
 incompleteness. As time passes, Java will grow significantly in 
 size and complexity. It will double or triple in size and grow 
 implementation-dependent extensions or libraries. That is the 
 way every commercially successful language has developed. Just 
 look at any language you consider successful on a large scale.


I think he is making excuses for himself. Both Java and C++ are 
rather close to Simula+C as a starting point and have added cruft 
that should have been predicted in the initial version.

Anyway, I think language specs often are too verbose, they 
probably all hit around 700-1000 pages eventually due to editors 
cutting it back to that book-like size before it is published 
(and when the book-sized limit has been reached they feel they 
have done enough ungrateful and unpaid work so they don't cut 
more even though they could have).
Dec 10 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 10 December 2014 at 15:15:59 UTC, Ola Fosheim
Grøstad wrote:

 On Tuesday, 9 December 2014 at 21:58:48 UTC, deadalnix wrote:

 That why i say they are linked. I don't think your way of 
 stating
 it contradict what I said.

 scope allow for manipulation of data without owning them.
 Whatever the owner is (be it the stack frame or anything else)
 doesn't really matter here.


 It does when you return parts of it, like if you pass in a 
 binary tree and return a node. Which you have to be able to do 
 for the concept to make sense.

 When you solve issues in language design you should address the 
 hard issues first, because the easy issues will then tend to 
 resolve themselves if the design is good. If you start with 
 "the low hanging fruit" you end up with pointless special 
 casing… D is already in that landscape and should try hard not 
 to sink deeper into the muddy waters.

 I posit that if you have a solution for ownership/retaining 
 references then the solution for scope will come from this as a 
 side-effect.


That is completely off topic. this is a function parameter, you
can return scope, and you don't need to know who own the
container for that to work.

Granted, the current proposal lack the horsepower to do so.
Dec 10 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 10 December 2014 at 22:05:10 UTC, deadalnix wrote:

 That is completely off topic. this is a function parameter, you
 can return scope, and you don't need to know who own the
 container for that to work.


You have many scopes, if two different scopes pass in "horses" to 
a function in a recursive chain then you need to know which horse 
the returned "leg" belongs to…

But by all means… go ahead if you think it is off topic. (it 
isn't)
Dec 10 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 10 December 2014 at 22:20:37 UTC, Ola Fosheim 
Grøstad wrote:

 On Wednesday, 10 December 2014 at 22:05:10 UTC, deadalnix wrote:

 That is completely off topic. this is a function parameter, you
 can return scope, and you don't need to know who own the
 container for that to work.


 You have many scopes, if two different scopes pass in "horses" 
 to a function in a recursive chain then you need to know which 
 horse the returned "leg" belongs to…

 But by all means… go ahead if you think it is off topic. (it 
 isn't)


It is always safe to consider scopeness of the retrun value (if 
marked scope) as being the intersection of the lifetime of 
parameters.

That should cover most bases, and we can still extends later if 
this is too limited (I suspect it is ok for most cases).
Dec 10 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
deadalnix:


 That should cover most bases, and we can still extends later if 
 this is too limited (I suspect it is ok for most cases).


What syntax do you suggest to use if you want to extend it that 
way later?

Bye,
bearophile
Dec 10 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 11 December 2014 at 00:48:32 UTC, bearophile wrote:

 deadalnix:


 That should cover most bases, and we can still extends later 
 if this is too limited (I suspect it is ok for most cases).


 What syntax do you suggest to use if you want to extend it that 
 way later?

 Bye,
 bearophile


scope[symbol_name] would get the same lifetime as symbol_name for
instance. This is only a proposal, I'm not convinced that this is
strictly necessary, and, as long as we know we can add it if
needed, we are good.
Dec 10 2014
prev sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 11 December 2014 at 00:35:46 UTC, deadalnix wrote:

 It is always safe to consider scopeness of the retrun value (if 
 marked scope) as being the intersection of the lifetime of 
 parameters.

 That should cover most bases, and we can still extends later if 
 this is too limited (I suspect it is ok for most cases).


Linear typing is already extremely limiting, by limiting it even 
further you end up with something annoying. You basically get a 
version of memory safety that does not solve any typical memory 
unsafe situations.

By having pointers that do scope-depth-tracking you do at least 
get a generic solution that can be optimized away when possible. 
The D authors have to accept that you need to embed ownership in 
pointers if you want memory safety and convenience, or that you 
have to provide means to guide the semantic analysis. You need 
one or the other, or both, but you cannot pretend that you can do 
without.

Arbitrary constraints are annoying, not convenient. If I as a 
programmer know that something is safe, then the compiler should 
accept it, and the language should allow me express it.
Dec 11 2014
next sibling parent reply Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 11/12/2014 09:07, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:

 If I as a programmer know that something is safe, then the compiler
 should accept it, and the language should allow me express it.


It will, use  system. Maybe this proposal will be supplemented later, 
but there are always costs to making the language more complicated. We 
need to be certain they are worth it before adding them. A cautious 
approach can be extended later, if the situation demands it.

Sometimes innovative workarounds are developed that are difficult to 
foresee in advance - e.g. in Rust their type system can be restrictive, 
but they rely on trusted library functions/types to make more things 
possible in a safe way.
Dec 11 2014
next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 11 December 2014 at 13:43:07 UTC, Nick Treleaven 
wrote:

 It will, use  system.


Yes…


 Maybe this proposal will be supplemented later, but there are 
 always costs to making the language more complicated. We need 
 to be certain they are worth it before adding them. A cautious 
 approach can be extended later, if the situation demands it.


Can it be extended later? Without breaking stuff?

You need to get it right from the start, otherwise you end up 
with N different, but incompatible ways of doing the same thing. 
That means you need to get ownership right first.
Dec 11 2014
prev sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Nick Treleaven:


 Sometimes innovative workarounds are developed that are 
 difficult to foresee in advance - e.g. in Rust their type 
 system can be restrictive, but they rely on trusted library 
 functions/types to make more things possible in a safe way.


Ideally a type system should be flexible enough, in practice 
giving it a high flexibility has significant costs (in 
compilation times, amount of code that implements the system, 
bugs in such implementation, costs for the final programmer in 
inventing a way to express the semantics, etc), so most languages 
avoid a too much complex type system (even Haskell does this) and 
accept reasonable workarounds...

Bye,
bearophile
Dec 11 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 11 December 2014 at 15:37:27 UTC, bearophile wrote:

 Nick Treleaven:


 Sometimes innovative workarounds are developed that are 
 difficult to foresee in advance - e.g. in Rust their type 
 system can be restrictive, but they rely on trusted library 
 functions/types to make more things possible in a safe way.


 Ideally a type system should be flexible enough, in practice 
 giving it a high flexibility has significant costs (in 
 compilation times, amount of code that implements the system, 
 bugs in such implementation, costs for the final programmer in 
 inventing a way to express the semantics, etc), so most 
 languages avoid a too much complex type system (even Haskell 
 does this) and accept reasonable workarounds...

 Bye,
 bearophile


What is important is how much expressiveness you get from the 
complexity. here we get almost none. Considering complexity alone 
is not going to yield good results.
Dec 11 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 11 December 2014 at 09:07:44 UTC, Ola Fosheim 
Grøstad wrote:

 On Thursday, 11 December 2014 at 00:35:46 UTC, deadalnix wrote:

 It is always safe to consider scopeness of the retrun value 
 (if marked scope) as being the intersection of the lifetime of 
 parameters.

 That should cover most bases, and we can still extends later 
 if this is too limited (I suspect it is ok for most cases).


 Linear typing is already extremely limiting, by limiting it 
 even further you end up with something annoying. You basically 
 get a version of memory safety that does not solve any typical 
 memory unsafe situations.

 By having pointers that do scope-depth-tracking you do at least 
 get a generic solution that can be optimized away when 
 possible. The D authors have to accept that you need to embed 
 ownership in pointers if you want memory safety and 
 convenience, or that you have to provide means to guide the 
 semantic analysis. You need one or the other, or both, but you 
 cannot pretend that you can do without.

 Arbitrary constraints are annoying, not convenient. If I as a 
 programmer know that something is safe, then the compiler 
 should accept it, and the language should allow me express it.


I have no idea what you are saying. It sounds like randomly 
generated gibberish.
Dec 11 2014
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 11 December 2014 at 23:26:33 UTC, deadalnix wrote:

 I have no idea what you are saying. It sounds like randomly 
 generated gibberish.


What is your problem?

Scope parameters are trying to address what Rust tried to do with 
borrowing. To get an idea of where you are going you should look 
at ordered linear and non-linear type systems.

Without a dynamic solution OR a proof system OR a way to provide 
a manual guarantee you will hit the ceiling for what you can do, 
which will be annoying for the user (programmer).
Dec 11 2014
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/5/14 3:55 PM, Walter Bright wrote:

 On 12/5/2014 7:27 AM, Steven Schveighoffer wrote:

 Can someone who knows what this new feature is supposed to do give
 some Ali
 Çehreli-like description on the feature? Basically, let's strip out
 the *proof*
 in the DIP (the how it works and why we have it), and focus on how it
 is to be
 used.

 I still am having a hard time wrapping my head around the benefits and
 when to
 use scope, scope ref, why I would use it. I'm worried that we are
 adding all
 this complication and it will confuse the shit out of users, to the
 point where
 they won't use it.


 The tl;dr version is when a declaration is tagged with 'scope', the
 contents of that variable will not escape the lifetime of that declaration.

 It means that this code will be safe:

     void foo(scope int* p);

     p = malloc(n);
     foo(p);
     free(p);

 The rest is all the nuts and bolts of making that work.


This is not what I was asking for. What I wanted to know was, when I see 
scope ref, why is it there? When should I use it? When should I use 
scope? What nasty things are prevented if I use it? Examples would be 
preferable.

Note, in your example above, marking foo pure solves the problem already.

The rules and statements give an inferred benefit. I'd like that benefit 
to be more fully explained.

-Steve
Dec 08 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2014 7:23 AM, Steven Schveighoffer wrote:

 What I wanted to know was, when I see scope
 ref, why is it there? When should I use it?


It's there to indicate a parameter is NOT returned by ref.




 When should I use scope? What nasty
 things are prevented if I use it? Examples would be preferable.


http://wiki.dlang.org/DIP69#Ref has examples of errors that are prevented.
Dec 14 2014
prev sibling parent zeljkog <zeljkog home.com> writes:
On 05.12.14 21:55, Walter Bright wrote:

 
 It means that this code will be safe:
 
     void foo(scope int* p);
 
     p = malloc(n);
     foo(p);
     free(p);
 
 The rest is all the nuts and bolts of making that work.
 


If it is main goal, opposite looks more natural.

void foo(int* p);
void foo1(escape int* p);
 
p = malloc(n);
p1 = malloc(n);
foo(p);
free(p);  // OK
foo1(p1);
free(p1);  // no
Dec 08 2014
prev sibling parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2014-12-05 15:27:41 +0000, Steven Schveighoffer said:


 On 12/4/14 4:24 AM, Walter Bright wrote:

 http://wiki.dlang.org/DIP69
 
 Despite its length, this is a fairly simple proposal. It adds the
 missing semantics for the 'scope' storage class in order to make it
 possible to pass a reference to a function without it being possible for
 it to escape.
 
 This, among other things, makes a ref counting type practical. It also
 makes it more practical to use other storage allocation schemes than
 garbage collection.
 
 It does not make scope into a type constructor, nor a general
 type-annotation system.
 
 It does not provide an ownership system, though it would complement one.


 
 Can we take a step back here?
 
 I read many people's comments and I understand only about half of them.
 
 Can someone who knows what this new feature is supposed to do give some 
 Ali Çehreli-like description on the feature? Basically, let's strip out 
 the *proof* in the DIP (the how it works and why we have it), and focus 
 on how it is to be used.
 
 I still am having a hard time wrapping my head around the benefits and 
 when to use scope, scope ref, why I would use it. I'm worried that we 
 are adding all this complication and it will confuse the shit out of 
 users, to the point where they won't use it.
 
 -Steve


This is exactly why this feature should be default behavior with 
compiler warnings generated when things escape scope.

The basic idea is that most things should be on the stack and go away 
when the stack goes away.   The proposal codifies how the compiler 
should infer that references to variables which are to be placed on the 
stack are not escaping their scope.   The benefit is that the GC has to 
do *way* less work from what it has to do now for most of the use cases 
of D.

-Shammah
Dec 10 2014
parent zeljkog <zeljkog home.com> writes:
On 11.12.14 08:25, Shammah Chancellor wrote:

 This is exactly why this feature should be default behavior with
 compiler warnings generated when things escape scope.


It can be compiler switch escape=I(gnore)|W(arning)|E(rror).
Dec 11 2014
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
There are limitations this proposal has in comparison to my 
original one. These limitations might of course be harmless and 
play no role in practice, but on the other hand, they may, so I 
think it's good to list them here.

Additionally I have to agree with Steven Schveighoffer: This DIP 
is very complicated to understand. It's not obvious how the 
various parts play together, and why/to which degree it "works", 
and which are the limitations. I don't think that's only because 
my brain is already locked on my proposal...

1) Escape detection is limited to `ref`.

     T* evil;
     ref T func(scope ref T t, ref T u)  safe {
       return t; // Error: escaping scope ref t
       return u; // ok
       evil = &u; // Error: escaping reference
     }

vs.

     T[] evil;
     T[] func(scope T[] t, T[] u)  safe {
       return t; // Error: cannot return scope
       return u; // ok
       evil = u; // !!! not good
     }

As can be seen, `ref T u` is protected from escaping (apart from 
returning it), while `T[] u` in the second example is not. 
There's no general way to express that `u` can only be returned 
from the function, but will not be retained otherwise by storing 
it in a global variable. Adding `pure` can express this in many 
cases, but is, of course, not always possible.

Another workaround is passing the parameters as `ref`, but this 
would introduce an additional indirection and has different 
semantics (e.g. when the lengths of the slices are modified).

2) `scope ref` return values cannot be stored.

     scope ref int foo();
     void bar(scope ref int a);

     foo().bar();        // allowed
     scope tmp = foo();  // not allowed
     tmp.bar();

Another example:

     struct Container(T) {
         scope ref T opIndex(size_t index);
     }

     void bar(scope ref int a);

     Container c;
     bar(c[42]);            // ok
     scope ref tmp = c[42]; // nope

Both cases should be fine theoretically; the "real" owner lives 
longer than `tmp`. Unfortunately the compiler doesn't know about 
this.

Both restrictions 1) and 2) are because there are no explicit 
lifetime/owner designations (the scope!identifier thingy in my 
proposal).

3) `scope` cannot be used for value types.

I can think of a few use cases for scoped value types (RC and 
file descriptors), but they might only be marginal.

4) No overloading on `scope`.

This is at least partially a consequence of `scope` inference. I 
think overloading can be made to work in the presence of 
inference, but I haven't thought it through.

5) `scope` is a storage class.

Manu complained about `ref` being a storage class. If I 
understand him right, one reason is that we have a large toolkit 
for dealing with type modifiers, but almost nothing for storage 
classes. I have to agree with him there. But I haven't understood 
his point fully, maybe he himself can post more about his 
problems with this?

6) There seem to be problems with chaining.

     scope ref int foo();
     scope ref int bar1(ref int a) {
         return a;
     }
     scope ref int bar2(scope ref int a) {
         return a;
     }
     ref int bar3(ref int a) {
         return a;
     }
     ref int bar4(scope ref int a) {
         return a;
     }
     void baz(scope ref int a);

Which of the following calls would work?

     foo().bar1().baz();
     foo().bar2().baz();
     foo().bar3().baz();
     foo().bar4().baz();

I'm not sure I understand this fully yet, but it could be that 
none of them work...
Dec 05 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/5/2014 8:48 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:

 There are limitations this proposal has in comparison to my original one. These
 limitations might of course be harmless and play no role in practice, but on
the
 other hand, they may, so I think it's good to list them here.


Good idea. Certainly, this is less powerful than your proposal. The question, 
obviously, is what is good enough to get the job done. By "the job", I mean 
reference counting, migrating many allocations to RAII (meaning the stack), and 
eliminating a lot of closure GC allocations.



 Additionally I have to agree with Steven Schveighoffer: This DIP is very
 complicated to understand. It's not obvious how the various parts play
together,
 and why/to which degree it "works", and which are the limitations. I don't
think
 that's only because my brain is already locked on my proposal...


I'm still looking for an easier way to explain it. The good news in this, 
however, is if it is correctly implement the compiler should be a big help in 
using scope correctly.



 1) Escape detection is limited to `ref`.

      T* evil;
      ref T func(scope ref T t, ref T u)  safe {
        return t; // Error: escaping scope ref t
        return u; // ok
        evil = &u; // Error: escaping reference
      }

 vs.

      T[] evil;
      T[] func(scope T[] t, T[] u)  safe {
        return t; // Error: cannot return scope
        return u; // ok
        evil = u; // !!! not good


right, although:
      evil = t;  // Error: not allowed


      }

 As can be seen, `ref T u` is protected from escaping (apart from returning it),
 while `T[] u` in the second example is not. There's no general way to express
 that `u` can only be returned from the function, but will not be retained
 otherwise by storing it in a global variable. Adding `pure` can express this in
 many cases, but is, of course, not always possible.


As you point out, 'ref' is designed for this.



 Another workaround is passing the parameters as `ref`, but this would introduce
 an additional indirection and has different semantics (e.g. when the lengths of
 the slices are modified).

 2) `scope ref` return values cannot be stored.

      scope ref int foo();
      void bar(scope ref int a);

      foo().bar();        // allowed
      scope tmp = foo();  // not allowed
      tmp.bar();


Right



 Another example:

      struct Container(T) {
          scope ref T opIndex(size_t index);
      }

      void bar(scope ref int a);

      Container c;
      bar(c[42]);            // ok
      scope ref tmp = c[42]; // nope

 Both cases should be fine theoretically; the "real" owner lives longer than
 `tmp`. Unfortunately the compiler doesn't know about this.


Right, though the compiler can optimize to produce the equivalent.


 Both restrictions 1) and 2) are because there are no explicit lifetime/owner
 designations (the scope!identifier thingy in my proposal).

 3) `scope` cannot be used for value types.

 I can think of a few use cases for scoped value types (RC and file
descriptors),
 but they might only be marginal.


I suspect that one can encapsulate such values in a struct where access to them 
is strictly controlled.



 4) No overloading on `scope`.

 This is at least partially a consequence of `scope` inference. I think
 overloading can be made to work in the presence of inference, but I haven't
 thought it through.


Right. Different overloads can have different semantic implementations, so what 
should inference do? I also suspect it is bad style to overload on 'scope'.



 5) `scope` is a storage class.

 Manu complained about `ref` being a storage class. If I understand him right,
 one reason is that we have a large toolkit for dealing with type modifiers, but
 almost nothing for storage classes. I have to agree with him there. But I
 haven't understood his point fully, maybe he himself can post more about his
 problems with this?


I didn't fully understand Manu's issue, but it was about 'ref' not being 
inferred by template type deduction. I didn't understand why 'auto ref' did not 
work for him. I got the impression that he was trying to program in D the same 
way he'd do things in C++, and that's where the trouble came in.



 6) There seem to be problems with chaining.

      scope ref int foo();
      scope ref int bar1(ref int a) {
          return a;
      }
      scope ref int bar2(scope ref int a) {
          return a;
      }
      ref int bar3(ref int a) {
          return a;
      }
      ref int bar4(scope ref int a) {
          return a;
      }
      void baz(scope ref int a);

 Which of the following calls would work?

      foo().bar1().baz();


yes


      foo().bar2().baz();


no - cannot return scope ref parameter


      foo().bar3().baz();


yes


      foo().bar4().baz();


no, cannot return scope ref parameter



 I'm not sure I understand this fully yet, but it could be that none of them
work...


Well, you're half right :-)
Dec 05 2014
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:


 The question, obviously, is what is good enough to get the job 
 done. By "the job", I mean reference counting, migrating many 
 allocations to RAII (meaning the stack), and eliminating a lot 
 of closure GC allocations.


Another essential purpose of a similar proposal is to avoid some 
bugs caused by using memory from vanished stack frames, etc, that 
is to have memory safety in D.

Bye,
bearophile
Dec 05 2014
prev sibling next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 6 December 2014 at 09:58, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 12/5/2014 8:48 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:

 There are limitations this proposal has in comparison to my original one.
 These
 limitations might of course be harmless and play no role in practice, but
 on the
 other hand, they may, so I think it's good to list them here.



 Good idea. Certainly, this is less powerful than your proposal. The
 question, obviously, is what is good enough to get the job done. By "the
 job", I mean reference counting, migrating many allocations to RAII (meaning
 the stack), and eliminating a lot of closure GC allocations.



 Additionally I have to agree with Steven Schveighoffer: This DIP is very
 complicated to understand. It's not obvious how the various parts play
 together,
 and why/to which degree it "works", and which are the limitations. I don't
 think
 that's only because my brain is already locked on my proposal...



 I'm still looking for an easier way to explain it. The good news in this,
 however, is if it is correctly implement the compiler should be a big help
 in using scope correctly.



 1) Escape detection is limited to `ref`.

      T* evil;
      ref T func(scope ref T t, ref T u)  safe {
        return t; // Error: escaping scope ref t
        return u; // ok
        evil = &u; // Error: escaping reference
      }

 vs.

      T[] evil;
      T[] func(scope T[] t, T[] u)  safe {
        return t; // Error: cannot return scope
        return u; // ok
        evil = u; // !!! not good



 right, although:
      evil = t;  // Error: not allowed


      }

 As can be seen, `ref T u` is protected from escaping (apart from returning
 it),
 while `T[] u` in the second example is not. There's no general way to
 express
 that `u` can only be returned from the function, but will not be retained
 otherwise by storing it in a global variable. Adding `pure` can express
 this in
 many cases, but is, of course, not always possible.



 As you point out, 'ref' is designed for this.



 Another workaround is passing the parameters as `ref`, but this would
 introduce
 an additional indirection and has different semantics (e.g. when the
 lengths of
 the slices are modified).

 2) `scope ref` return values cannot be stored.

      scope ref int foo();
      void bar(scope ref int a);

      foo().bar();        // allowed
      scope tmp = foo();  // not allowed
      tmp.bar();



 Right



 Another example:

      struct Container(T) {
          scope ref T opIndex(size_t index);
      }

      void bar(scope ref int a);

      Container c;
      bar(c[42]);            // ok
      scope ref tmp = c[42]; // nope

 Both cases should be fine theoretically; the "real" owner lives longer
 than
 `tmp`. Unfortunately the compiler doesn't know about this.



 Right, though the compiler can optimize to produce the equivalent.


 Both restrictions 1) and 2) are because there are no explicit
 lifetime/owner
 designations (the scope!identifier thingy in my proposal).

 3) `scope` cannot be used for value types.

 I can think of a few use cases for scoped value types (RC and file
 descriptors),
 but they might only be marginal.



 I suspect that one can encapsulate such values in a struct where access to
 them is strictly controlled.



 4) No overloading on `scope`.

 This is at least partially a consequence of `scope` inference. I think
 overloading can be made to work in the presence of inference, but I
 haven't
 thought it through.



 Right. Different overloads can have different semantic implementations, so
 what should inference do? I also suspect it is bad style to overload on
 'scope'.



 5) `scope` is a storage class.

 Manu complained about `ref` being a storage class. If I understand him
 right,
 one reason is that we have a large toolkit for dealing with type
 modifiers, but
 almost nothing for storage classes. I have to agree with him there. But I
 haven't understood his point fully, maybe he himself can post more about
 his
 problems with this?



 I didn't fully understand Manu's issue, but it was about 'ref' not being
 inferred by template type deduction. I didn't understand why 'auto ref' did
 not work for him. I got the impression that he was trying to program in D
 the same way he'd do things in C++, and that's where the trouble came in.


NO!!
I barely program C++ at all! I basically write C code with 'enum' and
'class'. NEVER 'template', and very rarely 'virtual'.
Your impression is dead wrong.

If I do things in any particular 'way', it is that at all times, I
have regard for, and control over the code generation, the ABI, and I
also value distribution of code as static libs, which means I must
retain tight control over where code is, and isn't, emitted.
I expect this from a native language.


It's exactly as Marc says, we have the best tools for dealing with
types of any language I know, and practically none for dealing with
'storage class'.
In my use of meta in D, 'ref' is the single greatest cause of
complexity, code bloat, duplication, and text mixins. I've been
banging on about this for years!

I've been over it so many times.
I'll start over if there is actually some possibility I can convince
you? Is there?

I've lost faith that I am able to have any meaningful impact on issues
that matter to me, and I'm fairly sure at this point that my
compounded resentment and frustration actually discredit my cause, and
certainly, my quality of debate.
I'm passionate about these things, but I'm also extremely frustrated.
To date, whenever I have engaged in a topic that I *really* care
about, it goes the other direction. To participate has proven to be
something of a (very time consuming) act of masochism.


I __absolutely objected__ to 'auro ref' when it appeared. I argued
that it was a massive mistake, and since it's introduction and
experience with it in the wild, I am more confident in that conviction
than ever.

As a programmer, I expect control over whether code is a template, or
not. 'ref' doesn't have anything to do with templates. Confusing these
2 concepts was such a big mistake.
auto ref makes a template out of something that shouldn't be a
template. It's a particularly crude hack to address a prior mistake.
ref should have been fixed at that time, not compounded with layers of
even more weird and special-case/non-uniform behaviours above it.


There has been a couple of instances where the situation has been
appropriate that I've tried to make use of auto ref, but in each case,
the semantics have never been what I want.
auto ref presumes to decide for you when something should be a ref or
not. It prescribes a bunch of rules on how that decision is made, but
it doesn't know what I'm doing, and it gets it wrong.
In my experience, auto ref has proven to be, at best, completely
useless. But in some cases I've found where I bump into it in 3rd
party code, it's been a nuisance, requiring me to wrap it away.


ref is a bad design. C++'s design isn't fantastic, and I appreciate
that D made effort to improve on it, but we need to recognise when the
experiment was a failure. D's design is terrible; it's basically
orthogonal to the rest of the language. It's created way more
complicated edge cases for me than C++ references ever have. Anyone
who says otherwise obviously hasn't really used it much!
Don't double down on that mistake with scope.


My intended reply to this post was to ask you to justify making it a
storage class, and why the design fails as a type constructor?
Can we explore that direction to it's point of failure?

As a storage class, it runs the risk of doubling out existing bloat
caused by ref. As a type constructor, I see no disadvantages.
It even addresses some of the awkward problems right at the face of
storage classes, like how/where the attributes actually apply. Type
constructors use parens; ie, const(T), and scope(T) would make that
matter a lot  more clear.

Apart from the storage class issue, it looks okay, but it gives me the
feeling that it kinda stops short.
Marc's proposal addressed more issues. I feel this proposal will
result in more edge cases than Marc's proposal.
The major edge case that I imagine is that since scope return values
can't be assigned to scope local's, that will result in some awkward
meta, requiring yet more special cases. I think Mark's proposal may be
a lot more relaxed in that way.
Dec 06 2014
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-12-06 10:50, Manu via Digitalmars-d wrote:


 I've been over it so many times.


I suggest you take the time and write down how your vision of "ref" 
looks like and the issue with the current implementation. A blog post, a 
DIP or similar. Then you can easily refer to that in cases like this. 
Then you don't have to repeat yourself so many times. That's what I did 
when there was a lot of talk about AST macros. I was tired of constantly 
repeating myself so I created a DIP. It has already saved me more time 
than it took to write the actual DIP.

-- 
/Jacob Carlborg
Dec 06 2014
parent reply "Piotrek" <pl pl.pl> writes:
On Saturday, 6 December 2014 at 11:06:16 UTC, Jacob Carlborg 
wrote:

 On 2014-12-06 10:50, Manu via Digitalmars-d wrote:


 I've been over it so many times.


 I suggest you take the time and write down how your vision of 
 "ref" looks like and the issue with the current implementation. 
 A blog post, a DIP or similar. Then you can easily refer to 
 that in cases like this. Then you don't have to repeat yourself 
 so many times. That's what I did when there was a lot of talk 
 about AST macros. I was tired of constantly repeating myself so 
 I created a DIP. It has already saved me more time than it took 
 to write the actual DIP.


 Manu
Seconded. Please create even short one. I coulnd'd find any 
example of use case you are referring to (as you said I don't use 
D"ref" so often) . I plan to apply D for embedded systems, so 
full control is a must. But so far Water still has the most 
accurate taste according to my experience.

BTW. I consider game devs to be the most underpaid programmers, 
so your perspective is very precious to me.

Cheers
Piotrek
Dec 06 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 6 December 2014 at 23:59:01 UTC, Piotrek wrote:

 On Saturday, 6 December 2014 at 11:06:16 UTC, Jacob Carlborg 
 wrote:

 On 2014-12-06 10:50, Manu via Digitalmars-d wrote:


 I've been over it so many times.


 I suggest you take the time and write down how your vision of 
 "ref" looks like and the issue with the current 
 implementation. A blog post, a DIP or similar. Then you can 
 easily refer to that in cases like this. Then you don't have 
 to repeat yourself so many times. That's what I did when there 
 was a lot of talk about AST macros. I was tired of constantly 
 repeating myself so I created a DIP. It has already saved me 
 more time than it took to write the actual DIP.


  Manu
 Seconded. Please create even short one. I coulnd'd find any 
 example of use case you are referring to (as you said I don't 
 use D"ref" so often) . I plan to apply D for embedded systems, 
 so full control is a must. But so far Water still has the most 
 accurate taste according to my experience.

 BTW. I consider game devs to be the most underpaid programmers, 
 so your perspective is very precious to me.

 Cheers
 Piotrek


I'd like to not polute this thread with the ref topic.

Long story short:
   - it is hard to know if something is ref, making it hard to
metaprogram.
   - you sometime want to switch ref on and off (for instance, you
may use ref to avoid copies, which is not worthwhile for small
values) which is complex.
   - auto ref do not cut it.

That is the extra short version, please start a thread on the 
subject. Please, please, this one is complicated enough.
Dec 08 2014
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 6 December 2014 at 09:51:15 UTC, Manu via 
Digitalmars-d wrote:

 I've been over it so many times.
 I'll start over if there is actually some possibility I can 
 convince
 you? Is there?

 I've lost faith that I am able to have any meaningful impact on 
 issues
 that matter to me, and I'm fairly sure at this point that my
 compounded resentment and frustration actually discredit my 
 cause, and
 certainly, my quality of debate.
 I'm passionate about these things, but I'm also extremely 
 frustrated.
 To date, whenever I have engaged in a topic that I *really* care
 about, it goes the other direction. To participate has proven 
 to be
 something of a (very time consuming) act of masochism.


That's very sad to hear. I think you have brought up some very 
important points, especially from a practical point of view.


 I __absolutely objected__ to 'auro ref' when it appeared. I 
 argued
 that it was a massive mistake, and since it's introduction and
 experience with it in the wild, I am more confident in that 
 conviction
 than ever.

 As a programmer, I expect control over whether code is a 
 template, or
 not. 'ref' doesn't have anything to do with templates. 
 Confusing these
 2 concepts was such a big mistake.
 auto ref makes a template out of something that shouldn't be a
 template. It's a particularly crude hack to address a prior 
 mistake.
 ref should have been fixed at that time, not compounded with 
 layers of
 even more weird and special-case/non-uniform behaviours above 
 it.


 There has been a couple of instances where the situation has 
 been
 appropriate that I've tried to make use of auto ref, but in 
 each case,
 the semantics have never been what I want.
 auto ref presumes to decide for you when something should be a 
 ref or
 not. It prescribes a bunch of rules on how that decision is 
 made, but
 it doesn't know what I'm doing, and it gets it wrong.
 In my experience, auto ref has proven to be, at best, completely
 useless. But in some cases I've found where I bump into it in 
 3rd
 party code, it's been a nuisance, requiring me to wrap it away.


I tend to agree with this. Here's an example of the dangers:

https://github.com/deadalnix/libd/pull/7

In this case, `auto ref` accepted a class by reference, because 
it chooses ref-ness based on whether you pass an lvalue or an 
rvalue. This is not very helpful. I agree that `ref` isn't 
something that should ever be inferred, because it affects the 
semantics of the function. Either the function requires `ref` 
semantics, or not. It doesn't depend on how it is called.
Dec 06 2014
parent "bearophile" <bearophileHUGS lycos.com> writes:
Marc Schütz:


 I agree that `ref` isn't something that should ever be 
 inferred, because it affects the semantics of the function. 
 Either the function requires `ref` semantics, or not. It 
 doesn't depend on how it is called.


(Perhaps this was already said) I think Ada used to have 
something like "auto ref" and later it was removed from the 
language.

Bye,
bearophile
Dec 06 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/6/2014 1:50 AM, Manu via Digitalmars-d wrote:

 I didn't fully understand Manu's issue, but it was about 'ref' not being
 inferred by template type deduction. I didn't understand why 'auto ref' did
 not work for him. I got the impression that he was trying to program in D
 the same way he'd do things in C++, and that's where the trouble came in.


 NO!!
 I barely program C++ at all! I basically write C code with 'enum' and
 'class'. NEVER 'template', and very rarely 'virtual'.
 Your impression is dead wrong.


I apologize for misunderstanding you.



 It's exactly as Marc says, we have the best tools for dealing with
 types of any language I know, and practically none for dealing with
 'storage class'.
 In my use of meta in D, 'ref' is the single greatest cause of
 complexity, code bloat, duplication, and text mixins. I've been
 banging on about this for years!

 I've been over it so many times.
 I'll start over if there is actually some possibility I can convince
 you? Is there?


I know there's no easy way to derive a storage class from an expression. The 
difficulty in my understanding is why this is a great cause of problems for you 
in particular (and by implication not for others). There's something about the 
way you write code that's different.



 I've lost faith that I am able to have any meaningful impact on issues
 that matter to me, and I'm fairly sure at this point that my
 compounded resentment and frustration actually discredit my cause, and
 certainly, my quality of debate.


This is incorrect, you had enormous influence over the vector type in D! And I 
wish you had more.



 There has been a couple of instances where the situation has been
 appropriate that I've tried to make use of auto ref, but in each case,
 the semantics have never been what I want.


I don't really know what you want. Well, perhaps a better statement is 'why', 
not what.



 ref is a bad design. C++'s design isn't fantastic, and I appreciate
 that D made effort to improve on it, but we need to recognise when the
 experiment was a failure. D's design is terrible; it's basically
 orthogonal to the rest of the language. It's created way more
 complicated edge cases for me than C++ references ever have. Anyone
 who says otherwise obviously hasn't really used it much!
 Don't double down on that mistake with scope.


Why does your code need to care so much about to ref or not to ref? That's the 
central point here, I think.



 My intended reply to this post was to ask you to justify making it a
 storage class, and why the design fails as a type constructor?
 Can we explore that direction to it's point of failure?

 As a storage class, it runs the risk of doubling out existing bloat
 caused by ref. As a type constructor, I see no disadvantages.
 It even addresses some of the awkward problems right at the face of
 storage classes, like how/where the attributes actually apply. Type
 constructors use parens; ie, const(T), and scope(T) would make that
 matter a lot  more clear.

 Apart from the storage class issue, it looks okay, but it gives me the
 feeling that it kinda stops short.
 Marc's proposal addressed more issues. I feel this proposal will
 result in more edge cases than Marc's proposal.
 The major edge case that I imagine is that since scope return values
 can't be assigned to scope local's, that will result in some awkward
 meta, requiring yet more special cases. I think Mark's proposal may be
 a lot more relaxed in that way.


The disadvantages of making it a type qualifier are:

1. far more complexity. Type constructors interact with everything, often in 
unanticipated ways. We spent *years* working out issues with the 'const' type 
qualifier, and are still doing so. Kenji just fixed another one.

2. we are never going to get users to use 'scope' qualifiers pervasively. It's 
been a long struggle to get 'const' used.

3. we added 'inout' as a type qualifier to avoid code duplication engendered by 
'const'. It hurts my brain to even think about how that might interact with 
'scope' qualifiers.

Yes, I agree unequivocably, that 'scope' as a type qualifier is more expressive 
and more powerful than as a storage class. Multiple inheritance is also more 
expressive and more powerful than single inheritance. But many times, more
power 
perhaps isn't better than redoing the program design to use something simpler 
and less complex.
Dec 06 2014
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 7 December 2014 at 08:15, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 12/6/2014 1:50 AM, Manu via Digitalmars-d wrote:

 I didn't fully understand Manu's issue, but it was about 'ref' not being
 inferred by template type deduction. I didn't understand why 'auto ref'
 did
 not work for him. I got the impression that he was trying to program in D
 the same way he'd do things in C++, and that's where the trouble came in.


 NO!!
 I barely program C++ at all! I basically write C code with 'enum' and
 'class'. NEVER 'template', and very rarely 'virtual'.
 Your impression is dead wrong.



 I apologize for misunderstanding you.



 It's exactly as Marc says, we have the best tools for dealing with
 types of any language I know, and practically none for dealing with
 'storage class'.
 In my use of meta in D, 'ref' is the single greatest cause of
 complexity, code bloat, duplication, and text mixins. I've been
 banging on about this for years!

 I've been over it so many times.
 I'll start over if there is actually some possibility I can convince
 you? Is there?



 I know there's no easy way to derive a storage class from an expression. The
 difficulty in my understanding is why this is a great cause of problems for
 you in particular (and by implication not for others). There's something
 about the way you write code that's different.


Perhaps it's the tasks I typically perform with meta?
I'm a fairly conservative user of templates, but one place where they
shine, and I'm always very tempted to use them, is the task of
serialisation, and cross-language bindings.
Those tasks typically involve mountains of boilerplate, and D is the
only language expressive enough to start to really automate that
mechanical mess.

subtly different tasks, but share a lot in common, and the major
characteristic which reveals my problems with things like ref, and
auto ref, is that these things aren't open to interpretation. They're
not cases where the compiler can make decisions for you (ie, auto ref
fails), and the ABI + API are strict (ie, ref must match correctly).

There's a lot more more to it, but perhaps I'm among few who have had
such extensive need to engage in these tasks?



 I've lost faith that I am able to have any meaningful impact on issues
 that matter to me, and I'm fairly sure at this point that my
 compounded resentment and frustration actually discredit my cause, and
 certainly, my quality of debate.



 This is incorrect, you had enormous influence over the vector type in D! And
 I wish you had more.


I appreciate that. That was uncontroversial though; I didn't need to
spend months or years trying to justify my claims on that issue. I
feel like that was a known item that was just somewhere slightly down
the list, and I was able to bring it forward.
I was never in a position where I had to argue against Andrei to sell that one.

I have std.simd sitting here, and I really want to finish it, but I
still don't have the tools to do so.
I need, at least, forceinline to complete it, but that one *is*
controversial - we've talked about this for years.

GDC and LDC both have a forceinline, so I could theoretically support
those compilers, but then I can't practically make use of them without
some sort of attribute aliasing system, otherwise I need to triplicate
the code for each compiler, just to insert a different (compiler
specific) forceinline attribute name. It'd be really great if we
agreed on just one.

There is also a practical problem with GCC (perhaps it's an
incompatbility with my std.simd design), where I need to change the
sse-level between functions.
There is a GCC attribute to do this ('target'), but it would rely on,
at least, a few subtle tweaks. In this case, I need to be able to feed
a template argument to a UDA, which doesn't work because UDA
declarations seem to be parsed prior to knowledge of functions
template args. There may be a further problem with GCC though which I
haven't been able to prove yet though.

The reason for this is that an important design goal for my work was
to be able to semi-automatically generate multiple code paths for
different SSE versions, which can be selected at runtime based on
available hardware.
It's a particularly awkward issue in C/C++, usually requiring you to
have multiple modules which are each built with different compile
flags, and then do some magic while linking to resolve the runtime
selection problem.

Anyway, I have been able to use SIMD directly in my own software with
the support we have, but I haven't been able to complete the library
that I want to produce.
I need a little more language support. I've reached out a few times,
but there hasn't been too much interest in the past. I really do wanna
finish it one of these days though!



 There has been a couple of instances where the situation has been
 appropriate that I've tried to make use of auto ref, but in each case,
 the semantics have never been what I want.



 I don't really know what you want. Well, perhaps a better statement is
 'why', not what.


Because a function is a function. It's a fundamental and ultimately
*simple* element of a language, with simple and reliable in behaviour;
you write a function, compiler emits some code with a matching symbol
name. There is perhaps nothing more simple or fundamental to the
language.

A template is not a function. It may *generate* a function, but now
we've added many extra details which require careful consideration and
handling, like where the code is, or isn't, emitted? How many
permutations are there? What about static/dynamic libraries? Calling
with various, or arbitrary types, always required careful
consideration, more so than an explicit type.
Templates are a powerful (often dangerous) **tool**, which should only
be applied deliberately and very carefully.
Obviously auto-ref gives hard answers to some of those questions, but
it is still a template, and that instantly makes it much more complex.

Making a function that has nothing to do with templates into a
template is not what I want... it couldn't be further from what I ever
wanted!

The ABI is different; there are 2 functions now (or zero, until one is
called). Thich means taking the address of a function with auto-ref is
a complex issue.
ref-ness isn't a decision I ever want the compiler to make for me. I
either want ref, or I don't, and I will state that to the compiler
explicitly.
This is especially true when interacting with other languages; while D
remains in relative infancy, I think that is an overwhelmingly common
situation when a project is modestly large.

In the situation where templates are involved, it would be nice to be
able to make that explicit statement that some type is ref or not at
the point of template instantiation, and the resolution should work
according to the well-defined rules that we are all familiar with.
Within a complex template, if logic should be performed, we have
powerful tools to do that already; nobody ever complains about
Unqual!, or PointerTarget!... these things are very easy to read and
understand. ref should fit right in there with the others, is(x ==
ref), UnRef!T, etc.
Instead, it's practically orthogonal to the language. Wrangling
anything involving storage classes is really, really tedious, and
almost always results in extensive code duplication, or text mixins
when duplication just get too much.

This issue really surprised me at the time, because the whole thing
was in response to a topic that I consistently raised and pushed.
Who was the customer welcoming that solution? I could never work it
out; there was nobody else that seemed particularly invested in the
problem, except the usual suite of language enthusiasts that took it
as an interesting intellectual problem to discuss. The person who
cared about that issue the most (afaict) was completely unsatisfied
with the solution. I said at the time that I would have preferred that
no action was taken, than to compound, and further set in stone, an
issue that I was already extremely critical of.


Don't double down on this mistake with scope!



 ref is a bad design. C++'s design isn't fantastic, and I appreciate
 that D made effort to improve on it, but we need to recognise when the
 experiment was a failure. D's design is terrible; it's basically
 orthogonal to the rest of the language. It's created way more
 complicated edge cases for me than C++ references ever have. Anyone
 who says otherwise obviously hasn't really used it much!
 Don't double down on that mistake with scope.



 Why does your code need to care so much about to ref or not to ref? That's
 the central point here, I think.


That same logic could ask why I want to have 'short', or 'int' in some
places, but not others. There's no absolute answer. As a software
engineer, and particularly, a native software engineer, it's a
fundamental part of the type system (sorry, not part of the type
system!) that I must have control over.

Why offer 'ref' if you don't intend people to use it?
I put to you, why do you hate 'ref' so much? Remove it from the
language if people aren't meant to use it.

ref is just a pointer with some semantics removed (pointer
re-assignment, indexing). It's use facilitates some forms of generic
code which would fail otherwise, and also reduces some possibilities
for end-user misuse or mistakes when using pointers (ie, user may
intend an assignment, but unsuspectingly re-assign a pointer; result
looks the same, but exposes bug somewhere else).

It's also used as an optimisation where I require control over the
ref-ness of things, but don't want to retrofit the code with a sea of
'*' and '&' operators.

It's also used when interfacing C++, where it appears in API's extensively.

I'm sure there are many more common and useful cases where ref is a good tool.



 My intended reply to this post was to ask you to justify making it a
 storage class, and why the design fails as a type constructor?
 Can we explore that direction to it's point of failure?

 As a storage class, it runs the risk of doubling out existing bloat
 caused by ref. As a type constructor, I see no disadvantages.
 It even addresses some of the awkward problems right at the face of
 storage classes, like how/where the attributes actually apply. Type
 constructors use parens; ie, const(T), and scope(T) would make that
 matter a lot  more clear.

 Apart from the storage class issue, it looks okay, but it gives me the
 feeling that it kinda stops short.
 Marc's proposal addressed more issues. I feel this proposal will
 result in more edge cases than Marc's proposal.
 The major edge case that I imagine is that since scope return values
 can't be assigned to scope local's, that will result in some awkward
 meta, requiring yet more special cases. I think Mark's proposal may be
 a lot more relaxed in that way.



 The disadvantages of making it a type qualifier are:

 1. far more complexity. Type constructors interact with everything, often in
 unanticipated ways. We spent *years* working out issues with the 'const'
 type qualifier, and are still doing so. Kenji just fixed another one.


What you describe is is the opposite of 'complexity'. Uniform
behaviour is what people expect from a language.
I understand complexity may arise as a result of interaction with
other features, but that's a worthwhile issue to explore if you ask
me. That's complexity that can actually be addressed, rather than
pushed to the side.
Edge cases which are orthogonal to the rest of the language (ie, ref),
are a terrible idea. You don't buffer against complexity by creating a
whole new class of complexity, which we have no effective tools to
manage or mitigate.



 2. we are never going to get users to use 'scope' qualifiers pervasively.
 It's been a long struggle to get 'const' used.


I don't think scope will be as important as const. I think we should
*try it*, to experimentally see how it plays out.
That said, scope should be able to be inferred quite effectively in
many most cases. Again, I think we will only be able to measure the
effectiveness of this when we try it.



 3. we added 'inout' as a type qualifier to avoid code duplication engendered
 by 'const'. It hurts my brain to even think about how that might interact
 with 'scope' qualifiers.


inout was an interesting idea (with a terrible name!). I'm still not
sure if I think it was a good idea or not, but I have found it very
useful.

What's the issue? I don't quite see how inout and scope overlap (no
differently than const or immutable?).



 Yes, I agree unequivocably, that 'scope' as a type qualifier is more
 expressive and more powerful than as a storage class. Multiple inheritance
 is also more expressive and more powerful than single inheritance. But many
 times, more power perhaps isn't better than redoing the program design to
 use something simpler and less complex.


I don't think that's a reasonable comparison. Multiple inheritance is
(I think quite well agreed) a wildly unpopular and super-complex
disaster.

Comparing scope-as-a-storage-class or scope-as-a-type-constructor is
nothing like comparing multiple inheritence and interfaces...
interfaces aren't orthogonal to the language for a start! And they're
well understood, and precedented in other languages.

Don't introduce red herrings which incite unrelated, yet strong
emotional response.

My argument is that scope as storage class is MORE COMPLEX in that it
is orthogonal to the language, than scope as type constructor. I think
users will also find it extremely unintuitive, when they realise that
all the usual tools for interacting with types in the language are
unavailable in this special case.
I think scope will also prove to be more popular than ref, so, while
you don't hear so much about how much of a disaster ref is, you'll
start to hear all the exact same problems arise when people are trying
to use scope, because it's a far more interesting type qualifier (and
should probably be the default).

...I can't wait for 'auto scope'! ;)
Dec 06 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/6/2014 4:49 PM, Manu via Digitalmars-d wrote:

 On 7 December 2014 at 08:15, Walter Bright via Digitalmars-d
 There's a lot more more to it, but perhaps I'm among few who have had
 such extensive need to engage in these tasks?


I don't know, but it's a mystery to me what you're doing that there's no 
reasonable alternative, or why this is such an extensive issue for you.



 This is incorrect, you had enormous influence over the vector type in D! And
 I wish you had more.


 I appreciate that. That was uncontroversial though; I didn't need to
 spend months or years trying to justify my claims on that issue. I
 feel like that was a known item that was just somewhere slightly down
 the list, and I was able to bring it forward.
 I was never in a position where I had to argue against Andrei to sell that one.


UDA was controversial, and was one of your initiatives.



 I have std.simd sitting here, and I really want to finish it, but I
 still don't have the tools to do so.
 I need, at least, forceinline to complete it, but that one *is*
 controversial - we've talked about this for years.


I proposed a DIP to fix that, but it could not get reasonable consensus.

   http://wiki.dlang.org/DIP56

You did, after all, convince me that we need an "always inline" and a "never 
inline" method.




 There is also a practical problem with GCC (perhaps it's an
 incompatbility with my std.simd design), where I need to change the
 sse-level between functions.
 There is a GCC attribute to do this ('target'), but it would rely on,
 at least, a few subtle tweaks. In this case, I need to be able to feed
 a template argument to a UDA, which doesn't work because UDA
 declarations seem to be parsed prior to knowledge of functions
 template args. There may be a further problem with GCC though which I
 haven't been able to prove yet though.


I don't know why this needs to be a deduced template argument.



 The reason for this is that an important design goal for my work was
 to be able to semi-automatically generate multiple code paths for
 different SSE versions, which can be selected at runtime based on
 available hardware.


No argument there.



 I need a little more language support. I've reached out a few times,
 but there hasn't been too much interest in the past. I really do wanna
 finish it one of these days though!


I want it done, too :-)



 I don't really know what you want. Well, perhaps a better statement is
 'why', not what.


 Because a function is a function. It's a fundamental and ultimately
 *simple* element of a language, with simple and reliable in behaviour;
 you write a function, compiler emits some code with a matching symbol
 name. There is perhaps nothing more simple or fundamental to the
 language.

 A template is not a function. It may *generate* a function, but now
 we've added many extra details which require careful consideration and
 handling, like where the code is, or isn't, emitted? How many
 permutations are there? What about static/dynamic libraries? Calling
 with various, or arbitrary types, always required careful
 consideration, more so than an explicit type.
 Templates are a powerful (often dangerous) **tool**, which should only
 be applied deliberately and very carefully.
 Obviously auto-ref gives hard answers to some of those questions, but
 it is still a template, and that instantly makes it much more complex.


I view a template function as a function that takes both compile-time and 
run-time arguments, not as a function generator. (The latter is an 
implementation artifact.)



 Making a function that has nothing to do with templates into a
 template is not what I want... it couldn't be further from what I ever
 wanted!


I don't see any problem with:

    T func()(runtime args ...) { ... }

That idiom has found various uses in practice.



 The ABI is different; there are 2 functions now (or zero, until one is
 called). Thich means taking the address of a function with auto-ref is
 a complex issue.
 ref-ness isn't a decision I ever want the compiler to make for me.


But why?


 I either want ref, or I don't, and I will state that to the compiler
 explicitly.
 This is especially true when interacting with other languages; while D
 remains in relative infancy, I think that is an overwhelmingly common
 situation when a project is modestly large.


The set of other languages that support ref parameters is [C++]. None others I 
know of that are accessible from D. Is it commonplace in C++ to have these 
overloads:

     T foo(S s);
     T foo(S& s);

? Such a practice would at least raise an eyebrow for me.



 In the situation where templates are involved, it would be nice to be
 able to make that explicit statement that some type is ref or not at
 the point of template instantiation, and the resolution should work
 according to the well-defined rules that we are all familiar with.
 Within a complex template, if logic should be performed, we have
 powerful tools to do that already; nobody ever complains about
 Unqual!, or PointerTarget!... these things are very easy to read and
 understand. ref should fit right in there with the others, is(x ==
 ref), UnRef!T, etc.


http://dlang.org/traits.html#isRef


 Instead, it's practically orthogonal to the language. Wrangling
 anything involving storage classes is really, really tedious, and
 almost always results in extensive code duplication, or text mixins
 when duplication just get too much.

 This issue really surprised me at the time, because the whole thing
 was in response to a topic that I consistently raised and pushed.
 Who was the customer welcoming that solution? I could never work it
 out; there was nobody else that seemed particularly invested in the
 problem, except the usual suite of language enthusiasts that took it
 as an interesting intellectual problem to discuss. The person who
 cared about that issue the most (afaict) was completely unsatisfied
 with the solution. I said at the time that I would have preferred that
 no action was taken, than to compound, and further set in stone, an
 issue that I was already extremely critical of.


 Don't double down on this mistake with scope!


The thing is, I don't understand *why* you want to wrangle storage classes.
What 
is the coding pattern?



 Why does your code need to care so much about to ref or not to ref? That's
 the central point here, I think.


 That same logic could ask why I want to have 'short', or 'int' in some
 places, but not others. There's no absolute answer. As a software
 engineer, and particularly, a native software engineer, it's a
 fundamental part of the type system (sorry, not part of the type
 system!) that I must have control over.


I can give practical technical reasons why for 'short' or 'int'.



 Why offer 'ref' if you don't intend people to use it?
 I put to you, why do you hate 'ref' so much? Remove it from the
 language if people aren't meant to use it.


That's the wrong question. I could ask "why do you hate ref as a storage class"?



 ref is just a pointer with some semantics removed (pointer
 re-assignment, indexing). It's use facilitates some forms of generic
 code which would fail otherwise, and also reduces some possibilities
 for end-user misuse or mistakes when using pointers (ie, user may
 intend an assignment, but unsuspectingly re-assign a pointer; result
 looks the same, but exposes bug somewhere else).

 It's also used as an optimisation where I require control over the
 ref-ness of things, but don't want to retrofit the code with a sea of
 '*' and '&' operators.

 It's also used when interfacing C++, where it appears in API's extensively.

 I'm sure there are many more common and useful cases where ref is a good tool.


And D's ref works for all that.



 The disadvantages of making it a type qualifier are:

 1. far more complexity. Type constructors interact with everything, often in
 unanticipated ways. We spent *years* working out issues with the 'const'
 type qualifier, and are still doing so. Kenji just fixed another one.


 What you describe is is the opposite of 'complexity'. Uniform
 behaviour is what people expect from a language.
 I understand complexity may arise as a result of interaction with
 other features, but that's a worthwhile issue to explore if you ask
 me. That's complexity that can actually be addressed, rather than
 pushed to the side.
 Edge cases which are orthogonal to the rest of the language (ie, ref),
 are a terrible idea. You don't buffer against complexity by creating a
 whole new class of complexity, which we have no effective tools to
 manage or mitigate.


Uniform behavior is a nice ideal, and sounds good, but it never happens in 
practice with programming languages. We can't even get 'int' to behave 
uniformly! (Quick: what is -int.min ?)

You might also consider the "uniformity" of the ref type in C++. It's awful - 
it's a special case EVERYWHERE in the C++ type system! It just does not fit as
a 
type qualifier.



 2. we are never going to get users to use 'scope' qualifiers pervasively.
 It's been a long struggle to get 'const' used.


 I don't think scope will be as important as const. I think we should
 *try it*, to experimentally see how it plays out.
 That said, scope should be able to be inferred quite effectively in
 many most cases. Again, I think we will only be able to measure the
 effectiveness of this when we try it.


'scope' is so important we may even consider it to be the default.



 3. we added 'inout' as a type qualifier to avoid code duplication engendered
 by 'const'. It hurts my brain to even think about how that might interact
 with 'scope' qualifiers.


 inout was an interesting idea (with a terrible name!). I'm still not
 sure if I think it was a good idea or not, but I have found it very
 useful.

 What's the issue? I don't quite see how inout and scope overlap (no
 differently than const or immutable?).


Every type will have scope and nonscope variants, combinatorialy with the other 
type qualifiers.



 Yes, I agree unequivocably, that 'scope' as a type qualifier is more
 expressive and more powerful than as a storage class. Multiple inheritance
 is also more expressive and more powerful than single inheritance. But many
 times, more power perhaps isn't better than redoing the program design to
 use something simpler and less complex.


 I don't think that's a reasonable comparison. Multiple inheritance is
 (I think quite well agreed) a wildly unpopular and super-complex
 disaster.

 Comparing scope-as-a-storage-class or scope-as-a-type-constructor is
 nothing like comparing multiple inheritence and interfaces...
 interfaces aren't orthogonal to the language for a start! And they're
 well understood, and precedented in other languages.

 Don't introduce red herrings which incite unrelated, yet strong
 emotional response.


Nobody has successfully introduced a type qualifier like scope.



 My argument is that scope as storage class is MORE COMPLEX in that it
 is orthogonal to the language, than scope as type constructor. I think
 users will also find it extremely unintuitive, when they realise that
 all the usual tools for interacting with types in the language are
 unavailable in this special case.
 I think scope will also prove to be more popular than ref, so, while
 you don't hear so much about how much of a disaster ref is, you'll
 start to hear all the exact same problems arise when people are trying
 to use scope, because it's a far more interesting type qualifier (and
 should probably be the default).

 ...I can't wait for 'auto scope'! ;)


I guess we'll see! I agree that this design is untried, and we don't really
know 
how it will work out.
Dec 06 2014
next sibling parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Walter Bright"  wrote in message news:m60oa4$kd5$1 digitalmars.com...


 I have std.simd sitting here, and I really want to finish it, but I
 still don't have the tools to do so.
 I need, at least, forceinline to complete it, but that one *is*
 controversial - we've talked about this for years.


 I proposed a DIP to fix that, but it could not get reasonable consensus.

    http://wiki.dlang.org/DIP56

 You did, after all, convince me that we need an "always inline" and a 
 "never inline" method.


From the DIP:


 If a pragma specifies always inline, whether or not the target function(s) 
 are actually inlined is
 implementation defined, although the implementation will be expected to 
 inline it if practical.
 Implementations will likely vary in their ability to inline.


I expect a proposal in which 'always' means "inline or give an error" would 
be much better received.


 The thing is, I don't understand *why* you want to wrangle storage 
 classes. What is the coding pattern?



 That's the wrong question. I could ask "why do you hate ref as a storage 
 class"?


I suspect the answer is that sometimes it is useful to ensure some 
parameters are passed by reference, not by value.  With ref as a type, this 
would be trivial:

template MyParamType(T)
{
    static if (someCriteria!T)
        alias MyParamType = ref(T);
    else
        alias MyParamType = T;
}

void myFunction(T, U, V)(const MyParamType!T, const MyParamType!U, const 
MyParamType!V)
{
    ...
}
Dec 06 2014
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 7 December 2014 at 05:24:20 UTC, Walter Bright wrote:

 I appreciate that. That was uncontroversial though; I didn't 
 need to
 spend months or years trying to justify my claims on that 
 issue. I
 feel like that was a known item that was just somewhere 
 slightly down
 the list, and I was able to bring it forward.
 I was never in a position where I had to argue against Andrei 
 to sell that one.


 UDA was controversial, and was one of your initiatives.


I don't think UDA is controversial, but the way it was done
certainly is controverted.
Dec 08 2014
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-12-07 01:49, Manu via Digitalmars-d wrote:


 I have std.simd sitting here, and I really want to finish it, but I
 still don't have the tools to do so.
 I need, at least, forceinline to complete it, but that one *is*
 controversial - we've talked about this for years.

 GDC and LDC both have a forceinline, so I could theoretically support
 those compilers, but then I can't practically make use of them without
 some sort of attribute aliasing system, otherwise I need to triplicate
 the code for each compiler, just to insert a different (compiler
 specific) forceinline attribute name. It'd be really great if we
 agreed on just one.


I don't know about LDC but at least GDC allows you to use UDA's instead 
of a pragma. Then you can create a dummy attribute for DMD (and LDC):

version (GNU)
     import gcc.attribute

else
{
     struct attribute
     {
         string attr;
     }
}

 attribute("forceinline") void foo ();

-- 
/Jacob Carlborg
Dec 07 2014
parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 7 Dec 2014 10:40, "Jacob Carlborg via Digitalmars-d" <
digitalmars-d puremagic.com> wrote:

 On 2014-12-07 01:49, Manu via Digitalmars-d wrote:


 I have std.simd sitting here, and I really want to finish it, but I
 still don't have the tools to do so.
 I need, at least, forceinline to complete it, but that one *is*
 controversial - we've talked about this for years.

 GDC and LDC both have a forceinline, so I could theoretically support
 those compilers, but then I can't practically make use of them without
 some sort of attribute aliasing system, otherwise I need to triplicate
 the code for each compiler, just to insert a different (compiler
 specific) forceinline attribute name. It'd be really great if we
 agreed on just one.



 I don't know about LDC but at least GDC allows you to use UDA's instead


of a pragma. Then you can create a dummy attribute for DMD (and LDC):

 version (GNU)
     import gcc.attribute

 else
 {
     struct attribute
     {
         string attr;
     }
 }

  attribute("forceinline") void foo ();


You can add shorthand aliases for them too. :)

 forceinline void foo ();
Dec 07 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-12-07 11:50, Iain Buclaw via Digitalmars-d wrote:


 You can add shorthand aliases for them too. :)

  forceinline void foo ();


Good point.

-- 
/Jacob Carlborg
Dec 07 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/6/14 4:49 PM, Manu via Digitalmars-d wrote:

 I need, at least, forceinline to complete it, but that one*is*
 controversial - we've talked about this for years.


I'm still 883 messages behind so here's a drive-by comment - it's time 
to revisit this, I think the need has become a lot clearer. -- Andrei
Dec 20 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/6/14 4:49 PM, Manu via Digitalmars-d wrote:

 In the situation where templates are involved, it would be nice to be
 able to make that explicit statement that some type is ref or not at
 the point of template instantiation, and the resolution should work
 according to the well-defined rules that we are all familiar with.


Another drive-by comment: I understand the motivation for this and the 
difficulties involved. There needs to be a clear understanding that 
adding new type qualifiers is extremely intrusive and expensive. Because 
of that, I think we should best address binding generation via a set of 
tactical tools i.e. standard library artifacts that do all that mixin 
business in an encapsulated and reusable manner.

(As an aside forcing a template instantiation to decide ref vs. no ref 
should be easy but currently can't be done, i.e. this code should work 
but currently doesn't:

T fun(T)(ref T x) { return x + 1; }

void main(string[] group)
{
     int function(int) f1 = &fun!int;
     int function(ref int) f2 = &fun!int;
}

)


Andrei
Dec 20 2014
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 5 December 2014 at 23:58:41 UTC, Walter Bright wrote:

 On 12/5/2014 8:48 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:

 1) Escape detection is limited to `ref`.

     T* evil;
     ref T func(scope ref T t, ref T u)  safe {
       return t; // Error: escaping scope ref t
       return u; // ok
       evil = &u; // Error: escaping reference
     }

 vs.

     T[] evil;
     T[] func(scope T[] t, T[] u)  safe {
       return t; // Error: cannot return scope
       return u; // ok
       evil = u; // !!! not good


 right, although:
      evil = t;  // Error: not allowed


     }

 As can be seen, `ref T u` is protected from escaping (apart 
 from returning it),
 while `T[] u` in the second example is not. There's no general 
 way to express
 that `u` can only be returned from the function, but will not 
 be retained
 otherwise by storing it in a global variable. Adding `pure` 
 can express this in
 many cases, but is, of course, not always possible.


 As you point out, 'ref' is designed for this.


I wouldn't call it "designed", but "repurposed"...


     struct Container(T) {
         scope ref T opIndex(size_t index);
     }

     void bar(scope ref int a);

     Container c;
     bar(c[42]);            // ok
     scope ref tmp = c[42]; // nope

 Both cases should be fine theoretically; the "real" owner 
 lives longer than
 `tmp`. Unfortunately the compiler doesn't know about this.


 Right, though the compiler can optimize to produce the 
 equivalent.


??? This is a problem on the semantic level, unrelated to 
optimization:

     // contrived example to illustrated the point
     Container c;
     scope ref x = c[42];   // not
     scope ref y = c[44];   // ...
     scope ref z = c[13];   // allowed
     foo(x, y, z, x+y, y+z, z+x, x+y+z);

     // workaround, but error-prone and has different semantics
     // (opIndex may have side effects, called multiple times)
     foo(c[42], c[44], c[13], c[42]+c[44], c[44]+c[13], 
c[13]+c[42], c[42]+c[44]+c[13]);

     // another workaround, same semantics, but ugly and unreadable
     (scope ref int x, scope ref int y, scope ref int z) {
         foo(x, y, z, x+y, y+z, z+x, x+y+z);
     }(c[42], c[44], c[13]);


 3) `scope` cannot be used for value types.

 I can think of a few use cases for scoped value types (RC and 
 file descriptors),
 but they might only be marginal.


 I suspect that one can encapsulate such values in a struct 
 where access to them is strictly controlled.


They can, but again at the cost of an indirection (ref).


 4) No overloading on `scope`.

 This is at least partially a consequence of `scope` inference. 
 I think
 overloading can be made to work in the presence of inference, 
 but I haven't
 thought it through.


 Right. Different overloads can have different semantic 
 implementations, so what should inference do? I also suspect it 
 is bad style to overload on 'scope'.


It only makes sense with scope value types (see the RC example). 
For references, I don't know any useful applications.


 6) There seem to be problems with chaining.

     scope ref int foo();
     scope ref int bar1(ref int a) {
         return a;
     }
     scope ref int bar2(scope ref int a) {
         return a;
     }
     ref int bar3(ref int a) {
         return a;
     }
     ref int bar4(scope ref int a) {
         return a;
     }
     void baz(scope ref int a);

 Which of the following calls would work?

     foo().bar1().baz();


 yes


     foo().bar2().baz();


 no - cannot return scope ref parameter


     foo().bar3().baz();


 yes


     foo().bar4().baz();


 no, cannot return scope ref parameter



 I'm not sure I understand this fully yet, but it could be that 
 none of them work...


 Well, you're half right :-)


Ok, so let's drop bar2() and bar4().

     scope ref int foo();
     scope ref int bar1(ref int a) {
         return a;
     }
     ref int bar3(ref int a) {
         return a;
     }
     ref int baz_noscope(/*scope*/ ref int a);

     foo().bar1().baz_noscope();
     foo().bar3().baz_noscope();

And now? In particular, will the return value of `bar3` be 
treated as if it were `scope ref`?
Dec 06 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/6/2014 5:30 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:

 On Friday, 5 December 2014 at 23:58:41 UTC, Walter Bright wrote:

 As you point out, 'ref' is designed for this.


 I wouldn't call it "designed", but "repurposed"...


Perhaps, but the original reason to even have 'ref' was so it could be a 
restricted pointer type that could be passed down a call hierarchy, but not up.



     struct Container(T) {
         scope ref T opIndex(size_t index);
     }

     void bar(scope ref int a);

     Container c;
     bar(c[42]);            // ok
     scope ref tmp = c[42]; // nope

 Both cases should be fine theoretically; the "real" owner lives longer than
 `tmp`. Unfortunately the compiler doesn't know about this.


 Right, though the compiler can optimize to produce the equivalent.


 ??? This is a problem on the semantic level, unrelated to optimization:

      // contrived example to illustrated the point
      Container c;
      scope ref x = c[42];   // not
      scope ref y = c[44];   // ...
      scope ref z = c[13];   // allowed
      foo(x, y, z, x+y, y+z, z+x, x+y+z);

      // workaround, but error-prone and has different semantics
      // (opIndex may have side effects, called multiple times)
      foo(c[42], c[44], c[13], c[42]+c[44], c[44]+c[13], c[13]+c[42],
 c[42]+c[44]+c[13]);

      // another workaround, same semantics, but ugly and unreadable
      (scope ref int x, scope ref int y, scope ref int z) {
          foo(x, y, z, x+y, y+z, z+x, x+y+z);
      }(c[42], c[44], c[13]);


You are correct, and it remains to be seen if these occur enough to be a
problem 
or not. The workarounds do exist, though. Another workaround:

     scope ref tmp = c[42];

becomes:

     auto scope ref tmp() { return c[42]; }



 Ok, so let's drop bar2() and bar4().

      scope ref int foo();
      scope ref int bar1(ref int a) {
          return a;
      }
      ref int bar3(ref int a) {
          return a;
      }
      ref int baz_noscope(/*scope*/ ref int a);

      foo().bar1().baz_noscope();
      foo().bar3().baz_noscope();

 And now? In particular, will the return value of `bar3` be treated as if it
were
 `scope ref`?


Yes.
Dec 06 2014
prev sibling next sibling parent reply "Sebastiaan Koppe" <mail skoppe.eu> writes:
On Friday, 5 December 2014 at 23:58:41 UTC, Walter Bright wrote:

 On 12/5/2014 8:48 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:

     scope ref int foo();
     scope ref int bar1(ref int a) {
         return a;
     }
     scope ref int bar2(scope ref int a) {
         return a;
     }
     ref int bar3(ref int a) {
         return a;
     }
     ref int bar4(scope ref int a) {
         return a;
     }
     void baz(scope ref int a);

 Which of the following calls would work?

     foo().bar1().baz();


 yes


     foo().bar2().baz();


 no - cannot return scope ref parameter


     foo().bar3().baz();


 yes


     foo().bar4().baz();


 no, cannot return scope ref parameter


I understand that scope will not allow the contents of the 
variable to escape the lifetime of a declaration. But can you 
explain why bar1() works, but bar2() doesn't? Isn't the body of 
bar2() in the line `foo().bar2();` part of the declaration?

Besides, what does it mean to return a `scope ref int`? Does it 
mean that the content of the variable that is returned is not 
allowed to escape the scope of the calling site? Huh?

It seemed so easy when you gave the example.

On Friday, 5 December 2014 at 20:55:55 UTC, Walter Bright wrote:

 It means that this code will be safe:

    void foo(scope int* p);

    p = malloc(n);
    foo(p);
    free(p);

 The rest is all the nuts and bolts of making that work.
Dec 07 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/7/2014 2:46 AM, Sebastiaan Koppe wrote:

 On Friday, 5 December 2014 at 23:58:41 UTC, Walter Bright wrote:

 On 12/5/2014 8:48 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:

     scope ref int foo();
     scope ref int bar1(ref int a) {
         return a;
     }
     scope ref int bar2(scope ref int a) {
         return a;
     }
     ref int bar3(ref int a) {
         return a;
     }
     ref int bar4(scope ref int a) {
         return a;
     }
     void baz(scope ref int a);

 Which of the following calls would work?

     foo().bar1().baz();


 yes


     foo().bar2().baz();


 no - cannot return scope ref parameter


     foo().bar3().baz();


 yes


     foo().bar4().baz();


 no, cannot return scope ref parameter


 I understand that scope will not allow the contents of the variable to escape
 the lifetime of a declaration. But can you explain why bar1() works, but bar2()
 doesn't?


A 'scope ref' parameter may not be returned as a 'ref' or a 'scope ref'.


 Isn't the body of bar2() in the line `foo().bar2();` part of the
 declaration?

 Besides, what does it mean to return a `scope ref int`? Does it mean that the
 content of the variable that is returned is not allowed to escape the scope of
 the calling site? Huh?


It means the reference itself (the pointer) does not escape.
Dec 08 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Monday, 8 December 2014 at 21:16:36 UTC, Walter Bright wrote:

 A 'scope ref' parameter may not be returned as a 'ref' or a 
 'scope ref'.


It can safely be returned if you consider its lifetime as the
intersection of the lifetime of the function's parameter.
Dec 08 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2014 3:21 PM, deadalnix wrote:

 On Monday, 8 December 2014 at 21:16:36 UTC, Walter Bright wrote:

 A 'scope ref' parameter may not be returned as a 'ref' or a 'scope ref'.


 It can safely be returned if you consider its lifetime as the
 intersection of the lifetime of the function's parameter.


The only purpose to a 'scope ref' parameter is to say it isn't being returned. 
'ref' itself does not escape in any way other than by returning.
Dec 09 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 10 December 2014 at 05:23:29 UTC, Walter Bright 
wrote:

 On 12/8/2014 3:21 PM, deadalnix wrote:

 On Monday, 8 December 2014 at 21:16:36 UTC, Walter Bright 
 wrote:

 A 'scope ref' parameter may not be returned as a 'ref' or a 
 'scope ref'.


 It can safely be returned if you consider its lifetime as the
 intersection of the lifetime of the function's parameter.


 The only purpose to a 'scope ref' parameter is to say it isn't 
 being returned. 'ref' itself does not escape in any way other 
 than by returning.


That is a completely useless feature. Also, you want to have 
scope return for container like thing.
Dec 09 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/9/2014 10:43 PM, deadalnix wrote:

 On Wednesday, 10 December 2014 at 05:23:29 UTC, Walter Bright wrote:

 On 12/8/2014 3:21 PM, deadalnix wrote:

 On Monday, 8 December 2014 at 21:16:36 UTC, Walter Bright wrote:

 A 'scope ref' parameter may not be returned as a 'ref' or a 'scope ref'.


 It can safely be returned if you consider its lifetime as the
 intersection of the lifetime of the function's parameter.


 The only purpose to a 'scope ref' parameter is to say it isn't being returned.
 'ref' itself does not escape in any way other than by returning.


 That is a completely useless feature. Also, you want to have scope return for
 container like thing.


I disagree. It's critical for chaining one function to the next.
Dec 10 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 11 December 2014 at 03:27:08 UTC, Walter Bright 
wrote:

 I disagree. It's critical for chaining one function to the next.


I one can't return, one can't chain.
Dec 10 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/10/2014 8:56 PM, deadalnix wrote:

 On Thursday, 11 December 2014 at 03:27:08 UTC, Walter Bright wrote:

 I disagree. It's critical for chaining one function to the next.


 I one can't return, one can't chain.


I guess I'm not seeing the problem.
Dec 10 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 11 December 2014 at 07:30:03 UTC, Walter Bright 
wrote:

 On 12/10/2014 8:56 PM, deadalnix wrote:

 On Thursday, 11 December 2014 at 03:27:08 UTC, Walter Bright 
 wrote:

 I disagree. It's critical for chaining one function to the 
 next.


 I one can't return, one can't chain.


 I guess I'm not seeing the problem.


a.foo().bar() is the same as bar(foo(a)), that is, using the 
returned value as first parameter. If you can't return, you can't 
chain.
Dec 10 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/10/2014 11:55 PM, deadalnix wrote:

 On Thursday, 11 December 2014 at 07:30:03 UTC, Walter Bright wrote:

 On 12/10/2014 8:56 PM, deadalnix wrote:

 On Thursday, 11 December 2014 at 03:27:08 UTC, Walter Bright wrote:

 I disagree. It's critical for chaining one function to the next.


 I one can't return, one can't chain.


 I guess I'm not seeing the problem.


 a.foo().bar() is the same as bar(foo(a)), that is, using the returned value as
 first parameter. If you can't return, you can't chain.


Right, and you can chain when the declarations are:

scope ref S foo(ref T a);
U bar(ref S s);

a.foo().bar(); // works
Dec 11 2014
prev sibling parent Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 05/12/2014 23:58, Walter Bright wrote:

 2) `scope ref` return values cannot be stored.

      scope ref int foo();
      void bar(scope ref int a);

      foo().bar();        // allowed
      scope tmp = foo();  // not allowed
      tmp.bar();


 Right


 From the DIP:

"The lifetime of a scope return value is the lifetime of an rvalue. It 
may not be copied in a way that extends its life."

With part of the example:

scope int* foo();
...
int* p = foo();       // Error, lifetime(p) is &infin;

Maybe the error should be 'scope return value cannot be stored', because 
otherwise p could be inferred as scope.
Dec 07 2014
prev sibling parent reply "Daniel N" <ufo orbiting.us> writes:
On Friday, 5 December 2014 at 16:48:45 UTC, Marc Schütz wrote:

 There are limitations this proposal has in comparison to my 
 original one. These limitations might of course be harmless and 
 play no role in practice, but on the other hand, they may, so I 
 think it's good to list them here.


One concern I had with your proposal was that it refers to a 
symbol before it's available.

scope!haystack(string) findSubstring(scope(string) haystack, 
scope(string) needle)

C++11 had similar issues and they solved it by introducing 
trailing return types.

ex:
"template<class T>
auto mul(T a, T b) -> decltype(a*b)"

Personally I would prefer not to go down that lane and DIP69 
avoids that problem.
Dec 06 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 6 December 2014 at 10:59:24 UTC, Daniel N wrote:

 On Friday, 5 December 2014 at 16:48:45 UTC, Marc Schütz wrote:

 There are limitations this proposal has in comparison to my 
 original one. These limitations might of course be harmless 
 and play no role in practice, but on the other hand, they may, 
 so I think it's good to list them here.


 One concern I had with your proposal was that it refers to a 
 symbol before it's available.

 scope!haystack(string) findSubstring(scope(string) haystack, 
 scope(string) needle)

 C++11 had similar issues and they solved it by introducing 
 trailing return types.

 ex:
 "template<class T>
 auto mul(T a, T b) -> decltype(a*b)"

 Personally I would prefer not to go down that lane and DIP69 
 avoids that problem.


For D, this wouldn't be necessary, because parsing and semantic 
analysis are strictly separated. The owners would only have to be 
evaluated very late during the semantic phase.

But let's see how DIP69 works out...
Dec 06 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:

 http://wiki.dlang.org/DIP69

 Despite its length, this is a fairly simple proposal. It adds 
 the missing semantics for the 'scope' storage class in order to 
 make it possible to pass a reference to a function without it 
 being possible for it to escape.

 This, among other things, makes a ref counting type practical. 
 It also makes it more practical to use other storage allocation 
 schemes than garbage collection.

 It does not make scope into a type constructor, nor a general 
 type-annotation system.

 It does not provide an ownership system, though it would 
 complement one.


Thanks a lot of trying to move forward with this. Also glad to 
see DIP36 didn't vanish completely useless :P It will take me 
some time to provide a detailed response so I'll post a summary 
of my impression first.

I recognize and respect your attempt to go for most simple 
solution that is still useful in practice. Can't say I am happy 
about it but it is better to have something working than awesome 
plans that never get implemented. In this context trying to get 
most of scope as storage class seems right.

But from existing cases it doesn't seem working good enough. For 
example, not being able to represent idiom of `scope ref int 
foo(scope ref int x) { return x; }` seems very limiting. There 
are also issues that pop up because of missing transitivity. 
Maybe this can be fixed within existing proposal, maybe not. 
Right now I don't have any strong opinion.

I also don't consider `ref` design as a storage class any kind of 
success at all and generally agree with Manu on this topic. At 
the same time alternative proposals that make it a qualifier 
(like Marc did) do impact existing language much more and this no 
small concern.

This won't be easy.
Dec 07 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/7/2014 6:12 AM, Dicebot wrote:

 But from existing cases it doesn't seem working good enough. For example, not
 being able to represent idiom of `scope ref int foo(scope ref int x) { return
x;
 }` seems very limiting.


   scope ref int foo(ref int x);

will do it.



 I also don't consider `ref` design as a storage class any kind of success at
all
 and generally agree with Manu on this topic. At the same time alternative
 proposals that make it a qualifier (like Marc did) do impact existing language
 much more and this no small concern.


My experience with C++ ref as type qualifier is very, very bad. It's a special 
case EVERYWHERE. Doing type deduction with it is an exercise in a completely 
baffling set of rules and a different rule for every occasion - Scott Meyers
has 
a great piece on this.

There are probably only a handful of people on the planet who actually 
understand C++ ref. I wished very hard to avoid that with D ref.
Dec 07 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:


 There are probably only a handful of people on the planet who 
 actually understand C++ ref. I wished very hard to avoid that 
 with D ref.


When C++ programmers say that D-style ranges can't do everything 
C++ iterators can do, they seem to miss that sometimes it's a 
good idea to adopt a simpler language feature, that doesn't cover 
100% usages, if it covers 80-90% of the cases, and has a simpler 
syntax, and simpler semantics to understand for the programmer.

(The comment above is not about DIP69).

Bye,
bearophile
Dec 07 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/7/2014 2:58 PM, bearophile wrote:

 When C++ programmers say that D-style ranges can't do everything C++ iterators
 can do, they seem to miss that sometimes it's a good idea to adopt a simpler
 language feature, that doesn't cover 100% usages, if it covers 80-90% of the
 cases, and has a simpler syntax, and simpler semantics to understand for the
 programmer.


I agree, but it's hard to find that sweet spot. I think Java definitely went
too 
far, and Go went too far for my taste.



 (The comment above is not about DIP69).


Yes, it is :-)
Dec 07 2014
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/7/14 4:29 PM, Walter Bright wrote:

 On 12/7/2014 6:12 AM, Dicebot wrote:

 But from existing cases it doesn't seem working good enough. For
 example, not
 being able to represent idiom of `scope ref int foo(scope ref int x) {
 return x;
 }` seems very limiting.


    scope ref int foo(ref int x);

 will do it.


So:

int x;

foo(x) += 1;

will compile? I was under the impression this would be disallowed.

If you do not connect the scope to the parameter, then the caller has to 
assume that variable can escape, and you can't use scoped variables as 
arguments at all.


 My experience with C++ ref as type qualifier is very, very bad. It's a
 special case EVERYWHERE. Doing type deduction with it is an exercise in
 a completely baffling set of rules and a different rule for every
 occasion - Scott Meyers has a great piece on this.

 There are probably only a handful of people on the planet who actually
 understand C++ ref. I wished very hard to avoid that with D ref.


D has so many features that did not exist when ref was created (as inout 
in D1), that you can ALMOST duplicate ref. The only thing we could not 
duplicate is the implicit address-taking on construction, which maybe is 
not such a bad thing.

-Steve
Dec 08 2014
parent reply Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 08/12/2014 15:53, Steven Schveighoffer wrote:

    scope ref int foo(ref int x);

 will do it.


 So:

 int x;

 foo(x) += 1;

 will compile?


Yes, because foo's argument is not scope, it can be returned.
Dec 09 2014
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/9/14 9:23 AM, Nick Treleaven wrote:

 On 08/12/2014 15:53, Steven Schveighoffer wrote:

    scope ref int foo(ref int x);

 will do it.


 So:

 int x;

 foo(x) += 1;

 will compile?


 Yes, because foo's argument is not scope, it can be returned.


But I thought if you take a reference from the stack, it's inferred as 
scope? I feel like there's a missing link somewhere if the scope is 
missing from the parameter.

Will ref just automatically bind to any scoped reference?

What about this:

auto y = &x;

foo(*y) += 1;

-Steve
Dec 09 2014
next sibling parent Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 09/12/2014 16:25, Steven Schveighoffer wrote:

 But I thought if you take a reference from the stack, it's inferred as
 scope? I feel like there's a missing link somewhere if the scope is
 missing from the parameter.


I think (with this DIP) values on the stack can safely be passed as a 
ref parameter, which is a bit more flexible than a scope parameter.


 Will ref just automatically bind to any scoped reference?

 What about this:


(for reference:)
int x;
scope ref int foo(ref int x);


 auto y = &x;

 foo(*y) += 1;


y gets inferred as scope, but *y is not scope, so it should work.
Dec 09 2014
prev sibling parent reply Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 09/12/2014 16:25, Steven Schveighoffer wrote:

 Will ref just automatically bind to any scoped reference?


A ref parameter essentially is a scope ref parameter, but it can also be 
returned:
http://forum.dlang.org/post/m64v3g$2bga$1 digitalmars.com
Dec 11 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 11 December 2014 at 14:12:16 UTC, Nick Treleaven 
wrote:

 On 09/12/2014 16:25, Steven Schveighoffer wrote:

 Will ref just automatically bind to any scoped reference?


 A ref parameter essentially is a scope ref parameter, but it 
 can also be returned:
 http://forum.dlang.org/post/m64v3g$2bga$1 digitalmars.com


Introducing a keyword to be able to add arbitrary restriction 
that do not make the code safer, more optimization or anything is 
not paying for itself.
Dec 11 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Sunday, 7 December 2014 at 21:29:50 UTC, Walter Bright wrote:

 On 12/7/2014 6:12 AM, Dicebot wrote:

 But from existing cases it doesn't seem working good enough. 
 For example, not
 being able to represent idiom of `scope ref int foo(scope ref 
 int x) { return x;
 }` seems very limiting.


   scope ref int foo(ref int x);

 will do it.


This isn't the same as it does not propagate scope but just 
restricts return value. Difference is that it cannot be chained. 
Let's consider practical example based on Phobos:

there was an issue with byLine range that it has reused same 
buffer internally which sometimes caught users off guard when 
trying to save slice. It is a natural fit for `scope` - make it 
return `scope string` instead to ensure that no slices get stored.

Two issues immediately pop up:

1) scope is not transitive thus it doesn't work at all - you 
still can store slice of `scope string` as only actual ptr+length 
struct is protected.

2) even if it worked, existing definition of scope return value 
makes it impossible to use in typical idiomatic pipeline: 
`file.byLine.algo1.algo2`. Either algoX is defined to take `scope 
ref` and thus can't return it or it is defined to take `ref` and 
can't take another `scope ref` as an argument.

At least this is what I get from reading existing examples in 
DIP69


 I also don't consider `ref` design as a storage class any kind 
 of success at all
 and generally agree with Manu on this topic. At the same time 
 alternative
 proposals that make it a qualifier (like Marc did) do impact 
 existing language
 much more and this no small concern.


 My experience with C++ ref as type qualifier is very, very bad. 
 It's a special case EVERYWHERE. Doing type deduction with it is 
 an exercise in a completely baffling set of rules and a 
 different rule for every occasion - Scott Meyers has a great 
 piece on this.

 There are probably only a handful of people on the planet who 
 actually understand C++ ref. I wished very hard to avoid that 
 with D ref.


While there is no argument that C++ ref is screwed, it is rather 
hard to say if this is inherent consequence of ref being a type 
qualifier or just C++ being C++. I mean how many C++ type system 
features in general are understood my more than a handful of 
people on the planet? For me `ref` is essentially just a 
different flavor of `*` - and if the latter can be part of type, 
I see no reasons why former can't
Dec 08 2014
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 8 December 2014 at 16:25:22 UTC, Dicebot wrote:

 On Sunday, 7 December 2014 at 21:29:50 UTC, Walter Bright wrote:

 On 12/7/2014 6:12 AM, Dicebot wrote:

 But from existing cases it doesn't seem working good enough. 
 For example, not
 being able to represent idiom of `scope ref int foo(scope ref 
 int x) { return x;
 }` seems very limiting.


  scope ref int foo(ref int x);

 will do it.


 This isn't the same as it does not propagate scope but just 
 restricts return value. Difference is that it cannot be 
 chained. Let's consider practical example based on Phobos:

 there was an issue with byLine range that it has reused same 
 buffer internally which sometimes caught users off guard when 
 trying to save slice. It is a natural fit for `scope` - make it 
 return `scope string` instead to ensure that no slices get 
 stored.

 Two issues immediately pop up:

 1) scope is not transitive thus it doesn't work at all - you 
 still can store slice of `scope string` as only actual 
 ptr+length struct is protected.

 2) even if it worked, existing definition of scope return value 
 makes it impossible to use in typical idiomatic pipeline: 
 `file.byLine.algo1.algo2`. Either algoX is defined to take 
 `scope ref` and thus can't return it or it is defined to take 
 `ref` and can't take another `scope ref` as an argument.


That's why I asked the question in 
http://forum.dlang.org/post/xdjsmwocbtxovjnathek forum.dlang.org 
. It seems Walter wants to allow passing `scope ref` to `ref`, 
but then automatically treat a normal `ref` of the second 
function as if it were `scope ref`. But I can't quite see through 
it :-(


 At least this is what I get from reading existing examples in 
 DIP69


 I also don't consider `ref` design as a storage class any 
 kind of success at all
 and generally agree with Manu on this topic. At the same time 
 alternative
 proposals that make it a qualifier (like Marc did) do impact 
 existing language
 much more and this no small concern.


 My experience with C++ ref as type qualifier is very, very 
 bad. It's a special case EVERYWHERE. Doing type deduction with 
 it is an exercise in a completely baffling set of rules and a 
 different rule for every occasion - Scott Meyers has a great 
 piece on this.

 There are probably only a handful of people on the planet who 
 actually understand C++ ref. I wished very hard to avoid that 
 with D ref.


 While there is no argument that C++ ref is screwed, it is 
 rather hard to say if this is inherent consequence of ref being 
 a type qualifier or just C++ being C++. I mean how many C++ 
 type system features in general are understood my more than a 
 handful of people on the planet? For me `ref` is essentially 
 just a different flavor of `*` - and if the latter can be part 
 of type, I see no reasons why former can't


I think most problems would come from type deduction. All the 
other qualifiers and function attributes, like const and pure, 
just either work, or they don't, so they can be tested easily, 
and you want to apply these qualifiers wherever possible. This 
also applies to scope. But for `ref`, it's presence influences 
behaviour, therefore I think it would be best if it were never 
inferred implicitly. Actually that too is just what we do with 
pointers. We don't change non-pointer types to pointers 
automatically.
Dec 08 2014
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 8 December 2014 at 16:57:44 UTC, Marc Schütz wrote:

 Two issues immediately pop up:

 1) scope is not transitive thus it doesn't work at all - you 
 still can store slice of `scope string` as only actual 
 ptr+length struct is protected.

 2) even if it worked, existing definition of scope return 
 value makes it impossible to use in typical idiomatic 
 pipeline: `file.byLine.algo1.algo2`. Either algoX is defined 
 to take `scope ref` and thus can't return it or it is defined 
 to take `ref` and can't take another `scope ref` as an 
 argument.


 That's why I asked the question in 
 http://forum.dlang.org/post/xdjsmwocbtxovjnathek forum.dlang.org 
 . It seems Walter wants to allow passing `scope ref` to `ref`, 
 but then automatically treat a normal `ref` of the second 
 function as if it were `scope ref`. But I can't quite see 
 through it :-(


Yes I have seen that thread and have no idea what Walter has 
meant either. His answer only makes sense if return value scope 
is supposed to be inferred from body - what when bodies are 
guaranteed to be there everything can be inferred anyway.
Dec 08 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2014 8:25 AM, Dicebot wrote:

 2) even if it worked, existing definition of scope return value makes it
 impossible to use in typical idiomatic pipeline: `file.byLine.algo1.algo2`.
 Either algoX is defined to take `scope ref` and thus can't return it or it is
 defined to take `ref` and can't take another `scope ref` as an argument.

 At least this is what I get from reading existing examples in DIP69


The difference between 'scope ref' and 'ref' parameters is that the former 
cannot be returned by reference.

The difference between 'scope ref' and 'ref' function returns is that the
former 
cannot be saved by the caller.

You can still safely pass the address of a stack variable by 'ref' - it will
not 
escape. Under the current proposal, as now, you cannot store a ref by ref, and 
cannot take the address of a ref variable.



 There are probably only a handful of people on the planet who actually
 understand C++ ref. I wished very hard to avoid that with D ref.


 While there is no argument that C++ ref is screwed, it is rather hard to say if
 this is inherent consequence of ref being a type qualifier or just C++ being
 C++.


It's not because C++ designers are idiots. It inevitably follows from what ref 
is. ref grew, brick by brick, into a monster, each brick being inevitable.



 I mean how many C++ type system features in general are understood my more
 than a handful of people on the planet? For me `ref` is essentially just a
 different flavor of `*` - and if the latter can be part of type, I see no
 reasons why former can't


I agree it's a seductively simple idea. The trouble starts happening when you 
start when making ref idempotent, when ref can only be at the 'head' of a data 
structure, when trying to do type deduction of a ref type (do you get the ref, 
or do you look 'through' the ref?), what happens with overloading, etc., and on 
and on.
Dec 08 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 8 December 2014 at 19:44:48 UTC, Walter Bright wrote:

 The difference between 'scope ref' and 'ref' parameters is that 
 the former cannot be returned by reference.

 The difference between 'scope ref' and 'ref' function returns 
 is that the former cannot be saved by the caller.

 You can still safely pass the address of a stack variable by 
 'ref' - it will not escape. Under the current proposal, as now, 
 you cannot store a ref by ref, and cannot take the address of a 
 ref variable.


Easier to go straight with pseudo-code:

struct ByLine
{
     scope string front();
     // ...
}

auto byLine(File file)
{
     return ByLine(file);
}

scope /* ref */ string foo(scope /* ref */ string input)
{
     return input[1..$];
}

void main()
{
     auto r = file.byLine.map!foo;
     string s = r.front; // this should not compile
     string s = r.front.dup; // this should compile

     // how foo signature should look like for this to work?
}



 I mean how many C++ type system features in general are 
 understood my more
 than a handful of people on the planet? For me `ref` is 
 essentially just a
 different flavor of `*` - and if the latter can be part of 
 type, I see no
 reasons why former can't


 I agree it's a seductively simple idea. The trouble starts 
 happening when you start when making ref idempotent, when ref 
 can only be at the 'head' of a data structure, when trying to 
 do type deduction of a ref type (do you get the ref, or do you 
 look 'through' the ref?), what happens with overloading, etc., 
 and on and on.


But was there any reason why those traits (alien to type 
qualifiers) were pursued? What is the problem with `ref` simply 
meaning `non-null pointer` and allowing non-idempotent 
ref(ref(int))?
Dec 08 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2014 12:54 PM, Dicebot wrote:

 On Monday, 8 December 2014 at 19:44:48 UTC, Walter Bright wrote:

 I agree it's a seductively simple idea. The trouble starts happening when you
 start when making ref idempotent, when ref can only be at the 'head' of a data
 structure, when trying to do type deduction of a ref type (do you get the ref,
 or do you look 'through' the ref?), what happens with overloading, etc., and
 on and on.


 But was there any reason why those traits (alien to type qualifiers) were
 pursued? What is the problem with `ref` simply meaning `non-null pointer` and
 allowing non-idempotent ref(ref(int))?


Because it isn't just a non-null pointer (and ref's can still be null in C++!), 
it's an auto-dereferencing pointer.
Dec 08 2014
next sibling parent reply =?UTF-8?Q?Tobias=20M=C3=BCller?= <troplin bluewin.ch> writes:
Walter Bright <newshound2 digitalmars.com> wrote:
[...]

 and ref's can still be null in C++!


AFAIK only if you dereference a NULL pointer, which is UB. So not really.

[...]

Tobi
Dec 08 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2014 1:59 PM, Tobias Müller wrote:

 Walter Bright <newshound2 digitalmars.com> wrote:
 [...]

 and ref's can still be null in C++!


 AFAIK only if you dereference a NULL pointer, which is UB. So not really.


Saying it's UB doesn't help in the slightest. You can still have null ref's,
and 
no compiler will prevent it.
Dec 08 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 8 December 2014 at 21:07:39 UTC, Walter Bright wrote:

 But was there any reason why those traits (alien to type 
 qualifiers) were
 pursued? What is the problem with `ref` simply meaning 
 `non-null pointer` and
 allowing non-idempotent ref(ref(int))?


 Because it isn't just a non-null pointer (and ref's can still 
 be null in C++!), it's an auto-dereferencing pointer.


Don't want to distract from topic that actually matters but would 
be nice to ask you few more question on topic at some point.
Dec 08 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2014 12:54 PM, Dicebot wrote:

 struct ByLine
 {
      scope string front();
      // ...
 }

 auto byLine(File file)
 {
      return ByLine(file);
 }

 scope /* ref */ string foo(scope /* ref */ string input)
 {
      return input[1..$];
 }

 void main()
 {
      auto r = file.byLine.map!foo;
      string s = r.front; // this should not compile
      string s = r.front.dup; // this should compile

      // how foo signature should look like for this to work?
 }


front() should return a 'scope ref string'.
Dec 08 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 8 December 2014 at 21:12:47 UTC, Walter Bright wrote:

 On 12/8/2014 12:54 PM, Dicebot wrote:

 struct ByLine
 {
     scope string front();
     // ...
 }

 auto byLine(File file)
 {
     return ByLine(file);
 }

 scope /* ref */ string foo(scope /* ref */ string input)
 {
     return input[1..$];
 }

 void main()
 {
     auto r = file.byLine.map!foo;
     string s = r.front; // this should not compile
     string s = r.front.dup; // this should compile

     // how foo signature should look like for this to work?
 }


 front() should return a 'scope ref string'.


That seems to contradict your other statement:


 A 'scope ref' parameter may not be returned as a 'ref' or a 
 'scope ref'.


Please check `foo()` once more - it needs to accept scope (ref) 
to be able to accept ByLine.front as an argument. And it also 
needs to pass it down the call chain - but returning `input` by 
reference is illegal according to abovementioned rule.
Dec 08 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2014 3:14 PM, Dicebot wrote:

 On Monday, 8 December 2014 at 21:12:47 UTC, Walter Bright wrote:

 On 12/8/2014 12:54 PM, Dicebot wrote:

 struct ByLine
 {
     scope string front();
     // ...
 }

 auto byLine(File file)
 {
     return ByLine(file);
 }

 scope /* ref */ string foo(scope /* ref */ string input)
 {
     return input[1..$];
 }

 void main()
 {
     auto r = file.byLine.map!foo;
     string s = r.front; // this should not compile
     string s = r.front.dup; // this should compile

     // how foo signature should look like for this to work?
 }


 front() should return a 'scope ref string'.


 That seems to contradict your other statement:


 A 'scope ref' parameter may not be returned as a 'ref' or a 'scope ref'.




Just make it a 'ref' parameter.



 Please check `foo()` once more - it needs to accept scope (ref) to be able to
 accept ByLine.front as an argument. And it also needs to pass it down the call
 chain - but returning `input` by reference is illegal according to
 abovementioned rule.


It can still be passed down as a 'ref' parameter.
Dec 09 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 December 2014 at 22:24:51 UTC, Walter Bright wrote:

 front() should return a 'scope ref string'.


 That seems to contradict your other statement:


 A 'scope ref' parameter may not be returned as a 'ref' or a 
 'scope ref'.




 Just make it a 'ref' parameter.



 Please check `foo()` once more - it needs to accept scope 
 (ref) to be able to
 accept ByLine.front as an argument. And it also needs to pass 
 it down the call
 chain - but returning `input` by reference is illegal 
 according to
 abovementioned rule.


 It can still be passed down as a 'ref' parameter.


But as far as I understand the spec it will result it this code 
failing too:

auto r = ["aaa", "bbb", "ccc"].map!foo;
// should compile but will fail because foo returns scope  ref:
string s = r.front;

What I mean is that in current proposal it is impossible to 
transfer scope information down the call chain - either function 
always returns scope ref or never. Which implies the necessity of 
something like `auto scope`...
Dec 09 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/9/2014 8:34 PM, Dicebot wrote:

 But as far as I understand the spec it will result it this code failing too:

 auto r = ["aaa", "bbb", "ccc"].map!foo;
 // should compile but will fail because foo returns scope  ref:
 string s = r.front;

 What I mean is that in current proposal it is impossible to transfer scope
 information down the call chain - either function always returns scope ref or
 never. Which implies the necessity of something like `auto scope`...


If you want data to 'escape' from r.front, then it shouldn't be marked as
scope. 
Definitely, using scope successfully will require some rethinking of how code
is 
written.
Dec 10 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 11 December 2014 at 03:30:07 UTC, Walter Bright 
wrote:

 If you want data to 'escape' from r.front, then it shouldn't be 
 marked as scope. Definitely, using scope successfully will 
 require some rethinking of how code is written.


Allowing or prohibiting escaping of r.front in that example 
depends on definition of mapped range. There is nothing to 
"rethink" about it - case is almost identical to const+inout.
Dec 10 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/10/2014 10:43 PM, Dicebot wrote:

 On Thursday, 11 December 2014 at 03:30:07 UTC, Walter Bright wrote:

 If you want data to 'escape' from r.front, then it shouldn't be marked as
 scope. Definitely, using scope successfully will require some rethinking of
 how code is written.


 Allowing or prohibiting escaping of r.front in that example depends on
 definition of mapped range. There is nothing to "rethink" about it - case is
 almost identical to const+inout.


One reason why with templates, they'll do scope inference.
Dec 11 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 11 December 2014 at 08:30:55 UTC, Walter Bright 
wrote:

 On 12/10/2014 10:43 PM, Dicebot wrote:

 On Thursday, 11 December 2014 at 03:30:07 UTC, Walter Bright 
 wrote:

 If you want data to 'escape' from r.front, then it shouldn't 
 be marked as
 scope. Definitely, using scope successfully will require some 
 rethinking of
 how code is written.


 Allowing or prohibiting escaping of r.front in that example 
 depends on
 definition of mapped range. There is nothing to "rethink" 
 about it - case is
 almost identical to const+inout.


 One reason why with templates, they'll do scope inference.


So the answer really is "current proposal does not support it"? 
Well this alone seems limiting enough for me to kills the whole 
DIP. "Use templates everywhere" is not really an answer, we could 
as well just prohibit separate compilation without .di files and 
enjoy full source availability as inherent requirement.
Dec 11 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/11/2014 10:04 PM, Dicebot wrote:

 On Thursday, 11 December 2014 at 08:30:55 UTC, Walter Bright wrote:

 On 12/10/2014 10:43 PM, Dicebot wrote:

 On Thursday, 11 December 2014 at 03:30:07 UTC, Walter Bright wrote:

 If you want data to 'escape' from r.front, then it shouldn't be marked as
 scope. Definitely, using scope successfully will require some rethinking of
 how code is written.


 Allowing or prohibiting escaping of r.front in that example depends on
 definition of mapped range. There is nothing to "rethink" about it - case is
 almost identical to const+inout.


 One reason why with templates, they'll do scope inference.


 So the answer really is "current proposal does not support it"? Well this alone
 seems limiting enough for me to kills the whole DIP. "Use templates everywhere"
 is not really an answer, we could as well just prohibit separate compilation
 without .di files and enjoy full source availability as inherent requirement.


When you're relying on the definition of an argument, it is inherently a
template.
Dec 11 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Monday, 8 December 2014 at 20:54:54 UTC, Dicebot wrote:

 But was there any reason why those traits (alien to type 
 qualifiers) were pursued? What is the problem with `ref` simply 
 meaning `non-null pointer` and allowing non-idempotent 
 ref(ref(int))?


Please no.

when you do int a; and then use a, you always either refer to a,
to memory storage (lvalue) or a, the value stored in that memory
(the rvalue).

When doing ref int a = xxx;

You specify that you don't create a new storage for a, but that
you must consider xxx as an lvalue, and bind the name a to that
same lvalue.

Once you get that, you get why ref(ref(int)) do not make any
sense, and is generally undesirable.
Dec 08 2014
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 8 December 2014 at 23:19:26 UTC, deadalnix wrote:

 On Monday, 8 December 2014 at 20:54:54 UTC, Dicebot wrote:

 But was there any reason why those traits (alien to type 
 qualifiers) were pursued? What is the problem with `ref` 
 simply meaning `non-null pointer` and allowing non-idempotent 
 ref(ref(int))?


 Please no.

 when you do int a; and then use a, you always either refer to a,
 to memory storage (lvalue) or a, the value stored in that memory
 (the rvalue).

 When doing ref int a = xxx;

 You specify that you don't create a new storage for a, but that
 you must consider xxx as an lvalue, and bind the name a to that
 same lvalue.


My thoughts have went totally different direction:

ref(int) a = &xxx; // ok
ref(int) a = xxx; // bad, can't initialize ref from value
ref(int) a = intptr; // ok, implicitly inject assert(intptr)
ref(int) a = null; // bad, can't initialize ref from literal

Now I see how issue may arise because of implicit conversion to 
matching value type (when passing to as function argument for 
example) but we do already have structs with `alias this` which 
may need similar semantics - it needs to be defined anyway.

This is still offtopic. I support attempt to make scope a storage 
class in general because this is an approach of minimal change 
and still can be useful. Problem with defining it so that it 
actually is useful.
Dec 08 2014
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 8 December 2014 at 16:25:22 UTC, Dicebot wrote:

 This isn't the same as it does not propagate scope but just 
 restricts return value. Difference is that it cannot be 
 chained. Let's consider practical example based on Phobos:

 there was an issue with byLine range that it has reused same 
 buffer internally which sometimes caught users off guard when 
 trying to save slice. It is a natural fit for `scope` - make it 
 return `scope string` instead to ensure that no slices get 
 stored.


That would only ensure that slice do not outline the range.


 Two issues immediately pop up:

 1) scope is not transitive thus it doesn't work at all - you 
 still can store slice of `scope string` as only actual 
 ptr+length struct is protected.


Yes, that is my whole point whith the scope flag in expression.
Without it, the proposal need hacks to work (&(*e)) and is too
restrictive to be useful.


 While there is no argument that C++ ref is screwed, it is 
 rather hard to say if this is inherent consequence of ref being 
 a type qualifier or just C++ being C++. I mean how many C++ 
 type system features in general are understood my more than a 
 handful of people on the planet? For me `ref` is essentially 
 just a different flavor of `*` - and if the latter can be part 
 of type, I see no reasons why former can't


No, a type constructor is not the right tool. This about it.

scope(int)[] do not make any sense as the slice cannot outlive
its content safely.

You need to have a flag on expression and declarations to make
this work.
Dec 08 2014
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 7 December 2014 at 21:29:50 UTC, Walter Bright wrote:

 My experience with C++ ref as type qualifier is very, very bad. 
 It's a special case EVERYWHERE. Doing type deduction with it is 
 an exercise in a completely baffling set of rules and a 
 different rule for every occasion - Scott Meyers has a great 
 piece on this.

 There are probably only a handful of people on the planet who 
 actually understand C++ ref. I wished very hard to avoid that 
 with D ref.


Type qualifier for scope is the wrong tool.

scope(int)[] do not make any sense. The slice cannot outlive its
content. scope as a flag on expressions/symbols is very useful.
Dec 08 2014
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 8 December 2014 at 23:05:49 UTC, deadalnix wrote:

 scope(int)[] do not make any sense. The slice cannot outlive its
 content. scope as a flag on expressions/symbols is very useful.


I'd say it strange for directly the opposite reason - pure value 
type is inherently scope so adding such annotation shouldn't 
matter in general.
Dec 08 2014
prev sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 12/7/2014 6:12 AM, Dicebot wrote:

 But from existing cases it doesn't seem working good enough. For example,
 not
 being able to represent idiom of `scope ref int foo(scope ref int x) {
 return x;
 }` seems very limiting.



   scope ref int foo(ref int x);

 will do it.


Will it? It looks like foo can't be called with scope data?



 I also don't consider `ref` design as a storage class any kind of success
 at all
 and generally agree with Manu on this topic. At the same time alternative
 proposals that make it a qualifier (like Marc did) do impact existing
 language
 much more and this no small concern.



 My experience with C++ ref as type qualifier is very, very bad. It's a
 special case EVERYWHERE. Doing type deduction with it is an exercise in a
 completely baffling set of rules and a different rule for every occasion -
 Scott Meyers has a great piece on this.

 There are probably only a handful of people on the planet who actually
 understand C++ ref. I wished very hard to avoid that with D ref.


I do completely sympathise with you on this point. I understand where
you were coming from on 'ref', and I do applaud the effort. Nobody
said that ref should be just the same as C++, but I am saying that the
D solution is not a success.
I'm all about a ref that improves on C++, but I don't think that's
what we have. We have something that's just complex and unwieldy in a
different (and for my money, even more frustrating) way.

I'm just saying, don't repeat that mistake with scope! There must be
another way...

I'm also quite uneasy with the fact that scope would not be transitive
as a storage class. What happens when it's applied to a value type,
like a struct, that contains some pointers? An adaptation wrapper for
a single pointer is super common; ie, a tiny struct passed by value
with scope needs to have it's contained pointer receive the scope-ness
of the argument.

I don't have the perfect proposal, but I feel very strongly about 2 things:
1. It must not be a storage class; the concept was a disaster with
ref, and I struggle with this more frequently than any other 'feature'
in D.
2. I feel it's a big mistake to separate it from the type system,
which I think most agree, is D's greatest asset by far. Manipulating
types is simple and convenient using the type system, and I think
manipulating scope will be just as important as any other attribute as
soon as even slightly complex use cases begin to arise.
Dec 11 2014
next sibling parent reply "ixid" <nuaccount gmail.com> writes:
Manu have you shown Walter some of the code where ref is 
problematic? He seems to have asked to see examples repeatedly as 
he's not convinced there is a problem.
Dec 11 2014
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 December 2014 at 23:22, ixid via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 Manu have you shown Walter some of the code where ref is problematic? He
 seems to have asked to see examples repeatedly as he's not convinced there
 is a problem.


I have discussed issues here in the past, but possibly not as
concretely as would be useful to this discussion.
I want to take some time to contribute value to this issue, but I have
virtually no time at all lately, and I can't even keep up with this
posts thread, let alone sit and consider+craft a bunch of
demonstration cases.
This thread is moving faster than I am able to, so I'm not sure what I can do.

There's no TL;DR conversation, and almost all discussion happens
outside of my timezone, which wasn't such a problem when I was
nocturnal, but I have had to change lifestyle recently :/
I have about 30 posts to try and catch up on and understand right now
in 5 minutes before I have to leave >_<
Dec 12 2014
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 11 December 2014 at 12:48:05 UTC, Manu via 
Digitalmars-d wrote:

 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 12/7/2014 6:12 AM, Dicebot wrote:

 But from existing cases it doesn't seem working good enough. 
 For example,
 not
 being able to represent idiom of `scope ref int foo(scope ref 
 int x) {
 return x;
 }` seems very limiting.



   scope ref int foo(ref int x);

 will do it.


 Will it? It looks like foo can't be called with scope data?


This is a point that most people don't seem to understand yet, 
and which wasn't obvious for me either, at the beginning:

* A `ref` parameter means that it cannot escape the function, 
_except_ by return.
* A `scope ref` parameter means that it cannot escape the 
function  _ever_, not even by return.
* A `scope ref` return means that it cannot leave the current 
statement.

Therefore, a `scope ref` return value can be passed on to the 
next function as a `ref` argument. If that function again returns 
a reference (even if not explicitly designated as `scope`), the 
compiler will treat it as if it were `scope ref`.


 I'm also quite uneasy with the fact that scope would not be 
 transitive
 as a storage class. What happens when it's applied to a value 
 type,
 like a struct, that contains some pointers? An adaptation 
 wrapper for
 a single pointer is super common; ie, a tiny struct passed by 
 value
 with scope needs to have it's contained pointer receive the 
 scope-ness
 of the argument.


I agree, this is important. In my proposal, this works without 
transitivity. The wrapper stores the pointer as a `scope` member, 
then by copying the wrapper, the pointer gets copied implicitly, 
to which the normal scope restrictions apply (`scope` on members 
means "will not outlive the aggregate"). If it stored it as 
normal non-scope pointer, it couldn't get assigned in the first 
place. (Additionally, some higher level tricks are possible if we 
allow scope for value types and overloading on scope.)
Dec 11 2014
next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 11 December 2014 at 13:55:55 UTC, Marc Schütz wrote:

 On Thursday, 11 December 2014 at 12:48:05 UTC, Manu via 
 Digitalmars-d wrote:

 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 12/7/2014 6:12 AM, Dicebot wrote:

 But from existing cases it doesn't seem working good enough. 
 For example,
 not
 being able to represent idiom of `scope ref int foo(scope 
 ref int x) {
 return x;
 }` seems very limiting.



  scope ref int foo(ref int x);

 will do it.


 Will it? It looks like foo can't be called with scope data?


 This is a point that most people don't seem to understand yet, 
 and which wasn't obvious for me either, at the beginning:

 * A `ref` parameter means that it cannot escape the function, 
 _except_ by return.
 * A `scope ref` parameter means that it cannot escape the 
 function  _ever_, not even by return.
 * A `scope ref` return means that it cannot leave the current 
 statement.

 Therefore, a `scope ref` return value can be passed on to the 
 next function as a `ref` argument. If that function again 
 returns a reference (even if not explicitly designated as 
 `scope`), the compiler will treat it as if it were `scope ref`.


OH! I had totally misunderstood that. Cheers for the explanation.
Dec 11 2014
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/11/14 8:55 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:

 On Thursday, 11 December 2014 at 12:48:05 UTC, Manu via Digitalmars-d
 wrote:

 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 12/7/2014 6:12 AM, Dicebot wrote:

 But from existing cases it doesn't seem working good enough. For
 example,
 not
 being able to represent idiom of `scope ref int foo(scope ref int x) {
 return x;
 }` seems very limiting.



   scope ref int foo(ref int x);

 will do it.


 Will it? It looks like foo can't be called with scope data?


 This is a point that most people don't seem to understand yet, and which
 wasn't obvious for me either, at the beginning:

 * A `ref` parameter means that it cannot escape the function, _except_
 by return.
 * A `scope ref` parameter means that it cannot escape the function
 _ever_, not even by return.
 * A `scope ref` return means that it cannot leave the current statement.

 Therefore, a `scope ref` return value can be passed on to the next
 function as a `ref` argument. If that function again returns a reference
 (even if not explicitly designated as `scope`), the compiler will treat
 it as if it were `scope ref`.


Please, put this in the DIP! This is exactly the thing I had been asking 
for here: http://forum.dlang.org/post/m5sitd$1ki0$1 digitalmars.com

Thanks, I'll re-read the proposal with this in mind.

-Steve
Dec 11 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/11/2014 8:26 AM, Steven Schveighoffer wrote:

 Please, put this in the DIP! This is exactly the thing I had been asking for
 here: http://forum.dlang.org/post/m5sitd$1ki0$1 digitalmars.com

 Thanks, I'll re-read the proposal with this in mind.


Marc is clearly better at explaining things than I am, as I've explained that 
point in this thread about 5 times now.
Dec 11 2014
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 11 December 2014 at 13:55:55 UTC, Marc Schütz wrote:

 On Thursday, 11 December 2014 at 12:48:05 UTC, Manu via 
 Digitalmars-d wrote:

 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 12/7/2014 6:12 AM, Dicebot wrote:

 But from existing cases it doesn't seem working good enough. 
 For example,
 not
 being able to represent idiom of `scope ref int foo(scope 
 ref int x) {
 return x;
 }` seems very limiting.



  scope ref int foo(ref int x);

 will do it.


 Will it? It looks like foo can't be called with scope data?


 This is a point that most people don't seem to understand yet, 
 and which wasn't obvious for me either, at the beginning:

 * A `ref` parameter means that it cannot escape the function, 
 _except_ by return.
 * A `scope ref` parameter means that it cannot escape the 
 function  _ever_, not even by return.
 * A `scope ref` return means that it cannot leave the current 
 statement.

 Therefore, a `scope ref` return value can be passed on to the 
 next function as a `ref` argument. If that function again 
 returns a reference (even if not explicitly designated as 
 `scope`), the compiler will treat it as if it were `scope ref`.


No, it understood. It is simply not useful.


 I agree, this is important. In my proposal, this works without 
 transitivity. The wrapper stores the pointer as a `scope` 
 member, then by copying the wrapper, the pointer gets copied 
 implicitly, to which the normal scope restrictions apply 
 (`scope` on members means "will not outlive the aggregate"). If 
 it stored it as normal non-scope pointer, it couldn't get 
 assigned in the first place.


Wut ? You cante store anything with grear lifetime, including non 
scope things (as they'll have infinite lifetime). Meaning the 
only thing you know, is that thing are possibly scoped.

Meaning you have to assume infinite lifetime with every 
indirection, which make this proposal useless.
Dec 11 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Friday, 12 December 2014 at 00:13:10 UTC, deadalnix wrote:

 Therefore, a `scope ref` return value can be passed on to the 
 next function as a `ref` argument. If that function again 
 returns a reference (even if not explicitly designated as 
 `scope`), the compiler will treat it as if it were `scope ref`.


 No, it understood. It is simply not useful.


So far I tend to agree, unfortunately. Considering all provided 
answers and explanations suggested semantics are simply not 
powerful enough to be useful even for relatively simple idiomatic 
D code and with no clear way of backward compatible improvements 
this DIP does not pull own weight.

"Those are not scopes you are looking for"
Dec 11 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/11/2014 10:10 PM, Dicebot wrote:

 So far I tend to agree, unfortunately. Considering all provided answers and
 explanations suggested semantics are simply not powerful enough to be useful
 even for relatively simple idiomatic D code and with no clear way of backward
 compatible improvements this DIP does not pull own weight.
 "Those are not scopes you are looking for"


The proposal provides escape proof passing of arguments. This is necessary in 
order to make rc safe, for example.

What are you looking for?
Dec 12 2014
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-12-13 01:09, Walter Bright wrote:


 The proposal provides escape proof passing of arguments. This is
 necessary in order to make rc safe, for example.

 What are you looking for?


Passing references to stack allocated data to functions safely is one 
idea that comes to mind.

-- 
/Jacob Carlborg
Dec 13 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/13/2014 8:01 AM, Jacob Carlborg wrote:

 On 2014-12-13 01:09, Walter Bright wrote:


 The proposal provides escape proof passing of arguments. This is
 necessary in order to make rc safe, for example.

 What are you looking for?


 Passing references to stack allocated data to functions safely is one idea that
 comes to mind.


'scope ref' does that.
Dec 13 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 13 December 2014 at 00:10:07 UTC, Walter Bright 
wrote:

 The proposal provides escape proof passing of arguments. This 
 is necessary in order to make rc safe, for example.

 What are you looking for?


I am looking for a tool to prevent escaping references to 
user-defined entities / resources from specific scope. RC case is 
not interesting at all to me though I recognize the potential 
benefits of tweaking the scope system to help with it. But there 
is a big difference between tweak for one specific use case or 
designing feature around it (so it becomes almost useless in 
other cases).

I have already provided you an example of code I want to be 
enforced with scope and two major issues with existing proposal 
that make it lacking. This example is akin to litmus test for any 
scope proposal - if it doesn't allow to express such design, that 
proposal is simply not good enough.

In case you have forgotten, I am reminding about two critical 
points that are necessary to make it fly:

1) inout analog for scope to be able to deduce borrowship of 
return values based on borrowship of input arguments. Current 
system is conservative and results in template bloat (complicated 
by the fact that it is storage class thus not actually part of a 
type)

2) at least optional transitivity to be able to express to 
protect with scope data referenced by slice or owned linked list 
referenced from root node.

------

In your tree example I would have never wanted scope protection 
of one specific node of such tree - but a transitive scope 
protection of whole tree view available through on of node 
pointers/references. It doesn't matter who and how owns the data 
for borrowship implementation - only thing that matters that _it 
is not me_.
Dec 14 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/14/2014 5:44 PM, Dicebot wrote:

 I am looking for a tool to prevent escaping references to user-defined entities
 / resources from specific scope. RC case is not interesting at all to me though
 I recognize the potential benefits of tweaking the scope system to help with
it.
 But there is a big difference between tweak for one specific use case or
 designing feature around it (so it becomes almost useless in other cases).

 I have already provided you an example of code I want to be enforced with scope
 and two major issues with existing proposal that make it lacking. This example
 is akin to litmus test for any scope proposal - if it doesn't allow to express
 such design, that proposal is simply not good enough.

 In case you have forgotten, I am reminding about two critical points that are
 necessary to make it fly:

 1) inout analog for scope to be able to deduce borrowship of return values
based
 on borrowship of input arguments. Current system is conservative and results in
 template bloat (complicated by the fact that it is storage class thus not
 actually part of a type)


1. All inout actually does is reduce code copy/pasta, it is not critical.
2. This is what scope inference is all about.



 2) at least optional transitivity to be able to express to protect with scope
 data referenced by slice or owned linked list referenced from root node.


1. that won't work unless scope is a type constructor
2. it can be achieved by using wrappers that only allow by-scope references to 
their data (RC is an example of such a wrapper)



 In your tree example I would have never wanted scope protection of one specific
 node of such tree - but a transitive scope protection of whole tree view
 available through on of node pointers/references. It doesn't matter who and how
 owns the data for borrowship implementation - only thing that matters that _it
 is not me_.


As I explained to Manu, transitive scope makes things like doing a symbol table 
lookup, then putting a pointer to the found symbol in an AST, not workable.
Dec 14 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 15 December 2014 at 02:45:04 UTC, Walter Bright wrote:

 1. All inout actually does is reduce code copy/pasta, it is not 
 critical.


Being forced to duplicate every single function in two flavors to 
actually make scope system usable? This is as much critical as it 
can be.


 2. This is what scope inference is all about.


Which only works with templates and lack of scope on arguments 
does not affect function body -> templates are not necessary, 
same as inout.


 2) at least optional transitivity to be able to express to 
 protect with scope
 data referenced by slice or owned linked list referenced from 
 root node.


 1. that won't work unless scope is a type constructor


I know and this is why I am leaning toward it being qualifier 
despite all related issues.


 2. it can be achieved by using wrappers that only allow 
 by-scope references to their data (RC is an example of such a 
 wrapper)


It is viral approach and backwards incompatible one - we can't 
change signatures of Phobos functions to return Wrapped!(char[]) 
instead of char[] for example. While theoretically possible I 
can't call this approach practical - not until see some 
convincing application example.


 In your tree example I would have never wanted scope 
 protection of one specific
 node of such tree - but a transitive scope protection of whole 
 tree view
 available through on of node pointers/references. It doesn't 
 matter who and how
 owns the data for borrowship implementation - only thing that 
 matters that _it
 is not me_.


 As I explained to Manu, transitive scope makes things like 
 doing a symbol table lookup, then putting a pointer to the 
 found symbol in an AST, not workable.


This sounds totally against my understanding of scope. I want 
scope exactly to prohibit such actions. However it is possible in 
slightly modified way - where you don't directly insert pointer 
to AST but use public method of AST control structure that checks 
if supplied scope pointer belongs to list of nodes it owns  and 
casts away scope after that.
Dec 14 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/14/2014 7:41 PM, Dicebot wrote:

 On Monday, 15 December 2014 at 02:45:04 UTC, Walter Bright wrote:

 1. All inout actually does is reduce code copy/pasta, it is not critical.


 Being forced to duplicate every single function in two flavors to actually make
 scope system usable? This is as much critical as it can be.


C++ seems to do fine without it for const. It's a convenience feature.



 2. This is what scope inference is all about.


 Which only works with templates and lack of scope on arguments does not affect
 function body -> templates are not necessary, same as inout.


It also works with all the lambdas, since source for them is always available.
I 
also wanted to make it (i.e. inference) work with auto functions, but Don 
Clugston was the primary objector :-)

I do view inference as something we need to extend to more functions.



 2) at least optional transitivity to be able to express to protect with scope
 data referenced by slice or owned linked list referenced from root node.


 1. that won't work unless scope is a type constructor


 I know and this is why I am leaning toward it being qualifier despite all
 related issues.


That would be a truly massive change to D, and I'm not at all sure it would be 
worth it. We've (i.e. Kenji) have been fixing bugs with inout for years, and 
idea that had seemed straightforward.



 2. it can be achieved by using wrappers that only allow by-scope references to
 their data (RC is an example of such a wrapper)


 It is viral approach and backwards incompatible one - we can't change
signatures
 of Phobos functions to return Wrapped!(char[]) instead of char[] for example.
 While theoretically possible I can't call this approach practical - not until
 see some convincing application example.


But you can change signatures to add annotations?

You're going to get one or the other.



 This sounds totally against my understanding of scope. I want scope exactly to
 prohibit such actions. However it is possible in slightly modified way - where
 you don't directly insert pointer to AST but use public method of AST control
 structure that checks if supplied scope pointer belongs to list of nodes it
 owns  and casts away scope after that.


Wrappers do that, too.

I know that 'wrapper' sounds bad to you, but one of the goals of D's UDT's is
to 
make it easy to adapt the behavior of a type by wrapping it in a struct, rather 
than build new behavior into the language.
Dec 14 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 15 December 2014 at 06:12:05 UTC, Walter Bright wrote:

 C++ seems to do fine without it for const. It's a convenience 
 feature.


C++ const does not really restrict or affect anything "for real", 
it is non-existent feature. `scope` as proposed would result in 
inability to store result of predicates if those are ever to 
accept scope data - unless you defined two versions for each 
function that may possibly accept scope data with absolutely 
identical body.


 2. This is what scope inference is all about.


 Which only works with templates and lack of scope on arguments 
 does not affect
 function body -> templates are not necessary, same as inout.


 It also works with all the lambdas, since source for them is 
 always available. I also wanted to make it (i.e. inference) 
 work with auto functions, but Don Clugston was the primary 
 objector :-)


This is not really answering my objections but side-stepping. I 
do support attribute inference. I would actually support full 
attribute inference in the language (and help convincing Don 
about it ;)) if it was introduced as foundation of the language 
and not arbitrary hack (== rethink the way .di interfaces and 
static libraries are defined)


 I do view inference as something we need to extend to more 
 functions.


That would help with many issues but it need careful design to be 
well-accepted. How about teaming up to do a DIP about it at some 
point where this discussion is over? :P


 2) at least optional transitivity to be able to express to 
 protect with scope
 data referenced by slice or owned linked list referenced 
 from root node.


 1. that won't work unless scope is a type constructor


 I know and this is why I am leaning toward it being qualifier 
 despite all
 related issues.


 That would be a truly massive change to D, and I'm not at all 
 sure it would be worth it. We've (i.e. Kenji) have been fixing 
 bugs with inout for years, and idea that had seemed 
 straightforward.


Yes, I know and this why I am leaning towards that path but 
willing to accept any storage-class based solution - as long as 
it fits my basic criteria. I wonder if some hybrid approach is 
possible - for example, keep it storage class but pretend 
anything transitively accessible though it as if it was of scope 
storage class on its own. But that quickly gets into "exceptions 
for special cases" land :(


 2. it can be achieved by using wrappers that only allow 
 by-scope references to
 their data (RC is an example of such a wrapper)


 It is viral approach and backwards incompatible one - we can't 
 change signatures
 of Phobos functions to return Wrapped!(char[]) instead of 
 char[] for example.
 While theoretically possible I can't call this approach 
 practical - not until
 see some convincing application example.


 But you can change signatures to add annotations?


Such annotation would only prevent from compiling code which was 
already undefined by documentation. Changing return type (which 
wasn't Voldemort type before) may break any program that has 
lines akin to `char[] data = foo()` - perfectly legal working 
code.


 This sounds totally against my understanding of scope. I want 
 scope exactly to
 prohibit such actions. However it is possible in slightly 
 modified way - where
 you don't directly insert pointer to AST but use public method 
 of AST control
 structure that checks if supplied scope pointer belongs to 
 list of nodes it
 owns  and casts away scope after that.


 Wrappers do that, too.

 I know that 'wrapper' sounds bad to you, but one of the goals 
 of D's UDT's is to make it easy to adapt the behavior of a type 
 by wrapping it in a struct, rather than build new behavior into 
 the language.


Problem is that our UDT are simply not that good enough yet. 
Wrappers are fine in general but I want to wrap for less common 
cases and support more common ones natively. I suppose this is 
your intention too but we disagree on what is actually the common 
case and what is the special case.
Dec 15 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/15/2014 1:28 AM, Dicebot wrote:

 On Monday, 15 December 2014 at 06:12:05 UTC, Walter Bright wrote:

 C++ seems to do fine without it for const. It's a convenience feature.


 C++ const does not really restrict or affect anything "for real", it is
 non-existent feature.


C++ programmers constantly copy/pasta functions for the sole purpose of having 
one with const and one without.



 `scope` as proposed would result in inability to store
 result of predicates if those are ever to accept scope data - unless you
defined
 two versions for each function that may possibly accept scope data with
 absolutely identical body.


If that is necessary, it is no worse than C++ is with const. It isn't a
critical 
issue, or C++ would not be usable. I'd like to wait and see how this plays out.



 2. This is what scope inference is all about.


 Which only works with templates and lack of scope on arguments does not affect
 function body -> templates are not necessary, same as inout.


 It also works with all the lambdas, since source for them is always available.
 I also wanted to make it (i.e. inference) work with auto functions, but Don
 Clugston was the primary objector :-)


 This is not really answering my objections but side-stepping.


Inference addresses the issue.



 That would help with many issues but it need careful design to be
well-accepted.
 How about teaming up to do a DIP about it at some point where this discussion
is
 over? :P


Ok, sure.
Dec 15 2014
prev sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 15 December 2014 at 06:12:05 UTC, Walter Bright wrote:

 On 12/14/2014 7:41 PM, Dicebot wrote:

 Being forced to duplicate every single function in two flavors 
 to actually make
 scope system usable? This is as much critical as it can be.


 C++ seems to do fine without it for const. It's a convenience 
 feature.


It's more than that. It has the potential (well, could have) to 
be used for type erasure, which is a useful and worthwhile 
concept by itself.


 I know and this is why I am leaning toward it being qualifier 
 despite all
 related issues.


 That would be a truly massive change to D, and I'm not at all 
 sure it would be worth it. We've (i.e. Kenji) have been fixing 
 bugs with inout for years, and idea that had seemed 
 straightforward.


To be honest, I think one reason for those bugs is the awkward 
implementation. I had a look at mtype.c to see how difficult it 
would be to add another type modifier. It's no fun - the current 
implementation requires a dozen groups of helper functions to be 
written for every possible combination of type modifiers. 
Therefore there are helpers like `makeConst`, `makeImmutable`, 
`makeShared`, `makeSharedConst`, `makeWild` (this is `inout`), 
`makeWildConst`, `makeSharedWild`, `makeSharedWildConst`, and 
`makeMutable`. There are also some monster switch statements in 
the same fashion.

No wonder this is error prone. I suspect that with a cleaner 
implementation most of these problems will go away.
Dec 15 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/15/2014 3:34 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:

 No wonder this is error prone. I suspect that with a cleaner implementation
most
 of these problems will go away.


Sometimes, it isn't possible to see a cleaner way unless the dirty version is 
written first. Sometimes, there isn't a cleaner way :-(

But there are more issues with inout than are in mtype.c. The template 
implementation is full of it.
Dec 15 2014
prev sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 15 December 2014 at 01:44:27 UTC, Dicebot wrote:

 On Saturday, 13 December 2014 at 00:10:07 UTC, Walter Bright 
 wrote:

 The proposal provides escape proof passing of arguments. This 
 is necessary in order to make rc safe, for example.

 What are you looking for?


 I am looking for a tool to prevent escaping references to 
 user-defined entities / resources from specific scope. RC case 
 is not interesting at all to me though I recognize the 
 potential benefits of tweaking the scope system to help with 
 it. But there is a big difference between tweak for one 
 specific use case or designing feature around it (so it becomes 
 almost useless in other cases).

 I have already provided you an example of code I want to be 
 enforced with scope and two major issues with existing proposal 
 that make it lacking. This example is akin to litmus test for 
 any scope proposal - if it doesn't allow to express such 
 design, that proposal is simply not good enough.

 In case you have forgotten, I am reminding about two critical 
 points that are necessary to make it fly:

 1) inout analog for scope to be able to deduce borrowship of 
 return values based on borrowship of input arguments. Current 
 system is conservative and results in template bloat 
 (complicated by the fact that it is storage class thus not 
 actually part of a type)


Is it be possible to make `inout` a more general tool for type 
erasure? I don't think yet another ad-hoc mechanism should be 
introduced.


 2) at least optional transitivity to be able to express to 
 protect with scope data referenced by slice or owned linked 
 list referenced from root node.

 ------

 In your tree example I would have never wanted scope protection 
 of one specific node of such tree - but a transitive scope 
 protection of whole tree view available through on of node 
 pointers/references. It doesn't matter who and how owns the 
 data for borrowship implementation - only thing that matters 
 that _it is not me_.


I think there are two cases that are relevant. 1) the tree nodes 
don't own their children, instead they are managed by - let's say 
- a region allocator, and 2) the tree nodes do own their 
children. In both cases, they can declare the children as scope 
members (assuming a proposal that allows that).

The only case I can think of where the children should not be 
scope members is when they are GC managed. But in this case, you 
don't need to worry about references to them escaping.

Therefore, I don't see a use case for transitivity of scope, from 
a conceptional point of view. If a particular design makes 
transitivity necessary, this points to a flaw in the design, IMO.
Dec 15 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 15 December 2014 at 11:23:28 UTC, Marc Schütz wrote:

 I think there are two cases that are relevant. 1) the tree 
 nodes don't own their children, instead they are managed by - 
 let's say - a region allocator, and 2) the tree nodes do own 
 their children. In both cases, they can declare the children as 
 scope members (assuming a proposal that allows that).

 The only case I can think of where the children should not be 
 scope members is when they are GC managed. But in this case, 
 you don't need to worry about references to them escaping.

 Therefore, I don't see a use case for transitivity of scope, 
 from a conceptional point of view. If a particular design makes 
 transitivity necessary, this points to a flaw in the design, 
 IMO.


Actually I think in neither case children should be declared as 
scope members. Instead those should be declared as private 
members but methods that return pointers/references to children 
would be declared to return (transitive) scope 
pointers/references.

For me "scopeness" is a property of "view", not object itself - 
this also makes ownership method of actual data irrelevant. Only 
difference between GC owned data and stack allocated one is that 
former can have scoped view optionally but for the latter 
compiler must force it as the only available.
Dec 15 2014
next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote:

 On Monday, 15 December 2014 at 11:23:28 UTC, Marc Schütz wrote:

 I think there are two cases that are relevant. 1) the tree 
 nodes don't own their children, instead they are managed by - 
 let's say - a region allocator, and 2) the tree nodes do own 
 their children. In both cases, they can declare the children 
 as scope members (assuming a proposal that allows that).

 The only case I can think of where the children should not be 
 scope members is when they are GC managed. But in this case, 
 you don't need to worry about references to them escaping.

 Therefore, I don't see a use case for transitivity of scope, 
 from a conceptional point of view. If a particular design 
 makes transitivity necessary, this points to a flaw in the 
 design, IMO.


 Actually I think in neither case children should be declared as 
 scope members. Instead those should be declared as private 
 members but methods that return pointers/references to children 
 would be declared to return (transitive) scope 
 pointers/references.


I see this as the same thing, conceptually (module the 
"transitive" in in your parentheses). It's just a different 
mechanism. Scope members are just a short-cut instead of writing 
an accessor method that returns scope.


 For me "scopeness" is a property of "view", not object itself - 
 this also makes ownership method of actual data irrelevant.


Yes, and this would be true for scope members just the same. (If 
the aggregate "owns" it's scope members, it will however get in 
trouble when it tries to free() them, therefore your mechanism is 
more appropriate in this case.)


 Only difference between GC owned data and stack allocated one 
 is that former can have scoped view optionally but for the 
 latter compiler must force it as the only available.


Agreed.
Dec 15 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote:

 For me "scopeness" is a property of "view", not object itself - 
 this also makes ownership method of actual data irrelevant. 
 Only difference between GC owned data and stack allocated one 
 is that former can have scoped view optionally but for the 
 latter compiler must force it as the only available.


Ha finally something start to make sense here.
Dec 15 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/15/2014 5:38 AM, deadalnix wrote:

 On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote:

 For me "scopeness" is a property of "view", not object itself - this also
 makes ownership method of actual data irrelevant. Only difference between GC
 owned data and stack allocated one is that former can have scoped view
 optionally but for the latter compiler must force it as the only available.


 Ha finally something start to make sense here.


Well, the DIP does defined scope in terms of a 'view' in just this manner.

I am obviously terrible at explaining things.
Dec 15 2014
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 16 December 2014 at 01:00:47 UTC, Walter Bright wrote:

 On 12/15/2014 5:38 AM, deadalnix wrote:

 On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote:

 For me "scopeness" is a property of "view", not object itself 
 - this also
 makes ownership method of actual data irrelevant. Only 
 difference between GC
 owned data and stack allocated one is that former can have 
 scoped view
 optionally but for the latter compiler must force it as the 
 only available.


 Ha finally something start to make sense here.


 Well, the DIP does defined scope in terms of a 'view' in just 
 this manner.

 I am obviously terrible at explaining things.


No, I actually think the concept of view in the proposal is a 
good one. However, the proposal itself is too limiting.
Dec 16 2014
prev sibling next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 16 December 2014 at 01:00:47 UTC, Walter Bright wrote:

 On 12/15/2014 5:38 AM, deadalnix wrote:

 On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote:

 For me "scopeness" is a property of "view", not object itself 
 - this also
 makes ownership method of actual data irrelevant. Only 
 difference between GC
 owned data and stack allocated one is that former can have 
 scoped view
 optionally but for the latter compiler must force it as the 
 only available.


 Ha finally something start to make sense here.


 Well, the DIP does defined scope in terms of a 'view' in just 
 this manner.

 I am obviously terrible at explaining things.


Don't you think there is a contradiction between this notion of 
"view" and being so focused on lifetimes? Does it make sense to 
explicitly throw away lifetime information and then spend a lot 
of effort on preventing lifetime-violating assignments to take 
place?

The current proposal is either too limiting or not limiting 
enough.

Too limiting, because programmers expect to gain something for 
making parameter types explicit. A naive "make no assumptions 
view" is more suitable as a default, then let the programmer 
specify the specifics where possible/needed, but you need to 
figure out the specifics first and let the "make no assumptions 
view default" be the leftovers…

The proposal is not limiting enough, because retaining knowledge 
about allocation order and aliasing brings important benefits:

An ordered "linear" parameter type would have properties such as:
- being aliasfree
- being on the stack, and cheaply checkable against the stack 
pointer

Sure, ref counting is fairly general for dynamic tracking, but it 
is also quite expensive. A check against the stack-pointer is 
much cheaper.

I think the uncanny valley metaphor that was brought up was quite 
good. If you are going to put in constraints on references, then 
do it in a manner that is easy to deal with and which brings 
benefits such as aliasfree references. If constraints become 
"arbitrary" then usability will suffer and it will seem pointless 
to put in extra work to use them.

You have already gone more or less wholesale for templates, so 
you might as well use it here as well, to bring substantial 
benefits to stack-allocation, because the current proposal is a 
very weak in terms of benefits.
Dec 16 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/16/2014 5:37 AM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:

 The current proposal is either too limiting or not limiting enough.


I'm afraid I don't understand at all what you wrote.
Dec 16 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:


 I'm afraid I don't understand at all what you wrote.


Perhaps reading about linear type systems could help:

http://en.wikipedia.org/w/index.php?title=Substructural_type_system&redirect=no#Linear_type_systems

Bye,
bearophile
Dec 16 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Wednesday, 17 December 2014 at 07:48:52 UTC, bearophile wrote:

 Walter Bright:


 I'm afraid I don't understand at all what you wrote.


 Perhaps reading about linear type systems could help:

 http://en.wikipedia.org/w/index.php?title=Substructural_type_system&redirect=no#Linear_type_systems


But note that we want a "relaxed" linear type system. In general, 
we are fine with multiple aliases, although there are some 
applications where strict uniqueness is useful. Deadalnix' 
proposal goes in this direction.
Dec 17 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 17 December 2014 at 11:13:02 UTC, Marc Schütz wrote:

 On Wednesday, 17 December 2014 at 07:48:52 UTC, bearophile 
 wrote:

 Walter Bright:


 I'm afraid I don't understand at all what you wrote.


 Perhaps reading about linear type systems could help:

 http://en.wikipedia.org/w/index.php?title=Substructural_type_system&redirect=no#Linear_type_systems


 But note that we want a "relaxed" linear type system. In 
 general, we are fine with multiple aliases, although there are


Pointer aliasing is an optimization killer, which is why C99 
added the «restrict» keyword. I think in most cases the use of 
stack allocated objects tend to be alias-free, so it is important 
that this knowledge is retained so that the compiler can make use 
of it.

http://en.wikipedia.org/wiki/Pointer_aliasing
Dec 17 2014
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Wednesday, 17 December 2014 at 12:29:21 UTC, Ola Fosheim 
Grøstad wrote:

 On Wednesday, 17 December 2014 at 11:13:02 UTC, Marc Schütz 
 wrote:

 On Wednesday, 17 December 2014 at 07:48:52 UTC, bearophile 
 wrote:

 Walter Bright:


 I'm afraid I don't understand at all what you wrote.


 Perhaps reading about linear type systems could help:

 http://en.wikipedia.org/w/index.php?title=Substructural_type_system&redirect=no#Linear_type_systems


 But note that we want a "relaxed" linear type system. In 
 general, we are fine with multiple aliases, although there are


 Pointer aliasing is an optimization killer, which is why C99 
 added the «restrict» keyword. I think in most cases the use of 
 stack allocated objects tend to be alias-free, so it is 
 important that this knowledge is retained so that the compiler 
 can make use of it.

 http://en.wikipedia.org/wiki/Pointer_aliasing


I know, this is one of those applications. The one I was thinking 
of is const-borrowing (useful to prevent iterator invalidation 
and a bunch of other things).

I think that enforcing alias-freeness by default (the way a 
strict linear type system requires) doesn't fit too well into the 
existing language. Some way of signalling this to the compiler 
will therefore be required. It might be as simple as C's 
`restrict` keyword (but it needs to be enforced!), or better yet, 
an `unrestrict` keyword. Parameters would be alias-free by 
default unless explicitly specified otherwise. Local aliases 
(inside one function) are however ok, as long as not more than 
one is passed to another function at the same time.
Dec 17 2014
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 17 December 2014 at 13:35:53 UTC, Marc Schütz wrote:

 I think that enforcing alias-freeness by default (the way a 
 strict linear type system requires) doesn't fit too well into 
 the existing language. Some way of signalling this to the 
 compiler will therefore be required. It might be as simple as 
 C's `restrict` keyword (but it needs to be enforced!), or 
 better yet, an `unrestrict` keyword. Parameters would be 
 alias-free by default unless explicitly specified otherwise. 
 Local aliases (inside one function) are however ok, as long as 
 not more than one is passed to another function at the same 
 time.


Hmm, I kind of feel that D is having too many parameter 
keywords/types/qualifiers. Parameter specification is primarily 
useful where it provides information about constraints that the 
programmer need to know about when using a library.

"restricted scope ref" could be a useful default for 
non-templated functions, then use templated ownership reference 
typing or inference to go beyond it.

I think that if one improved the templating system so that it was 
capable of analysing similarities between instantiations and use 
heuristics to analyse when a new instance is profitable then you 
could make functions templated by default and propagate knowledge 
from the call site to the template-instantiation mechanism of the 
compiler. So if the reference is known to be aliasfree then the 
compiler can use heuristics to determine if it should instantiate 
a "restricted" version or use a slower general version of the 
templated function.

Similarily, if refcounters always are on offset 0 and destructors 
are virtual, then you only need a single container method 
template instance for refcouting references, even if you have 
many types using the same method. If refcounters are big enough, 
you can have a unique_ptr that just set it to a large value and 
use the same template instance for a unique_ptr and 
refcounted_ptr behind the scene when referencing objects that 
provide a refcount field.

There are also many other things that could be done to enhance 
the template system, like querying properties of the block in 
loops that iterate over a "range", so that you can instantiate 
different "optimized ranges" depending on whether the loop 
contains a "break" statement or not. Or like Jonathan Blow's 
language, where you can issue a "remove element and continue" 
like statement within a foreach-loop,  which might require a 
different "range iteration" implementation.
Dec 17 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 17 December 2014 at 12:29:21 UTC, Ola Fosheim 
Grøstad wrote:

 On Wednesday, 17 December 2014 at 11:13:02 UTC, Marc Schütz 
 wrote:

 On Wednesday, 17 December 2014 at 07:48:52 UTC, bearophile 
 wrote:

 Walter Bright:


 I'm afraid I don't understand at all what you wrote.


 Perhaps reading about linear type systems could help:

 http://en.wikipedia.org/w/index.php?title=Substructural_type_system&redirect=no#Linear_type_systems


 But note that we want a "relaxed" linear type system. In 
 general, we are fine with multiple aliases, although there are


 Pointer aliasing is an optimization killer, which is why C99 
 added the «restrict» keyword. I think in most cases the use of 
 stack allocated objects tend to be alias-free, so it is 
 important that this knowledge is retained so that the compiler 
 can make use of it.

 http://en.wikipedia.org/wiki/Pointer_aliasing


If you have owner ship, you have free. If you have a pair 
alloc/free then you can promote on stack.

This is a much more powerful way to handle things, as this take 
advantage of inlining.
Dec 17 2014
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 17 December 2014 at 13:48:24 UTC, deadalnix wrote:

 If you have owner ship, you have free. If you have a pair 
 alloc/free then you can promote on stack.

 This is a much more powerful way to handle things, as this take 
 advantage of inlining.


More powerful than what?

Rather than thinking in terms of alloc/free it might be helpful 
to think about it more generally as resources, use, reuse and 
side-effects. On the stack you get to reuse memory when the 
variable is not longer live, meaning you junk writes if the 
results cannot be read and reuse the memory "prematurely".

http://en.wikipedia.org/wiki/Live_variable_analysis

If you are saying that turning GC allocations automatically into 
stack allocations is a good direction for D2, then I agree.
Dec 17 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 17 December 2014 at 05:28:35 UTC, Walter Bright 
wrote:

 On 12/16/2014 5:37 AM, "Ola Fosheim Grøstad" 
 <ola.fosheim.grostad+dlang gmail.com>" wrote:

 The current proposal is either too limiting or not limiting 
 enough.


 I'm afraid I don't understand at all what you wrote.


Well, look at it this way: D1 took 1990s style C++ application 
programming model and made it more convenient with a GC. Having a 
GC-memory model with the possibility of stack/heap allocations as 
optimizations can keep the language simple. D2 is more of a 
library author's language, but without GC you need to deal with 
different types of ownership.

If you toss out the GC, and want the same level of safety and 
generality you either:

1. need to track ownership add dynamic runtime checks
2. provide a full blown proof system (not realistic)
3. provide means for the programmer to "help" the semantic 
analysis

What you want is a combination of 1 and 3, but you don't need to 
keep them separate. Since refcounting is expensive, you don't 
want that everywhere. For stack allocated objects you have an 
implicit "refcounting" in the nesting of scopes, you only need to 
track scope-depth.

If the compiler knows that a returned object is stack allocated, 
then it can check that by comparing against the stack frame 
address since the stack grows downwards in an ordered fashion. 
And in most cases it should be able to elide the check.

If you also maintain information about order (stack depth) among 
the stack allocated objects a function receives through reference 
parameters (for every function call), then you can safely combine 
the stack allocated objects you receive through parameters in 
aggregates, like inserting an stack allocated element into a 
container.

Since D is increasingly becoming a library author's language, you 
probably also would benefit more from strengthening the template 
support by having a templated ref-type rather than "scope ref". 
You could define protocols (e.g. predefined UDAs) that tells the 
compiler what kind of key properties the library type provides, 
such as ownership. Then use that for a more generic approach to 
optimization and escape analysis and let the 
ownership-reference-types be library types. If semantic analysis 
gets rid of the dynamic aspects of a referencs (such as reference 
counting) then the compiler should be able to use the same 
function body, and in other cases just add a inferred wrapper to 
a generic function body. So the code bloat can be limited.

If all references to external resources (like the heap) in the 
stack allocated object is owned by the object, then you can treat 
those as a whole. But you need the compiler to understand what 
ownership is if it is to guarantee memory safety in the kind of 
situations where memory safety is known to be difficult to 
achieve.

The alternative is to state that  safe is not covering non-GC 
memory and that one should stick to vetted library code when 
writing  safe code that does not use the GC. I actually find that 
acceptable for D2. In D2 you would be better off spending the 
effort at finding ways to turn automatically turn GC allocations 
into stack allocations as optimizations. Then you can find a more 
general and powerful ownership-oriented approach for non-GC 
applications for a new major version of D.

It is not realistic that people will annotate with "scope ref", 
and compiler inference could make updates to libraries break 
application code if you get false negatives. Function signatures 
are contracts, they ought to be explicit for libraries. So having 
library functions act as "scope ref" without it being explicit in 
the signature is not necessarily a good idea.
Dec 17 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 16 December 2014 at 01:00:47 UTC, Walter Bright wrote:

 On 12/15/2014 5:38 AM, deadalnix wrote:

 On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote:

 For me "scopeness" is a property of "view", not object itself 
 - this also
 makes ownership method of actual data irrelevant. Only 
 difference between GC
 owned data and stack allocated one is that former can have 
 scoped view
 optionally but for the latter compiler must force it as the 
 only available.


 Ha finally something start to make sense here.


 Well, the DIP does defined scope in terms of a 'view' in just 
 this manner.

 I am obviously terrible at explaining things.


Such notion of "view" requires at least some elements of 
transitivity to be practical in my opinion. Also with my 
definition in mind your example of tree that stores scope nodes 
makes absolutely no sense unless whole tree itself is scoped (and 
nodes are thus scoped transitively). Such view is always assumes 
worst case about ownership and shouldn't persist in any form (as 
that would require some serious ownership tracking).

I don't think you have explained your case that bad - we simply 
have a very different use cases in mind as "primary" use case.
Dec 19 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/19/2014 9:44 PM, Dicebot wrote:

 Such notion of "view" requires at least some elements of transitivity to be
 practical in my opinion.


I have no idea how "some elements of transitivity" can even work. It's either 
transitive or its not. Please don't think of scope in terms of ownership, 
ownership is an orthogonal issue.



 Also with my definition in mind your example of tree
 that stores scope nodes makes absolutely no sense unless whole tree itself is
 scoped (and nodes are thus scoped transitively). Such view is always assumes
 worst case about ownership and shouldn't persist in any form (as that would
 require some serious ownership tracking).


This is definitely conflating scope and ownership.
Dec 20 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 December 2014 at 21:39:44 UTC, Walter Bright 
wrote:

 On 12/19/2014 9:44 PM, Dicebot wrote:

 Such notion of "view" requires at least some elements of 
 transitivity to be
 practical in my opinion.


 I have no idea how "some elements of transitivity" can even 
 work. It's either transitive or its not. Please don't think of 
 scope in terms of ownership, ownership is an orthogonal issue.


.. and here I was about you to do exactly the same :P What in 
example I show makes you think of ownership?

When I was speaking about "some elements of transitivity" I was 
thinking in a way of keeping scope storage class but transitively 
applying same restrictions to all data accessible through it AS 
IF it had scope storage class on its own - while still making 
illegal to use scope as a separate type qualifier.


 Also with my definition in mind your example of tree
 that stores scope nodes makes absolutely no sense unless whole 
 tree itself is
 scoped (and nodes are thus scoped transitively). Such view is 
 always assumes
 worst case about ownership and shouldn't persist in any form 
 (as that would
 require some serious ownership tracking).


 This is definitely conflating scope and ownership.


No, it is exactly the other way around. The very point of what I 
am saying is that you DOESN'T CARE about ownership as long as 
worst case scenario is assumed.   I have zero idea why you 
identify it is conflating as ownership when it is explicitly 
designed to be distinct.
Dec 21 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/21/2014 2:06 AM, Dicebot wrote:

 No, it is exactly the other way around. The very point of what I am saying is
 that you DOESN'T CARE about ownership as long as worst case scenario is
 assumed.   I have zero idea why you identify it is conflating as ownership when
 it is explicitly designed to be distinct.


The point of transitive scoping would be if the root owned the data reachable 
through the root.
Dec 21 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 22 December 2014 at 03:07:53 UTC, Walter Bright wrote:

 On 12/21/2014 2:06 AM, Dicebot wrote:

 No, it is exactly the other way around. The very point of what 
 I am saying is
 that you DOESN'T CARE about ownership as long as worst case 
 scenario is
 assumed.   I have zero idea why you identify it is conflating 
 as ownership when
 it is explicitly designed to be distinct.


 The point of transitive scoping would be if the root owned the 
 data reachable through the root.


Quoting myself:


 For me "scopeness" is a property of "view", not object itself - 
 this also makes ownership method of actual data irrelevant. Only
 difference between GC owned data and stack allocated one is that
 former can have scoped view optionally but for the latter
 compiler must force it as the only available.


It doesn't matter of root owns the data. We _assume_ that as 
worst case scenario and allowed actions form a strict subset of 
allowed actions for any other ownership situation. Such `scope` 
for stack/GC is same as `const` for mutable/immutable  - common 
denominator.

Point of transitive scope is to make easy to expose complex 
custom data structures without breaking memory safety.
Dec 22 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/22/2014 12:04 AM, Dicebot wrote:

 Point of transitive scope is to make easy to expose complex custom data
 structures without breaking memory safety.


I do understand that. Making it work with the type system is another matter 
entirely - it's far more complex than just adding a qualifier. 'inout' looks 
simple but still has ongoing issues.

And the thing is, wrappers can be used instead of qualifiers, in the same
places 
in the same way. It's much simpler.
Dec 22 2014
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 22 December 2014 at 20:51:46 UTC, Walter Bright wrote:

 On 12/22/2014 12:04 AM, Dicebot wrote:

 Point of transitive scope is to make easy to expose complex 
 custom data
 structures without breaking memory safety.


 I do understand that. Making it work with the type system is 
 another matter entirely - it's far more complex than just 
 adding a qualifier. 'inout' looks simple but still has ongoing 
 issues.

 And the thing is, wrappers can be used instead of qualifiers, 
 in the same places in the same way. It's much simpler.


Ok, lets consider some examples of code I'd want to work with 
wrapper approach.

Basic tree structure:

struct Node
{
     Node* left, right;
     Payload payload;
}

struct Tree
{
     Node* root;
}

Restricting public access to nodes to prevent storing persistent 
node pointers.

Your proposed approach:

struct Wrapper(T)
{
     scope T wrapped;
     alias wrapped this;
}

struct Node
{
     private:
         Node* _left, _right;
     public:
         auto left()  property { return Wrapper!(Node*)(_left); }
         auto right()  property { return Wrapper!(Node*)(_right); }

         Payload payload;
}

struct Tree
{
     private:
         Node* _root;

     public:
         auto root()  property { return Wrapper!(Node*)(_root); }
}

Am I correct?

If yes, this is exactly the breakage issue I was speaking about : 
any code that looked like `is(typeof(Tree.root) == Node*)` will 
stop working.

With my latest proposal (pseudo-transitive storage class which is 
not part of a type) getters would look simply like `scope Node* 
root()  property { return _root; }` and `is(typeof(Tree.root) == 
Node*)` still remained true. Only impact on existing would be 
restriction of operations to take address or assign a pointer.

I am pretty sure I am missing many complicated parts here but 
this seems like a big deal in retro-fitting existing libraries to 
use scope after it was added.
Dec 29 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 12 December 2014 at 00:13:10 UTC, deadalnix wrote:

 On Thursday, 11 December 2014 at 13:55:55 UTC, Marc Schütz 
 wrote:

 This is a point that most people don't seem to understand yet, 
 and which wasn't obvious for me either, at the beginning:

 * A `ref` parameter means that it cannot escape the function, 
 _except_ by return.
 * A `scope ref` parameter means that it cannot escape the 
 function  _ever_, not even by return.
 * A `scope ref` return means that it cannot leave the current 
 statement.

 Therefore, a `scope ref` return value can be passed on to the 
 next function as a `ref` argument. If that function again 
 returns a reference (even if not explicitly designated as 
 `scope`), the compiler will treat it as if it were `scope ref`.


 No, it understood.


Steven hadn't, evidently.


 It is simply not useful.


I'm not convinced either.


 I agree, this is important. In my proposal, this works without 
 transitivity. The wrapper stores the pointer as a `scope` 
 member, then by copying the wrapper, the pointer gets copied 
 implicitly, to which the normal scope restrictions apply 
 (`scope` on members means "will not outlive the aggregate"). 
 If it stored it as normal non-scope pointer, it couldn't get 
 assigned in the first place.


 Wut ? You cante store anything with grear lifetime, including 
 non scope things (as they'll have infinite lifetime). Meaning 
 the only thing you know, is that thing are possibly scoped.

 Meaning you have to assume infinite lifetime with every 
 indirection, which make this proposal useless.


My comments above don't refer to this proposal, but my original 
one:

     struct Wrapper(T) {
         scope T payload;
         // used to be:
         // scope!this T payload;
     }

     scope int a;
     auto w = Wrapper!(int*)(&a); // ok
     scope int b;
     w.payload = &b;              // error, w (and therefore 
w.payload)
                                  // lives longer than b
     Wrapper!(int*) w2 = w;       // ok, lifetime(w2) < lifetime(w)
     w = w2;                      // error, equivalent to
                                  // w.payload = w2.payload

Therefore, wrapping is safe without transitivity of scope.
Dec 12 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/11/2014 4:13 PM, deadalnix wrote:

 On Thursday, 11 December 2014 at 13:55:55 UTC, Marc Schütz wrote:

 On Thursday, 11 December 2014 at 12:48:05 UTC, Manu via Digitalmars-d wrote:

 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 12/7/2014 6:12 AM, Dicebot wrote:

 But from existing cases it doesn't seem working good enough. For example,
 not
 being able to represent idiom of `scope ref int foo(scope ref int x) {
 return x;
 }` seems very limiting.



  scope ref int foo(ref int x);

 will do it.


 Will it? It looks like foo can't be called with scope data?


 This is a point that most people don't seem to understand yet, and which
 wasn't obvious for me either, at the beginning:

 * A `ref` parameter means that it cannot escape the function, _except_ by
return.
 * A `scope ref` parameter means that it cannot escape the function  _ever_,
 not even by return.
 * A `scope ref` return means that it cannot leave the current statement.

 Therefore, a `scope ref` return value can be passed on to the next function as
 a `ref` argument. If that function again returns a reference (even if not
 explicitly designated as `scope`), the compiler will treat it as if it were
 `scope ref`.


 No, it understood. It is simply not useful.


The use case is described in the DIP under the 'Scope Ref' heading.



 I agree, this is important. In my proposal, this works without transitivity.
 The wrapper stores the pointer as a `scope` member, then by copying the
 wrapper, the pointer gets copied implicitly, to which the normal scope
 restrictions apply (`scope` on members means "will not outlive the
 aggregate"). If it stored it as normal non-scope pointer, it couldn't get
 assigned in the first place.


 Wut ? You cante store anything with grear lifetime, including non scope things
 (as they'll have infinite lifetime). Meaning the only thing you know, is that
 thing are possibly scoped.

 Meaning you have to assume infinite lifetime with every indirection, which make
 this proposal useless.


It means you'll have to wrap the next level of indirection in a ref counting
(or 
equivalent) manner.

The great thing about GC is you can leave everything to the GC. But when 
explicitly managing memory, you have to decide for EVERY pointer who owns it, 
when it can be released, etc. Having a transitive scope is NOT going to resolve 
that for you, it isn't even going to help.
Dec 12 2014
prev sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 December 2014 at 23:55, via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Thursday, 11 December 2014 at 12:48:05 UTC, Manu via Digitalmars-d wrote:

 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 12/7/2014 6:12 AM, Dicebot wrote:

 But from existing cases it doesn't seem working good enough. For
 example,
 not
 being able to represent idiom of `scope ref int foo(scope ref int x) {
 return x;
 }` seems very limiting.




   scope ref int foo(ref int x);

 will do it.



 Will it? It looks like foo can't be called with scope data?



 This is a point that most people don't seem to understand yet, and which
 wasn't obvious for me either, at the beginning:

 * A `ref` parameter means that it cannot escape the function, _except_ by
 return.
 * A `scope ref` parameter means that it cannot escape the function  _ever_,
 not even by return.
 * A `scope ref` return means that it cannot leave the current statement.

 Therefore, a `scope ref` return value can be passed on to the next function
 as a `ref` argument. If that function again returns a reference (even if not
 explicitly designated as `scope`), the compiler will treat it as if it were
 `scope ref`.


Ummm. I reckon there's a good reason that people don't seem to
understand this... because it would have difficulty being any more
unintuitive!
It's contrary to the behaviour of basically everything else in D!
const, pure, nothrow, etc... they all work a reliable and consistent way.

Also, I still don't think I get it... a scope-ref return, which has a
restriction as you say, can be subverted by being piped through a
function that receives and returns it as ref?
What is the point?


 I'm also quite uneasy with the fact that scope would not be transitive
 as a storage class. What happens when it's applied to a value type,
 like a struct, that contains some pointers? An adaptation wrapper for
 a single pointer is super common; ie, a tiny struct passed by value
 with scope needs to have it's contained pointer receive the scope-ness
 of the argument.



 I agree, this is important. In my proposal, this works without transitivity.
 The wrapper stores the pointer as a `scope` member, then by copying the
 wrapper, the pointer gets copied implicitly, to which the normal scope
 restrictions apply (`scope` on members means "will not outlive the
 aggregate"). If it stored it as normal non-scope pointer, it couldn't get
 assigned in the first place. (Additionally, some higher level tricks are
 possible if we allow scope for value types and overloading on scope.)


How can a member be marked scope? Apparently it's a storage class, not
a type constructor... we can only attribute storage classes to
function args/results(?).
Dec 12 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 12 December 2014 at 08:42:22 UTC, Manu via 
Digitalmars-d wrote:

 On 11 December 2014 at 23:55, via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 This is a point that most people don't seem to understand yet, 
 and which
 wasn't obvious for me either, at the beginning:

 * A `ref` parameter means that it cannot escape the function, 
 _except_ by
 return.
 * A `scope ref` parameter means that it cannot escape the 
 function  _ever_,
 not even by return.
 * A `scope ref` return means that it cannot leave the current 
 statement.

 Therefore, a `scope ref` return value can be passed on to the 
 next function
 as a `ref` argument. If that function again returns a 
 reference (even if not
 explicitly designated as `scope`), the compiler will treat it 
 as if it were
 `scope ref`.


 Ummm. I reckon there's a good reason that people don't seem to
 understand this... because it would have difficulty being any 
 more
 unintuitive!
 It's contrary to the behaviour of basically everything else in 
 D!
 const, pure, nothrow, etc... they all work a reliable and 
 consistent way.


Unfortunately I have to agree.


 Also, I still don't think I get it... a scope-ref return, which 
 has a
 restriction as you say, can be subverted by being piped through 
 a
 function that receives and returns it as ref?
 What is the point?


It cannot be subverted. As soon as you pass something into a 
function G() as `ref` that's been returned from a function F() 
via `scope ref`, the compiler needs to treat what is returned 
from G() as `scope ref`, even if it's only declared as `ref`.

As you say, it's unintuitive and non-obvious.


 I'm also quite uneasy with the fact that scope would not be 
 transitive
 as a storage class. What happens when it's applied to a value 
 type,
 like a struct, that contains some pointers? An adaptation 
 wrapper for
 a single pointer is super common; ie, a tiny struct passed by 
 value
 with scope needs to have it's contained pointer receive the 
 scope-ness
 of the argument.



 I agree, this is important. In my proposal, this works without 
 transitivity.
 The wrapper stores the pointer as a `scope` member, then by 
 copying the
 wrapper, the pointer gets copied implicitly, to which the 
 normal scope
 restrictions apply (`scope` on members means "will not outlive 
 the
 aggregate"). If it stored it as normal non-scope pointer, it 
 couldn't get
 assigned in the first place. (Additionally, some higher level 
 tricks are
 possible if we allow scope for value types and overloading on 
 scope.)


 How can a member be marked scope? Apparently it's a storage 
 class, not
 a type constructor... we can only attribute storage classes to
 function args/results(?).


I was talking about my proposal here, where it's a type modifier, 
not a storage class.

deadalnix also brought up the problem of transitivity. This, too, 
would simply go away if it's a type modifier.
Dec 12 2014
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 December 2014 at 22:18, via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Friday, 12 December 2014 at 08:42:22 UTC, Manu via Digitalmars-d wrote:

 How can a member be marked scope? Apparently it's a storage class, not
 a type constructor... we can only attribute storage classes to
 function args/results(?).



 I was talking about my proposal here, where it's a type modifier, not a
 storage class.


Oh okay. Is that still on the table?



 deadalnix also brought up the problem of transitivity. This, too, would
 simply go away if it's a type modifier.


Just for clarity, does type modifier mean the same thing as type constructor?
Dec 12 2014
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 12 December 2014 at 12:34:40 UTC, Manu via 
Digitalmars-d wrote:

 On 12 December 2014 at 22:18, via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On Friday, 12 December 2014 at 08:42:22 UTC, Manu via 
 Digitalmars-d wrote:

 How can a member be marked scope? Apparently it's a storage 
 class, not
 a type constructor... we can only attribute storage classes to
 function args/results(?).



 I was talking about my proposal here, where it's a type 
 modifier, not a
 storage class.


 Oh okay. Is that still on the table?


Dunno, maybe? DIP69 doesn't look very well-received at the 
moment...


 deadalnix also brought up the problem of transitivity. This, 
 too, would
 simply go away if it's a type modifier.


 Just for clarity, does type modifier mean the same thing as 
 type constructor?


Yes. "Type modifier" = const, shared, immutable, etc.; "type 
constructor" = the syntactical construct by which a type is 
constructed/modified. Andrei suggested this differentiation some 
time ago (sorry, don't have a reference at hand).
Dec 12 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 12 December 2014 at 13:38:10 UTC, Marc Schütz wrote:

 Yes. "Type modifier" = const, shared, immutable, etc.; "type 
 constructor" = the syntactical construct by which a type is 
 constructed/modified. Andrei suggested this differentiation 
 some time ago (sorry, don't have a reference at hand).


That won't fly. scope turtle left, type qualifier turtle right.
Dec 12 2014
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 12 December 2014 at 12:34:40 UTC, Manu via
Digitalmars-d wrote:

 deadalnix also brought up the problem of transitivity. This, 
 too, would
 simply go away if it's a type modifier.


 Just for clarity, does type modifier mean the same thing as 
 type constructor?


It appears to me that scopeness is a completely different beast
than type qualifier, and type qualifier won't work.

I think type modifier was just made up here. And why not ? It is
more like some metadata (lifetime) being associated with value
and symbols, like a type would, but that isn't really per se a
type as D understands it right now.
Dec 12 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/11/2014 4:47 AM, Manu via Digitalmars-d wrote:

 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 12/7/2014 6:12 AM, Dicebot wrote:

 But from existing cases it doesn't seem working good enough. For example,
 not
 being able to represent idiom of `scope ref int foo(scope ref int x) {
 return x;
 }` seems very limiting.



    scope ref int foo(ref int x);

 will do it.


 Will it? It looks like foo can't be called with scope data?


Yes, it can be.



 I don't have the perfect proposal, but I feel very strongly about 2 things:
 1. It must not be a storage class; the concept was a disaster with
 ref, and I struggle with this more frequently than any other 'feature'
 in D.


I simply do not understand why distinguishing beteen ref and not-ref is a 
cornerstone of everything you do.



 2. I feel it's a big mistake to separate it from the type system,
 which I think most agree, is D's greatest asset by far. Manipulating
 types is simple and convenient using the type system, and I think
 manipulating scope will be just as important as any other attribute as
 soon as even slightly complex use cases begin to arise.


Consider a ref counted type, RC!T. If scope were transitive, then you could not 
have, say, a tree where the edges were RC!T. I.e., the payload of an RC type 
should not be forced to be scope.
Dec 11 2014
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
12-Dec-2014 00:41, Walter Bright пишет:

 On 12/11/2014 4:47 AM, Manu via Digitalmars-d wrote:

 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:





 I don't have the perfect proposal, but I feel very strongly about 2
 things:
 1. It must not be a storage class; the concept was a disaster with
 ref, and I struggle with this more frequently than any other 'feature'
 in D.


 I simply do not understand why distinguishing beteen ref and not-ref is
 a cornerstone of everything you do.


I would hazard a guess that const T& is something lots of C++ got 
addicted to as the most common way of parameter passing. There is 
nothing exactly like that in D, auto ref was quite close but only for 
templates and it instantiates 2 distinct bodies for ref and non-ref at 
the moment.

It allows easy pass by ref for both r-values and l-values with logical 
const and is likely what Manu is referring(!) to.

Truth be told it's not always the fastest way, in fact Boost C++ has 
template for optimal parameter passing - call_traits<T>::param_type  but 
it must be too cumbersome for actual use as I never seen it used in the 
wild.
http://www.boost.org/doc/libs/1_57_0/libs/utility/call_traits.htm

-- 
Dmitry Olshansky
Dec 11 2014
prev sibling next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 11 December 2014 at 21:41:11 UTC, Walter Bright 
wrote:

 On 12/11/2014 4:47 AM, Manu via Digitalmars-d wrote:

 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 12/7/2014 6:12 AM, Dicebot wrote:

 But from existing cases it doesn't seem working good enough. 
 For example,
 not
 being able to represent idiom of `scope ref int foo(scope 
 ref int x) {
 return x;
 }` seems very limiting.



   scope ref int foo(ref int x);

 will do it.


 Will it? It looks like foo can't be called with scope data?


 Yes, it can be.



 I don't have the perfect proposal, but I feel very strongly 
 about 2 things:
 1. It must not be a storage class; the concept was a disaster 
 with
 ref, and I struggle with this more frequently than any other 
 'feature'
 in D.


 I simply do not understand why distinguishing beteen ref and 
 not-ref is a cornerstone of everything you do.


Because he requires control over function ABIs for both 
inter-language communication and performance.

In binding D to IDL (Interactive Data, not Interface Description) 
I found ref often required special casing.
Dec 11 2014
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 December 2014 at 08:53, John Colvin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Thursday, 11 December 2014 at 21:41:11 UTC, Walter Bright wrote:

 On 12/11/2014 4:47 AM, Manu via Digitalmars-d wrote:

 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 12/7/2014 6:12 AM, Dicebot wrote:

 But from existing cases it doesn't seem working good enough. For
 example,
 not
 being able to represent idiom of `scope ref int foo(scope ref int x) {
 return x;
 }` seems very limiting.




   scope ref int foo(ref int x);

 will do it.



 Will it? It looks like foo can't be called with scope data?



 Yes, it can be.



 I don't have the perfect proposal, but I feel very strongly about 2
 things:
 1. It must not be a storage class; the concept was a disaster with
 ref, and I struggle with this more frequently than any other 'feature'
 in D.



 I simply do not understand why distinguishing beteen ref and not-ref is a
 cornerstone of everything you do.



 Because he requires control over function ABIs for both inter-language
 communication and performance.

 In binding D to IDL (Interactive Data, not Interface Description) I found
 ref often required special casing.


Thank you. I feel like I'm just repeating myself over and over again.
Dec 12 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 4:20 AM, Manu via Digitalmars-d wrote:

 On 12 December 2014 at 08:53, John Colvin via Digitalmars-d

 Because he requires control over function ABIs for both inter-language
 communication and performance.

 In binding D to IDL (Interactive Data, not Interface Description) I found
 ref often required special casing.


 Thank you. I feel like I'm just repeating myself over and over again.



Example, please.
Dec 12 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 11 December 2014 at 21:41:11 UTC, Walter Bright 
wrote:

 Consider a ref counted type, RC!T. If scope were transitive, 
 then you could not have, say, a tree where the edges were RC!T. 
 I.e., the payload of an RC type should not be forced to be 
 scope.


I don't see how this is related. It would be perfectly ok to 
declare root of such tree scope if it was transitive (as long as 
it only controls access and does not attempt early destruction).

You may want to explain this in details in DIP if this is 
motivating case.
Dec 11 2014
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 12 December 2014 at 06:06:40 UTC, Dicebot wrote:

 On Thursday, 11 December 2014 at 21:41:11 UTC, Walter Bright 
 wrote:

 Consider a ref counted type, RC!T. If scope were transitive, 
 then you could not have, say, a tree where the edges were 
 RC!T. I.e., the payload of an RC type should not be forced to 
 be scope.


 I don't see how this is related. It would be perfectly ok to 
 declare root of such tree scope if it was transitive (as long 
 as it only controls access and does not attempt early 
 destruction).


That is my proposal. However, it is not as simply as you think it 
is without making it a type qualifier (which is not desirable).

In effect, that mean that you can see something that has infinite 
lifetime through a scope reference, so you need to track lifetime 
rvalue and lvalue differently.

I'm still for it. The current proposal is not powerful enough to 
pull its weight.
Dec 11 2014
parent "Dicebot" <public dicebot.lv> writes:
On Friday, 12 December 2014 at 06:57:54 UTC, deadalnix wrote:

 On Friday, 12 December 2014 at 06:06:40 UTC, Dicebot wrote:

 On Thursday, 11 December 2014 at 21:41:11 UTC, Walter Bright 
 wrote:

 Consider a ref counted type, RC!T. If scope were transitive, 
 then you could not have, say, a tree where the edges were 
 RC!T. I.e., the payload of an RC type should not be forced to 
 be scope.


 I don't see how this is related. It would be perfectly ok to 
 declare root of such tree scope if it was transitive (as long 
 as it only controls access and does not attempt early 
 destruction).


 That is my proposal. However, it is not as simply as you think 
 it is without making it a type qualifier (which is not 
 desirable).

 In effect, that mean that you can see something that has 
 infinite lifetime through a scope reference, so you need to 
 track lifetime rvalue and lvalue differently.

 I'm still for it. The current proposal is not powerful enough 
 to pull its weight.


I don't hope it will be simple. It is all about making scope as 
simple as possible to keep it useful for idiomatic D code - but 
not simpler.
Dec 11 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/11/2014 10:06 PM, Dicebot wrote:

 On Thursday, 11 December 2014 at 21:41:11 UTC, Walter Bright wrote:

 Consider a ref counted type, RC!T. If scope were transitive, then you could
 not have, say, a tree where the edges were RC!T. I.e., the payload of an RC
 type should not be forced to be scope.


 I don't see how this is related. It would be perfectly ok to declare root of
 such tree scope if it was transitive (as long as it only controls access and
 does not attempt early destruction).


Are you suggesting two kinds of scope - transitive and non-transitive?

The more I think about it, the more ref counting is the definitive case, as
just 
about everything else can be explained in terms of how ref counting works.
Dec 11 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Friday, 12 December 2014 at 07:48:21 UTC, Walter Bright wrote:

 I don't see how this is related. It would be perfectly ok to 
 declare root of
 such tree scope if it was transitive (as long as it only 
 controls access and
 does not attempt early destruction).


 Are you suggesting two kinds of scope - transitive and 
 non-transitive?

 The more I think about it, the more ref counting is the 
 definitive case, as just about everything else can be explained 
 in terms of how ref counting works.


I don't see applicability of non-transitive scope because I don't 
understand the problem with the tree or reference counting you 
have mentioned - that is why I suggested to amend DIP to put 
explanation there.
Dec 11 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/11/2014 11:55 PM, Dicebot wrote:

 On Friday, 12 December 2014 at 07:48:21 UTC, Walter Bright wrote:

 I don't see how this is related. It would be perfectly ok to declare root of
 such tree scope if it was transitive (as long as it only controls access and
 does not attempt early destruction).


 Are you suggesting two kinds of scope - transitive and non-transitive?

 The more I think about it, the more ref counting is the definitive case, as
 just about everything else can be explained in terms of how ref counting works.


 I don't see applicability of non-transitive scope because I don't understand
the
 problem with the tree or reference counting you have mentioned - that is why I
 suggested to amend DIP to put explanation there.


Consider every pointer in a tree to be a ref counted pointer. There is no 
purpose to making scope transitive - traveling from one node to the next goes 
through the usual ref counting mechanism, and the ref counting wrapper can 
control that.

Also, consider a data structure:

A -> B
C -> B

If scope is transitive, how is C going to simultaneously access B?
Dec 12 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 12 December 2014 at 07:48:21 UTC, Walter Bright wrote:

 Are you suggesting two kinds of scope - transitive and 
 non-transitive?


Non transitive scope can be added without any language extension. 
There is no point doing much with the scope keyword if 
indirection aren't in the loop.


 The more I think about it, the more ref counting is the 
 definitive case, as just about everything else can be explained 
 in terms of how ref counting works.


ref counting is just one form of ownership.
Dec 12 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 12:16 AM, deadalnix wrote:

 On Friday, 12 December 2014 at 07:48:21 UTC, Walter Bright wrote:

 Are you suggesting two kinds of scope - transitive and non-transitive?


 Non transitive scope can be added without any language extension.


In order for it to work, the holes in the language that enabled escapes had to 
be plugged. That's what this proposal does - plug the holes.



 ref counting is just one form of ownership.


My point is if that works with this proposal, then the other forms can work,
too.
Dec 12 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 12 December 2014 at 08:44:56 UTC, Walter Bright wrote:

 On 12/12/2014 12:16 AM, deadalnix wrote:

 On Friday, 12 December 2014 at 07:48:21 UTC, Walter Bright 
 wrote:

 Are you suggesting two kinds of scope - transitive and 
 non-transitive?


 Non transitive scope can be added without any language 
 extension.


 In order for it to work, the holes in the language that enabled 
 escapes had to be plugged. That's what this proposal does - 
 plug the holes.


The holes covered by a non transitive scope are already undefined 
behavior.


 ref counting is just one form of ownership.


 My point is if that works with this proposal, then the other 
 forms can work, too.


No, unless we wrap every single indirection (pointer, slice, 
classes, delegates) into a wrapper. That is not going to fly very 
far.
Dec 12 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 2:28 AM, deadalnix wrote:

 On Friday, 12 December 2014 at 08:44:56 UTC, Walter Bright wrote:

 On 12/12/2014 12:16 AM, deadalnix wrote:

 On Friday, 12 December 2014 at 07:48:21 UTC, Walter Bright wrote:

 Are you suggesting two kinds of scope - transitive and non-transitive?


 Non transitive scope can be added without any language extension.


 In order for it to work, the holes in the language that enabled escapes had to
 be plugged. That's what this proposal does - plug the holes.


 The holes covered by a non transitive scope are already undefined behavior.


Right, but since the current compiler cannot detect them, it is a hole.



 ref counting is just one form of ownership.


 My point is if that works with this proposal, then the other forms can work,
too.


 No, unless we wrap every single indirection (pointer, slice, classes,
delegates)
 into a wrapper. That is not going to fly very far.


The beauty of GC is that you don't have to do any of this. To have safe other 
methods of allocation, there has to be annotation everywhere or use a wrapped 
type in the same places.
Dec 12 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 12 December 2014 at 22:05:21 UTC, Walter Bright wrote:

 forms can work, too.


 No, unless we wrap every single indirection (pointer, slice, 
 classes, delegates)
 into a wrapper. That is not going to fly very far.


 The beauty of GC is that you don't have to do any of this. To 
 have safe other methods of allocation, there has to be 
 annotation everywhere or use a wrapped type in the same places.


The whole point of scope is to reduce that cost, both in term of
runtime (as copy can be elided) but also on the user.
Dec 12 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 2:09 PM, deadalnix wrote:

 On Friday, 12 December 2014 at 22:05:21 UTC, Walter Bright wrote:

 forms can work, too.


 No, unless we wrap every single indirection (pointer, slice, classes,
delegates)
 into a wrapper. That is not going to fly very far.


 The beauty of GC is that you don't have to do any of this. To have safe other
 methods of allocation, there has to be annotation everywhere or use a wrapped
 type in the same places.


 The whole point of scope is to reduce that cost, both in term of
 runtime (as copy can be elided) but also on the user.


Yes, I do understand that :-) but currently doing a safe refcounted type is 
impossible in D. This proposal fixes that.
Dec 12 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 12 December 2014 at 22:34:27 UTC, Walter Bright wrote:

 On 12/12/2014 2:09 PM, deadalnix wrote:
 Yes, I do understand that :-) but currently doing a safe 
 refcounted type is impossible in D. This proposal fixes that.


Actually, no it doesn't. It allow to ensure that this is safe,
module the RC system is the only owner.

This proposal allow for easy to use and fast RC, but certainly do
not make it safe.
Dec 12 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 2:48 PM, deadalnix wrote:

 On Friday, 12 December 2014 at 22:34:27 UTC, Walter Bright wrote:

 On 12/12/2014 2:09 PM, deadalnix wrote:
 Yes, I do understand that :-) but currently doing a safe refcounted type is
 impossible in D. This proposal fixes that.


 Actually, no it doesn't. It allow to ensure that this is safe,
 module the RC system is the only owner.

 This proposal allow for easy to use and fast RC, but certainly do
 not make it safe.


How not?
Dec 12 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright 
wrote:

 How not?


scope a = new Stuff();
scope b = a;

auto rc1 = RC(a);
auto rc2 = RC(b);

// Enjoy the free when it comes !
Dec 12 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 4:07 PM, deadalnix wrote:

 On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright wrote:

 How not?


 scope a = new Stuff();
 scope b = a;

 auto rc1 = RC(a);
 auto rc2 = RC(b);

 // Enjoy the free when it comes !


RC needs to take a type as a parameter, not a symbol.
Dec 12 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 4:07 PM, deadalnix wrote:

 On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright wrote:

 How not?


 scope a = new Stuff();
 scope b = a;

 auto rc1 = RC(a);
 auto rc2 = RC(b);

 // Enjoy the free when it comes !


What's the type signature of RC ?
Dec 12 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 13 December 2014 at 00:52:46 UTC, Walter Bright 
wrote:

 On 12/12/2014 4:07 PM, deadalnix wrote:

 On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright 
 wrote:

 How not?


 scope a = new Stuff();
 scope b = a;

 auto rc1 = RC(a);
 auto rc2 = RC(b);

 // Enjoy the free when it comes !


 What's the type signature of RC ?


RCWrapper!T RC(T)(scope T t) or somethign along these lines.
Dec 12 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 4:59 PM, deadalnix wrote:

 On Saturday, 13 December 2014 at 00:52:46 UTC, Walter Bright wrote:

 On 12/12/2014 4:07 PM, deadalnix wrote:

 On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright wrote:

 How not?


 scope a = new Stuff();
 scope b = a;

 auto rc1 = RC(a);
 auto rc2 = RC(b);

 // Enjoy the free when it comes !


 What's the type signature of RC ?


 RCWrapper!T RC(T)(scope T t) or somethign along these lines.


And how does RC assign it to the wrapper? (It will fail to compile, and so
won't 
be a safety hole.)
Dec 12 2014
prev sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 December 2014 at 07:41, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 12/11/2014 4:47 AM, Manu via Digitalmars-d wrote:

 On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 12/7/2014 6:12 AM, Dicebot wrote:

 But from existing cases it doesn't seem working good enough. For
 example,
 not
 being able to represent idiom of `scope ref int foo(scope ref int x) {
 return x;
 }` seems very limiting.




    scope ref int foo(ref int x);

 will do it.



 Will it? It looks like foo can't be called with scope data?



 Yes, it can be.



 I don't have the perfect proposal, but I feel very strongly about 2
 things:
 1. It must not be a storage class; the concept was a disaster with
 ref, and I struggle with this more frequently than any other 'feature'
 in D.



 I simply do not understand why distinguishing beteen ref and not-ref is a
 cornerstone of everything you do.


I've said so many times, it's the single greatest regular frustration
I encounter, by far.
That is of course a relative measure. It's not the 'cornerstone of
everything I do'; I don't start discussions about things that are not
broken and otherwise painless.
It is the thing that is *the most broken*, and leads to the most edge
cases, general bloat, and text mixins.
It comes up the most frequently, and by that metric alone, I consider
it highest priority on my list.

I've also said many times before, it possibly stems from the fact that
one of the key cornerstones of almost everything I do in D is interact
with other languages.
This is a practical reality, I can't write all my code in D, and have
mountains of existing code to interact with.
Boilerplate is the result. D has powerful systems to automate
boilerplate, like a carrot dangling right in front of my face, but
it's almost always thwarted by ref, and almost exclusively so.

My recent work updating LuaD to support all the features I required
wasted about 80% of my time dealing with ref issues.
In my past C++ bindings solutions, much code, which was otherwise
simple, readable, and elegant, turned into a mess of text mixins,
almost exclusively because ref is broken.



 2. I feel it's a big mistake to separate it from the type system,
 which I think most agree, is D's greatest asset by far. Manipulating
 types is simple and convenient using the type system, and I think
 manipulating scope will be just as important as any other attribute as
 soon as even slightly complex use cases begin to arise.



 Consider a ref counted type, RC!T. If scope were transitive, then you could
 not have, say, a tree where the edges were RC!T. I.e., the payload of an RC
 type should not be forced to be scope.


I'm not sure I quite visualise this correctly...

It sounds like you're talking about the same problem where people
complain with respect to 'const' and you happily tell them to stop
whingeing and deal with it?
(FWIW, I think you made the right call on const. It's never caused me
any hair loss.)

So you're saying that a pointer itself shouldn't be able to escape a
call tree, but the thing it points to should be able to escape just
fine?
It feels like that kinda defeats the purpose... or at least makes the
concept about as reliable as 'const' is in C++.

I guess you're seeing a situation where 'scope' is almost exclusively
useful as a mechanism to tell the RC that it doesn't need to worry
about ref-fiddling, and the thing we're passing isn't interested in
scope restrictions at any level other than that RC optimisation?
I guess I can see your angle... but my reaction is if that's all you
are concerned about, maybe scope is the wrong tool for eliding ref
fiddling in that case :/

I can see how you arrived at the opinion from your other comment
(wherever it was) that you are kinda likening this whole problem to
the RC case, if that's how you're looking at it.
It sounds like what you want is something like head-scope, in the same
way that people often want head-const?
Dec 12 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 4:19 AM, Manu via Digitalmars-d wrote:

 I simply do not understand why distinguishing beteen ref and not-ref is a
 cornerstone of everything you do.


 I've said so many times, it's the single greatest regular frustration
 I encounter, by far.
 That is of course a relative measure. It's not the 'cornerstone of
 everything I do'; I don't start discussions about things that are not
 broken and otherwise painless.
 It is the thing that is *the most broken*, and leads to the most edge
 cases, general bloat, and text mixins.
 It comes up the most frequently, and by that metric alone, I consider
 it highest priority on my list.

 I've also said many times before, it possibly stems from the fact that
 one of the key cornerstones of almost everything I do in D is interact
 with other languages.
 This is a practical reality, I can't write all my code in D, and have
 mountains of existing code to interact with.
 Boilerplate is the result. D has powerful systems to automate
 boilerplate, like a carrot dangling right in front of my face, but
 it's almost always thwarted by ref, and almost exclusively so.

 My recent work updating LuaD to support all the features I required
 wasted about 80% of my time dealing with ref issues.
 In my past C++ bindings solutions, much code, which was otherwise
 simple, readable, and elegant, turned into a mess of text mixins,
 almost exclusively because ref is broken.


You've said this before, many times, but what is lacking is an explanation of 
WHY. What is the pattern, and why do you need that pattern? You say you don't 
like auto ref because it doesn't give you exact control, but what are the cases 
where auto ref is wrong?



 Consider a ref counted type, RC!T. If scope were transitive, then you could
 not have, say, a tree where the edges were RC!T. I.e., the payload of an RC
 type should not be forced to be scope.


 I'm not sure I quite visualise this correctly...


struct Tree {
    RefCount!(Tree*) left;
    RefCount!(Tree*) right;
    ...
}



 So you're saying that a pointer itself shouldn't be able to escape a
 call tree, but the thing it points to should be able to escape just
 fine?


Yes.



 It feels like that kinda defeats the purpose...


You're arguing that a data structure with only one access point is the only
kind 
of data structure in use. With transitive scope, such a data structure would be 
the only one possible!



 I guess you're seeing a situation where 'scope' is almost exclusively
 useful as a mechanism to tell the RC that it doesn't need to worry
 about ref-fiddling, and the thing we're passing isn't interested in
 scope restrictions at any level other than that RC optimisation?
 I guess I can see your angle... but my reaction is if that's all you
 are concerned about, maybe scope is the wrong tool for eliding ref
 fiddling in that case :/


'scope' is a way to say that this use of a pointer does not escape this scope. 
That is incredibly useful.

Recall my statements that pervasive refcounting is a terrible performance 
problem because of all the inc/dec? Knowing that references cannot escape means 
an inc/dec pair can be elided.
Dec 12 2014
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 December 2014 at 08:14, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 12/12/2014 4:19 AM, Manu via Digitalmars-d wrote:

 I simply do not understand why distinguishing beteen ref and not-ref is a
 cornerstone of everything you do.



 I've said so many times, it's the single greatest regular frustration
 I encounter, by far.
 That is of course a relative measure. It's not the 'cornerstone of
 everything I do'; I don't start discussions about things that are not
 broken and otherwise painless.
 It is the thing that is *the most broken*, and leads to the most edge
 cases, general bloat, and text mixins.
 It comes up the most frequently, and by that metric alone, I consider
 it highest priority on my list.

 I've also said many times before, it possibly stems from the fact that
 one of the key cornerstones of almost everything I do in D is interact
 with other languages.
 This is a practical reality, I can't write all my code in D, and have
 mountains of existing code to interact with.
 Boilerplate is the result. D has powerful systems to automate
 boilerplate, like a carrot dangling right in front of my face, but
 it's almost always thwarted by ref, and almost exclusively so.

 My recent work updating LuaD to support all the features I required
 wasted about 80% of my time dealing with ref issues.
 In my past C++ bindings solutions, much code, which was otherwise
 simple, readable, and elegant, turned into a mess of text mixins,
 almost exclusively because ref is broken.



 You've said this before, many times, but what is lacking is an explanation
 of WHY. What is the pattern, and why do you need that pattern? You say you
 don't like auto ref because it doesn't give you exact control, but what are
 the cases where auto ref is wrong?


I did just give some examples, I'll repeat; auto ref fails when the
function is extern.

It is also wrong that when I pass an int or float, it is passed by ref
instead of by value... that is never what I want!

What do you get when you take a pointer of a function with an auto-ref
arg? ...an error, because it's not actually a function!
So in a context where I'm dealing with functions and function pointers
(very, very frequent), I suddenly have something that's not a function
find it's way into my meta and I have to special-case the hell out of
it.
The solution in this case is to wrap it in a non-auto-ref shim with
the ref-ness explicitly stated in the way I expect... which is in
itself another problem, because 'ref' is not part of the type! So the
annoying de-auto-ref-ing shim must be a text mixin with some very
complex, practically unreadable, definitely unmaintainable logic
surrounding it. It's insanity on top of insanity!

I also run the invisible risk of generating 2^num_args instances of
the code for functions containing auto-ref args.

When I'm writing a function that's not a template, I intend, and
expect, to write a function that's _not a template_.
Templates and functions are different things. I think it's a massive
mistake to have created a way to write a template that looks nothing
like a template.

auto-ref is not, and never has been a tool I have wanted. I don't have
any use for auto-ref, and it has only proven to make an already severe
problem worse. I've tried to use it before in some instances, but it
made ref's out of int's and floats, so even in the rare potentially
useful cases, I had to reject it.

At the time, you introduced auto-ref as a 'solution' to an issue that
I was the primary complainant (although the solution was mainly pushed
by Andrei iirc?). I absolutely rejected it then, and said it would
have the disastrous effect of setting ref in stone, which I was
already very critical about. I reject it even more now that I have had
some experience with it.
Who was the customer that you designed it for? (It wasn't me!)
What issue was it designed to address?



 Consider a ref counted type, RC!T. If scope were transitive, then you
 could
 not have, say, a tree where the edges were RC!T. I.e., the payload of an
 RC
 type should not be forced to be scope.



 I'm not sure I quite visualise this correctly...



 struct Tree {
    RefCount!(Tree*) left;
    RefCount!(Tree*) right;
    ...
 }


... I don't think I'd ever have a use for this code.
I've been using trees for a long time, and I can't imagine a situation
where each node in a tree would want to be ref counted.
It sounds like a disaster for performance, and I can't imagine any
advantage it would offer?
Perhaps in some complex graph structure... but I expect that sort of
thing would be far more specialised.

I can see common cases where nodes may contain a refcounted object as
node data, but that's different than the situation you demonstrate
here.
I'll need to think about how that situation would be affected in this case.



 So you're saying that a pointer itself shouldn't be able to escape a
 call tree, but the thing it points to should be able to escape just
 fine?



 Yes.



 It feels like that kinda defeats the purpose...



 You're arguing that a data structure with only one access point is the only
 kind of data structure in use. With transitive scope, such a data structure
 would be the only one possible!


Surely the possibility remains for such a data structure that is NOT
scope... but I think I can see your point.

There is a common case, ie, refcounting, where we want to be able to
elide rc fiddling, and that may be the sole call for scope in those
cases.
It seems like that situation calls for something like head-scope. I
think transitive scope has a different application, useful in
different situations.

It seems a problem that a struct that contains a single pointer, which
is passed by value, can't have scope applied under your proposal(?).
I use this pattern in almost every interaction with other languages
that I encounter. Some external API which operates via opaque pointer,
wants to be wrapped in such a way that it maintains it's behaviour as
a reference type, but also gains convenient member-style access. It's
kinda making a pseudo-class out of something that's not a class, and
scope needs to support this.



 I guess you're seeing a situation where 'scope' is almost exclusively
 useful as a mechanism to tell the RC that it doesn't need to worry
 about ref-fiddling, and the thing we're passing isn't interested in
 scope restrictions at any level other than that RC optimisation?
 I guess I can see your angle... but my reaction is if that's all you
 are concerned about, maybe scope is the wrong tool for eliding ref
 fiddling in that case :/



 'scope' is a way to say that this use of a pointer does not escape this
 scope. That is incredibly useful.

 Recall my statements that pervasive refcounting is a terrible performance
 problem because of all the inc/dec? Knowing that references cannot escape
 means an inc/dec pair can be elided.


I know this. I'm not arguing against any proposition that we need
scope to make RC practical. I've been arguing in favour of that for at
least as long as I've been batting for the ARC team, and even then
some more, because it would also address rvalue->ref, which is another
massive pain point!
But there are more things than pointers which need to be protected
against escaping their scope, in particular, things that contain
pointers...

Maybe there's a compromise. If we say scope isn't 'transitive', but it
is transitive when it comes to aggregates?
Ie, you can apply scope to a by-val struct, and all aggregate members
have scope applied.
It is not transitive in the sense that it does not follow through
pointer members, but any such pointer members may not escape as if it
were a pointer argument alone.

This is unlike any other behaviour in D, but without that, I think
this proposal fails to suit my most common use cases.
Dec 12 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2014 6:55 PM, Manu via Digitalmars-d wrote:

 I did just give some examples, I'll repeat; auto ref fails when the
 function is extern.


Don't make it extern, then.



 It is also wrong that when I pass an int or float, it is passed by ref
 instead of by value... that is never what I want!


If there's source to the function, it'll often be inlined which will remove the 
indirection.



 What do you get when you take a pointer of a function with an auto-ref
 arg? ...an error, because it's not actually a function!
 So in a context where I'm dealing with functions and function pointers
 (very, very frequent), I suddenly have something that's not a function
 find it's way into my meta and I have to special-case the hell out of
 it.


Why are function pointers and ints going to the same argument of a function? I 
thought you weren't using templates?



 The solution in this case is to wrap it in a non-auto-ref shim with
 the ref-ness explicitly stated in the way I expect... which is in
 itself another problem, because 'ref' is not part of the type! So the
 annoying de-auto-ref-ing shim must be a text mixin with some very
 complex, practically unreadable, definitely unmaintainable logic
 surrounding it. It's insanity on top of insanity!

 I also run the invisible risk of generating 2^num_args instances of
 the code for functions containing auto-ref args.


I wonder what is the need for the code that you are writing.



 When I'm writing a function that's not a template, I intend, and
 expect, to write a function that's _not a template_.
 Templates and functions are different things. I think it's a massive
 mistake to have created a way to write a template that looks nothing
 like a template.


A function template is a function that takes both compile-time args and
run-time 
args. C++ tried to make them completely different animals, when they are not. 
Don't think "template", think "compile-time argument to a function". I
convinced 
Andrei to never mention "template" in his D book, as it brings forth all kinds 
of connotations and expectations that impede understanding what D templates 
actually are. I think the result was successful.



 auto-ref is not, and never has been a tool I have wanted. I don't have
 any use for auto-ref, and it has only proven to make an already severe
 problem worse. I've tried to use it before in some instances, but it
 made ref's out of int's and floats, so even in the rare potentially
 useful cases, I had to reject it.


If it's a rare useful case, why is it a pervasive problem?



 At the time, you introduced auto-ref as a 'solution' to an issue that
 I was the primary complainant (although the solution was mainly pushed
 by Andrei iirc?). I absolutely rejected it then, and said it would
 have the disastrous effect of setting ref in stone, which I was
 already very critical about. I reject it even more now that I have had
 some experience with it.
 Who was the customer that you designed it for? (It wasn't me!)
 What issue was it designed to address?


I still have no idea what code you are developing that you need to send ints
and 
function pointers to the same argument of a function template, yet you don't
use 
function templates. Nor do I understand what pattern you need that simply must 
mix up ref and value parameters, and why that pattern appears pervasively in 
your code.



 struct Tree {
     RefCount!(Tree*) left;
     RefCount!(Tree*) right;
     ...
 }


 ... I don't think I'd ever have a use for this code.


You have no use for tree structures?



 I've been using trees for a long time, and I can't imagine a situation
 where each node in a tree would want to be ref counted.


You have more than one parent of a node. You never write data structures like 
that? dmd uses such pervasively (Type and Expression).



 It sounds like a disaster for performance, and I can't imagine any
 advantage it would offer?


How do you propose to manage the memory explicitly otherwise? Use a GC? :-)



 I can see common cases where nodes may contain a refcounted object as
 node data, but that's different than the situation you demonstrate
 here.


That would fail if scope were transitive.



 But there are more things than pointers which need to be protected
 against escaping their scope, in particular, things that contain
 pointers...


Solve it in the general case (this proposal) and RC now works.



 Maybe there's a compromise. If we say scope isn't 'transitive', but it
 is transitive when it comes to aggregates?


One thing I've tried very hard to make work in D is have basic types / 
aggregages / arrays be interchangeable.



 Ie, you can apply scope to a by-val struct, and all aggregate members
 have scope applied.
 It is not transitive in the sense that it does not follow through
 pointer members, but any such pointer members may not escape as if it
 were a pointer argument alone.

 This is unlike any other behaviour in D, but without that, I think
 this proposal fails to suit my most common use cases.


I have no idea what your use cases are, but when you're explicitly managing 
memory, every damn pointer needs to be carefully evaluated as to what exactly 
it's pointing two and who owns it. Scope is not a magic solution to this, 
neither is shared_ptr, neither are Rust's annotations.

The only scheme that absolves the user of having to deal with this is - GC.
Dec 12 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:


 struct Tree {
    RefCount!(Tree*) left;
    RefCount!(Tree*) right;
    ...
 }


 ... I don't think I'd ever have a use for this code.


 You have no use for tree structures?


Giving a reference counter to every pointer in a binary tree 
sounds a bit too much.

Bye,
bearophile
Dec 13 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/13/2014 2:10 AM, bearophile wrote:

 Walter Bright:


 struct Tree {
    RefCount!(Tree*) left;
    RefCount!(Tree*) right;
    ...
 }


 ... I don't think I'd ever have a use for this code.


 You have no use for tree structures?


 Giving a reference counter to every pointer in a binary tree sounds a bit too
much.


DMD, for example, uses ASTs where the nodes have multiple parents.
Dec 13 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 13 December 2014 at 20:45:53 UTC, Walter Bright 
wrote:

 On 12/13/2014 2:10 AM, bearophile wrote:

 Walter Bright:


 struct Tree {
   RefCount!(Tree*) left;
   RefCount!(Tree*) right;
   ...
 }


 ... I don't think I'd ever have a use for this code.


 You have no use for tree structures?


 Giving a reference counter to every pointer in a binary tree 
 sounds a bit too much.


 DMD, for example, uses ASTs where the nodes have multiple 
 parents.


It is a DAG then.
Dec 15 2014
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-12-13 06:11, Walter Bright wrote:


 Don't make it extern, then.


He's said a couple of time he's interfacing with C and/or C++ code.


 Why are function pointers and ints going to the same argument of a
 function? I thought you weren't using templates?


I think he meant taking the address of a function with auto ref.

-- 
/Jacob Carlborg
Dec 13 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/13/2014 8:19 AM, Jacob Carlborg wrote:

 On 2014-12-13 06:11, Walter Bright wrote:


 Don't make it extern, then.


 He's said a couple of time he's interfacing with C and/or C++ code.


Yes, and C++ code is either ref or not ref, and it is not a problem to insert 
'ref' or not, manually.



 Why are function pointers and ints going to the same argument of a
 function? I thought you weren't using templates?


 I think he meant taking the address of a function with auto ref.


Right, but why is that even being attempted?
Dec 13 2014
prev sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 December 2014 at 15:11, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 12/12/2014 6:55 PM, Manu via Digitalmars-d wrote:

 I did just give some examples, I'll repeat; auto ref fails when the
 function is extern.



 Don't make it extern, then.


Do you think I just interact with other languages for fun or something?



 It is also wrong that when I pass an int or float, it is passed by ref
 instead of by value... that is never what I want!



 If there's source to the function, it'll often be inlined which will remove
 the indirection.


Not if it's extern (applicable to ref, not auto-ref), or wrapped, or
if I ever want to capture a function pointer.
It's also semantically different; I can change the caller's value.



 What do you get when you take a pointer of a function with an auto-ref
 arg? ...an error, because it's not actually a function!
 So in a context where I'm dealing with functions and function pointers
 (very, very frequent), I suddenly have something that's not a function
 find it's way into my meta and I have to special-case the hell out of
 it.



 Why are function pointers and ints going to the same argument of a function?
 I thought you weren't using templates?


I'm confused. I'm talking about capturing function pointers. Not about
function arguments.
Capturing pointers of functions with auto-ref args (aka, template
functions) is a serious nuisance, and impossible outside the site of
instantiation.
You need information that's only available at the point of
instantiation. That changes the landscape quite a bit.



 The solution in this case is to wrap it in a non-auto-ref shim with
 the ref-ness explicitly stated in the way I expect... which is in
 itself another problem, because 'ref' is not part of the type! So the
 annoying de-auto-ref-ing shim must be a text mixin with some very
 complex, practically unreadable, definitely unmaintainable logic
 surrounding it. It's insanity on top of insanity!

 I also run the invisible risk of generating 2^num_args instances of
 the code for functions containing auto-ref args.



 I wonder what is the need for the code that you are writing.


General function adaptation. There are lots of reasons to wrap
functions. Adaptation to existing or external API's is the most common
in my experience.

Functions are what programs are! I really struggle to understand why
you have such trouble sympathising with my view here. Languages
generate code, which is packaged into blocks we call 'functions'...
and they require to adhere to strict ABI requirements. That is
programming in a nutshell. Surely it's reasonable to want to retain
complete control over this most primitive and basic of tasks.
One important aspect of that control is where the code is generated;
is it 'here', or 'there'? And that's a critical distinction between
functions and templates.


Look at LuaD. Lots of awkward cases have emerged there. There are many
more to come too; the known bug list are mostly nasty issues of this
nature that I've been putting off.
I can't offer any insight into my commercial code sadly, and I no
longer have access to it :/

Situations like this appear frequently:
https://github.com/JakobOvrum/LuaD/pull/76/files#diff-bcb370a5bc6fe75a9d5c04f2e1c17eb0R178

And something like this tends to appear as one aspect of the solution:
https://github.com/JakobOvrum/LuaD/pull/76/files#diff-bcb370a5bc6fe75a9d5c04f2e1c17eb0R68
'struct Ref(T)' leads to its own problems though, in that it's a
localised concept. No external code anywhere understands it as a
'ref', so if 3rd party code has any special treatment for ref, that
code now fails.

Then more magic like this:
https://github.com/JakobOvrum/LuaD/pull/76/files#diff-ec8c532aeca798240de4d70ee639fc16R90
Since we need to recognise 'ref' and substitute it for our magic
'struct Ref(T)'.


I've probably spent more hours wrangling D meta of this sort than most
people. This sort of thing always happens.
If Ref!T is the tool that resolves the situation, then it's clear
demonstration that ref should be part of the type.



 When I'm writing a function that's not a template, I intend, and
 expect, to write a function that's _not a template_.
 Templates and functions are different things. I think it's a massive
 mistake to have created a way to write a template that looks nothing
 like a template.



 A function template is a function that takes both compile-time args and
 run-time args. C++ tried to make them completely different animals, when
 they are not. Don't think "template", think "compile-time argument to a
 function". I convinced Andrei to never mention "template" in his D book, as
 it brings forth all kinds of connotations and expectations that impede
 understanding what D templates actually are. I think the result was
 successful.


You can spin it however you like, but it's exactly the same thing.
They are completely different animals. A function template is not a
function at all until it's instantiated. When it's instantiated, it
becomes a function, or even, one of a suite of functions. And it's not
known to the author where the function is.
(Note: I define 'function' in this context to mean 'some code that is emitted')

This one:many relationship between definition and code creates some
sorts of problems that aren't present with functions, which relate 1:1
with their codegen.
It is possible to take a pointer to a function. It is not possible to
take a pointer to a template, unless your code exists at the callsite
where the parameters for instantiation are known.
This makes certain things more complicated.

I reject that 'a template is a function with compile time args'.
Template functions have rather different characteristics, which result
in special consideration, and some restrictions on usage.
It's a cute idea, and something that might sound nice in a book... but
it's not the reality.
'functions' are a much simpler, more fundamental concept, and also one
that is easily portable between languages.



 auto-ref is not, and never has been a tool I have wanted. I don't have
 any use for auto-ref, and it has only proven to make an already severe
 problem worse. I've tried to use it before in some instances, but it
 made ref's out of int's and floats, so even in the rare potentially
 useful cases, I had to reject it.



 If it's a rare useful case, why is it a pervasive problem?


I never argued that auto-ref wasn't rare, but it was *a mistake*. What
it did was cement an already very shaky language feature into the
foundation.
Once there is layers of language built on top of something, it becomes
that much harder to refactor at some later time.

I'm not saying auto-ref is a pervasive problem, I'm saying (over and
over again) that *ref* is a pervasive problem, and auto-ref turns out
to be a further (fairly rare) nuisance. It can't be presented as a
solution to any problem that I have (how we got onto this tangent),
because it's not.
I also don't have any idea why it exists! What's it for?
You love to make people justify exactly what things are for; you're
doing it to me in almost every paragraph.

In my case here though, I'm not asking you to invent a language
feature like auto-ref, I'm just commenting on the usefulness of an
existing feature. Trouble is, my cases are really hard to define and
highly context specific.
I'm sharing my anecdotal experience that D's ref is an awkward pain in
the arse at best, and by my judgement, a failed experiment.



 At the time, you introduced auto-ref as a 'solution' to an issue that
 I was the primary complainant (although the solution was mainly pushed
 by Andrei iirc?). I absolutely rejected it then, and said it would
 have the disastrous effect of setting ref in stone, which I was
 already very critical about. I reject it even more now that I have had
 some experience with it.
 Who was the customer that you designed it for? (It wasn't me!)
 What issue was it designed to address?



 I still have no idea what code you are developing that you need to send ints
 and function pointers to the same argument of a function template, yet you
 don't use function templates. Nor do I understand what pattern you need that
 simply must mix up ref and value parameters, and why that pattern appears
 pervasively in your code.


I was talking about taking function pointers of functions. Not passing
function pointers to functions.
The auto-ref detail was on a slight tangent; explaining that it's not
useful to me to solve the problem of dealing with ref-ness of incoming
functions args or results.

I think I said that clearly: "What do you get when you take a pointer
of a function with an auto-ref
arg? ...an error, because it's not actually a function!"

WRT to mixing ref and value parameters; I'm wrapping/adapting existing
api's. I have no control over the code/api that exists. If that api
uses ref, for whatever reason that it makes sense to do so, I need to
handle the case.
There's no 'pattern', only a mechanical process.



 struct Tree {
     RefCount!(Tree*) left;
     RefCount!(Tree*) right;
     ...
 }



 ... I don't think I'd ever have a use for this code.



 You have no use for tree structures?


I don't have use for a tree where every node is ref counted... tree
nodes should only have one reference; their parent.



 I've been using trees for a long time, and I can't imagine a situation
 where each node in a tree would want to be ref counted.



 You have more than one parent of a node. You never write data structures
 like that? dmd uses such pervasively (Type and Expression).


That's not a tree, that's a graph.
I made a comment on such graph structures: "Perhaps in some complex
graph structure... but I expect that sort of thing would be far more
specialised."
And I still think that.

I'm not going to rule out that an RC graph node might be useful, but I
was responding to your example, and that wasn't your example.



 It sounds like a disaster for performance, and I can't imagine any
 advantage it would offer?



 How do you propose to manage the memory explicitly otherwise? Use a GC? :-)


Tree elements only have one reference; their parent. Management of
trees is very simple.

Graph's may be a little more complex, and I can see that an RC node
might be useful... but I've been using various graph structures for a
long time, and never wanted to RC the nodes.
I expect traditional patterns would remain. I don't think an RC is
required to know if a graph node is still referenced within the graph;
graph nodes typically have bi-directional linkage, so when there are
no links to the node, then it is unreferenced, and can easily be
destroyed/returned to a pool.
I think an RC mechanism is effectively implicit in graph structures I
can imagine.


The cases where I use RC are for general 'resources'. In my line of
work, this is textures, mesh, renderstate, sounds, etc... things that
find themselves being shared around arbitrarily.
It is important that tree/graph nodes may contain a ref-counted thing,
but I have trouble visualising how that affects 'scope' in this case.



 I can see common cases where nodes may contain a refcounted object as
 node data, but that's different than the situation you demonstrate
 here.



 That would fail if scope were transitive.


I'm not sure I see why...
The situation is: a whole transitive scope graph is received as scope,
how does that inhibit my accessing a resource contained in the graph?
It inhibits my _escaping_ said resource... as it should. Right?

The problem you're addressing is that, if the graph isn't received as
scope (ie, I want the ability to escape it's internals), then RC
doesn't work efficiently.
My reaction is that RC optimisation is perhaps not strongly associated
with 'scope'. scope may offer a mechanism by which it can work some of
the time, but it's not precisely the same thing.

I kinda feel like you're trying to make scope into RC optimisation,
rather than RC optimisation into scope...?

I was once arguing for: int^ rcInt;
I think I'm going back in that direction. That's what other languages
do. And scope would create additional opportunity here; it would allow
implicit casting of T^ -> T*.



 But there are more things than pointers which need to be protected
 against escaping their scope, in particular, things that contain
 pointers...



 Solve it in the general case (this proposal) and RC now works.


Perhaps scope and RC are different things?
Solve scope purely for RC, and you've broken scope for other purposes
that it's useful; that is, inhibiting escaping of data.



 Maybe there's a compromise. If we say scope isn't 'transitive', but it
 is transitive when it comes to aggregates?



 One thing I've tried very hard to make work in D is have basic types /
 aggregages / arrays be interchangeable.


I'm looking for ways to make scope fit your proposed mould and be useful to me.
If I can't do this then my most common use case is unsatisfied:

struct Wrap
{
  OpaqueThing *ptr;

  this() {}
  ~this() {}  // <- constructors that twiddle RC effectively.
  this(this) {}

  this() scope {}
  ~this() scope {}  // <- Overloadable for 'scope', in which case,
don't twiddle the RC!
  this(this) scope {}

  // lots of operations on OpaqueThing wrapped up here...
}

void f(scope Wrap x) <-- RC twiddling is effectively elided here due
to constructor overloads
{
}


If you can propose an alternative solution?
This is representative of my 99% case. Sometimes the struct is more
than a single pointer though.



 Ie, you can apply scope to a by-val struct, and all aggregate members
 have scope applied.
 It is not transitive in the sense that it does not follow through
 pointer members, but any such pointer members may not escape as if it
 were a pointer argument alone.

 This is unlike any other behaviour in D, but without that, I think
 this proposal fails to suit my most common use cases.



 I have no idea what your use cases are, but when you're explicitly managing
 memory, every damn pointer needs to be carefully evaluated as to what
 exactly it's pointing two and who owns it. Scope is not a magic solution to
 this, neither is shared_ptr, neither are Rust's annotations.

 The only scheme that absolves the user of having to deal with this is - GC.


scope just needs to say "I will not let this memory escape". Then it
finally allows us to safely put data on the stack, something we can't
do in D today. I don't think scope should aim to do anything more than
that.

If scope can be useful to RC, that's great! But from this, it's
starting to look to me like scope might be partially useful to RC, but
scope and RC aren't exactly the same thing.
RC seems to require something like head-scope in order to not place
awkward restrictions on usage of RC objects. That said, full-scope
isn't without use cases; it allows us to safely use the stack,
potentially eliminating much GC load.
Dec 13 2014
next sibling parent reply "Paolo Invernizzi" <paolo.invernizzi no.address> writes:
On Sunday, 14 December 2014 at 00:45:06 UTC, Manu via 
Digitalmars-d wrote:

 On 13 December 2014 at 15:11, Walter Bright via Digitalmars-d

 A function template is a function that takes both compile-time 
 args and
 run-time args. C++ tried to make them completely different 
 animals, when
 they are not. Don't think "template", think "compile-time 
 argument to a
 function". I convinced Andrei to never mention "template" in 
 his D book, as
 it brings forth all kinds of connotations and expectations 
 that impede understanding what D templates actually are. I 
 think the result was successful.


 You can spin it however you like, but it's exactly the same 
 thing.
 They are completely different animals. A function template is  
 not a
 function at all until it's instantiated. When it's 
 instantiated, it
 becomes a function, or even, one of a suite of functions. And 
 it's not known to the author where the function is.
 (Note: I define 'function' in this context to mean 'some code 
 that is emitted')


Philippe Sigaud Excellent "D Templates: A Tutorial"...
the MANTRA...

"XXX templates are not XXXs, they are templates. With XXX being 
any of (function, struct, class, interface, union)."

---
/Paolo
Dec 14 2014
parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Sun, 14 Dec 2014 09:04:58 +0000
Paolo Invernizzi via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 Philippe Sigaud Excellent "D Templates: A Tutorial"...
 the MANTRA...
=20
 "XXX templates are not XXXs, they are templates. With XXX being=20
 any of (function, struct, class, interface, union)."


and that still doesn't work. ;-)

as a somehow related example i recall my first expirience with C. my
first "serious" language was Pascal, and it has that nice range-checking
feature for arrays. it was very hard for me to understand that C has no
arrays at all, not to say range checking. why there is no arrays if i
have this nice indexing and all that?! what do you mean by that "buffer
overflow" nonsence? if it looks like an array it must be an array!
Dec 14 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Sunday, 14 December 2014 at 00:45:06 UTC, Manu via 
Digitalmars-d wrote:

 Perhaps scope and RC are different things?
 Solve scope purely for RC, and you've broken scope for other 
 purposes
 that it's useful; that is, inhibiting escaping of data.


Exactly. It shouldn't matter which type this data has - both 
references and value types are valid. I would define `scope` like 
this:

"A value designated with `scope` may not be assigned to a 
variable (or parameter, ...) with a lifetime longer than its 
owner's lifetime."


 Maybe there's a compromise. If we say scope isn't 
 'transitive', but it
 is transitive when it comes to aggregates?



 One thing I've tried very hard to make work in D is have basic 
 types /
 aggregages / arrays be interchangeable.




It should only be applicable to whatever is explicitly marked as 
`scope`. For references, that's the reference itself, but not 
what it points to. For aggregates, it's the entire aggregate with 
all it's members. That way, all types will be treated uniformly.

Of course, this can only work with a type modifier, not with a 
storage class.


 I'm looking for ways to make scope fit your proposed mould and 
 be useful to me.
 If I can't do this then my most common use case is unsatisfied:

 struct Wrap
 {
   OpaqueThing *ptr;

   this() {}
   ~this() {}  // <- constructors that twiddle RC effectively.
   this(this) {}

   this() scope {}
   ~this() scope {}  // <- Overloadable for 'scope', in which 
 case,
 don't twiddle the RC!
   this(this) scope {}

   // lots of operations on OpaqueThing wrapped up here...
 }

 void f(scope Wrap x) <-- RC twiddling is effectively elided 
 here due
 to constructor overloads
 {
 }


 If you can propose an alternative solution?
 This is representative of my 99% case. Sometimes the struct is 
 more
 than a single pointer though.


Unfortunately Walter rejected this when I had proposed it.

In the current proposal, we cannot overload on scope. The 
proposed solution is to not "borrow" the wrapper, but only its 
payload.

But this is restrictive; even in the case of RC, there are 
use-cases that cannot be served with this technique, namely when 
the callee can only decide at runtime whether it wants to make a 
copy of an RC value it received. The correct solution would be to 
pass the RC wrapper itself as scope, as you show in your example, 
thereby delegating the decision of whether to adjust the refcount 
from the caller to the callee.
Dec 14 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/13/2014 4:44 PM, Manu via Digitalmars-d wrote:

 On 13 December 2014 at 15:11, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 12/12/2014 6:55 PM, Manu via Digitalmars-d wrote:

 I did just give some examples, I'll repeat; auto ref fails when the
 function is extern.



 Don't make it extern, then.


 Do you think I just interact with other languages for fun or something?


I'm not trying to insult you - but why need 'auto ref' for extern functions?
The 
extern functions are ref or not-ref, and declare the interface to match.
There's 
no point to auto ref there.



 If there's source to the function, it'll often be inlined which will remove
 the indirection.


 Not if it's extern (applicable to ref, not auto-ref), or wrapped, or
 if I ever want to capture a function pointer.
 It's also semantically different; I can change the caller's value.


Again, if it is extern it has a fixed definition of ref or not. The D side 
doesn't decide if it's ref or not, it just follows whatever the extern 
declaration is.

This is why I do not understand why you are running into this issue everywhere.



 What do you get when you take a pointer of a function with an auto-ref
 arg? ...an error, because it's not actually a function!
 So in a context where I'm dealing with functions and function pointers
 (very, very frequent), I suddenly have something that's not a function
 find it's way into my meta and I have to special-case the hell out of
 it.



 Why are function pointers and ints going to the same argument of a function?
 I thought you weren't using templates?


 I'm confused. I'm talking about capturing function pointers. Not about
 function arguments.
 Capturing pointers of functions with auto-ref args (aka, template
 functions) is a serious nuisance, and impossible outside the site of
 instantiation.


WHY are you doing this? I have no idea what coding pattern would need to pass 
functions by reference.



 I wonder what is the need for the code that you are writing.


 General function adaptation. There are lots of reasons to wrap
 functions. Adaptation to existing or external API's is the most common
 in my experience.


Are those API's all templated? My guess is no. If they're not, you don't need 
templates or auto-ref to interface to them.



 Functions are what programs are! I really struggle to understand why
 you have such trouble sympathising with my view here.


Because you don't provide any examples of why you need this stuff. You say "I 
need this feature because I'm doing X", but I have no idea what X is or why you 
need X.

I.e. you present and advocate for particular solutions, but not the problems.



 Languages generate code, which is packaged into blocks we call 'functions'...
 and they require to adhere to strict ABI requirements. That is
 programming in a nutshell. Surely it's reasonable to want to retain
 complete control over this most primitive and basic of tasks.
 One important aspect of that control is where the code is generated;
 is it 'here', or 'there'?


Why is that important?



 And that's a critical distinction between functions and templates.


You can control that with function templates, too. The compiler won't 
instantiate a template locally if the import instantiates it.



 Look at LuaD. Lots of awkward cases have emerged there. There are many
 more to come too; the known bug list are mostly nasty issues of this
 nature that I've been putting off.
 I can't offer any insight into my commercial code sadly, and I no
 longer have access to it :/

 Situations like this appear frequently:
 https://github.com/JakobOvrum/LuaD/pull/76/files#diff-bcb370a5bc6fe75a9d5c04f2e1c17eb0R178

 And something like this tends to appear as one aspect of the solution:
 https://github.com/JakobOvrum/LuaD/pull/76/files#diff-bcb370a5bc6fe75a9d5c04f2e1c17eb0R68
 'struct Ref(T)' leads to its own problems though, in that it's a
 localised concept. No external code anywhere understands it as a
 'ref', so if 3rd party code has any special treatment for ref, that
 code now fails.

 Then more magic like this:
 https://github.com/JakobOvrum/LuaD/pull/76/files#diff-ec8c532aeca798240de4d70ee639fc16R90
 Since we need to recognise 'ref' and substitute it for our magic
 'struct Ref(T)'.


 I've probably spent more hours wrangling D meta of this sort than most
 people. This sort of thing always happens.
 If Ref!T is the tool that resolves the situation, then it's clear
 demonstration that ref should be part of the type.


I'll take a look at your references in a moment.



 When I'm writing a function that's not a template, I intend, and
 expect, to write a function that's _not a template_.
 Templates and functions are different things. I think it's a massive
 mistake to have created a way to write a template that looks nothing
 like a template.



 A function template is a function that takes both compile-time args and
 run-time args. C++ tried to make them completely different animals, when
 they are not. Don't think "template", think "compile-time argument to a
 function". I convinced Andrei to never mention "template" in his D book, as
 it brings forth all kinds of connotations and expectations that impede
 understanding what D templates actually are. I think the result was
 successful.


 You can spin it however you like, but it's exactly the same thing.
 They are completely different animals. A function template is not a
 function at all until it's instantiated. When it's instantiated, it
 becomes a function, or even, one of a suite of functions. And it's not
 known to the author where the function is.
 (Note: I define 'function' in this context to mean 'some code that is emitted')


That's the C++ view of templates, which is outdated and unnecessary.



 This one:many relationship between definition and code creates some
 sorts of problems that aren't present with functions, which relate 1:1
 with their codegen.
 It is possible to take a pointer to a function. It is not possible to
 take a pointer to a template, unless your code exists at the callsite
 where the parameters for instantiation are known.


It's not possible to take a pointer to a function unless that function exists, 
either.



 This makes certain things more complicated.


I don't see a problem with taking the address of an instantiated template
function.



 I reject that 'a template is a function with compile time args'.
 Template functions have rather different characteristics, which result
 in special consideration, and some restrictions on usage.
 It's a cute idea, and something that might sound nice in a book... but
 it's not the reality.


I've found templates to be far more useful when viewed that way.



 'functions' are a much simpler, more fundamental concept, and also one
 that is easily portable between languages.


It's always going to be true that in order to interface D with X, you'll have
to 
use a D subset that X understands. BTW, D can interface to C++ template
functions!



 I also don't have any idea why it exists! What's it for?


So you can make a template work with both lvalues and rvalues - and implement 
perfect forwarding.


 You love to make people justify exactly what things are for;


It's not a question of loving it, it's a question of practicality.



 I think I said that clearly: "What do you get when you take a pointer
 of a function with an auto-ref
 arg? ...an error, because it's not actually a function!"


What's baffling to me is why even declare a function pointer parameter as 
auto-ref? I don't use floating point to index strings, either, that doesn't
mean 
floating point is a failed concept.



 WRT to mixing ref and value parameters; I'm wrapping/adapting existing
 api's. I have no control over the code/api that exists. If that api
 uses ref, for whatever reason that it makes sense to do so, I need to
 handle the case.
 There's no 'pattern', only a mechanical process.


If the existing code uses ref, use ref on the D side. If it does not, do not
use 
ref on the D side. What's the problem?



 That's not a tree, that's a graph.


Sure. Do you want a scope design that precludes graphs? And even with trees,
are 
you sure you want a design where nobody but the root can point to anything in 
the tree?

How would you do a symbol table? You couldn't ever return a pointer to the
found 
symbol if it was all scoped.



 Tree elements only have one reference; their parent. Management of
 trees is very simple.


Again, what about the contents of the tree?



 That would fail if scope were transitive.


 I'm not sure I see why...
 The situation is: a whole transitive scope graph is received as scope,
 how does that inhibit my accessing a resource contained in the graph?
 It inhibits my _escaping_ said resource... as it should. Right?


How would you manage a symbol table lookup, and then store a pointer to the 
found symbol in the AST?



 I was once arguing for: int^ rcInt;
 I think I'm going back in that direction. That's what other languages
 do. And scope would create additional opportunity here; it would allow
 implicit casting of T^ -> T*.


scope actually does that. (int^ rcInt; isn't transitive, either.)



 Perhaps scope and RC are different things?
 Solve scope purely for RC, and you've broken scope for other purposes
 that it's useful; that is, inhibiting escaping of data.


RC is the canonical use of scope. The beauty is the compiler does not have to 
recognize RC as special. It becomes just ordinary library code.



 struct Wrap
 {
    OpaqueThing *ptr;

    this() {}
    ~this() {}  // <- constructors that twiddle RC effectively.
    this(this) {}

    this() scope {}
    ~this() scope {}  // <- Overloadable for 'scope', in which case,
 don't twiddle the RC!
    this(this) scope {}

    // lots of operations on OpaqueThing wrapped up here...
 }

 void f(scope Wrap x) <-- RC twiddling is effectively elided here due
 to constructor overloads
 {
 }


First off, get rid of the scope this overloads. Second, add an 'alias this' to 
implement a default conversion of Wrap to OpaqueThing*. Third, declare f as:

     void f(scope ref OpaqueThing* x);



 scope just needs to say "I will not let this memory escape". Then it
 finally allows us to safely put data on the stack, something we can't
 do in D today. I don't think scope should aim to do anything more than
 that.


That's EXACTLY what this proposal is!



 If scope can be useful to RC, that's great! But from this, it's
 starting to look to me like scope might be partially useful to RC, but
 scope and RC aren't exactly the same thing.


Wheels are cars aren't the same thing, either, but cars need wheels.



 RC seems to require something like head-scope in order to not place
 awkward restrictions on usage of RC objects. That said, full-scope
 isn't without use cases; it allows us to safely use the stack,
 potentially eliminating much GC load.


This proposal IS head-scope.
Dec 14 2014
parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Sunday, 14 December 2014 at 23:27:15 UTC, Walter Bright wrote:

 I think I said that clearly: "What do you get when you take a 
 pointer
 of a function with an auto-ref
 arg? ...an error, because it's not actually a function!"


 What's baffling to me is why even declare a function pointer 
 parameter as auto-ref? I don't use floating point to index 
 strings, either, that doesn't mean floating point is a failed 
 concept.


That's not what he's talking about. My understanding is that Manu 
is talking about something like this:

void foo(auto ref someConcreteType) { ... }

auto fooP = &foo; //error, foo is not a function
Dec 15 2014
prev sibling parent reply "Max Samukha" <maxsamukha gmail.com> writes:
On Saturday, 13 December 2014 at 02:55:50 UTC, Manu via 
Digitalmars-d wrote:


 Templates and functions are different things. I think it's a 
 massive
 mistake to have created a way to write a template that looks 
 nothing
 like a template.


That is a misconception spread by C++. Templates are pure 
functions applied at compile time. Function templates are 
functions partially applied at compile time.
Dec 13 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/13/2014 6:10 AM, Max Samukha wrote:

 On Saturday, 13 December 2014 at 02:55:50 UTC, Manu via Digitalmars-d wrote:


 Templates and functions are different things. I think it's a massive
 mistake to have created a way to write a template that looks nothing
 like a template.


 That is a misconception spread by C++. Templates are pure functions applied at
 compile time. Function templates are functions partially applied at compile
time.


It took me an embarrassingly long time to get past the C++ notion that
templates 
are something orthogonal from 'normal' functions and data declarations.

Even the C++ notion of a 'primary' template is wholly unnecessary, but is
deeply 
embedded in how it handles the semantics of templates.

C++ did get right the partial ordering of templates, and D applied partial 
ordering to 'normal' functions as well.
Dec 13 2014
prev sibling parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2014-12-04 09:24:13 +0000, Walter Bright said:


 http://wiki.dlang.org/DIP69
 
 Despite its length, this is a fairly simple proposal. It adds the 
 missing semantics for the 'scope' storage class in order to make it 
 possible to pass a reference to a function without it being possible 
 for it to escape.
 
 This, among other things, makes a ref counting type practical. It also 
 makes it more practical to use other storage allocation schemes than 
 garbage collection.
 
 It does not make scope into a type constructor, nor a general 
 type-annotation system.
 
 It does not provide an ownership system, though it would complement one.


I like the basics of the proposal and I think it's the right direction. 
 HOWEVER, I strongly believe that function arguments should be scoped 
by default and `impure` when they take reference which they will keep.

-Shammah
Dec 10 2014
parent Shammah Chancellor <anonymous coward.com> writes:
On 2014-12-11 07:03:58 +0000, Shammah Chancellor said:


 On 2014-12-04 09:24:13 +0000, Walter Bright said:
 

 http://wiki.dlang.org/DIP69
 
 Despite its length, this is a fairly simple proposal. It adds the 
 missing semantics for the 'scope' storage class in order to make it 
 possible to pass a reference to a function without it being possible 
 for it to escape.
 
 This, among other things, makes a ref counting type practical. It also 
 makes it more practical to use other storage allocation schemes than 
 garbage collection.
 
 It does not make scope into a type constructor, nor a general 
 type-annotation system.
 
 It does not provide an ownership system, though it would complement one.


 
 I like the basics of the proposal and I think it's the right direction. 
  HOWEVER, I strongly believe that function arguments should be scoped 
 by default and `impure` when they take reference which they will keep.
 
 -Shammah


Also, more feedback.   I would suggest that scope variable are the 
default inside of functions as well as parameters.   Using escape 
analysis, instead of them being errors, make them warnings and 
implicitly make them impure.

-Shammah
Dec 10 2014