www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - auto ref is on the docket

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Walter and I discussed what auto ref for templates should look like and 
reached the conclusion that an approach based on lowering would be best. 
I added a proposed lowering to 
https://github.com/D-Programming-Language/dmd/pull/4717.

Andrei
Jun 21 2015
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/22/15 12:11 AM, Andrei Alexandrescu wrote:

 Walter and I discussed what auto ref for templates


You mean *non-templates*? auto ref for templates has a very well defined 
meaning.

And reading your post, I'm unclear what exactly happens. Does this 
generate 2 functions from one, and then call the wrapper for auto-ref?

So for instance:

ref int fun(auto ref int x);

What happens here?

auto x = &fun;
x(5);

-Steve
Jun 21 2015
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/21/2015 10:08 PM, Steven Schveighoffer wrote:

 On 6/22/15 12:11 AM, Andrei Alexandrescu wrote:

 Walter and I discussed what auto ref for templates


 You mean *non-templates*? auto ref for templates has a very well defined
meaning.

 And reading your post, I'm unclear what exactly happens. Does this generate 2
 functions from one, and then call the wrapper for auto-ref?

 So for instance:

 ref int fun(auto ref int x);

 What happens here?

 auto x = &fun;
 x(5);

 -Steve


The idea is that fun(5) would be lowered to:

    auto tmp = 5;
    fun(tmp);

But when talking to Andrei I didn't realize that it would be subtly different 
behavior than 'auto ref' for template functions, which makes me concerned that 
this is not a good idea.

Note that one can always rewrite:

     ref int fun(ref int x);

into:

     ref int fun()(auto ref int x);

if auto ref is desired.
Jun 21 2015
next sibling parent "Temtaime" <temtaime gmail.com> writes:
On Monday, 22 June 2015 at 05:25:57 UTC, Walter Bright wrote:

 On 6/21/2015 10:08 PM, Steven Schveighoffer wrote:

 On 6/22/15 12:11 AM, Andrei Alexandrescu wrote:

 Walter and I discussed what auto ref for templates


 You mean *non-templates*? auto ref for templates has a very 
 well defined meaning.

 And reading your post, I'm unclear what exactly happens. Does 
 this generate 2
 functions from one, and then call the wrapper for auto-ref?

 So for instance:

 ref int fun(auto ref int x);

 What happens here?

 auto x = &fun;
 x(5);

 -Steve


 The idea is that fun(5) would be lowered to:

    auto tmp = 5;
    fun(tmp);

 But when talking to Andrei I didn't realize that it would be 
 subtly different behavior than 'auto ref' for template 
 functions, which makes me concerned that this is not a good 
 idea.

 Note that one can always rewrite:

     ref int fun(ref int x);

 into:

     ref int fun()(auto ref int x);

 if auto ref is desired.


I think too that it's unclear and misleading. What will happen if 
auto ref function will be templated? How it interacts with 
xurrent logic ?
And why not simply allow "ref in" to pass a temporary object?
Jun 21 2015
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Monday, 22 June 2015 at 05:25:57 UTC, Walter Bright wrote:

 The idea is that fun(5) would be lowered to:

    auto tmp = 5;
    fun(tmp);


That's not what Andrei described (though it's what I would have 
expected us to do). Rather, Andrei seems to be suggesting that 
having an auto ref parameter would result in defining both a ref 
and non-ref overload of the function (with the non-ref one 
forwarding to the ref one), which would unfortunately result in a 
combinatorial explosion of function definitions (just like the 
templated version does, except that all of the overloads would 
just forward to the one which took all refs, and all of the 
overloads would always exist regardless of whether they were 
called). So, I'm inclined to think that what you just suggested 
makes a lot more sense than what Andrei suggested.


 But when talking to Andrei I didn't realize that it would be 
 subtly different behavior than 'auto ref' for template 
 functions, which makes me concerned that this is not a good 
 idea.


I don't see how there's any way around that given that the 
behavior of the templated auto ref pretty much depends on the 
function being templated, though Andrei's suggestion comes a lot 
closer than having the compiler insert a variable for you. Also, 
by reusing auto ref for this rather than coming up with a new 
attribute, we can't declare templated functions with the same 
behavior as the non-templated ones would get for auto ref (though 
that's a lot more important IMHO if we implement it via 
introducing a variable rather than introducing overloads, since 
then we'd be able to avoid the template bloat in the cases where 
you're just trying to accept both lvalues and rvalues and don't 
care about forwarding refness).


 Note that one can always rewrite:

     ref int fun(ref int x);

 into:

     ref int fun()(auto ref int x);

 if auto ref is desired.


Yes, but to be honest, the way that Andrei is suggesting that we 
implement this is so close to just templatizing the function that 
the only benefit to it I see over the status quo is that it would 
work with virtual functions, whereas the templated version 
obviously won't. I really think that we should consider 
implementing this via _one_ function where all of the auto ref 
parameters are lowered to ref, and variables are added where 
necessary by the compiler rather than implementing it via 
lowering it to a combinatorial explosion of overloads.

And if we _do_ implement this via a single function with the 
compiler adding variables rather than having the compiler add 
overloads, I really think that we should consider introducing a 
new attribute for this so that we can use it with templated 
functions and avoid all of the template bloat that we get with 
auto ref right now. There are cases where auto ref is exactly 
what you want, but there are plenty of cases where folks use it 
just so that they can have their function accept both lvalues and 
rvalues, and in that case, all of the extra template 
instantiations are just bloat. So, if we had  anyref or whatever 
we wanted to call it, we could use it both with templated and 
non-templated functions, and we would avoid the confusion 
introduced by having auto ref mean different things for templated 
and non-templated functions. Yes, that would mean introducing 
another attribute, which kind of sucks, but its use is specific 
enough that I think that it would be worth having, and I fully 
expect that it would be a big help in reducing template bloat in 
many programs - especially from those folks who keep clamoring 
for C++'s const& in D.

- Jonathan M Davis
Jun 21 2015
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/21/15 10:25 PM, Walter Bright wrote:

 On 6/21/2015 10:08 PM, Steven Schveighoffer wrote:

 On 6/22/15 12:11 AM, Andrei Alexandrescu wrote:

 Walter and I discussed what auto ref for templates


 You mean *non-templates*? auto ref for templates has a very well
 defined meaning.

 And reading your post, I'm unclear what exactly happens. Does this
 generate 2
 functions from one, and then call the wrapper for auto-ref?

 So for instance:

 ref int fun(auto ref int x);

 What happens here?

 auto x = &fun;
 x(5);

 -Steve


 The idea is that fun(5) would be lowered to:

     auto tmp = 5;
     fun(tmp);


I don't think that lowering is recommended - it prolongs the lifetime of 
the temporary through the end of the caller. But that may be actually a 
good thing.


 But when talking to Andrei I didn't realize that it would be subtly
 different behavior than 'auto ref' for template functions, which makes
 me concerned that this is not a good idea.


That's fine.


 Note that one can always rewrite:

      ref int fun(ref int x);

 into:

      ref int fun()(auto ref int x);

 if auto ref is desired.


I don't understand this. My lowering scheme doesn't allow for that.


Andrei
Jun 21 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/21/15 11:31 PM, Andrei Alexandrescu wrote:

 On 6/21/15 10:25 PM, Walter Bright wrote:

 The idea is that fun(5) would be lowered to:

     auto tmp = 5;
     fun(tmp);


 I don't think that lowering is recommended - it prolongs the lifetime of
 the temporary through the end of the caller. But that may be actually a
 good thing.


On second thought - Walter's lowering, which makes the rvalue last more 
than strictly necessary, may be the most flexible of all at the cost of 
more resource consumption (for types that define destructors). -- Andrei
Jun 21 2015
next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Monday, 22 June 2015 at 06:38:57 UTC, Andrei Alexandrescu 
wrote:

 On 6/21/15 11:31 PM, Andrei Alexandrescu wrote:

 On 6/21/15 10:25 PM, Walter Bright wrote:

 The idea is that fun(5) would be lowered to:

     auto tmp = 5;
     fun(tmp);


 I don't think that lowering is recommended - it prolongs the 
 lifetime of
 the temporary through the end of the caller. But that may be 
 actually a
 good thing.


 On second thought - Walter's lowering, which makes the rvalue 
 last more than strictly necessary, may be the most flexible of 
 all at the cost of more resource consumption (for types that 
 define destructors). -- Andrei


I would think in the general case, the variable would need to 
stick around until the statement was completed in order to avoid 
problems with it going away prematurely, though maybe the recent 
changes with return as an attribute fix that problem. But as long 
as its legal for the function to return an auto ref argument via 
ref or auto ref, keeping the variable around for the duration of 
the call - or even just the expression - rather than the 
statement would risk using the variable after it was destroyed.

- Jonathan M Davis
Jun 21 2015
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 22 June 2015 at 06:38:57 UTC, Andrei Alexandrescu 
wrote:

 On 6/21/15 11:31 PM, Andrei Alexandrescu wrote:

 On 6/21/15 10:25 PM, Walter Bright wrote:

 The idea is that fun(5) would be lowered to:

     auto tmp = 5;
     fun(tmp);


 I don't think that lowering is recommended - it prolongs the 
 lifetime of
 the temporary through the end of the caller. But that may be 
 actually a
 good thing.


 On second thought - Walter's lowering, which makes the rvalue 
 last more than strictly necessary, may be the most flexible of 
 all at the cost of more resource consumption (for types that 
 define destructors). -- Andrei


Why would that be desirable? Resource consumption is the least of 
the problems. It is simply surprising, inconsistent with the 
lifetime of other temporaries, and it is unnecessary. Just lower 
it to:

     {
         auto tmp = 5;
         fun(tmp);
     }
Jun 22 2015
next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 22 June 2015 at 10:04:28 UTC, Marc Schütz wrote:

 On Monday, 22 June 2015 at 06:38:57 UTC, Andrei Alexandrescu 
 wrote:

 On 6/21/15 11:31 PM, Andrei Alexandrescu wrote:

 On 6/21/15 10:25 PM, Walter Bright wrote:

 The idea is that fun(5) would be lowered to:

     auto tmp = 5;
     fun(tmp);


 I don't think that lowering is recommended - it prolongs the 
 lifetime of
 the temporary through the end of the caller. But that may be 
 actually a
 good thing.


 On second thought - Walter's lowering, which makes the rvalue 
 last more than strictly necessary, may be the most flexible of 
 all at the cost of more resource consumption (for types that 
 define destructors). -- Andrei


 Why would that be desirable? Resource consumption is the least 
 of the problems. It is simply surprising, inconsistent with the 
 lifetime of other temporaries, and it is unnecessary. Just 
 lower it to:

     {
         auto tmp = 5;
         fun(tmp);
     }


I see now that Andrei already wrote in the PR "creating a named 
temporary in a scope immediately enclosing the call". I would 
like to add that AFAIK other temporaries live until the end of 
the entire statement they appear in, not just for the duration of 
the call. In any case, the temporaries involved here should not 
behave different from normal temporaries; in particular, they 
should have the same lifetime as with pass-by-value.
Jun 22 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/22/15 3:04 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:

 Just lower it to:

      {
          auto tmp = 5;
          fun(tmp);
      }


You need to lower an expression to an expression, not a statement. (e.g. 
what if fun returns a result?)

I considered this lowering for "int fun(ref int);"

fun(42)

==>>

(function int(int a) { return fun(a); })(42)

This does work, but if fun returns a ref int, I found no way to 
syntactically express that lambda. This does not parse:

(function ref int(int a) { return fun(a); })(42)

Is this a bug in the grammar?


Andrei
Jun 22 2015
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 22 June 2015 at 15:39:38 UTC, Andrei Alexandrescu 
wrote:

 On 6/22/15 3:04 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:

 Just lower it to:

      {
          auto tmp = 5;
          fun(tmp);
      }


 You need to lower an expression to an expression, not a 
 statement. (e.g. what if fun returns a result?)

 I considered this lowering for "int fun(ref int);"

 fun(42)

 ==>>

 (function int(int a) { return fun(a); })(42)

 This does work, but if fun returns a ref int, I found no way to 
 syntactically express that lambda. This does not parse:

 (function ref int(int a) { return fun(a); })(42)

 Is this a bug in the grammar?


Probably. But for lowering, the resulting AST doesn't really need 
to be representable in the language's syntax. I'm sure the AST 
_can_ express a lambda returning by reference.

Anway, I really think we should not deviate from the normal 
lifetime rules for temporaries.
Jun 22 2015
prev sibling parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 22 Jun 2015 08:40, "Andrei Alexandrescu via Digitalmars-d" <
digitalmars-d puremagic.com> wrote:

 On 6/21/15 11:31 PM, Andrei Alexandrescu wrote:

 On 6/21/15 10:25 PM, Walter Bright wrote:

 The idea is that fun(5) would be lowered to:

     auto tmp = 5;
     fun(tmp);



 I don't think that lowering is recommended - it prolongs the lifetime of
 the temporary through the end of the caller. But that may be actually a
 good thing.



 On second thought - Walter's lowering, which makes the rvalue last more


than strictly necessary, may be the most flexible of all at the cost of
more resource consumption (for types that define destructors). -- Andrei


I think keeping the lifetime of objects strictly in the call expression is
the behaviour that will give least surprise.

int i = 42;
struct S { ~this() { i++; } }

// auto __autoreftmp = S();
foo(S());  // __autoreftmp
// __dtor(__autoreftmp);
assert(i == 43);

As for optimisations, I think it should be possible to assert that the
reference never escapes, and use more aggressive optimisations based on
that, something that is still not possible with 'scope ref' or 'in ref'
parameters ...

Iain.



 Jun 24 2015



  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Wednesday, 24 June 2015 at 07:12:53 UTC, Iain Buclaw wrote:

 On 22 Jun 2015 08:40, "Andrei Alexandrescu via Digitalmars-d" < 
 digitalmars-d puremagic.com> wrote:

 On 6/21/15 11:31 PM, Andrei Alexandrescu wrote:

 On 6/21/15 10:25 PM, Walter Bright wrote:

 The idea is that fun(5) would be lowered to:

     auto tmp = 5;
     fun(tmp);



 I don't think that lowering is recommended - it prolongs the 
 lifetime of the temporary through the end of the caller. But 
 that may be actually a good thing.



 On second thought - Walter's lowering, which makes the rvalue 
 last more


 than strictly necessary, may be the most flexible of all at the 
 cost of more resource consumption (for types that define 
 destructors). -- Andrei


 I think keeping the lifetime of objects strictly in the call 
 expression is the behaviour that will give least surprise.


Call expression, or statement containing the call expression? For 
normal temporaries, it's the latter AFAIK.


 int i = 42;
 struct S { ~this() { i++; } }

 // auto __autoreftmp = S();
 foo(S());  // __autoreftmp
 // __dtor(__autoreftmp);
 assert(i == 43);


It would make a difference for:

     auto x = foo(S()) + foo(S());


 As for optimisations, I think it should be possible to assert 
 that the reference never escapes, and use more aggressive 
 optimisations based on that, something that is still not 
 possible with 'scope ref' or 'in ref' parameters ...


Not possible because it's not implemented, or are there 
fundamental reasons against it?



 Jun 24 2015



  parent  Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 24 June 2015 at 19:48, via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Wednesday, 24 June 2015 at 07:12:53 UTC, Iain Buclaw wrote:

 On 22 Jun 2015 08:40, "Andrei Alexandrescu via Digitalmars-d" <
 digitalmars-d puremagic.com> wrote:

 On 6/21/15 11:31 PM, Andrei Alexandrescu wrote:

 On 6/21/15 10:25 PM, Walter Bright wrote:

 The idea is that fun(5) would be lowered to:

     auto tmp = 5;
     fun(tmp);




 I don't think that lowering is recommended - it prolongs the lifetime of
 the temporary through the end of the caller. But that may be actually a good
 thing.




 On second thought - Walter's lowering, which makes the rvalue last more


 than strictly necessary, may be the most flexible of all at the cost of
 more resource consumption (for types that define destructors). -- Andrei


 I think keeping the lifetime of objects strictly in the call expression is
 the behaviour that will give least surprise.



 Call expression, or statement containing the call expression? For normal
 temporaries, it's the latter AFAIK.


I suppose DMD doesn't have an equivalent, but they are called
BIND_EXPR in GCC, where you have a new local block with assigned
temporaries that expire outside of the given bind.


 int i = 42;
 struct S { ~this() { i++; } }

 // auto __autoreftmp = S();
 foo(S());  // __autoreftmp
 // __dtor(__autoreftmp);
 assert(i == 43);



 It would make a difference for:

     auto x = foo(S()) + foo(S());


It works like this, the last result is binded back to the lvalue.

auto x =  {
    tmpA = S();
    tmpB = S();

    foo(tmpA) + foo(tmpB);
}


 As for optimisations, I think it should be possible to assert that the
 reference never escapes, and use more aggressive optimisations based on
 that, something that is still not possible with 'scope ref' or 'in ref'
 parameters ...



 Not possible because it's not implemented, or are there fundamental reasons
 against it?


Ignoring the 'ref' part of my comment, so as we are not confusing
DIP36 (which I've only just learned about). The last time I checked,
the use of 'scope' and 'in' on parameters are ignored by escape
analysis, so you can quite happily escape a reference as you see fit
and the language will allow you to do that.

So, I cannot optimised if the language allows you to break that
optimisation without warning or erroring the user.  If I did, then
ultimately you'll end up seeing your programs becoming silently
corrupted.



 Jun 24 2015



prev sibling  parent reply "Daniel N" <ufo orbiting.us>  writes:


On Monday, 22 June 2015 at 05:25:57 UTC, Walter Bright wrote:

 The idea is that fun(5) would be lowered to:

    auto tmp = 5;
    fun(tmp);

 But when talking to Andrei I didn't realize that it would be 
 subtly different behavior than 'auto ref' for template 
 functions, which makes me concerned that this is not a good 
 idea.


Considering we already have working 'Sealed references', why not 
simply allow rvalues for plain ref?

Bonus: No risk of mixup with template auto ref after refactoring 
etc.
Save auto for other potential future use.



 Jun 22 2015



 next sibling parent reply "Namespace" <rswhite4 gmail.com>  writes:


On Monday, 22 June 2015 at 20:42:58 UTC, Daniel N wrote:

 On Monday, 22 June 2015 at 05:25:57 UTC, Walter Bright wrote:

 The idea is that fun(5) would be lowered to:

    auto tmp = 5;
    fun(tmp);

 But when talking to Andrei I didn't realize that it would be 
 subtly different behavior than 'auto ref' for template 
 functions, which makes me concerned that this is not a good 
 idea.


 Considering we already have working 'Sealed references', why 
 not simply allow rvalues for plain ref?

 Bonus: No risk of mixup with template auto ref after 
 refactoring etc.
 Save auto for other potential future use.


That would be horrible. How would you distinguish between lvalues 
and rvalues? What if you want to store a pointer to an lvalue? If 
ref accept both you cannot do that.



 Jun 22 2015



  parent  "Daniel N" <ufo orbiting.us>  writes:


On Monday, 22 June 2015 at 20:50:23 UTC, Namespace wrote:

 That would be horrible. How would you distinguish between 
 lvalues and rvalues? What if you want to store a pointer to an 
 lvalue? If ref accept both you cannot do that.


storing requires 'return ref'



 Jun 22 2015



prev sibling  parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Monday, 22 June 2015 at 20:42:58 UTC, Daniel N wrote:

 Considering we already have working 'Sealed references', why 
 not simply allow rvalues for plain ref?


Please, please, no. That would be horrible. ref indicates an 
intention to mutate the argument, which makes no sense with 
rvalues. And if ref is used simply so that you can accept both 
rvalues and lvalues without copying lvalues, then there's zero 
indication in the function signature when the point of the ref is 
to actually set what's being passed in.

It works in C++ with const&, because you know that the function 
won't mutate the argument, but if you allow it with ref in 
general, then you stand no chance of being able to look at a 
function signature and deduce whether the function intends to 
mutate an argument or not. And even if Andrei wasn't so set 
against const ref that we could go that route in D as well, it 
wouldn't solve the problem either, because const in D is just too 
restrictive to use in many cases. We need a way to indicate that 
a parameter accepts both lvalues and rvalues without intending to 
mutate the argument but without actually guaranteeing it via 
const. And that does _not_ work with naked ref.

- Jonathan M Davis



 Jun 22 2015



  parent reply "Daniel N" <ufo orbiting.us>  writes:


On Monday, 22 June 2015 at 20:56:12 UTC, Jonathan M Davis wrote:

 won't mutate the argument, but if you allow it with ref in 
 general, then you stand no chance of being able to look at a 
 function signature and deduce whether the function intends to 
 mutate an argument or not.

 - Jonathan M Davis


It's no worse than auto ref. It's only by convention that auto 
ref functions doesn't mutate, at least I follow that convention, 
but when looking at someone else's function, all bets are off.



 Jun 22 2015



  parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Monday, 22 June 2015 at 21:30:45 UTC, Daniel N wrote:

 On Monday, 22 June 2015 at 20:56:12 UTC, Jonathan M Davis wrote:

 won't mutate the argument, but if you allow it with ref in 
 general, then you stand no chance of being able to look at a 
 function signature and deduce whether the function intends to 
 mutate an argument or not.

 - Jonathan M Davis


 It's no worse than auto ref. It's only by convention that auto 
 ref functions doesn't mutate, at least I follow that 
 convention, but when looking at someone else's function, all 
 bets are off.


No, you can't guarantee that an auto ref parameter won't be 
mutated, but auto ref clearly indicates that you intend to accept 
both lvalues and rvalues, meaning that it would be stupid to be 
writing the function with the idea that you would be passing in 
an argument which would be mutated. ref on the other hand clearly 
indicates the intention to mutate the argument.

- Jonathan M Davis



 Jun 22 2015



  parent reply "Daniel N" <ufo orbiting.us>  writes:


On Tuesday, 23 June 2015 at 01:07:17 UTC, Jonathan M Davis wrote:

 No, you can't guarantee that an auto ref parameter won't be 
 mutated, but auto ref clearly indicates that you intend to 
 accept both lvalues and rvalues, meaning that it would be 
 stupid to be writing the function with the idea that you would 
 be passing in an argument which would be mutated. ref on the 
 other hand clearly indicates the intention to mutate the 
 argument.

 - Jonathan M Davis


That is why, as a good API designer, you'd express that fact by 
adding const ontop of it, i.e. const ref.

Only for the few rare cases where you need to account for 
lazy-caching or similar mutating constructs, you'd be grateful to 
have the nice escape hatch, plain ref.

Daniel N



 Jun 22 2015



  parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Tuesday, 23 June 2015 at 06:49:50 UTC, Daniel N wrote:

 On Tuesday, 23 June 2015 at 01:07:17 UTC, Jonathan M Davis 
 wrote:

 No, you can't guarantee that an auto ref parameter won't be 
 mutated, but auto ref clearly indicates that you intend to 
 accept both lvalues and rvalues, meaning that it would be 
 stupid to be writing the function with the idea that you would 
 be passing in an argument which would be mutated. ref on the 
 other hand clearly indicates the intention to mutate the 
 argument.

 - Jonathan M Davis


 That is why, as a good API designer, you'd express that fact by 
 adding const ontop of it, i.e. const ref.


Which does not generally work in D. It can in some cases, but 
const is so restrictive in D that you simply cannot use it just 
because you don't intend to mutate a variable. Too many types 
won't work with const, and many types _can't_ work with const. 
const has its uses, but you have to be careful with it, and in 
the general case, that means that you can't put it on function 
parameters just to indicate that the function argument is not 
going to mutated.

- Jonathan M Davis



 Jun 23 2015



  parent reply "Daniel N" <ufo orbiting.us>  writes:


On Tuesday, 23 June 2015 at 11:03:10 UTC, Jonathan M Davis wrote:

 Which does not generally work in D. It can in some cases, but 
 const is so restrictive in D that you simply cannot use it just 
 because you don't intend to mutate a variable. Too many types 
 won't work with const, and many types _can't_ work with const. 
 const has its uses, but you have to be careful with it, and in 
 the general case, that means that you can't put it on function 
 parameters just to indicate that the function argument is not 
 going to mutated.

 - Jonathan M Davis


Thus the solution cannot require const.

auto is worse because, if you later decide you need to add a 
template parameter then the meaning is changed and you get a 
_hidden_ performance issue.

Thus requiring neither auto nor const is the least evil of the 
above.

You could always use comments, or even dummy aliases to make the 
code self-documenting:

alias view(T...) = T;
void example(ref view!int i) {}



 Jun 23 2015



  parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Tuesday, 23 June 2015 at 13:52:19 UTC, Daniel N wrote:

 On Tuesday, 23 June 2015 at 11:03:10 UTC, Jonathan M Davis 
 wrote:

 Which does not generally work in D. It can in some cases, but 
 const is so restrictive in D that you simply cannot use it 
 just because you don't intend to mutate a variable. Too many 
 types won't work with const, and many types _can't_ work with 
 const. const has its uses, but you have to be careful with it, 
 and in the general case, that means that you can't put it on 
 function parameters just to indicate that the function 
 argument is not going to mutated.

 - Jonathan M Davis


 Thus the solution cannot require const.

 auto is worse because, if you later decide you need to add a 
 template parameter then the meaning is changed and you get a 
 _hidden_ performance issue.


And how does it introduce a hidden performance issue. You'd end 
up with some template bloat, but that's it, and that's just going 
to increase the memory footprint of your program slightly, which 
isn't generally going to be a performance issue. It's just going 
to make your program slightly larger.

- Jonathan M Davis



 Jun 23 2015



  parent reply "John Colvin" <john.loughran.colvin gmail.com>  writes:


On Tuesday, 23 June 2015 at 16:27:34 UTC, Jonathan M Davis wrote:

 On Tuesday, 23 June 2015 at 13:52:19 UTC, Daniel N wrote:

 On Tuesday, 23 June 2015 at 11:03:10 UTC, Jonathan M Davis 
 wrote:

 [...]


 Thus the solution cannot require const.

 auto is worse because, if you later decide you need to add a 
 template parameter then the meaning is changed and you get a 
 _hidden_ performance issue.


 And how does it introduce a hidden performance issue. You'd end 
 up with some template bloat, but that's it, and that's just 
 going to increase the memory footprint of your program 
 slightly, which isn't generally going to be a performance 
 issue. It's just going to make your program slightly larger.

 - Jonathan M Davis


instruction cache misses can really hurt



 Jun 23 2015



 next sibling parent  "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Tuesday, 23 June 2015 at 17:05:41 UTC, John Colvin wrote:

 On Tuesday, 23 June 2015 at 16:27:34 UTC, Jonathan M Davis 
 wrote:

 On Tuesday, 23 June 2015 at 13:52:19 UTC, Daniel N wrote:

 On Tuesday, 23 June 2015 at 11:03:10 UTC, Jonathan M Davis 
 wrote:

 [...]


 Thus the solution cannot require const.

 auto is worse because, if you later decide you need to add a 
 template parameter then the meaning is changed and you get a 
 _hidden_ performance issue.


 And how does it introduce a hidden performance issue. You'd 
 end up with some template bloat, but that's it, and that's 
 just going to increase the memory footprint of your program 
 slightly, which isn't generally going to be a performance 
 issue. It's just going to make your program slightly larger.

 - Jonathan M Davis


 instruction cache misses can really hurt


Well, if you're caring about stuff on that level, then you really 
need to understand what kind of code the compiler is generating, 
and most anything you do to the program risks changing its 
performance. So, I don't see much reason to worry about how a 
slight increase in program size from templatizing a function 
using auto ref could introduce cache misses. The fact alone that 
you've templatized the function could change its code gen enough 
to do that given that the compiler is then guaranteed to have the 
code available and will be doing attribute inference and whatnot. 
And most code probably won't see a performance difference anyway. 
Performance-intensive stuff may care, but you need to be very 
careful when tweaking that sort of code anyway, simply because 
all kinds of small tweaks can have unexpected impacts on 
performance.

- Jonathan M Davis



 Jun 23 2015



prev sibling  parent  "weaselcat" <weaselcat gmail.com>  writes:


On Tuesday, 23 June 2015 at 17:05:41 UTC, John Colvin wrote:

 On Tuesday, 23 June 2015 at 16:27:34 UTC, Jonathan M Davis 
 wrote:

 On Tuesday, 23 June 2015 at 13:52:19 UTC, Daniel N wrote:

 [...]


 And how does it introduce a hidden performance issue. You'd 
 end up with some template bloat, but that's it, and that's 
 just going to increase the memory footprint of your program 
 slightly, which isn't generally going to be a performance 
 issue. It's just going to make your program slightly larger.

 - Jonathan M Davis


 instruction cache misses can really hurt


any time spent optimizing for this would be better spent making 
GDC/LDC 'good enough' to replace dmd.



 Jun 23 2015



prev sibling  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 6/21/15 10:08 PM, Steven Schveighoffer wrote:

 On 6/22/15 12:11 AM, Andrei Alexandrescu wrote:

 Walter and I discussed what auto ref for templates


 You mean *non-templates*? auto ref for templates has a very well defined
 meaning.


Yes, non-tempaltes.


 And reading your post, I'm unclear what exactly happens. Does this
 generate 2 functions from one, and then call the wrapper for auto-ref?

 So for instance:

 ref int fun(auto ref int x);

 What happens here?

 auto x = &fun;
 x(5);


What you'd expect after lowering: error.


Andrei



 Jun 21 2015



prev sibling next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Monday, 22 June 2015 at 04:11:41 UTC, Andrei Alexandrescu 
wrote:

 Walter and I discussed what auto ref for templates should look 
 like and reached the conclusion that an approach based on 
 lowering would be best. I added a proposed lowering to 
 https://github.com/D-Programming-Language/dmd/pull/4717.


So, you're basically suggesting that have an auto ref parameter 
on a non-templated function basically does the combinatorial 
explosion of function definitions that you get with templated 
functions except that all of the combinations are instantiated 
(the one difference then being that all of the overloads except 
the one with all refs would just be wrappers calling the one with 
all refs)? That doesn't sound particularly desirable me. It's 
better than writing all of those wrappers by hand, but that's 
still likely to be a fair bit of code bloat if you're using auto 
ref parameters much - and if the folks coming from C++ who have 
been begging for this get it, they _will_ start using it all over 
the place.

I would have thought that we'd define this such that the compiler 
introduced a variable restricted to the scope of the statement 
for each rvalue passed as an auto ref parameter rather than 
introducing function overloads. No, that's not quite as nice a 
lowering, but it would avoid a lot of extraneous function 
definitions.

- Jonathan M Davis



 Jun 21 2015



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 6/21/15 11:11 PM, Jonathan M Davis wrote:

 No, that's not quite as nice a lowering, but it would avoid a lot of
 extraneous function definitions.


I think a lowering is a must. Matters are complicated as they are, I 
don't want to add new rules.

Combinatorial issues can be addressed by either generating the functions 
on demand, or by lowering at the client side.


Andrei



 Jun 21 2015



prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Monday, 22 June 2015 at 04:11:41 UTC, Andrei Alexandrescu 
wrote:

 Walter and I discussed what auto ref for templates should look 
 like and reached the conclusion that an approach based on 
 lowering would be best. I added a proposed lowering to 
 https://github.com/D-Programming-Language/dmd/pull/4717.


I have to concur with Manu's statement in the PR. IMO `auto ref` 
(i.e. the status quo) is a hack introduced because of the lack of 
usable `scope`. We shouldn't spread it to other parts of the 
language, at least not for rvalue references. `scope ref` is the 
perfect fit for those.

(There may still be a use for `auto ref` with the meaning "pass 
this as efficiently as possible, I don't care whether it's a 
reference or not", but I'm not convinced of that either.)



 Jun 22 2015



  parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Monday, 22 June 2015 at 13:23:47 UTC, Marc Schütz wrote:

 On Monday, 22 June 2015 at 04:11:41 UTC, Andrei Alexandrescu 
 wrote:

 Walter and I discussed what auto ref for templates should look 
 like and reached the conclusion that an approach based on 
 lowering would be best. I added a proposed lowering to 
 https://github.com/D-Programming-Language/dmd/pull/4717.


 I have to concur with Manu's statement in the PR. IMO `auto 
 ref` (i.e. the status quo) is a hack introduced because of the 
 lack of usable `scope`. We shouldn't spread it to other parts 
 of the language, at least not for rvalue references. `scope 
 ref` is the perfect fit for those.

 (There may still be a use for `auto ref` with the meaning "pass 
 this as efficiently as possible, I don't care whether it's a 
 reference or not", but I'm not convinced of that either.)


I'm afraid that I don't understand this. What on earth does scope 
have to do with this issue? We need a way to indicate that a 
parameter should accept both lvalues and rvalues. auto ref was 
supposed to do that originally, but Walter misunderstood what 
Andrei was proposing and implemented what we have now, which is 
great for forwarding, but it doesn't solve the general case, 
because it only works with templates. That means that if we want 
that functionality we then need to either implement auto ref for 
non-templated functions as originally intended, or we need to 
come up with a new attribute which does that and which could be 
used with both templated and non-templated functions (thus not 
requiring the extra template bloat of the current auto ref 
implementation with templates if all you want is to accept both 
lvalues and rvalues, not have the refness of the argument 
forwarded).

What on earth does _any_ of that have to do with scope? 
Theoretically, scope indicates that a variable isn't supposed to 
escape the function - whatever that really means (it's never 
actually been defined outside of delegates, and even there, I'm 
not sure that it's defined all that well). What does that have to 
do with accepting lvalues or rvalues?

- Jonathan M Davis



 Jun 22 2015



 next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 22 June 2015 at 23:49, Jonathan M Davis via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 22 June 2015 at 13:23:47 UTC, Marc Schütz wrote:

 On Monday, 22 June 2015 at 04:11:41 UTC, Andrei Alexandrescu wrote:

 Walter and I discussed what auto ref for templates should look like and
 reached the conclusion that an approach based on lowering would be best. I
 added a proposed lowering to
 https://github.com/D-Programming-Language/dmd/pull/4717.



 I have to concur with Manu's statement in the PR. IMO `auto ref` (i.e. the
 status quo) is a hack introduced because of the lack of usable `scope`. We
 shouldn't spread it to other parts of the language, at least not for rvalue
 references. `scope ref` is the perfect fit for those.

 (There may still be a use for `auto ref` with the meaning "pass this as
 efficiently as possible, I don't care whether it's a reference or not", but
 I'm not convinced of that either.)



 I'm afraid that I don't understand this. What on earth does scope have to do
 with this issue? We need a way to indicate that a parameter should accept
 both lvalues and rvalues.


This is what I don't understand; why does this have anything to do
with rvalues/lvalues specifically?
An rvalue is just a temporary. It can safely be passed to anything
that we can be sure won't keep a reference to it; that's why escape
analysis seems to be the best approach.
This problem extends beyond rvalues, there are other forms of
temporaries which exhibit the exact same problem, ie, an rvalue
written to the stack explicitly on the line prior to the function
call:

f(vec3(1,2,3)); // onoes, comple error!

vec3 v = vec3(1,2,3);
f(v); // no worries bro!

This is accepted today, but the safety of the operation is identical
to the rejected line above. This is the de facto means to 'work
around' the problem, and it's just a pointless nuisance.
Both problems should be dealt with in the same way.


 auto ref was supposed to do that originally, but
 Walter misunderstood what Andrei was proposing and implemented what we have
 now, which is great for forwarding, but it doesn't solve the general case,
 because it only works with templates. That means that if we want that
 functionality we then need to either implement auto ref for non-templated
 functions as originally intended, or we need to come up with a new attribute
 which does that and which could be used with both templated and
 non-templated functions (thus not requiring the extra template bloat of the
 current auto ref implementation with templates if all you want is to accept
 both lvalues and rvalues, not have the refness of the argument forwarded).


It's been proposed repeatedly for years. 'scope' as a means of
expressing borrowing or escape analysis would solve the problem
elegantly, in addition to other uses.

There was also an idea Walter had at dconf 13, where he could do
runtime checking similar to range checking for safety validation,
thereby allowing unqualified ref args to receive temporaries safely.


 What on earth does _any_ of that have to do with scope? Theoretically, scope
 indicates that a variable isn't supposed to escape the function - whatever
 that really means (it's never actually been defined outside of delegates,
 and even there, I'm not sure that it's defined all that well). What does
 that have to do with accepting lvalues or rvalues?


If a parameter was scope, it can safely receive an rvalue.



 Jun 22 2015



  parent reply "Namespace" <rswhite4 gmail.com>  writes:


Rather than raising the matter of scope again and again, we 
should be thankful that a solution for this nasty problem is 
accepted and could be merged. How scope and escape analysis could 
do a better job is unclear and if you want to solve the problem 
this way you will wait a very long time. So please let us 
concentrate and discuss how we could solve this problem with auto 
ref. :)

To repeat my statement from the PR:
I would also prefer to avoid the generation of further functions, 
because I want to avoid the code bloat. I like the way of the 
current implementation, because it is following the way C++ does 
and will be understandable for every person which comes from C++.



 Jun 22 2015



  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Monday, 22 June 2015 at 16:10:10 UTC, Namespace wrote:

 Rather than raising the matter of scope again and again, we 
 should be thankful that a solution for this nasty problem is 
 accepted and could be merged.


I disagree strongly with this. A bad solution is worse than no 
solution.


 How scope and escape analysis could do a better job is unclear 
 and if you want to solve the problem this way you will wait a 
 very long time.


Not at all. `auto ref` without a working `scope` implementation 
is unsafe. If that is acceptable for now, then it should be just 
as acceptable to already use `scope ref` as a syntax for the same 
semantics, even if the required escape-proofing of `scope` is not 
yet implemented.

If/when we will finally get a working `scope`, using those 
functions will then be actually verified by the compiler. In the 
meantime, the `scope` keyword will at least express the intention 
to the caller and will serve as a reminder to the callee's author.

In contrast, with `auto ref` accepting rvalues, we will have the 
same keywords `auto ref` mean two very different things for 
templates and non-templates, we still won't have a way to write 
rvalue-ref-accepting template functions, and when we will finally 
get `scope`, `auto ref` will be a pointless alternative syntax 
for `scope ref` that will have to be kept around forever.


 So please let us concentrate and discuss how we could solve 
 this problem with auto ref. :)


Use `scope ref` instead of `auto ref` now, and just ignore that 
it's unsafe for the moment. Your PR will become even shorter with 
it, because you no longer need `STCrvref`. Conceptually, just do 
"#define STCrvref (STCref | STCscope)".


 To repeat my statement from the PR:
 I would also prefer to avoid the generation of further 
 functions, because I want to avoid the code bloat.


That's what my suggestion does, too.


 I like the way of the current implementation, because it is 
 following the way C++ does and will be understandable for every 
 person which comes from C++.


With the difference that C++ requires const-ness.



 Jun 22 2015



 next sibling parent reply "Namespace" <rswhite4 gmail.com>  writes:


We had this discussion already with DIP 36. A PR still exists 
AFAIK, but because it was rejected I don't think that's going to 
happen. :) So auto ref is obviously the choice. If you insist on 
scope/in ref, please make your own thread and ask why it was 
rejected or read the corresponding thread.



 Jun 22 2015



  parent reply "Temtaime" <temtaime gmail.com>  writes:


I see no reasons why « ref in » is bad. Maybe someone explain ?
It's also natural for those who came from C++.
In C++ there's no problem with const&, so why they will be in D?



 Jun 22 2015



  parent reply "Namespace" <rswhite4 gmail.com>  writes:


On Monday, 22 June 2015 at 18:03:43 UTC, Temtaime wrote:

 I see no reasons why « ref in » is bad. Maybe someone explain ?
 It's also natural for those who came from C++.
 In C++ there's no problem with const&, so why they will be in D?


Because const is transitive in D and therefore more restrictive.



 Jun 22 2015



  parent reply "kinke" <noone nowhere.com>  writes:


On Monday, 22 June 2015 at 18:34:37 UTC, Namespace wrote:

 On Monday, 22 June 2015 at 18:03:43 UTC, Temtaime wrote:

 I see no reasons why « ref in » is bad. Maybe someone explain ?
 It's also natural for those who came from C++.
 In C++ there's no problem with const&, so why they will be in 
 D?


 Because const is transitive in D and therefore more restrictive.


That's not a reason. It's just an additional restriction imposed 
by D and only prevents you from using the proposed `in ref 
T`/`const auto ref T` *shudder* equivalents if 
const-transitiveness would be violated. In my day-to-day C++ 
work, I almost never have to do that.

C++ "simply" ;) relies on the callee not escaping rvalue 
references bound to `const T&`. D's `(const/immutable) ref` only 
allows lvalue arguments. It simply says: 'Hey stupid, don't give 
me an rvalue reference, it might escape!' So to circumvent this 
common problem, you'll currently have to either (a) create an 
`auto ref` template or (b) declare a new variable, potentially 
having to introduce a new scope.

Is it ugly?
(a) Leads to code bloat, but provides for in-place parameter 
construction (and no indirections) for rvalue arguments of suited 
types.
(b) Definitely ugly.

Is it safe now?
(a) Ideally, the rvalue argument was in-place constructed in the 
function parameters stack. Noone but you, the dev, prevents the 
callee from escaping a reference to the parameter though.
(b) Well, the code is more verbose and emphasizes the original 
function argument expression. But it's obviously still all your 
responsibility to take care of proper lifetime for escaping 
references. Just declaring the variable right before the function 
call isn't safer than passing the rvalue reference directly.

So none of this really solves the underlying problem. One 
solution consists in classifying function parameter references as 
non-escaping ones (proposed `scope ref`/non-template `auto ref` 
accepting rvalues too, to be verified by compiler) and escaping 
ones (`ref`), the bad ones to watch out for. I'd like having to 
syntactically annotate an lvalue argument passed by non-scope 

catch the eye.



 Jun 22 2015



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 06/23/2015 12:18 AM, kinke wrote:

 In C++ there's no problem with const&, so why they will be in D?


 Because const is transitive in D and therefore more restrictive.


 That's not a reason.


Yes it is.


 It's just an additional restriction imposed by D and only prevents
 you from using the proposed `in ref T`/`const auto ref T` *shudder*
 equivalents if const-transitiveness would be violated. In my
 day-to-day C++ work, I almost never have to do that.


Note "almost".



 Jun 22 2015



  parent reply "kinke" <noone nowhere.com>  writes:


On Monday, 22 June 2015 at 23:04:14 UTC, Timon Gehr wrote:

 On 06/23/2015 12:18 AM, kinke wrote:

 In C++ there's no problem with const&, so why they will be 
 in D?


 Because const is transitive in D and therefore more 
 restrictive.


 That's not a reason.


 Yes it is.


 It's just an additional restriction imposed by D and only 
 prevents
 you from using the proposed `in ref T`/`const auto ref T` 
 *shudder*
 equivalents if const-transitiveness would be violated. In my
 day-to-day C++ work, I almost never have to do that.


 Note "almost".


Ah, I remember you from another rvalue discussion some years ago. 
;)
Care to share an example where transitive const is a problem 
apart from obvious lazily caching stuff? I don't think I've 
encountered any lately.



 Jun 22 2015



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 06/23/2015 01:27 AM, kinke wrote:

 On Monday, 22 June 2015 at 23:04:14 UTC, Timon Gehr wrote:

 On 06/23/2015 12:18 AM, kinke wrote:

 In C++ there's no problem with const&, so why they will be in D?


 Because const is transitive in D and therefore more restrictive.


 That's not a reason.


 Yes it is.


 It's just an additional restriction imposed by D and only prevents
 you from using the proposed `in ref T`/`const auto ref T` *shudder*
 equivalents if const-transitiveness would be violated. In my
 day-to-day C++ work, I almost never have to do that.


 Note "almost".


 ...
 Care to share an example where transitive const is a problem apart from
 obvious lazily caching stuff?


One obvious problem ought to be enough. Some of the remaining cases are 
obvious as well. Others are not. One way to avoid non-obvious issues is 
to keep language features orthogonal.


 I don't think I've encountered any lately.


The problems with transitive const to be encountered in day-to-day C++ 
work are few.



 Jun 22 2015



  parent  "kink" <noone nowhere.com>  writes:


On Monday, 22 June 2015 at 23:43:07 UTC, Timon Gehr wrote:

 The problems with transitive const to be encountered in 
 day-to-day C++ work are few.


Trolling as always. You know what I meant. Haven't had to resort 
to mutable fields and/or const_casts in a long time. Mostly a 
matter of good design imho.

As different behavior for templated `auto ref` and non-template 
`auto ref` seems to be an issue for more people here, why don't 
just rename the non-templated one `scope ref`, so that it's 
usable for templates too and also enables the `in ref` shortcut 
for const? I find `const auto ref` extremely clumsy.



 Jun 23 2015



prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 6/22/15 9:54 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:

 `auto ref` without a working `scope` implementation is unsafe.


I'm unclear on this. With the existence of the return attribute, what's 
unsafe about auto ref? -- Andrei



 Jun 22 2015



  parent  "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Monday, 22 June 2015 at 18:10:11 UTC, Andrei Alexandrescu 
wrote:

 On 6/22/15 9:54 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:

 `auto ref` without a working `scope` implementation is unsafe.


 I'm unclear on this. With the existence of the return 
 attribute, what's unsafe about auto ref? -- Andrei


DIP25 _is_ a working `scope` implementation, for that matter, 
albeit incomplete. (Strictly speaking, it's not the `return` 
attribute that makes it safe, but the fact that under this 
proposal `ref` implies what `scope` was originally supposed to 
mean.)

You're right that if DIP25 is all we'll ever get (i.e. the 
`scope` keyword will never have any meaning), then `auto ref` 
might not be so bad a choice, although it still leaves the fact 
that it does something different for template and non-template 
functions. But it would create other inconsistencies with other 
kinds of references, e.g.

     void bar(ref MyStruct s);
     void foo(MyClass c);

     void baz() {
         MyStruct s;
         Scoped!MyClass c;
         bar(s); // safe
         foo(c); // not safe, reference may escape
     }

For consistency, I would much prefer if `ref` and scope-ness were 
kept independent, because they are orthogonal concepts. Why 
should one kind of reference imply `scope`, but not others?



 Jun 23 2015



prev sibling  parent reply "kinke" <noone nowhere.com>  writes:


On Monday, 22 June 2015 at 16:54:50 UTC, Marc Schütz wrote:

 On Monday, 22 June 2015 at 16:10:10 UTC, Namespace wrote:

 Rather than raising the matter of scope again and again, we 
 should be thankful that a solution for this nasty problem is 
 accepted and could be merged.


 I disagree strongly with this. A bad solution is worse than no 
 solution.


 How scope and escape analysis could do a better job is unclear 
 and if you want to solve the problem this way you will wait a 
 very long time.


 Not at all. `auto ref` without a working `scope` implementation 
 is unsafe. If that is acceptable for now, then it should be 
 just as acceptable to already use `scope ref` as a syntax for 
 the same semantics, even if the required escape-proofing of 
 `scope` is not yet implemented.

 If/when we will finally get a working `scope`, using those 
 functions will then be actually verified by the compiler. In 
 the meantime, the `scope` keyword will at least express the 
 intention to the caller and will serve as a reminder to the 
 callee's author.

 In contrast, with `auto ref` accepting rvalues, we will have 
 the same keywords `auto ref` mean two very different things for 
 templates and non-templates, we still won't have a way to write 
 rvalue-ref-accepting template functions, and when we will 
 finally get `scope`, `auto ref` will be a pointless alternative 
 syntax for `scope ref` that will have to be kept around forever.


 So please let us concentrate and discuss how we could solve 
 this problem with auto ref. :)


 Use `scope ref` instead of `auto ref` now, and just ignore that 
 it's unsafe for the moment. Your PR will become even shorter 
 with it, because you no longer need `STCrvref`. Conceptually, 
 just do "#define STCrvref (STCref | STCscope)".


I definitely see your point and very much agree!


 I like the way of the current implementation, because it is 
 following the way C++ does and will be understandable for 
 every person which comes from C++.


 With the difference that C++ requires const-ness.


Yep, so `in ref T` translating to `scope const ref T` would be 
D's convenient and safe counterpart to C++' `const T&` 
parameters, for their main use case: passing pure-input arguments 
of types which are or may be costly to copy (post-blit ctor, 
dtor, or simply big). I explicitly mention 'may be' here because 
in templates one often doesn't know (containers...).

As I have already pointed out in another thread, I'd go one step 
further and propose an extremely convenient `in T` for this very 
common use case:

* The argument is passed by value (`const T`) if the compiler 
assumes moving/copying is more efficient than passing a reference 
(with its indirection on the callee side) for the particular 
target environment (hardware, ABI), e.g., for plain-old-datatypes 
T fitting into 1-2 registers, and Object references obviously.
* Otherwise, the argument is passed by-ref (`in ref T`). As `in 
T` doesn't mention any ref at all, it's clear that the hidden 
reference cannot escape.

Let me give you an example. The Win64 ABI only allows passing POD 
types <= 64 bit by value (exception: vector/SSE types). All other 
types need to be passed by ref, which means that D's otherwise 
cool in-place construction of arguments on the callee's function 
parameters stack isn't possible. The current `auto ref` 
implementation for templates is in vain on Win64 for these 
byref-types, as the ABI forces both versions to take the argument 
byref anyway.

I'm pretty sure I could live without an `auto ref` for templates 
once I can use `in T` for all my pure-input parameters. If the 
function may change the argument, I'd go with `scope ref T`, 
accepting rvalues too. If the function may even let the argument 
escape, it'll have to be `ref T` or `const/immutable ref T`. And 
if I just want to mutate the parameter inside my function, but 
not touch the argument, it'd be `T`. I can't think of any other 
use cases for function parameters atm.



 Jun 22 2015



  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Monday, 22 June 2015 at 19:05:28 UTC, kinke wrote:

 On Monday, 22 June 2015 at 16:54:50 UTC, Marc Schütz wrote:

 With the difference that C++ requires const-ness.


 Yep, so `in ref T` translating to `scope const ref T` would be 
 D's convenient and safe counterpart to C++' `const T&` 
 parameters, for their main use case: passing pure-input 
 arguments of types which are or may be costly to copy 
 (post-blit ctor, dtor, or simply big). I explicitly mention 
 'may be' here because in templates one often doesn't know 
 (containers...).


To clarify: What I meant by my comment was that const-ness should 
not be a precondition for allowing rvalue refs. Mutable rvalue 
refs are fine.


 As I have already pointed out in another thread, I'd go one 
 step further and propose an extremely convenient `in T` for 
 this very common use case:

 * The argument is passed by value (`const T`) if the compiler 
 assumes moving/copying is more efficient than passing a 
 reference (with its indirection on the callee side) for the 
 particular target environment (hardware, ABI), e.g., for 
 plain-old-datatypes T fitting into 1-2 registers, and Object 
 references obviously.
 * Otherwise, the argument is passed by-ref (`in ref T`). As `in 
 T` doesn't mention any ref at all, it's clear that the hidden 
 reference cannot escape.


Theoretically, `immutable ref` by itself would already allow 
these semantics (without the `scope` that `in` implies). Because 
(disregarding identity/addresses) for an immutable object there 
is no observable difference between pass-by-value and 
pass-by-reference. The same is not always true for `const ref`, 
whenever aliasing is possible:

     void foo(const ref int a, ref int b) {
         int x = a;
         b++;
         assert(x == a); // can fail if both refer to the same 
variable
         // similar with global variables
     }

To guarantee this from the caller's POV, the callee must be pure 
and the parameters must be known not to alias each other.



 Jun 23 2015



  parent reply "kink" <noone nowhere.com>  writes:


On Tuesday, 23 June 2015 at 09:57:26 UTC, Marc Schütz wrote:

 On Monday, 22 June 2015 at 19:05:28 UTC, kinke wrote:
 [...]
 To clarify: What I meant by my comment was that const-ness 
 should not be a precondition for allowing rvalue refs. Mutable 
 rvalue refs are fine.


I know and see it the same way.


 As I have already pointed out in another thread, I'd go one 
 step further and propose an extremely convenient `in T` for 
 this very common use case:

 * The argument is passed by value (`const T`) if the compiler 
 assumes moving/copying is more efficient than passing a 
 reference (with its indirection on the callee side) for the 
 particular target environment (hardware, ABI), e.g., for 
 plain-old-datatypes T fitting into 1-2 registers, and Object 
 references obviously.
 * Otherwise, the argument is passed by-ref (`in ref T`). As 
 `in T` doesn't mention any ref at all, it's clear that the 
 hidden reference cannot escape.


 Theoretically, `immutable ref` by itself would already allow 
 these semantics (without the `scope` that `in` implies). 
 Because (disregarding identity/addresses) for an immutable 
 object there is no observable difference between pass-by-value 
 and pass-by-reference. The same is not always true for `const 
 ref`, whenever aliasing is possible:

     void foo(const ref int a, ref int b) {
         int x = a;
         b++;
         assert(x == a); // can fail if both refer to the same 
 variable
         // similar with global variables
     }

 To guarantee this from the caller's POV, the callee must be 
 pure and the parameters must be known not to alias each other.


This is obviously true. Rvalues aren't affected as they cannot 
alias another parameter by definition. Lvalues are if passed by 
ref and the same instance is accessible by mutable ref from 
another parameter or global. But as shown by your example, that 
danger is always there. The proposed `in` semantics make it less 
obvious, that's true, but I still think it'd be worth it, as 
these aliasing bugs are a pain to track down, but in my 
experience extremely rare.



 Jun 23 2015



  parent  "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Tuesday, 23 June 2015 at 12:45:25 UTC, kink wrote:

 On Tuesday, 23 June 2015 at 09:57:26 UTC, Marc Schütz wrote:

 To guarantee this from the caller's POV, the callee must be 
 pure and the parameters must be known not to alias each other.


 This is obviously true. Rvalues aren't affected as they cannot 
 alias another parameter by definition.


You're right, I didn't think of that! They are not _transitively_ 
unique, but there can indeed be no other references to the 
rvalues themselves. So for rvalues, const is enough. Nice :-)


 Lvalues are if passed by ref and the same instance is 
 accessible by mutable ref from another parameter or global. But 
 as shown by your example, that danger is always there. The 
 proposed `in` semantics make it less obvious, that's true, but 
 I still think it'd be worth it, as these aliasing bugs are a 
 pain to track down, but in my experience extremely rare.





 Jun 23 2015



prev sibling  parent  "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Monday, 22 June 2015 at 13:49:31 UTC, Jonathan M Davis wrote:

 On Monday, 22 June 2015 at 13:23:47 UTC, Marc Schütz wrote:

 On Monday, 22 June 2015 at 04:11:41 UTC, Andrei Alexandrescu 
 wrote:

 Walter and I discussed what auto ref for templates should 
 look like and reached the conclusion that an approach based 
 on lowering would be best. I added a proposed lowering to 
 https://github.com/D-Programming-Language/dmd/pull/4717.


 I have to concur with Manu's statement in the PR. IMO `auto 
 ref` (i.e. the status quo) is a hack introduced because of the 
 lack of usable `scope`. We shouldn't spread it to other parts 
 of the language, at least not for rvalue references. `scope 
 ref` is the perfect fit for those.

 (There may still be a use for `auto ref` with the meaning 
 "pass this as efficiently as possible, I don't care whether 
 it's a reference or not", but I'm not convinced of that 
 either.)


 I'm afraid that I don't understand this. What on earth does 
 scope have to do with this issue? We need a way to indicate 
 that a parameter should accept both lvalues and rvalues.


`scope` is a precondition for doing this safely. As Manu pointed 
out, rvalues are merely a special case of a more general problem, 
but with them, the risk of misuse is much higher than with local 
variables. That was probably the reason why `ref` originally was 
designed not to accept rvalues. Given a working `scope` feature, 
that reason is no longer relevant, therefore `scope ref` can 
safely accept both rvalues and lvalues.



 Jun 22 2015



prev sibling next sibling parent  "Dennis Ritchie" <dennis.ritchie mail.ru>  writes:


On Monday, 22 June 2015 at 04:11:41 UTC, Andrei Alexandrescu 
wrote:

 Walter and I discussed what auto ref for templates should look 
 like and reached the conclusion that an approach based on 
 lowering would be best. I added a proposed lowering to 
 https://github.com/D-Programming-Language/dmd/pull/4717.


`For example, consider the declaration:`

ref int fun(auto ref int x);

`That should lower to:`

ref int fun(int' x);   /* or */   ref int fun('int x);



 Jun 22 2015



prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 06/22/2015 06:11 AM, Andrei Alexandrescu wrote:

 Walter and I discussed what auto ref for templates should look like and
 reached the conclusion that an approach based on lowering would be best.
 I added a proposed lowering to
 https://github.com/D-Programming-Language/dmd/pull/4717.

 Andrei


" WalterBright auto ref for templates and non-templates do different 
things by necessity. ..."

The proposed lowering also works for template functions.

"... This is not an issue."

Yes it is. Walter is right.



 Jun 22 2015



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 6/22/15 3:06 PM, Timon Gehr wrote:

 On 06/22/2015 06:11 AM, Andrei Alexandrescu wrote:

 Walter and I discussed what auto ref for templates should look like and
 reached the conclusion that an approach based on lowering would be best.
 I added a proposed lowering to
 https://github.com/D-Programming-Language/dmd/pull/4717.

 Andrei


 " WalterBright auto ref for templates and non-templates do different
 things by necessity. ..."

 The proposed lowering also works for template functions.

 "... This is not an issue."

 Yes it is. Walter is right.


For templates, auto ref generates two copies of the function. Inside, 
which was generated can be interrogated by using is(param == ref). For 
non-templates, only one copy of the function is generated. Using 
is(param == ref) inside should be an error.

The two mechanisms bear many similarities and some differences, too.


Andrei



 Jun 22 2015



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 06/23/2015 12:40 AM, Andrei Alexandrescu wrote:

 On 6/22/15 3:06 PM, Timon Gehr wrote:

 On 06/22/2015 06:11 AM, Andrei Alexandrescu wrote:

 Walter and I discussed what auto ref for templates should look like and
 reached the conclusion that an approach based on lowering would be best.
 I added a proposed lowering to
 https://github.com/D-Programming-Language/dmd/pull/4717.

 Andrei


 " WalterBright auto ref for templates and non-templates do different
 things by necessity. ..."

 The proposed lowering also works for template functions.

 "... This is not an issue."

 Yes it is. Walter is right.


 For templates, auto ref generates two copies of the function. Inside,
 which was generated can be interrogated by using is(param == ref). For
 non-templates, only one copy of the function is generated. Using
 is(param == ref) inside should be an error.

 The two mechanisms bear many similarities and some differences, too.
 ...


There is no reason to prevent templates from using the mechanism that 
generates only one copy. The two mechanisms shouldn't share the same 
syntax, because then there is no way to tell them apart for template 
functions.



 Jun 22 2015



 next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 6/22/15 4:09 PM, Timon Gehr wrote:

 There is no reason to prevent templates from using the mechanism that
 generates only one copy. The two mechanisms shouldn't share the same
 syntax, because then there is no way to tell them apart for template
 functions.


I understand. For my money I'd be okay with what's being proposed 
instead of complicating the language yet again for the perfect solution. 
-- Andrei



 Jun 22 2015



 next sibling parent  "Namespace" <rswhite4 gmail.com>  writes:


On Tuesday, 23 June 2015 at 01:01:26 UTC, Andrei Alexandrescu 
wrote:

 On 6/22/15 4:09 PM, Timon Gehr wrote:

 There is no reason to prevent templates from using the 
 mechanism that
 generates only one copy. The two mechanisms shouldn't share 
 the same
 syntax, because then there is no way to tell them apart for 
 template
 functions.


 I understand. For my money I'd be okay with what's being 
 proposed instead of complicating the language yet again for the 
 perfect solution. -- Andrei


There is no perfect solution. :) Some are for in/scope ref, some 
are against it. Some are for auto ref, some are against it and 
even a few are for const ref but most are against it. So the 
perfect solution does not exist. And since scope/in ref was 
already rejected (DIP 36) I think auto ref is the only choice to 
introduce this feature without introducing a new attribute. And 
since you reverted the introducing of virtual because it has not 
enough value, I'm sure that introducing a new attribute just to 
accept both, lvalues and rvalues, wouldn't have enough value as 
well. I think the way it is implemented right now is good enough, 
but what matters is what you and Walter think.



 Jun 23 2015



prev sibling  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 06/23/2015 03:01 AM, Andrei Alexandrescu wrote:

 On 6/22/15 4:09 PM, Timon Gehr wrote:

 There is no reason to prevent templates from using the mechanism that
 generates only one copy. The two mechanisms shouldn't share the same
 syntax, because then there is no way to tell them apart for template
 functions.


 I understand. For my money I'd be okay with what's being proposed
 instead of complicating the language yet again for


We're introducing a new feature either way. I don't agree that the 
proposed hack complicates the language any less.


 the perfect solution.


We might as well do it right. Why promote unnecessary template bloat?



 Jun 23 2015



  parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Wednesday, 24 June 2015 at 00:23:42 UTC, Timon Gehr wrote:

 On 06/23/2015 03:01 AM, Andrei Alexandrescu wrote:

 the perfect solution.


 We might as well do it right. Why promote unnecessary template 
 bloat?


I'm fine with it either way, but given all of the template bloat 
that auto ref creates and how common templates are in D (and will 
only become more so as more stuff is range-based and uses design 
by introspection), the cost of auto ref generating extra template 
instantiations will only increase. So, I think that we'd be 
better off simply adding a new attribute like  anyref or whatever 
we want to call it. But I don't know how much it will ultimately 
matter.

- Jonathan M Davis



 Jun 24 2015



  parent reply "Temtaime" <temtaime gmail.com>  writes:


Simply give a possibility to "ref in" allowing use rvalues.
Don't add any new strange attributes.



 Jun 24 2015



  parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Wednesday, 24 June 2015 at 09:26:49 UTC, Temtaime wrote:

 Simply give a possibility to "ref in" allowing use rvalues.


That has already been rejected.

- Jonathan M Davis



 Jun 24 2015



 next sibling parent reply "Namespace" <rswhite4 gmail.com>  writes:


On Wednesday, 24 June 2015 at 09:54:01 UTC, Jonathan M Davis 
wrote:

 On Wednesday, 24 June 2015 at 09:26:49 UTC, Temtaime wrote:

 Simply give a possibility to "ref in" allowing use rvalues.


 That has already been rejected.

 - Jonathan M Davis


How many times did I say that already? :)



 Jun 24 2015



  parent  "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Wednesday, 24 June 2015 at 10:02:12 UTC, Namespace wrote:

 On Wednesday, 24 June 2015 at 09:54:01 UTC, Jonathan M Davis 
 wrote:

 On Wednesday, 24 June 2015 at 09:26:49 UTC, Temtaime wrote:

 Simply give a possibility to "ref in" allowing use rvalues.


 That has already been rejected.

 - Jonathan M Davis


 How many times did I say that already? :)


Indeed.

- Jonathan M Davis



 Jun 24 2015



prev sibling  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Wednesday, 24 June 2015 at 09:54:01 UTC, Jonathan M Davis 
wrote:

 On Wednesday, 24 June 2015 at 09:26:49 UTC, Temtaime wrote:

 Simply give a possibility to "ref in" allowing use rvalues.


 That has already been rejected.


Then we need to reconsider it. If that was several years ago, 
much has changed since then.

Can you point me to that decision? Who made it, and on what 
basis? I can't find anything in the threads linked in 
http://wiki.dlang.org/DIP36



 Jun 24 2015



  parent reply "Namespace" <rswhite4 gmail.com>  writes:


On Wednesday, 24 June 2015 at 17:45:15 UTC, Marc Schütz wrote:

 On Wednesday, 24 June 2015 at 09:54:01 UTC, Jonathan M Davis 
 wrote:

 On Wednesday, 24 June 2015 at 09:26:49 UTC, Temtaime wrote:

 Simply give a possibility to "ref in" allowing use rvalues.


 That has already been rejected.


 Then we need to reconsider it. If that was several years ago, 
 much has changed since then.

 Can you point me to that decision? Who made it, and on what 
 basis? I can't find anything in the threads linked in 
 http://wiki.dlang.org/DIP36


Read the thread: 
http://forum.dlang.org/thread/ylebrhjnrrcajnvtthtt forum.dlang.org?page=1



 Jun 24 2015



  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Wednesday, 24 June 2015 at 17:47:51 UTC, Namespace wrote:

 On Wednesday, 24 June 2015 at 17:45:15 UTC, Marc Schütz wrote:

 On Wednesday, 24 June 2015 at 09:54:01 UTC, Jonathan M Davis 
 wrote:

 On Wednesday, 24 June 2015 at 09:26:49 UTC, Temtaime wrote:

 Simply give a possibility to "ref in" allowing use rvalues.


 That has already been rejected.


 Then we need to reconsider it. If that was several years ago, 
 much has changed since then.

 Can you point me to that decision? Who made it, and on what 
 basis? I can't find anything in the threads linked in 
 http://wiki.dlang.org/DIP36


 Read the thread: 
 http://forum.dlang.org/thread/ylebrhjnrrcajnvtthtt forum.dlang.org?page=1


Thanks. There was no link to this thread on the wiki page.

But I can't really find a definite rejection in it, in the sense 
of someone with authority saying "This DIP is rejected". The 
closest I can find is Andrei making some vague statements about 
what is proposed being a new feature (I don't know though whether 
this is an argument against the DIP), and it being unsafe because 
scope is not implemented (which was true at the time, but we now 
have DIP25 as a mechanism, albeit incomplete).

If you can find anything definite, can you please add it to the 
wiki page? I.e. officially rejected by NN on YYYY-MM-DD 
(including link), summary of the arguments for rejection, etc.



 Jun 25 2015



  parent  "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Thursday, 25 June 2015 at 08:10:06 UTC, Marc Schütz wrote:

 On Wednesday, 24 June 2015 at 17:47:51 UTC, Namespace wrote:

 On Wednesday, 24 June 2015 at 17:45:15 UTC, Marc Schütz wrote:

 On Wednesday, 24 June 2015 at 09:54:01 UTC, Jonathan M Davis 
 wrote:

 On Wednesday, 24 June 2015 at 09:26:49 UTC, Temtaime wrote:

 Simply give a possibility to "ref in" allowing use rvalues.


 That has already been rejected.


 Then we need to reconsider it. If that was several years ago, 
 much has changed since then.

 Can you point me to that decision? Who made it, and on what 
 basis? I can't find anything in the threads linked in 
 http://wiki.dlang.org/DIP36


 Read the thread: 
 http://forum.dlang.org/thread/ylebrhjnrrcajnvtthtt forum.dlang.org?page=1


 Thanks. There was no link to this thread on the wiki page.

 But I can't really find a definite rejection in it, in the 
 sense of someone with authority saying "This DIP is rejected". 
 The closest I can find is Andrei making some vague statements 
 about what is proposed being a new feature (I don't know though 
 whether this is an argument against the DIP), and it being 
 unsafe because scope is not implemented (which was true at the 
 time, but we now have DIP25 as a mechanism, albeit incomplete).

 If you can find anything definite, can you please add it to the 
 wiki page? I.e. officially rejected by NN on YYYY-MM-DD 
 (including link), summary of the arguments for rejection, etc.


If it helps, at dconf, two years ago, when Manu tried to 
repeatedly convince Walter and Andrei to have scope ref do 
something similar to what we're talking about auto ref doing, 
Walter and Andrei repeatedly shot it down. They had no interest 
in having scope ref as any kind of special attribute. So, as of 
the beginning of May 2013 (which was less than a month after what 
the DIP lists as its last modification date), it was being 
refused by Walter and Andrei in person. And if you look at its 
revision history, Dicebot (who is one of the two authors of the 
DIP) has marked it as rejected.

- Jonathan M Davis



 Jun 25 2015



prev sibling  parent reply bitwise <bitwise.pvt gmail.com>  writes:


On Mon, 22 Jun 2015 19:09:45 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:


 There is no reason to prevent templates from using the mechanism that  
 generates only one copy. The two mechanisms shouldn't share the same  
 syntax, because then there is no way to tell them apart for template  
 functions.


You would be losing the optimization of passing primitive types by value,  
wouldn't you?

   Bit



 Jun 23 2015



  parent reply "kinke" <noone nowhere.com>  writes:


On Tuesday, 23 June 2015 at 18:11:21 UTC, bitwise wrote:

 On Mon, 22 Jun 2015 19:09:45 -0400, Timon Gehr 
 <timon.gehr gmx.ch> wrote:


 There is no reason to prevent templates from using the 
 mechanism that generates only one copy. The two mechanisms 
 shouldn't share the same syntax, because then there is no way 
 to tell them apart for template functions.


 You would be losing the optimization of passing primitive types 
 by value, wouldn't you?


Not if you keep the current `auto ref` template syntax AND 
introduce another syntax `scope ref` for non-escapable references 
also accepting rvalues.



 Jun 23 2015



  parent reply bitwise <bitwise.pvt gmail.com>  writes:


On Tue, 23 Jun 2015 14:33:49 -0400, kinke <noone nowhere.com> wrote:


 On Tuesday, 23 June 2015 at 18:11:21 UTC, bitwise wrote:

 On Mon, 22 Jun 2015 19:09:45 -0400, Timon Gehr <timon.gehr gmx.ch>  
 wrote:


 There is no reason to prevent templates from using the mechanism that  
 generates only one copy. The two mechanisms shouldn't share the same  
 syntax, because then there is no way to tell them apart for template  
 functions.


 You would be losing the optimization of passing primitive types by  
 value, wouldn't you?


 Not if you keep the current `auto ref` template syntax AND introduce  
 another syntax `scope ref` for non-escapable references also accepting  
 rvalues.


I don't think 'scope ref' is on the table at this point(although I do  
believe it should be).

And I still think Timon's statement is untrue. There is a reason, which is  
that the new auto ref syntax forces reference parameters for all  
types(even primitives), where the old template approach does not.

   Bit



 Jun 23 2015



 next sibling parent  "kinke" <noone nowhere.com>  writes:


On Tuesday, 23 June 2015 at 19:13:28 UTC, bitwise wrote:

 And I still think Timon's statement is untrue. There is a 
 reason, which is that the new auto ref syntax forces reference 
 parameters for all types(even primitives), where the old 
 template approach does not.


Timon suggested using another syntax/name for the new refs, as 
one couldn't use them in templates [and the different `auto ref` 
behavior is just prone to confuse beginners, my 2 cents]. One 
reason for preferring `scope ref` over `auto ref` in templates 
would be situations where code bloat is more important than 
runtime performance, another reason the fact that `in ref` is 
more convenient than `const auto ref`. ;)



 Jun 23 2015



prev sibling  parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Tuesday, 23 June 2015 at 19:13:28 UTC, bitwise wrote:

 And I still think Timon's statement is untrue. There is a 
 reason, which is that the new auto ref syntax forces reference 
 parameters for all types(even primitives), where the old 
 template approach does not.


The "old template approach" isn't going anywhere. When used with 
templated functions, auto ref will continue to function the same 
way, and if you want that behavior, you templatize your function. 
AFAIK, there are really only two ways to tackle having a function 
accept both lvalues and rvalues for the same parameter:

1. Duplicate the entire function and have one overload take an 
lvalue and the other take an rvalue. This is what the the 
templated auto ref does (though which overloads get generated 
depends on what arguments are passed to the function).

2. Assign rvalues to an lvalue and pass that in to the function 
by ref. This is what's going to happen with non-templated 
functions. It's just a question of whether that's going to be 
done by introducing a temporary variable or by having the 
compiler add overloads which take rvalues and then call the 
primary function which takes its arguments by ref.

And at this point, I think that there are really only three 
options that we might take in making it so that we can have 
non-templated functions accept both lvalues and rvalues with the 
same function declaration:

1. Leave things as they are and not support it except with 
templated functions (not desirable, but that may be the result of 
all of this).

2. Implement auto ref for non-templated functions as currently 
proposed, leaving auto ref on templated functions exactly as it 
is and providing no way to use the non-templated behavior with 
templated functions (though _maybe_ we could add compiler 
optimizations later to use the non-templated version in cases 
where the compiler can detect that using the non-templated 
version with a templated function would have the same semantics 
as the templated version).

3. Add a new attribute which does what's being proposed for auto 
ref for non-templated functions, in which case, we can use the 
non-templated behavior with templates as well and thus avoid 
template bloat when all you want is for your templated function 
to accept both lvalues and rvalues. auto ref, of course, then 
stays exactly as it is now.



especially when you consider how much D code tends to be 
templated and how much code bloat auto ref is likely to generate 
with templated functions.

- Jonathan M Davis



 Jun 24 2015



  parent reply "kink" <noone nowhere.com>  writes:


On Wednesday, 24 June 2015 at 11:19:04 UTC, Jonathan M Davis 
wrote:

 [...]

 3. Add a new attribute which does what's being proposed for 
 auto ref for non-templated functions, in which case, we can use 
 the non-templated behavior with templates as well and thus 
 avoid template bloat when all you want is for your templated 
 function to accept both lvalues and rvalues. auto ref, of 
 course, then stays exactly as it is now.



 especially when you consider how much D code tends to be 
 templated and how much code bloat auto ref is likely to 
 generate with templated functions.

 - Jonathan M Davis



`scope ref` and simplify the PR a lil' bit as suggested by Marc 
in an earlier post 
[http://forum.dlang.org/post/ricvtchihgzyisbkzcgl forum.dlang.org].



 Jun 24 2015



  parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Wednesday, 24 June 2015 at 16:39:22 UTC, kink wrote:

 On Wednesday, 24 June 2015 at 11:19:04 UTC, Jonathan M Davis 
 wrote:

 [...]

 3. Add a new attribute which does what's being proposed for 
 auto ref for non-templated functions, in which case, we can 
 use the non-templated behavior with templates as well and thus 
 avoid template bloat when all you want is for your templated 
 function to accept both lvalues and rvalues. auto ref, of 
 course, then stays exactly as it is now.



 especially when you consider how much D code tends to be 
 templated and how much code bloat auto ref is likely to 
 generate with templated functions.

 - Jonathan M Davis



 `scope ref` and simplify the PR a lil' bit as suggested by Marc 
 in an earlier post 
 [http://forum.dlang.org/post/ricvtchihgzyisbkzcgl forum.dlang.org].


But this has _nothing_ to do with scope, and scope ref was 
already rejected. The whole point of this is support having a 
function accept both rvalues and lvalues, not to do anything with 
scope.

And given that what scope does has never even been properly 
defined - all that the spec says about scope parameters is 
"references in the parameter cannot be escaped (e.g. assigned to 
a global variable)" - and that the only place that scope does 
anything at all is with delegates, trying to expand it with 
"scope ref" as if that were simply an extension of scope makes no 
sense. Before we can even consider what something like scope ref 
might mean, we'd have to properly define what scope means. And 
all we have for it is the basic idea of what it's supposed to do 
- none of the details - and trying to define scope ref before 
defining what scope means in general could totally hamstring us 
when properly defining scope later.

- Jonathan M Davis



 Jun 24 2015



 next sibling parent reply "kink" <noone nowhere.com>  writes:


On Wednesday, 24 June 2015 at 23:30:53 UTC, Jonathan M Davis 
wrote:

 But this has _nothing_ to do with scope, and scope ref was 
 already rejected. The whole point of this is support having a 
 function accept both rvalues and lvalues, not to do anything 
 with scope.

 And given that what scope does has never even been properly 
 defined - all that the spec says about scope parameters is 
 "references in the parameter cannot be escaped (e.g. assigned 
 to a global variable)".


Yeah right. And all we need to safely pass in rvalue references 
is exactly the constraint that they won't escape. And scope alone 
doesn't imply pass-by-ref (`in` params aren't currently passed by 
ref).


 [...] Before we can even consider what something like scope ref 
 might mean, we'd have to properly define what scope means. And 
 all we have for it is the basic idea of what it's supposed to 
 do - none of the details - and trying to define scope ref 
 before defining what scope means in general could totally 
 hamstring us when properly defining scope later.


Is there a roadmap for "later"? It seems like these things always 
just get postponed, further postponed and never really done. What 
about the original argument 2 years ago, when rejecting DIP36, 
where Andrei explained that the idea was to make `ref` itself not 
escapable and delegating escaping refs to pointers? I don't see 
how that could be implemented without a breaking change, so I 
guess that may be something for D3. But we have been needing a 
solution for rvalue refs for years, and, as you know, are waiting 
since then, as `auto ref` for templates alone is simply not 
enough.

To quote Manu from the DIP36 thread:

 It is, without doubt, the single biggest complaint I've heard
 by virtually every programmer I've introduced D to.


+1 kazillion



 Jun 25 2015



  parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Thursday, 25 June 2015 at 08:04:09 UTC, kink wrote:

 On Wednesday, 24 June 2015 at 23:30:53 UTC, Jonathan M Davis 
 wrote:

 But this has _nothing_ to do with scope, and scope ref was 
 already rejected. The whole point of this is support having a 
 function accept both rvalues and lvalues, not to do anything 
 with scope.

 And given that what scope does has never even been properly 
 defined - all that the spec says about scope parameters is 
 "references in the parameter cannot be escaped (e.g. assigned 
 to a global variable)".


 Yeah right. And all we need to safely pass in rvalue references 
 is exactly the constraint that they won't escape. And scope 
 alone doesn't imply pass-by-ref (`in` params aren't currently 
 passed by ref).


Whether a reference escapes is an orthogonal issue. The return 
attribute is for dealing with that. The function should be able 
to return by ref or not and still accept both rvalues and lvalues 
for its parameters. As long as the temporary variable created for 
holding an rvalue exists for the duration of the statement that 
the function call is in, it doesn't matter. It should be 
perfectly fine to have the function return one of its ref 
arguments by ref (and thus avoid a copy), even if it was 
originally an rvalue, because it have been assigned to an lvalue 
that would be around long enough. The only issue arises when the 
caller then returns the return value by ref, in which case that's 
going to be illegal without the return attribute. But in general, 
the fact that a ref to an argument could be returned by ref is 
orthogonal to being able to pass in both lvalues and rvalues, 
because the rvalues all have to become lvalues to be passed in 
anyway, in which case, you're in exactly the same place with 
regards to safety that you are with normal ref.


 [...] Before we can even consider what something like scope 
 ref might mean, we'd have to properly define what scope means. 
 And all we have for it is the basic idea of what it's supposed 
 to do - none of the details - and trying to define scope ref 
 before defining what scope means in general could totally 
 hamstring us when properly defining scope later.


 Is there a roadmap for "later"? It seems like these things 
 always just get postponed, further postponed and never really 
 done.


Perhaps so, but just because scope hasn't properly defined yet 
isn't a reason to use it for something else entirely right now - 
especially when whether the lvalue that gets passed into the 
function is returned by ref is an orthogonal issue to having a 
type of ref which accepts an rvalue and silently converts it to 
an lvalue so that it can be passed by ref.

- Jonathan M Davis



 Jun 25 2015



  parent reply "kinke" <noone nowhere.com>  writes:


On Thursday, 25 June 2015 at 10:10:42 UTC, Jonathan M Davis wrote:

 Whether a reference escapes is an orthogonal issue. The return 
 attribute is for dealing with that. The function should be able 
 to return by ref or not and still accept both rvalues and 
 lvalues for its parameters. As long as the temporary variable 
 created for holding an rvalue exists for the duration of the 
 statement that the function call is in, it doesn't matter.


When I was talking about escaping references, I didn't mean 
returned references; I meant storing a pointer to a ref parameter 
somewhere outside (global/instance variable).

This snippet is a rough sketch for an approach based on `scope 
ref`, to prevent escaping rvalues:

<CODE>

struct S
{
     int a;
     ref S chain() return { return this; } // returns a `scope 
ref` for rvalues, `ref` for lvalues
}

// pointers derived from `scope ref` params can safely be passed 
down to scope pointer params only
void manipulateWithoutEscaping(scope ref S s) { manipulate(&s, 
S.sizeof); }

// doesn't escape any part of its `s` reference, hence `scope ref`
// analog to S.chain(), returns a `scope ref` for rvalues, `ref` 
for lvalues
ref S forward(return scope ref S s) { return s; }

// low-level manipulation functions not escaping any parts of 
their pointees
void manipulate(scope void* ptr, size_t size) { ptr[0..size] = 0; 
}
void manipulate(scope void[] data) { manipulate(data.ptr, 
data.length); }

struct State
{
     S* s;        // external ownership; should most likely 
actually be shared ownership
     void[] data; // ditto
     void initByEscaping(ref S s)
     {
         this.s = &s; // obvious
         data = (cast(void*)&s.a)[0 .. int.sizeof]; // less obvious
     }
     void manipulate() { if (s) manipulate(data); }
}

void foo()
{
     // forward rvalue ref and invoke chain() on it twice (each 
call forwarding the rvalue ref)
     // rvalue is destructed at the end of the statement
     forward(S(1)).chain().chain();

     // forward rvalue ref and let it be manipulated before 
destruction
     manipulateWithoutEscaping(forward(S(2)));

     // promote forwarded rvalue ref to lvalue
     S s3 = forward(S(3));
     // let the lvalue escape and manipulate it indirectly
     auto state = new State();
     state.initByEscaping(forward(s3)); // requires lvalue `ref`, 
forward() returns that for lvalue arg
     state.manipulate();
     // GC-managed `state` outlives `s3` => dangling pointers 
`state.s` and `state.data.ptr`!
}

</CODE>

The first observation is that a `scope ref` system would probably 
need to introduce scope pointers/slices too, to allow taking the 
address of `scope ref` params.
There'd probably be quite a few `scope` usages, cluttering code 
and introducing additional complexity.

I do see a point in trying to let `ref` alone and the compiler 
work it out. After all, that scope system would only allow us to 
prevent ourselves from escaping rvalues (if we cared to type 
'scope'). With escape analysis, the compiler could disallow 
attempts to bind rvalues to escapable references, by inferring 
scope-ness for ref and pointer params automatically. In the 
future, escaping params could be disallowed in  safe functions, 
and otherwise require an explicit `ref` keyword when supplying 
the lvalue argument, hinting at the potential for dangling 
pointers.

In the meantime, what about making all `ref` params accept 
rvalues via lowering (at least for  system)? As a first step 
until escape analysis is sufficiently implemented and the 
compiler will help us out with compile errors for escaping 
rvalues..



 Jun 26 2015



  parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Saturday, 27 June 2015 at 00:13:08 UTC, kinke wrote:

 On Thursday, 25 June 2015 at 10:10:42 UTC, Jonathan M Davis 
 wrote:

 Whether a reference escapes is an orthogonal issue. The return 
 attribute is for dealing with that. The function should be 
 able to return by ref or not and still accept both rvalues and 
 lvalues for its parameters. As long as the temporary variable 
 created for holding an rvalue exists for the duration of the 
 statement that the function call is in, it doesn't matter.


 When I was talking about escaping references, I didn't mean 
 returned references; I meant storing a pointer to a ref 
 parameter somewhere outside (global/instance variable).


That's a completely orthogonal issue to ref or auto ref. That's 
an  system operation (since taking the address of a local 
variable is  system), and it's up to you to not screw it up. 
scope might cover that if it were fully ironed out, since that 
does involve escaping, but it also might not, since it's an 
 system operation and thus up to you not to screw it up. 
Regardless, it has nothing to do with whether the function 
accepts lvalues and rvalues with the same parameter. You have a 
safety issue regardless if you're talking about taking a pointer 
to an argument (or to some portion of the argument), since 
there's no guarantee that the lifetime of the pointer is shorter 
than that of the argument. That depends entirely on what the 
argument was or what you do with the pointer, and very few 
variables have a global lifetime, so whether the argument passed 
to the ref parameter referred to a local variable is pretty much 
irrelevant. You have to know what you're doing with you take the 
address of a variable and make sure that you do it right.

Safety with regards to ref is an entirely different issue, 
because that's  safe code. Also, whether the function might 
return its ref parameter by ref is orthogonal to whether it 
accepted an rvalue by ref or not, since whether you want to do 
that or not doesn't necessarily have anything to with what's 
being returned from the function (rather, it has to do with what 
you want the function to accept), and you have a safety issue 
even without having any kind of ref which accepts rvalues, 
because all it requires is multiple layers of functions which 
return by ref, and you're screwed anyway. e.g.

ref int id(ref int i)
{
     return i;
}

ref int foo(ref int i)
{
     ++i;
     return id(i);
}

ref int bar()
{
     int i;
     return foo(i);
}

We introduced the return attribute (currently only with -dip25) 
to fix this problem. So, with that, we eliminate the safety 
problem with ref itself, and we can safely have auto ref accept 
rvalues by having it assign them to a temporary variable first. 
The rules with returning by ref would then be the same with auto 
ref as ref and require the return attribute to return a 
parameter. And all of this is within  safe code.

Any issues with taking pointers to arguments is firmly in  system 
territory and is thus completely orthogonal. Even if scope were 
to be implemented in a way that prevented it, it would still have 
nothing to do with whether the function accepted both rvalues and 
lvalues with the same parameter.


 In the meantime, what about making all `ref` params accept 
 rvalues


Please, please, no. ref indicates that your intention is to 
mutate the argument. Having it accept rvalues completely destroys 
that and makes it that much harder to understand what a function 
is supposed to do. By having a separate attribute such as auto 
ref which also accepts rvalues, we cleanly separate out the code 
which just wants to accept both rvalues and lvalues for 
efficiency purposes and the code that actually wants to take its 
argument by ref so that it can mutate it.

- Jonathan M Davis



 Jun 26 2015



  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Saturday, 27 June 2015 at 01:18:19 UTC, Jonathan M Davis wrote:

 On Saturday, 27 June 2015 at 00:13:08 UTC, kinke wrote:

 When I was talking about escaping references, I didn't mean 
 returned references; I meant storing a pointer to a ref 
 parameter somewhere outside (global/instance variable).


 That's a completely orthogonal issue to ref or auto ref. That's 
 an  system operation (since taking the address of a local 
 variable is  system), and it's up to you to not screw it up. 
 scope might cover that if it were fully ironed out, since that 
 does involve escaping, but it also might not, since it's an 
  system operation and thus up to you not to screw it up.


The point is that with scope, it _doesn't_ need to be  system 
anymore. As an example where this can be useful, see:
https://github.com/D-Programming-Language/phobos/blob/41d2027c22d6f8d22edd2df688c3cfd08fb20228/std/digest/hmac.d#L252
We currently can't make this function  safe, although it is in 
fact memory safe.


 Regardless, it has nothing to do with whether the function 
 accepts lvalues and rvalues with the same parameter. You have a 
 safety issue regardless if you're talking about taking a 
 pointer to an argument (or to some portion of the argument), 
 since there's no guarantee that the lifetime of the pointer is 
 shorter than that of the argument.


No, not with scope. It seems you're using the status quo to argue 
against what could be. But that's not valid, because the problems 
you mention will no longer exist with scope.


 We introduced the return attribute (currently only with -dip25) 
 to fix this problem. So, with that, we eliminate the safety 
 problem with ref itself, and we can safely have auto ref accept 
 rvalues by having it assign them to a temporary variable first.


Yes this is now possible, but why in all world do you want the 
keyword that enables it to be `auto ref`?


 The rules with returning by ref would then be the same with 
 auto ref as ref and require the return attribute to return a 
 parameter. And all of this is within  safe code.

 Any issues with taking pointers to arguments is firmly in 
  system territory and is thus completely orthogonal. Even if 
 scope were to be implemented in a way that prevented it, it 
 would still have nothing to do with whether the function 
 accepted both rvalues and lvalues with the same parameter.


 In the meantime, what about making all `ref` params accept 
 rvalues


 Please, please, no. ref indicates that your intention is to 
 mutate the argument. Having it accept rvalues completely 
 destroys that and makes it that much harder to understand what 
 a function is supposed to do.


I can accept that argument, although I see cases where it's still 
justified. For example, if the author of the function decided it 
wants to mutate the argument, but you (the caller) are not 
interested in the new value. But still, explicitly using a 
temporary named `dummy` is preferable because it even documents 
that you're going to ignore it.


 By having a separate attribute such as auto ref which also 
 accepts rvalues, we cleanly separate out the code which just 
 wants to accept both rvalues and lvalues for efficiency 
 purposes and the code that actually wants to take its argument 
 by ref so that it can mutate it.


Okay, then you mainly want to use special syntax to opt-in to 
rvalue references. I simply think that `auto ref` is completely 
inadequate for this purpose, because its existing meaning with 
templates doesn't have anything to do with rvalue refs.

`scope ref` on the other hand is more fitting IMO. It's not that 
`scope` necessarily implies rvalue refs, but rvalue refs _do_ 
imply scope! If we go with DIP25 where `ref` by itself already 
implies that, `scope ref` would then be redundant, and therefore 
the syntax would be available for the special purpose of 
accepting rvalue refs. This has multiple advantages:

- No new keywords or  attributes.
- Rvalue refs are still opt-in, you don't get them accidentally 
just by writing `ref`.
- It reinforces the guarantee that the argument will not escape.
- It's usable in templates, too, with the same meaning as in 
normal functions.



 Jun 27 2015



 next sibling parent  "kinke" <noone nowhere.com>  writes:


On Saturday, 27 June 2015 at 11:10:49 UTC, Marc Schütz wrote:

 On Saturday, 27 June 2015 at 01:18:19 UTC, Jonathan M Davis 
 wrote:

 That's a completely orthogonal issue to ref or auto ref. 
 That's an  system operation (since taking the address of a 
 local variable is  system), and it's up to you to not screw it 
 up. scope might cover that if it were fully ironed out, since 
 that does involve escaping, but it also might not, since it's 
 an  system operation and thus up to you not to screw it up.


 The point is that with scope, it _doesn't_ need to be  system 
 anymore.


 Regardless, it has nothing to do with whether the function 
 accepts lvalues and rvalues with the same parameter. You have 
 a safety issue regardless if you're talking about taking a 
 pointer to an argument (or to some portion of the argument), 
 since there's no guarantee that the lifetime of the pointer is 
 shorter than that of the argument.


 No, not with scope. [...] the problems you mention will no 
 longer exist with scope.


Exactly, that's the whole point.
The relation with lvalue/rvalue reference parameters is that the 
compiler should be able to statically detect and refuse escaping 
rvalue bugs. Escaping lvalues with potential for dangling 
pointers should be  system only, but preventing those bugs is a 
whole different beast.

Still, to make (a) taking the address of ref params, locals & 
globals and (b) using pointer params  safe, the compiler needs to 
know whether the parameter escapes or not. I tend to let the 
compiler figure that out in the future rather than having to use 
`scoped` a lot for ref and pointer params.


 We introduced the return attribute (currently only with 
 -dip25) to fix this problem. So, with that, we eliminate the 
 safety problem with ref itself, and we can safely have auto 
 ref accept rvalues by having it assign them to a temporary 
 variable first.


 Yes this is now possible, but why in all world do you want the 
 keyword that enables it to be `auto ref`?


Exactly. For a function returning a ref parameter, it doesn't 
matter whether it's an rvalue or lvalue. The important thing is 
to propagate that distinction (automatically by the compiler), to 
disallow binding a returned rvalue reference to an escaping ref 
parameter (i.e., no `escape(forward(S(1)))`).
Where the distinction between lvalues and rvalues matters for a 
function returning a ref parameter, as for any other function 
with ref params, is whether the parameter escapes (i.e., by 
storing a pointer somewhere). If it does, it should be `return 
ref`, otherwise `return scope ref` (either explicitly or inferred 
automatically by the compiler).


 Any issues with taking pointers to arguments is firmly in 
  system territory and is thus completely orthogonal. Even if 
 scope were to be implemented in a way that prevented it, it 
 would still have nothing to do with whether the function 
 accepted both rvalues and lvalues with the same parameter.




Of course it would if one made actually use of a scope system 
(again, either explicit or as a result of compiler escape 
analysis) to (a) rule out all escaping rvalue bugs and (b) 
possibly warn about or highlight escaping lvalue params in 
 system code. The main aim of the scope system would be to 
enhance the flexibility of  safe code though.


 In the meantime, what about making all `ref` params accept 
 rvalues


 Please, please, no. ref indicates that your intention is to 
 mutate the argument. Having it accept rvalues completely 
 destroys that and makes it that much harder to understand what 
 a function is supposed to do.




If that was the only reason for not accepting rvalues as `ref` 
params, then rvalues should have been allowed for `const ref` 
right from the beginning, as there's surely absolutely no 
intention to mutate a `const ref` argument.


 I can accept that argument, although I see cases where it's 
 still justified. For example, if the author of the function 
 decided it wants to mutate the argument, but you (the caller) 
 are not interested in the new value. But still, explicitly 
 using a temporary named `dummy` is preferable because it even 
 documents that you're going to ignore it.


I find letting the callee decide whether the caller needs to 
provide an lvalue (`ref`) or not (non-templated `auto ref`), 
based solely on making sure the mutations are directly visible by 
the caller after the call, too much. I think it should be left to 
the caller to decide what mutations it's interested in.

But now I see why some would oppose calling that `scope ref`, as 
that really has nothing to do with scope or safety, that'd just 
be an arbitrary way of classifying ref params in 'essential 
output' (mutable lvalue `ref`s, mutations need to be visible by 
caller) and some other category (lvalue/rvalue `auto ref`s, not 
just for efficiency purposes). That really is an orthogonal and 
incompatible semantic compared to what `scope ref` is all about, 
and doesn't interest me at all. I find flexible  safe code and 
detecting escaping bugs a kazillion times more important than 
making sure a caller sees all mutations the callee deems 
essential after the call.



 Jun 27 2015



prev sibling  parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Saturday, 27 June 2015 at 11:10:49 UTC, Marc Schütz wrote:

 On Saturday, 27 June 2015 at 01:18:19 UTC, Jonathan M Davis 
 wrote:



 No, not with scope. It seems you're using the status quo to 
 argue against what could be. But that's not valid, because the 
 problems you mention will no longer exist with scope.


If we want to define what scope does and implement it, that's a 
separate issue from having a function accept both rvalues and 
lvalues with the same parameter. It would be trivial to have a 
function where you want to guarantee that no pointer to a ref 
parameter escapes but you still don't want to accept rvalues, 
because the whole point is to mutate the argument. Wanting to 
have a function accept lvalues and rvalues is thus fundamentally 
different from caring about issues with escaping. Yes, there is 
some overlap in terms of safety issues when it comes to returning 
by ref, but that's it, and the return attribute solves that 
problem, but whether you escape pointers to a parameter is 
fundamentally separate from whether a function should accept 
rvalues, because escaping is not the only issue, and ref itself 
has all of the same issues with escaping whether it accepts 
rvalues or not.

So, it really doesn't matter what scope might protect against in 
terms of escaping and what it might not if we were to fully 
define it (e.g. maybe it would protect against taking the address 
of a variable, and maybe not; that could be simply considered 
 system, or it could be protected against). We have to actually 
define it to know what it would or wouldn't do. All we have is 
the basic idea of what it's supposed to have been for. The _only_ 
thing that it applies to at this point is delegates. There's 
little point in arguing about what it could do until we're 
talking about actually defining it. But regardless, the issue of 
accepting rvalues by ref is orthogonal, because that is not 
solely an issue of escaping. It's also an issue of _why_ the 
argument is passed by ref in the first place.


 We introduced the return attribute (currently only with 
 -dip25) to fix this problem. So, with that, we eliminate the 
 safety problem with ref itself, and we can safely have auto 
 ref accept rvalues by having it assign them to a temporary 
 variable first.


 Yes this is now possible, but why in all world do you want the 
 keyword that enables it to be `auto ref`?


That's what auto ref was originally introduced to do. The reason 
that it works only with templates is because Walter misunderstood 
what Andrei was proposing. So, it makes sense to use it for what 
it was originally intended for and implement it as proposed with 
non-templated functions. Now, that has the downside that we can't 
use it with templated functions in a fashion that avoids template 
bloat, in which case, using a new attribute would make sense, but 
this _is_ what auto ref was originally for. And if we're going to 
use a new attribute, scope ref makes no sense precisely because 
the escaping issue is only part of the problem with accepting 
lvalues and rvalues with the same parameter, since you could 
easily have a function which is not supposed to escape the 
argument in any way, and yet it needs to be mutating the argument 
via ref for the caller, and it should not be accepting rvalues. 
So, in that case, if scope were fully ironed out, it might make 
sense to use scope ref when the intention was that the argument 
be mutated and not accept rvalues and that in not escape the 
function. It could also be that scope ref prevents returning by 
ref, since that's escaping the function, whereas when you pass an 
rvalue to the non-templated auto ref, you may very well want to 
return it by ref. So, if we were to use scope ref for this, we'd 
be mixing two largely orthogonal issues - escaping arguments and 
having a function accept both lvalues and rvalues - and it would 
cause us problems if and when we actually, fully defined scope.

We need to either use auto ref for this as originally intended or 
use a new attribute which does so so that we can use it with both 
templated and non-templated functions. I think that it's quite 
clear that scope ref does _not_ make sense in this case, and it's 
already been rejected by Walter and Andrei anyway.

- Jonathan M Davis



 Jun 27 2015



 next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 28 June 2015 at 08:50, Jonathan M Davis via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 [..]



 That's what auto ref was originally introduced to do. The reason that it
 works only with templates is because Walter misunderstood what Andrei was
 proposing. So, it makes sense to use it for what it was originally intended
 for and implement it as proposed with non-templated functions. Now, that has
 the downside that we can't use it with templated functions in a fashion that
 avoids template bloat,


Stop repeating that. It's not got anything to do with code bloat, you
can't use it because template auto ref doesn't pass rvalues by ref, it
passes them by value, which means it's not a ref in any way, and it's
not what the user wants.
I can easily get the same behaviour by not-typing 'auto ref', which I
happily do in all cases that I want something to be passed by value.
But we're not talking about by-val, we're talking about ref.

It's simple; I want to pass something by value, or I want it to pass
by ref... there is no middle ground, it's an explicit binary statement
about how the parameter shall be passed. I don't care about lvalues or
rvalues. There is no case where I want the compiler to not do the
thing I told it to. And if there were a hypothetical case - where
ref-ness were dependent on some conditions, I'm confident I could
easily express that with existing mechanisms in D. I don't need a
weird compiler defined semantic to try and do something smart that
turns out never to be what I want.



 in which case, using a new attribute would make
 sense, but this _is_ what auto ref was originally for. And if we're going to
 use a new attribute, scope ref makes no sense precisely because the escaping
 issue is only part of the problem with accepting lvalues and rvalues with
 the same parameter, since you could easily have a function which is not
 supposed to escape the argument in any way, and yet it needs to be mutating
 the argument via ref for the caller, and it should not be accepting rvalues.
 So, in that case, if scope were fully ironed out, it might make sense to use
 scope ref when the intention was that the argument be mutated and not accept
 rvalues and that in not escape the function. It could also be that scope ref
 prevents returning by ref, since that's escaping the function, whereas when
 you pass an rvalue to the non-templated auto ref, you may very well want to
 return it by ref. So, if we were to use scope ref for this, we'd be mixing
 two largely orthogonal issues - escaping arguments and having a function
 accept both lvalues and rvalues - and it would cause us problems if and when
 we actually, fully defined scope.

 We need to either use auto ref for this as originally intended or use a new
 attribute which does so so that we can use it with both templated and
 non-templated functions. I think that it's quite clear that scope ref does
 _not_ make sense in this case, and it's already been rejected by Walter and
 Andrei anyway.


scope ref is only related to the safety of passing temporaries by
reference to functions that may retain a pointer, it has nothing to do
with lvalues or rvalues; stack locals are equally unsafe right now as
rvalues would be.

const is the thing you're looking for that says "this isn't to be
changed", such that it makes sense to pass an rvalue.

scope was proposed as a means to address the "rvalues -> ref is
unsafe, the callee might sequester a pointer somehow" argument. If
we're not interested in scope (ie, escape analysis), then we're not
interested in ref safety in general (proposed scope applies to a much
wider set of cases than just rvalues).
That's fine, I don't really care, but then you can no longer make the
argument above that safety affects this debate, at which point you
arrive at C++ where rvalues can pass to const ref, as should be the
case here. End this auto-ref nonsense, const ref is enough.



 Jun 27 2015



  parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Sunday, 28 June 2015 at 05:46:11 UTC, Manu wrote:

 On 28 June 2015 at 08:50, Jonathan M Davis via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:

 [..]



 That's what auto ref was originally introduced to do. The 
 reason that it works only with templates is because Walter 
 misunderstood what Andrei was proposing. So, it makes sense to 
 use it for what it was originally intended for and implement 
 it as proposed with non-templated functions. Now, that has the 
 downside that we can't use it with templated functions in a 
 fashion that avoids template bloat,


 Stop repeating that. It's not got anything to do with code 
 bloat, you
 can't use it because template auto ref doesn't pass rvalues by 
 ref, it
 passes them by value, which means it's not a ref in any way, 
 and it's
 not what the user wants.
 I can easily get the same behaviour by not-typing 'auto ref', 
 which I
 happily do in all cases that I want something to be passed by 
 value.
 But we're not talking about by-val, we're talking about ref.

 It's simple; I want to pass something by value, or I want it to 
 pass by ref... there is no middle ground, it's an explicit 
 binary statement about how the parameter shall be passed. I 
 don't care about lvalues or rvalues. There is no case where I 
 want the compiler to not do the thing I told it to. And if 
 there were a hypothetical case - where ref-ness were dependent 
 on some conditions, I'm confident I could easily express that 
 with existing mechanisms in D. I don't need a weird compiler 
 defined semantic to try and do something smart that turns out 
 never to be what I want.


It makes no sense to pass rvalues by ref. The ref has to refer to 
a memory location so that a pointer can be passed underneath the 
hood, and rvalues don't qualify for that. A temporary variable 
_must_ be created in order to pass an rvalue by ref, even if the 
compiler is able to move the rvalue into the temporary variable 
rather than copy it. So, I don't understand why you would be 
looking to pass rvalues by ref. I can see why you would want to 
pass both rvalues and lvalues to the same function efficiently 
without caring about whether they're rvalues or lvalues, but that 
doesn't necessarily mean ref, and in the case of rvalues, it 
definitely doesn't mean ref.

The most efficient way to pass rvalues is by value, because the 
value can be moved or even simply constructed directly into the 
right place on the stack to begin with. That's why C++11 added 
move constructors and why D has postlblit constructors. That's 
also why it's no longer the advice in C++-land for folks to 
simply always use const& as much as possible (not that it 
shouldn't be used, but whether it should be used or not becomes 
much more complicated). D has moving built-in without the need 
for move constructors, so our situation is simpler, but 
regardless, ref really isn't the answer when it comes to rvalues. 
You _want_ to be passing rvalues by value. It's the lvalues that 
you don't want to pass by value if you're worried about 
efficiency, because they're the ones that are going to have to be 
copied. But still, if what you want is to have rvalues put into a 
place on the stack where they can be passed by ref and then have 
them passed by ref, that's what you'll get with auto ref for 
non-templated functions. It'll pass lvalues by ref, and it'll 
create temporary variables for the rvalues so that they can be 
passed by ref. So, as far as I can tell, auto ref for 
non-templated functions does exactly what you're looking for. 
It's just that it only works for non-templated functions, and 
with the templated ones, rvalues are passed efficiently - but not 
by ref.

- Jonathan M Davis



 Jun 28 2015



 next sibling parent  "Temtaime" <temtaime gmail.com>  writes:


All values must be passed by ref because otherwise it will be 
leading for code bloat : we want eliminate by having only one 
function that accepts both rvalues and lvalues.



 Jun 28 2015



prev sibling  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Sunday, 28 June 2015 at 07:36:41 UTC, Jonathan M Davis wrote:

 It makes no sense to pass rvalues by ref. The ref has to refer 
 to a memory location so that a pointer can be passed underneath 
 the hood, and rvalues don't qualify for that.


I strongly disagree with that. This is an implementation detail. 
`ref` describes a specific semantic, namely that no copying 
(observable by a postblit being called) takes place, and that it 
refers to the same "thing" as the original (observable by 
mutations being visible on the original object, if relevant). How 
a value is stored is completely up to the compiler. It can be in 
a register, a memory location, or even - if the compiler knows 
the value in advance - as an immediate of a machine instruction. 
The same is true for passing values: the compiler is free to 
implement it as it wants, as long as it keeps the required 
semantics.



 Jun 28 2015



  parent reply "rsw0x" <anonymous anonymous.com>  writes:


On Sunday, 28 June 2015 at 11:06:01 UTC, Marc Schütz wrote:

 On Sunday, 28 June 2015 at 07:36:41 UTC, Jonathan M Davis wrote:

 It makes no sense to pass rvalues by ref. The ref has to refer 
 to a memory location so that a pointer can be passed 
 underneath the hood, and rvalues don't qualify for that.


 I strongly disagree with that. This is an implementation 
 detail. `ref` describes a specific semantic, namely that no 
 copying (observable by a postblit being called) takes place,


no, it means pass by reference.
this sounds like some convoluted C++ doublespeak definition.



 Jun 28 2015



  parent  "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Sunday, 28 June 2015 at 13:08:48 UTC, rsw0x wrote:

 On Sunday, 28 June 2015 at 11:06:01 UTC, Marc Schütz wrote:

 On Sunday, 28 June 2015 at 07:36:41 UTC, Jonathan M Davis 
 wrote:

 It makes no sense to pass rvalues by ref. The ref has to 
 refer to a memory location so that a pointer can be passed 
 underneath the hood, and rvalues don't qualify for that.


 I strongly disagree with that. This is an implementation 
 detail. `ref` describes a specific semantic, namely that no 
 copying (observable by a postblit being called) takes place,


 no, it means pass by reference.
 this sounds like some convoluted C++ doublespeak definition.


Well, try experimenting with LDC with -O3 -output-s on some test 
programs. You will be surprised ;-) There will often be no 
"reference" in the resulting output, especially with inlining.

The definition I used is not necessarily the best one, but it 
needs to be based on observable behaviour, not implementation 
specifics, otherwise compilers would have no freedom in 
optimizing.



 Jun 28 2015



prev sibling  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


Thank you Jonathan, I think I (like kinke in his post above) 
finally understand your stance. So I guess from your POV the 
following would be an ideal situation (disregarding the need to 
avoid breaking changes):

1) `auto ref` for non-templates is required to make a function 
accept rvalue and lvalue refs alike.
2) `auto ref` for templates is changed to mean the same thing as 
1).
3) `scope ref` prevents escaping of the reference.

Neither of `auto ref` and `scope ref` would imply the other, 
because you want to treat them as orthogonal. Do I understand you 
right?

The main problems with this is that they aren't in fact 
orthogonal: `auto ref` without `scope ref` almost only makes 
sense if you want to deliberately break something. Furthermore, 
`auto ref` is already taken for something else, and I'm not sure 
Walter would be too happy if we wanted to change its meaning for 
templates.



 Jun 28 2015



  parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Sunday, 28 June 2015 at 10:50:10 UTC, Marc Schütz wrote:

 Thank you Jonathan, I think I (like kinke in his post above) 
 finally understand your stance. So I guess from your POV the 
 following would be an ideal situation (disregarding the need to 
 avoid breaking changes):

 1) `auto ref` for non-templates is required to make a function 
 accept rvalue and lvalue refs alike.
 2) `auto ref` for templates is changed to mean the same thing 
 as 1).
 3) `scope ref` prevents escaping of the reference.

 Neither of `auto ref` and `scope ref` would imply the other, 
 because you want to treat them as orthogonal. Do I understand 
 you right?

 The main problems with this is that they aren't in fact 
 orthogonal: `auto ref` without `scope ref` almost only makes 
 sense if you want to deliberately break something. Furthermore, 
 `auto ref` is already taken for something else, and I'm not 
 sure Walter would be too happy if we wanted to change its 
 meaning for templates.


We definitely don't want to lose the current auto ref that we 
have with templates. It's critical for forwarding refness, even 
if we got that ability by accident. IIRC, we don't quite manage 
perfect forwarding with what we have (though off the top of my 
head, I don't recall what we're missing), but without auto ref, 
we definitely don't get there. So, auto ref, as it stands has 
proven to be surprisingly useful completely aside from efficiency 
concerns. Also, the fact that it takes rvalues by value can 
actually be more efficient than taking them by ref, depending on 
the code, since it can just move the value in that case, whereas 
it's tied to a temporary variable in the case of what would be 
the non-templated auto ref, and that could require copying the 
value at some point when having it be passed by value could have 
avoided the copy - though obviously, all of that template bloat 
comes at a cost, so it's definitely a tradeoff. But we don't want 
to lose the current auto ref regardless.

So, if we want to have an attribute which accepts both lvalues 
and rvalues for non-templated functions and be able to use that 
same attribute for templated functions, a new attribute is 
required (even if originally, auto ref was intended to work for 
both templated and non-templated functions). Otherwise, if we're 
fine with the template bloat, we can just use the proposed auto 
ref with non-templated functions and not worry about adding a new 
attribute. So, ideally, at this point, we'd probably add a new 
attribute, simply because it's the most flexible, but that does 
come at the cost of adding a new attribute, so it may not be 
worth it.

Regardless, all of that is orthogonal to the escaping issue, 
because the escaping issue exists for all ref variables (not just 
those which refer to temporary variables that rvalues were 
assigned to). So, whether a ref parameter accepts both lvalues 
and rvalues is separate from whether they escape. And if scope 
really prevented the ref from escaping, then you couldn't even 
return it, and we need to be able to return by ref, so scope ref 
really doesn't make sense in the context of preventing ref 
parameters from existing longer than their arguments. It's 
related, but it would be too restrictive if it truly prevented 
all escaping.

And as far as escaping references to ref parameters by returning 
them by ref goes, the recently added return attribute already 
takes care of that. So, arguably, it overlaps with what scope 
conceivably could or should do, but it does what's required for 
solving the escape problem for ref, and a more general scope 
solution is not required for that. It also allows us to 
distinguish between ensuring that a ref parameter does not exist 
longer than is safe (i.e. that it's only returned by ref if it's 
safe to do so) and preventing a ref parameter from escaping the 
function at all. So, if we were to ever add scope ref, then 
presumably that would have to do with other types of escaping 
than simply returning by ref. And whether you wanted to prevent a 
ref parameter from escaping - either by being returned by ref or 
by other means - isn't necessarily related to whether it refers 
to a temporary variable that was an rvalue, much as it's clearly 
more dangerous to escape anything which refers to a ref 
parameters if it refers to a temporary variable that won't last 
beyond the statement that the function was called in.

- Jonathan M Davis



 Jun 29 2015



 next sibling parent  "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Monday, 29 June 2015 at 17:19:48 UTC, Jonathan M Davis wrote:

 On Sunday, 28 June 2015 at 10:50:10 UTC, Marc Schütz wrote:

 The main problems with this is that they aren't in fact 
 orthogonal: `auto ref` without `scope ref` almost only makes 
 sense if you want to deliberately break something. 
 Furthermore, `auto ref` is already taken for something else, 
 and I'm not sure Walter would be too happy if we wanted to 
 change its meaning for templates.


 We definitely don't want to lose the current auto ref that we 
 have with templates. It's critical for forwarding refness, even 
 if we got that ability by accident.
 - snip -


For the record, to avoid further misunderstandings: When I talked 
about "auto ref" I meant the proposed rvalue refs, not the 
existing syntax for templates. I don't have such a strong opinion 
about the latter, except that it is - and should stay - distinct 
from rvalue refs.


 Regardless, all of that is orthogonal to the escaping issue, 
 because the escaping issue exists for all ref variables (not 
 just those which refer to temporary variables that rvalues were 
 assigned to). So, whether a ref parameter accepts both lvalues 
 and rvalues is separate from whether they escape. And if scope 
 really prevented the ref from escaping, then you couldn't even 
 return it, and we need to be able to return by ref, so scope 
 ref really doesn't make sense in the context of preventing ref 
 parameters from existing longer than their arguments. It's 
 related, but it would be too restrictive if it truly prevented 
 all escaping.


It's not as strict as that. Returning an rvalue ref is not per se 
unsafe, because the temporary that it refers to lives until the 
end of the current statement, which can be longer than the 
current call. It can be safely passed to another function as an 
rvalue ref, e.g. in a UFCS chain. This is what the `return` 
attribute enables: it's a means of telling the callee to losen 
the restrictions on a scoped reference a bit, because the caller 
is aware of it and can handle it in a safe way.


 And as far as escaping references to ref parameters by 
 returning them by ref goes, the recently added return attribute 
 already takes care of that. So, arguably, it overlaps with what 
 scope conceivably could or should do, but it does what's 
 required for solving the escape problem for ref, and a more 
 general scope solution is not required for that. It also allows 
 us to distinguish between ensuring that a ref parameter does 
 not exist longer than is safe (i.e. that it's only returned by 
 ref if it's safe to do so) and preventing a ref parameter from 
 escaping the function at all. So, if we were to ever add scope 
 ref, then presumably that would have to do with other types of 
 escaping than simply returning by ref. And whether you wanted 
 to prevent a ref parameter from escaping - either by being 
 returned by ref or by other means - isn't necessarily related 
 to whether it refers to a temporary variable that was an 
 rvalue, much as it's clearly more dangerous to escape anything 
 which refers to a ref parameters if it refers to a temporary 
 variable that won't last beyond the statement that the function 
 was called in.


I already addressed this point in the last section of 
http://forum.dlang.org/post/udnrkqmfqeunsfgwscnk forum.dlang.org



 Jun 29 2015



prev sibling  parent reply "Atila Neves" <atila.neves gmail.com>  writes:


On Monday, 29 June 2015 at 17:19:48 UTC, Jonathan M Davis wrote:

 On Sunday, 28 June 2015 at 10:50:10 UTC, Marc Schütz wrote:

 Thank you Jonathan, I think I (like kinke in his post above) 
 finally understand your stance. So I guess from your POV the 
 following would be an ideal situation (disregarding the need 
 to avoid breaking changes):

 1) `auto ref` for non-templates is required to make a function 
 accept rvalue and lvalue refs alike.
 2) `auto ref` for templates is changed to mean the same thing 
 as 1).
 3) `scope ref` prevents escaping of the reference.

 Neither of `auto ref` and `scope ref` would imply the other, 
 because you want to treat them as orthogonal. Do I understand 
 you right?

 The main problems with this is that they aren't in fact 
 orthogonal: `auto ref` without `scope ref` almost only makes 
 sense if you want to deliberately break something. 
 Furthermore, `auto ref` is already taken for something else, 
 and I'm not sure Walter would be too happy if we wanted to 
 change its meaning for templates.


 We definitely don't want to lose the current auto ref that we 
 have with templates. It's critical for forwarding refness, even 
 if we got that ability by accident. IIRC, we don't quite manage 
 perfect forwarding with what we have (though off the top of my 
 head, I don't recall what we're missing), but without auto ref, 
 we definitely don't get there. So, auto ref, as it stands has 
 proven to be surprisingly useful completely aside from 
 efficiency concerns.


I thought perfect forwarding had everything to do with efficiency 
concerns. At least, that's the motivation for in in C++ 
AFAICT/AFAIR. If not, (since C++ const& can bind to rvalues), 
passing by value would be perfectly reasonable.

It seems to me that there's a lot of confusion in this thread. I 
think I understand what you're saying but I'm not sure everyone 
else does. So correct me if I'm wrong: your stance (which is how 
I understand how things should be done in D) is that if a person 
wants to bind an rvalue to a ref parameter for performance, they 
shouldn't; they should pass it by value since it'll either get 
constructed in place or moved, not copied.

Essentially, what this guy said about C++ (not the original I 
read, that's now a 404):

http://m.blog.csdn.net/blog/CPP_CHEN/17528669

The reason rvalue references even exist in C++11/14 is because 
rvalues can bind to const&. I remember hearing/reading Andrei say 
that D doesn't need them precisely because in D, they can't. Pass 
by value and you get the C++ behaviour when rvalue references are 
used: a move.

Atila



 Jun 29 2015



 next sibling parent  "kink" <noone nowhere.com>  writes:


On Monday, 29 June 2015 at 19:10:07 UTC, Atila Neves wrote:

 It seems to me that there's a lot of confusion in this thread. 
 I think I understand what you're saying but I'm not sure 
 everyone else does. So correct me if I'm wrong: your stance 
 (which is how I understand how things should be done in D) is 
 that if a person wants to bind an rvalue to a ref parameter for 
 performance, they shouldn't; they should pass it by value since 
 it'll either get constructed in place or moved, not copied.

 Essentially, what this guy said about C++ (not the original I 
 read, that's now a 404):

 http://m.blog.csdn.net/blog/CPP_CHEN/17528669

 The reason rvalue references even exist in C++11/14 is because 
 rvalues can bind to const&. I remember hearing/reading Andrei 
 say that D doesn't need them precisely because in D, they 
 can't. Pass by value and you get the C++ behaviour when rvalue 
 references are used: a move.


In some cases even better, in-place construction. That's perfect 
- for rvalues, as you said. But for lvalues, a copy must be made 
- afaik, you don't have the option to explicitly move an lvalue 
into a byval-param in D as you can in C++. And even moving can be 
expensive if the type is big.
So for efficiency, template auto-ref is a nice option, but 
requires a template and leads to code-bloat if both lvalues and 
rvalues are used as arguments. For these reasons, we want to have 
the additional option to pass both lvalues and rvalues by ref.

Jonathan wants a non-templated function to specify explicitly 
whether it wants to accept an rvalue by ref, apparently to 
discriminate between 'output parameters' (mutable `ref`, lvalues 
only) and another category of `auto ref` parameters which also 
accept rvalues via lowering (I don't know how one would describe 
that category - to me it seems completely arbitrary and adds 
complexity, not to mention different interpretations by different 
developers).

I'm solely interested in the safety of binding rvalues to refs, 
meaning that the compiler should prevent these rvalue refs from 
being escaped (I'm not talking about returning the ref via 
`return ref`, which is safe anyway when propagating the 
rvalue/lvalue origin, but storing derived pointers in external 
state). That's where `scope ref` (unescapable reference) comes 
in. But I prefer the compiler to infer scope-ness automatically 
in the future and would thus like to allow binding rvalues to 
`ref` in general - for not-so-performance-critical functions 
taking a value type by reference (for some value types, copying 
just doesn't make sense - streams, loggers etc.).



 Jun 30 2015



prev sibling  parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Monday, 29 June 2015 at 19:10:07 UTC, Atila Neves wrote:

 On Monday, 29 June 2015 at 17:19:48 UTC, Jonathan M Davis wrote:

 On Sunday, 28 June 2015 at 10:50:10 UTC, Marc Schütz wrote:

 Thank you Jonathan, I think I (like kinke in his post above) 
 finally understand your stance. So I guess from your POV the 
 following would be an ideal situation (disregarding the need 
 to avoid breaking changes):

 1) `auto ref` for non-templates is required to make a 
 function accept rvalue and lvalue refs alike.
 2) `auto ref` for templates is changed to mean the same thing 
 as 1).
 3) `scope ref` prevents escaping of the reference.

 Neither of `auto ref` and `scope ref` would imply the other, 
 because you want to treat them as orthogonal. Do I understand 
 you right?

 The main problems with this is that they aren't in fact 
 orthogonal: `auto ref` without `scope ref` almost only makes 
 sense if you want to deliberately break something. 
 Furthermore, `auto ref` is already taken for something else, 
 and I'm not sure Walter would be too happy if we wanted to 
 change its meaning for templates.


 We definitely don't want to lose the current auto ref that we 
 have with templates. It's critical for forwarding refness, 
 even if we got that ability by accident. IIRC, we don't quite 
 manage perfect forwarding with what we have (though off the 
 top of my head, I don't recall what we're missing), but 
 without auto ref, we definitely don't get there. So, auto ref, 
 as it stands has proven to be surprisingly useful completely 
 aside from efficiency concerns.


 I thought perfect forwarding had everything to do with 
 efficiency concerns. At least, that's the motivation for in in 
 C++ AFAICT/AFAIR. If not, (since C++ const& can bind to 
 rvalues), passing by value would be perfectly reasonable.


I don't remember. It's been a while since I looked at it, so I'd 
have to look it up again to know quite what it is or exactly why 
it's needed. I do remember that without auto ref though, we're 
definitely not there.


 It seems to me that there's a lot of confusion in this thread. 
 I think I understand what you're saying but I'm not sure 
 everyone else does. So correct me if I'm wrong: your stance 
 (which is how I understand how things should be done in D) is 
 that if a person wants to bind an rvalue to a ref parameter for 
 performance, they shouldn't; they should pass it by value since 
 it'll either get constructed in place or moved, not copied.

 Essentially, what this guy said about C++ (not the original I 
 read, that's now a 404):

 http://m.blog.csdn.net/blog/CPP_CHEN/17528669

 The reason rvalue references even exist in C++11/14 is because 
 rvalues can bind to const&. I remember hearing/reading Andrei 
 say that D doesn't need them precisely because in D, they 
 can't. Pass by value and you get the C++ behaviour when rvalue 
 references are used: a move.


Yeah. In general, it's more efficient to pass rvalues by value, 
because the compiler can avoid copying them and simply move them 
(and at least some of the time, it doesn't even need to move 
them, though I don't know enough about how stuff is implemented 
at that level to know whether it can always avoid even having to 
move rvalue arguments; it can certainly avoid copying them 
though). If you pass in an rvalue by reference (via const& in C++ 
or by the proposed non-templated auto ref in D), then you 
actually increase the odds that the argument will have to be 
copied. And if no copy is needed anywhere, then I believe that 
passing an rvalue by ref and passing it by value end up being 
more or less equivalent. So, the end result is that it's not 
actually beneficial to pass an rvalue by reference rather than by 
value.

The problem that you run into is that if you want a parameter to 
accept both lvalues and rvalues, then the more efficient thing 
for rvalues is to pass by value, whereas the more efficient thing 
to do for lvalues is to pass by reference, and const& in C++ 
forces you to pass by ref, whereas passing by value... forces 
passing by value. The same would go with the non-templated auto 
ref. However, the templated auto ref avoids that by inferring the 
refness of the argument and thus automatically accepts rvalues by 
value and lvalues by reference. So, you end up passing arguments 
efficiently regardless of whether they're lvalues or rvalues; 
it's just that it costs you a lot of template bloat in the 
process.

But a lot of programmers are used to simply using const& 
everywhere in C++, because prior to C++11, rvalues couldn't 
generally be moved, because there were no move constructors, and 
C++ doesn't disallow stuff like having a class/struct on the 
stack having a pointer to one of its members (which D does 
disallow - or at least assumes you won't do), and so the best way 
to avoid extra copies was to ensure that you passed by const& as 
much as possible. But with C++11, the situation becomes much more 
complicated, and figuring out what the most efficient thing to do 
is not as straightforward. And AFAIK, C++ has nothing like our 
templated auto ref that allows you to efficiently accept both 
lvalues and rvalues (but maybe that's where perfect forwarding 
comes in; I don't know; I really need to read up on it), leaving 
you to figure out whether having a function take its arguments by 
value or const& is more efficient based on what your function 
does.

Ultimately, it really doesn't seem like there's a simple rule of 
them that is going to give you efficiency in general. You need to 
understand the various pros and cons of how arguments are passed 
if you want to eke out extra efficiency.

Though in general, if you simply avoid having particularly large 
structs on the stack, I would think that just passing arguments 
by value as the default would be the best way to go and worrying 
about how to pass more efficiently when profiling shows that it's 
necessary. But having auto ref for non-templated functions will 
give us another tool in the toolbox for adjusting how function 
arguments are passed in code that cares.

Of course, if you have large structs for whatever reason, that 
makes the situation more complicated, but the best way to handle 
that will likely depend on your code.

The only real question I see is whether it's worth using a new 
attribute (as opposed to auto ref) so that we can use it with 
templated functions as well, allowing folks to pass rvalues by 
reference with templated functions if they need to avoid the 
template bloat or it actually turns out that that's more 
efficient in that case. As I understand it though, it's pretty 
much always more efficient to pass rvalues by value. And if 
that's indeed the case, the only real gain that I see by using a 
new attribute is that it allows us to avoid template bloat when 
it needs to be avoided.

- Jonathan M Davis



 Jun 30 2015



  parent  "Daniel N" <ufo orbiting.us>  writes:


This is semi-off-topic, but I was playing around... and line 12 
broke my personal record for the most insane line ever written, 
maybe someone else finds it amusing also:

[asm.dlang.org]
http://goo.gl/ChdbwD



 Jun 30 2015



prev sibling  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Wednesday, 24 June 2015 at 23:30:53 UTC, Jonathan M Davis 
wrote:

 On Wednesday, 24 June 2015 at 16:39:22 UTC, kink wrote:

 On Wednesday, 24 June 2015 at 11:19:04 UTC, Jonathan M Davis 
 wrote:

 [...]

 3. Add a new attribute which does what's being proposed for 
 auto ref for non-templated functions, in which case, we can 
 use the non-templated behavior with templates as well and 
 thus avoid template bloat when all you want is for your 
 templated function to accept both lvalues and rvalues. auto 
 ref, of course, then stays exactly as it is now.


 that's probably fine, but I do wonder if we'd be better off 

 to be templated and how much code bloat auto ref is likely to 
 generate with templated functions.

 - Jonathan M Davis



 `scope ref` and simplify the PR a lil' bit as suggested by 
 Marc in an earlier post 
 [http://forum.dlang.org/post/ricvtchihgzyisbkzcgl forum.dlang.org].


 But this has _nothing_ to do with scope, and scope ref was 
 already rejected. The whole point of this is support having a 
 function accept both rvalues and lvalues, not to do anything 
 with scope.


It everything to do with scope, and I already explained it 
several times. As for the rejection, see my answer to Namespace:
http://forum.dlang.org/post/yxruhtwjllozgvpbhgik forum.dlang.org


 And given that what scope does has never even been properly 
 defined - all that the spec says about scope parameters is 
 "references in the parameter cannot be escaped (e.g. assigned 
 to a global variable)"


Even this current vague definition is already enough of a 
guarantee to allow rvalue references with it. Of course I'd be 
the last to object against a more detailed specification, 
obviously.


 - and that the only place that scope does anything at all is 
 with delegates,


That's simply the status quo of the implementation and therefore 
can't be used as an argument.


 trying to expand it with "scope ref" as if that were simply an 
 extension of scope makes no sense. Before we can even consider 
 what something like scope ref might mean, we'd have to properly 
 define what scope means. And all we have for it is the basic 
 idea of what it's supposed to do - none of the details - and 
 trying to define scope ref before defining what scope means in 
 general could totally hamstring us when properly defining scope 
 later.


I can assure you that it will not limit us. The very concept of 
borrowing/scope already requires some very specific restrictions 
with regards to what a function is allowed to do with a scope 
parameter. These same restrictions guarantee that it's allowed to 
pass an rvalue to it. Every working scope proposal will always 
guarantee that, or it wouldn't be usable.

If you still fear that it will impede us later, then at least 
this current proposal needs to be deferred until we have a scope 
proposal and have decided on it.



 Jun 25 2015



  parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 06/25/2015 10:28 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
<schuetzm gmx.net>" wrote:

 trying to expand it with "scope ref" as if that were simply an
 extension of scope makes no sense. Before we can even consider what
 something like scope ref might mean, we'd have to properly define what
 scope means. And all we have for it is the basic idea of what it's
 supposed to do - none of the details - and trying to define scope ref
 before defining what scope means in general could totally hamstring us
 when properly defining scope later.


 I can assure you that it will not limit us. The very concept of
 borrowing/scope already requires some very specific restrictions with
 regards to what a function is allowed to do with a scope parameter.
 These same restrictions guarantee that it's allowed to pass an rvalue to
 it. Every working scope proposal will always guarantee that, or it
 wouldn't be usable.

 If you still fear that it will impede us later, then at least this
 current proposal needs to be deferred until we have a scope proposal and
 have decided on it.


I think the arguments against allowing rvalues with scope ref are the 
same as those against allowing rvalues with ref, modulo accidental 
escaping. (It has been argued that 'ref' signifies an intent to modify, 
which I wouldn't necessarily agree with.)



 Jun 25 2015



prev sibling next sibling parent  "Jesse Phillips" <Jesse.K.Phillips+D gmail.com>  writes:


On Monday, 22 June 2015 at 04:11:41 UTC, Andrei Alexandrescu 
wrote:

 Walter and I discussed what auto ref for templates should look 
 like and reached the conclusion that an approach based on 
 lowering would be best. I added a proposed lowering to 
 https://github.com/D-Programming-Language/dmd/pull/4717.

 Andrei


It seems to me that this may be an important target to seek:

auto foo(auto ref T bar) {}

Should be callable in the same manner

auto foo()(auto ref T bar) {}

is callable. That is to say it would be unfortunate if turning it 
into a template would cause different behaviors.

If nothing else, deviations should be well documented and 
explained.



 Jun 22 2015



prev sibling  parent  "ixid" <nuaccount gmail.com>  writes:


On Monday, 22 June 2015 at 04:11:41 UTC, Andrei Alexandrescu 
wrote:

 Walter and I discussed what auto ref for templates should look 
 like and reached the conclusion that an approach based on 
 lowering would be best. I added a proposed lowering to 
 https://github.com/D-Programming-Language/dmd/pull/4717.

 Andrei


A question from a place of ignorance- from the thread people say 
normal function and template function auto ref have different 
rules. This seems like a really bad idea. Is this necessary and 
why should this be acceptable? We should surely have normal 
functions be a simple subset of templates.



 Jul 02 2015