digitalmars.dip.ideas - ` rvalue ref` and ` universal ref`
- Quirin Schroll (68/68) Jul 03 Add two function parameter attributes `@rvalue` and `@universal`
- Meta (5/74) Jul 08 You briefly mentioned `auto ref`, but didn't say how it interacts
- Meta (6/75) Jul 15 You mentioned `auto ref` briefly, but did not describe how it
- Meta (4/4) Jul 15 It looks like a mailing list bug is setting the subject line to
- Quirin Schroll (6/9) Jul 16 The `@rvalue` could be used without `ref` to enforce an argument
- IchorDev (16/19) Jul 28 This would be really useful for sure, as long as escaping rvalue
- Quirin Schroll (86/105) Aug 06 An `@rvalue ref` parameter definitely could be `return`. Like
- IchorDev (26/58) Aug 06 And it still does. Just this week I’ve had to dissuade two people
- Quirin Schroll (49/112) Aug 07 The issue is, if we make `in` mean something that doesn’t include
Add two function parameter attributes ` rvalue` and ` universal` which must be used together with `ref` (similar as `auto` is only valid as `auto ref`). * An ` rvalue ref` parameter only binds rvalues, which are caller-side materialized. * A ` universal ref` parameter binds both lvalues and rvalues. * As before: Plain `ref` only binds lvalues. For function returns, there will be no ` universal ref`, but ` rvalue ref`. This is, however, equivalent to `ref` inside the function, meaning it must return an lvalue. The only difference between `ref` and ` rvalue ref` returning functions is caller-side. The result of a ` rvalue ref` returning function is considered an rvalue: A move instead of a copy is issued, it cannot have its address taken, etc. For `extern(C++)` functions, ` rvalue ref T` mangles like C++’s `T&&`; ` universal ref` is not allowed for `extern(C++)` functions. --- **Rationale:** `ref` can be used to make mutations transparent to the caller. There, only lvalue arguments make sense. But `ref` can also be used to alleviate copies. While there is `in` (with `-preview=in`), that also makes the parameter `const`, and some types simply don’t work well with `const`. A ` universal ref` binds arguments with whatever qualifier specified, including none (i.e. mutable). ```d struct BigStruct; int getN(BigStruct big) => big.n; // ❌ copies lvalues // ❌ result is not a reference ``` ```d ref inout(int) getN(ref inout(BigStruct) big) => big.n; // ❌ no rvalue arguments allowed ``` ```d int getN(in BigStruct big) => big.n; // ❌ can’t return by ref: big is scope // ❌ if it could, result would be const ``` ```d int getN(const(BigStruct) big) => big.n; // for rvalues ref inout(int) getN(ref inout(BigStruct) big) => big.n; // for lvalues // ❌ can’t simply take address anymore // ❌ for rvalues: result is not an lvalue ``` ```d ref inout(int) getN( universal ref inout(BigStruct) big) => big.n; // ✔️ no copies // ✔️ returns by ref // ✔️ allows rvalue arguments // ✔️ single function, `auto fp = &getN` works // ✔️ for rvalue argument: return value has lifetime to the end of the statement // ✔️ for mutable argument: result is mutable ``` As far as conversions are concerned, an `R function( universal ref T)` implicitly converts to `R function(ref T)` and `R function( rvalue ref T)`. The reasoning being that a ` universal ref` parameter behaves exactly like a `ref` parameter for lvalue arguments, so the conversion merely “forgets” that rvalue arguments would be expected and allowed. The conversion to ` rvalue ref` similarly forgets that lvalue arguments were allowed. `ref` and ` rvalue ref` returns are conversion-incompatible for implicit conversions. As they are not binary-incompatible, an explicit cast can be used, which is a ` system` operation.
Jul 03
Meta <jared771 gmail.com> On Wednesday, 3 July 2024 at 11:50:19 UTC, Quirin Schroll wrote:Add two function parameter attributes ` rvalue` and ` universal` which must be used together with `ref` (similar as `auto` is only valid as `auto ref`). * An ` rvalue ref` parameter only binds rvalues, which are caller-side materialized. * A ` universal ref` parameter binds both lvalues and rvalues. * As before: Plain `ref` only binds lvalues. For function returns, there will be no ` universal ref`, but ` rvalue ref`. This is, however, equivalent to `ref` inside the function, meaning it must return an lvalue. The only difference between `ref` and ` rvalue ref` returning functions is caller-side. The result of a ` rvalue ref` returning function is considered an rvalue: A move instead of a copy is issued, it cannot have its address taken, etc. For `extern(C++)` functions, ` rvalue ref T` mangles like C++’s `T&&`; ` universal ref` is not allowed for `extern(C++)` functions. --- **Rationale:** `ref` can be used to make mutations transparent to the caller. There, only lvalue arguments make sense. But `ref` can also be used to alleviate copies. While there is `in` (with `-preview=in`), that also makes the parameter `const`, and some types simply don’t work well with `const`. A ` universal ref` binds arguments with whatever qualifier specified, including none (i.e. mutable). ```d struct BigStruct; int getN(BigStruct big) => big.n; // ❌ copies lvalues // ❌ result is not a reference ``` ```d ref inout(int) getN(ref inout(BigStruct) big) => big.n; // ❌ no rvalue arguments allowed ``` ```d int getN(in BigStruct big) => big.n; // ❌ can’t return by ref: big is scope // ❌ if it could, result would be const ``` ```d int getN(const(BigStruct) big) => big.n; // for rvalues ref inout(int) getN(ref inout(BigStruct) big) => big.n; // for lvalues // ❌ can’t simply take address anymore // ❌ for rvalues: result is not an lvalue ``` ```d ref inout(int) getN( universal ref inout(BigStruct) big) => big.n; // ✔️ no copies // ✔️ returns by ref // ✔️ allows rvalue arguments // ✔️ single function, `auto fp = &getN` works // ✔️ for rvalue argument: return value has lifetime to the end of the statement // ✔️ for mutable argument: result is mutable ``` As far as conversions are concerned, an `R function( universal ref T)` implicitly converts to `R function(ref T)` and `R function( rvalue ref T)`. The reasoning being that a ` universal ref` parameter behaves exactly like a `ref` parameter for lvalue arguments, so the conversion merely “forgets” that rvalue arguments would be expected and allowed. The conversion to ` rvalue ref` similarly forgets that lvalue arguments were allowed. `ref` and ` rvalue ref` returns are conversion-incompatible for implicit conversions. As they are not binary-incompatible, an explicit cast can be used, which is a ` system` operation.You briefly mentioned `auto ref`, but didn't say how it interacts with this new feature. I assume it defaults to an lvalue ref, but what if the programmer wants it to default to an rvalue ref, or universal? It seems they don't work well together.
Jul 08
On Wednesday, 3 July 2024 at 11:50:19 UTC, Quirin Schroll wrote:Add two function parameter attributes ` rvalue` and ` universal` which must be used together with `ref` (similar as `auto` is only valid as `auto ref`). * An ` rvalue ref` parameter only binds rvalues, which are caller-side materialized. * A ` universal ref` parameter binds both lvalues and rvalues. * As before: Plain `ref` only binds lvalues. For function returns, there will be no ` universal ref`, but ` rvalue ref`. This is, however, equivalent to `ref` inside the function, meaning it must return an lvalue. The only difference between `ref` and ` rvalue ref` returning functions is caller-side. The result of a ` rvalue ref` returning function is considered an rvalue: A move instead of a copy is issued, it cannot have its address taken, etc. For `extern(C++)` functions, ` rvalue ref T` mangles like C++’s `T&&`; ` universal ref` is not allowed for `extern(C++)` functions. --- **Rationale:** `ref` can be used to make mutations transparent to the caller. There, only lvalue arguments make sense. But `ref` can also be used to alleviate copies. While there is `in` (with `-preview=in`), that also makes the parameter `const`, and some types simply don’t work well with `const`. A ` universal ref` binds arguments with whatever qualifier specified, including none (i.e. mutable). ```d struct BigStruct; int getN(BigStruct big) => big.n; // ❌ copies lvalues // ❌ result is not a reference ``` ```d ref inout(int) getN(ref inout(BigStruct) big) => big.n; // ❌ no rvalue arguments allowed ``` ```d int getN(in BigStruct big) => big.n; // ❌ can’t return by ref: big is scope // ❌ if it could, result would be const ``` ```d int getN(const(BigStruct) big) => big.n; // for rvalues ref inout(int) getN(ref inout(BigStruct) big) => big.n; // for lvalues // ❌ can’t simply take address anymore // ❌ for rvalues: result is not an lvalue ``` ```d ref inout(int) getN( universal ref inout(BigStruct) big) => big.n; // ✔️ no copies // ✔️ returns by ref // ✔️ allows rvalue arguments // ✔️ single function, `auto fp = &getN` works // ✔️ for rvalue argument: return value has lifetime to the end of the statement // ✔️ for mutable argument: result is mutable ``` As far as conversions are concerned, an `R function( universal ref T)` implicitly converts to `R function(ref T)` and `R function( rvalue ref T)`. The reasoning being that a ` universal ref` parameter behaves exactly like a `ref` parameter for lvalue arguments, so the conversion merely “forgets” that rvalue arguments would be expected and allowed. The conversion to ` rvalue ref` similarly forgets that lvalue arguments were allowed. `ref` and ` rvalue ref` returns are conversion-incompatible for implicit conversions. As they are not binary-incompatible, an explicit cast can be used, which is a ` system` operation.You mentioned `auto ref` briefly, but did not describe how it interacts with this feature. I assume it's an lvalue ref by default, but would it be more useful if it's a universal ref? Or what if the user wants it to degrade to an rvalue ref instead? How would this work?
Jul 15
It looks like a mailing list bug is setting the subject line to "Martin Nowak is officially MIA", even when I manually set it to something else. This is a test post - please delete it when my other post with the incorrect subject line gets deleted as well.
Jul 15
On Wednesday, 3 July 2024 at 11:50:19 UTC, Quirin Schroll wrote:Add two function parameter attributes ` rvalue` and ` universal` which must be used together with `ref` (similar as `auto` is only valid as `auto ref`).The ` rvalue` could be used without `ref` to enforce an argument is moved into the parameter. “For example, I would want […] a parameter storage class that enforces the parameter was moved” in [this forum post](https://forum.dlang.org/post/v70ajp$2ecn$1 digitalmars.com).
Jul 16
On Wednesday, 3 July 2024 at 11:50:19 UTC, Quirin Schroll wrote:Add two function parameter attributes ` rvalue` and ` universal` which must be used together with `ref` (similar as `auto` is only valid as `auto ref`).This would be really useful for sure, as long as escaping rvalue references isn’t possible, so I guess it’d have to work like scope? Also I’m not sure about the at-attribute-based syntax. It makes the feature feel like an afterthought, even though this functionality is very useful. Perhaps a better solution syntax-wise could be to use `in` as the ‘rvalue’ attribute, and `auto in` could be the ‘universal’ attribute. So then: - ` rvalue` —> `in` - ` rvalue ref` —> `in ref` - ` universal ref` —> `auto in ref` - ` universal auto ref` —> `auto in auto ref` This way it’s both shorter to write, more consistent with existing function attributes, and we can **finally** make `in` do something useful by default.
Jul 28
On Monday, 29 July 2024 at 05:52:40 UTC, IchorDev wrote:On Wednesday, 3 July 2024 at 11:50:19 UTC, Quirin Schroll wrote:An ` rvalue ref` parameter definitely could be `return`. Like every `ref`, you can’t actually escape it as it might local, in fact, even an rvalue: ```d // Use -dip1000 struct MaterializeL(T) { T* _value; property ref T value() => *_value; } struct MaterializeR(T) { T value; } MaterializeL!T materializeU(T)(return ref T value) { return MaterializeL!T(&value); } MaterializeR!T materializeU(T)(T value) { import core.lifetime : move; return MaterializeR!T(move(value)); } disable MaterializeL!T materializeR(T)(ref T value) nogc nothrow pure safe { return MaterializeL!T(&value); } MaterializeR!T materializeR(T)(T value) { import core.lifetime : move; return MaterializeR!T(move(value)); } ref int f(return ref int x) safe { x = 10; return x; } ref int g(return ref int x) safe => x += 1; void main() safe { int x = (0).materializeU.value.f.g; assert(x == 11); // int* p = &(0).materializeU.value.f.g; // Error, escapes reference to temporary // On lvalues, materializeU.value is a no-op: int* p = &x.materializeU.value.f.g; int y = (0).materializeR.value.f.g; // int* q = &y.materializeR.value.f.g; // Error: materializeR disabled for lvalues } ```Add two function parameter attributes ` rvalue` and ` universal` which must be used together with `ref` (similar as `auto` is only valid as `auto ref`).This would be really useful for sure, as long as escaping rvalue references isn’t possible, so I guess it’d have to work like scope?Also I’m not sure about the at-attribute-based syntax. It makes the feature feel like an afterthought, even though this functionality is very useful. Perhaps a better solution syntax-wise could be to use `in` as the ‘rvalue’ attribute, and `auto in` could be the ‘universal’ attribute. So then: - ` rvalue` —> `in` - ` rvalue ref` —> `in ref` - ` universal ref` —> `auto in ref` - ` universal auto ref` —> `auto in auto ref`I have a DIP draft on my old computer which proposed something very similar to that: `in` and `ref` both mean *bind by reference*, in particular, `in` would have meant *allow rvalues* and `ref` *allow lvalues* (so both means allow both, none means bind-by-value). The idea of making `in` mean rvalues and `ref` mean lvalues isn’t new. The issue is, it can’t be done anymore. For all the time it was there, `in` meant `const`. Also, and this is the key killer: Conceptually, `in` means input parameter, i.e. something that supplies your function with data to make decisions on, call it configuration if you like, and this is what informed the `-preview=in` semantics. Another aspect I don’t like stylistically, is `auto in`. As a parameter storage class, `auto` means infer and requires templates, plus one could introspect what the inference determined. Although it doesn’t work like that, `auto ref` for variables could be used in non-template contexts, where `ref` is inferred from the initializer. Contrary to that, ` universal` requires materializing rvalues. Essentially, binding `arg` to a ` universal ref` parameter would be exactly like passing `arg.materializeU.value` to a `ref` parameter, and likewise passing `arg.materializeR.value` for binding to an ` rvalue ref` parameter. When writing DIPs, I take the future state of the language into account, that includes the new `in` semantics. It makes no sense proposing something that would be incompatible with something that’s going to be in the language; e.g. in my Primary Type Syntax DIP, I took care it won’t conflict (conceptually) with `ref` variables, and lo and behold, `ref` variables are here to stay. `in` is going to mean `scope const universal ref` with the caveat that for small trivially copyable types, ignore the ` universal ref`. Maybe that could also go into the DIP: ` optimized ( rvalue/ universal) ref` which is defined to be a copy for small trivially copyable types. That would render `in` an abbreviation for `scope const optimized universal ref`. It’s been the only thing I disliked about the new `in` semantics. It can do things in a bundle that aren’t available as single items. Same with non-static struct member functions binding the instance by reference, no matter if it’s called on an lvalue or rvalue. It makes little sense that this exclusive to the implicit `this` parameter.This way it’s both shorter to write[.]That I consider the wordiness a win. Not only is it clearer and a pure addition, those parameter attributes are an advanced tool.[…] [A]nd we can **finally** make `in` do something useful by default.But `-preview=in` already does that.
Aug 06
On Tuesday, 6 August 2024 at 15:26:49 UTC, Quirin Schroll wrote:On Monday, 29 July 2024 at 05:52:40 UTC, IchorDev wrote:And it still does. Just this week I’ve had to dissuade two people from using `in` instead of `const`.Also I’m not sure about the at-attribute-based syntax. It makes the feature feel like an afterthought, even though this functionality is very useful. Perhaps a better solution syntax-wise could be to use `in` as the ‘rvalue’ attribute, and `auto in` could be the ‘universal’ attribute. So then: - ` rvalue` —> `in` - ` rvalue ref` —> `in ref` - ` universal ref` —> `auto in ref` - ` universal auto ref` —> `auto in auto ref`The issue is, it can’t be done anymore. For all the time it was there, `in` meant `const`.Also, and this is the key killer: Conceptually, `in` means input parameter, i.e. something that supplies your function with data to make decisions on, call it configuration if you likeThis meaning is nebulous at best. What would we actually make `in` *do* to have it make parameters inputs? `-preview=in` making `in` into `scope ref const` that binds to rvalues is just as related to a parameter being an input as `in` binding to rvalues in general is.and this is what informed the `-preview=in` semantics.And time has proven that `-preview=in` is: 1. never going to become the default; and 2. is too limiting to even be very useful. Your DIP would make `-preview=in` an obsolete dinosaur, so just replace it.Another aspect I don’t like stylistically, is `auto in`. As a parameter storage class, `auto` means infer and requires templates, plus one could introspect what the inference determined.Is that a huge problem? You could always try suggest a better syntax for that case though, I was just invoking a recognisable idiom.Contrary to that, ` universal` requires materializing rvalues. Essentially, binding `arg` to a ` universal ref` parameter would be exactly like passing `arg.materializeU.value` to a `ref` parameter, and likewise passing `arg.materializeR.value` for binding to an ` rvalue ref` parameter.I think you should try rephrasing this because it doesn’t make sense.It’s harder to read more text, particularly when it overflows the available horizontal space. It’s not ‘clearer‘ at all.This way it’s […] shorter to writeThat I consider the wordiness a win[sic]. Not only is it clearer […]`-preview=in` is not the default. Saying that ‘as long as you change the defaults, the defaults are good’ is misleading. `-preview=in` is not the default. No matter how we move forwards with `in`, something is going to break; but I think making `in` a useful feature instead of dead-on-arrival would be a welcome change rather than a disappointing one.[…] [A]nd we can **finally** make `in` do something useful by default.But `-preview=in` already does that.
Aug 06
On Wednesday, 7 August 2024 at 03:47:24 UTC, IchorDev wrote:On Tuesday, 6 August 2024 at 15:26:49 UTC, Quirin Schroll wrote:The issue is, if we make `in` mean something that doesn’t include `const`, it’ll be confusing for the simple reason of what it meant since forever and what it’s supposed to be.On Monday, 29 July 2024 at 05:52:40 UTC, IchorDev wrote:And it still does. Just this week I’ve had to dissuade two people from using `in` instead of `const`.Also I’m not sure about the at-attribute-based syntax. It makes the feature feel like an afterthought, even though this functionality is very useful. Perhaps a better solution syntax-wise could be to use `in` as the ‘rvalue’ attribute, and `auto in` could be the ‘universal’ attribute. So then: - ` rvalue` —> `in` - ` rvalue ref` —> `in ref` - ` universal ref` —> `auto in ref` - ` universal auto ref` —> `auto in auto ref`The issue is, it can’t be done anymore. For all the time it was there, `in` meant `const`.I can only say that I disagree. An input parameter is for the function to only take information from to base decisions on. Everything it is follows logically from that: - Can’t mutate it. - Can’t escape it. - Can bind it in any way it likes (by reference or by value). If `in` binds rvalues so that they’re being moved from, it’s basically the opposite of an input parameter. Of course, there could be a different meaning of “input” in the sense that it goes in and never gets out, but that was never intended.Also, and this is the key killer: Conceptually, `in` means input parameter, i.e. something that supplies your function with data to make decisions on, call it configuration if you likeThis meaning is nebulous at best. What would we actually make `in` *do* to have it make parameters inputs? `-preview=in` making `in` into `scope ref const` that binds to rvalues is just as related to a parameter being an input as `in` binding to rvalues in general is.Preview-`in` gives you an optimization based on the size of the type. ` universal ref` wouldn’t do that. The reason it’s behind a preview switch is the same reason as `fieldwise`, `fixAliasThis`, `nosharedaccess`, `inclusiveincontracts`, and `fixImmutableConv` are: They are breaking changes and some silently change behavior. If you have a ` system` function that uses an `in` slice parameter, with `-preview=in`, that parameter becomes `scope`, but unchecked. With `DIP1000`, a caller that passes a literal can allocate the literal on the stack and that introduces a bug. My bet is that all but `rvaluerefparam` are going to be the default in the next Edition.and this is what informed the `-preview=in` semantics.And time has proven that `-preview=in` is: 1. never going to become the default; and 2. is too limiting to even be very useful. Your DIP would make `-preview=in` an obsolete dinosaur, so just replace it.“Contrary to that, ` universal` requires materializing rvalues. Essentially, binding `arg` to a ` universal ref` parameter would be exactly like passing `arg.materializeU.value` to a `ref` parameter. Binding `arg` to an ` rvalue ref` would be exactly like passing passing `arg.materializeR.value` for binding to to a `ref` parameter.”Another aspect I don’t like stylistically, is `auto in`. As a parameter storage class, `auto` means infer and requires templates, plus one could introspect what the inference determined.Is that a huge problem? You could always try suggest a better syntax for that case though, I was just invoking a recognisable idiom.Contrary to that, ` universal` requires materializing rvalues. Essentially, binding `arg` to a ` universal ref` parameter would be exactly like passing `arg.materializeU.value` to a `ref` parameter, and likewise passing `arg.materializeR.value` for binding to an ` rvalue ref` parameter.I think you should try rephrasing this because it doesn’t make sense.I don’t understand what you’re meaning here.It’s harder to read more text, particularly when it overflows the available horizontal space. It’s not ‘clearer‘ at all.This way it’s […] shorter to writeThat I consider the wordiness a win[sic]. Not only is it clearer […]`-preview=in` is not the default. Saying that ‘as long as you change the defaults, the defaults are good’ is misleading. `-preview=in` is not the default.[…] [A]nd we can **finally** make `in` do something useful by default.But `-preview=in` already does that.No matter how we move forwards with `in`, something is going to break;Not with Editions. The whole idea of Editions is that an Edition is a set of Previews that are enabled together, and that modules opt into an Edition. Because every Edition starts with zero code written for it, it can’t break code.but I think making `in` a useful feature instead of dead-on-arrival would be a welcome change rather than a disappointing one.The reason `in` isn’t useful as-is is because it’s just `const` and `const` isn’t that much longer than `in` while both being clear and not having a preview that is likely to change its semantics. The reason `in` isn’t useful in Preview is because almost no-one uses this preview switch. It’s kind of niche, and most code works great without it. The difference between `scope const` pass-by-value and `in` (under preview) is an optimization in almost all cases (aliasing is the exception), and if a type is small, it’s not even that. Also, preview-`in` is actively dangerous with DIP1000 unless ` safe` isn’t the default.
Aug 07