digitalmars.D - aliasing expressions and identifiers
- deed (24/24) May 23 2016 As I've been finding myself constantly desiring to alias
- Nick Treleaven (13/22) May 23 2016 If we had local refs, we could use this instead:
- Marc =?UTF-8?B?U2Now7x0eg==?= (9/29) May 23 2016 At some point during the `scope` discussion, Walter wanted to
- Nick Treleaven (13/33) May 24 2016 OK. I suppose supporting local refs is a good reason not to allow
- Nick Treleaven (13/28) May 25 2016 Here I think local refs must be prevented from initialization by return
- Marc =?UTF-8?B?U2Now7x0eg==?= (25/46) May 26 2016 To elaborate: neither `scope` nor reference counting can ever
- Nick Treleaven (16/55) May 27 2016 This is the same situation. There's nothing special about
- Marc =?UTF-8?B?U2Now7x0eg==?= (24/66) May 27 2016 If this is what destroy does, then yes, that's ok. But it is a
- Nick Treleaven (7/48) May 29 2016 What about if the RCArray (of ref count 1) is assigned to a
- Marc =?UTF-8?B?U2Now7x0eg==?= (13/40) May 30 2016 Exactly, and then one of the two suggested approaches will have
- Nick Treleaven (11/28) May 30 2016 Well my solution does work, disallowing the problematic local
- Nick Treleaven (4/15) May 30 2016 Sorry, I meant using array(scope R range) would not affect the
- Marc =?UTF-8?B?U2Now7x0eg==?= (3/11) May 26 2016 You're calling `destroy` explicitly, what else would you expect
- deed (17/28) May 23 2016 Just to be clear: I'm not looking for anything touching the
- Nick Treleaven (11/16) May 24 2016 I think it's more useful to support local refs than alias
As I've been finding myself constantly desiring to alias expressions and identifiers for many years, I'd like to ask again whether it would be possible to extend either the alias-statement or the with-statement. It would be great to relieve the constant tension between descriptiveness, (often outside your control), and brevity in a straightforward manner. There is something missing when your fields and parameters start out with proper names but end up as cryptic abbreviations when you start using them. Sometimes you want to look at an identifier and understand its purpose, other times you want to look at a bunch of expressions and statements and understand their pattern. I think an expressive language should support both needs without tension. It would also isolate names and reduce editing when names change. Some thoughts about extending the with-statement were brought up here earlier: http://forum.dlang.org/post/txpifmwpmmhsvcpbcijw forum.dlang.org I don't care much whether it would be with, alias or possibly something clever already existing, but a solution should be easy to use, easy to read and shouldn't introduce any possible overhead at runtime. 1) Are there any technical blockers for extending the alias- or with-statement? 2) Does it carry its own weight?
May 23 2016
On Monday, 23 May 2016 at 14:05:43 UTC, deed wrote:Some thoughts about extending the with-statement were brought up here earlier: http://forum.dlang.org/post/txpifmwpmmhsvcpbcijw forum.dlang.org I don't care much whether it would be with, alias or possibly something clever already existing, but a solution should be easy to use, easy to read and shouldn't introduce any possible overhead at runtime.From the linked thread:ref M() { return matrix.rawArr; } ref Ex1() { return e1.someProperties.someModulusX; }If we had local refs, we could use this instead: ref m = matrix.rawArr; The difference is m is already computed, it is not recomputed each time m is read, unlike M(). I think the reason D doesn't support local refs is because it would make it harder to design safe, particularly with the planned rc ref-counting. Because M() above is only a return reference, it can't live longer than the data it references. My ref m could persist longer than matrix.rawArr using (naive) reference counting. BTW local refs are possible in system code: http://forum.dlang.org/post/lmuokynffgljzvrpvkvd forum.dlang.org
May 23 2016
On Monday, 23 May 2016 at 15:18:51 UTC, Nick Treleaven wrote:On Monday, 23 May 2016 at 14:05:43 UTC, deed wrote:Note that this wouldn't work with rvalues, which `with` supports.Some thoughts about extending the with-statement were brought up here earlier: http://forum.dlang.org/post/txpifmwpmmhsvcpbcijw forum.dlang.org I don't care much whether it would be with, alias or possibly something clever already existing, but a solution should be easy to use, easy to read and shouldn't introduce any possible overhead at runtime.From the linked thread:ref M() { return matrix.rawArr; } ref Ex1() { return e1.someProperties.someModulusX; }If we had local refs, we could use this instead: ref m = matrix.rawArr;The difference is m is already computed, it is not recomputed each time m is read, unlike M(). I think the reason D doesn't support local refs is because it would make it harder to design safe, particularly with the planned rc ref-counting. Because M() above is only a return reference, it can't live longer than the data it references. My ref m could persist longer than matrix.rawArr using (naive) reference counting.At some point during the `scope` discussion, Walter wanted to allow local `ref`s, so I guess he's not opposed to the idea. As far as I understand, the compiler already supports them internally, as they can result from lowering certain constructs, there's just no syntax for them. They wouldn't pose a problem for lifetime tracking, because they can never be assigned to, only initialized once.
May 23 2016
On Monday, 23 May 2016 at 17:03:32 UTC, Marc Schütz wrote:On Monday, 23 May 2016 at 15:18:51 UTC, Nick Treleaven wrote:OK. I suppose supporting local refs is a good reason not to allow rvalues to be passed as const ref arguments, if the function also returns by ref.If we had local refs, we could use this instead: ref m = matrix.rawArr;Note that this wouldn't work with rvalues, which `with` supports.Great :-)I think the reason D doesn't support local refs is because it would make it harder to design safe, particularly with the planned rc ref-counting. Because M() above is only a return reference, it can't live longer than the data it references. My ref m could persist longer than matrix.rawArr using (naive) reference counting.At some point during the `scope` discussion, Walter wanted to allow local `ref`s, so I guess he's not opposed to the idea. Asfar as I understand, the compiler already supports them internally, as they can result from lowering certain constructs, there's just no syntax for them. They wouldn't pose a problem for lifetime tracking, because they can never be assigned to, only initialized once.What about: safe unittest { RCArray!int arr; ref r = arr[0]; arr.destroy; // refcount drops to zero, arr.impl memory freed r++; // writes to unallocated memory }
May 24 2016
On 24/05/2016 14:48, Nick Treleaven wrote:On Monday, 23 May 2016 at 17:03:32 UTC, Marc Schütz wrote:+ arr.length = 1;On Monday, 23 May 2016 at 15:18:51 UTC, Nick Treleaven wrote:What about: safe unittest { RCArray!int arr;I think the reason D doesn't support local refs is because it would make it harder to design safe, particularly with the planned rc ref-counting.They wouldn't pose a problem for lifetime tracking, because they can never be assigned to, only initialized once.ref r = arr[0]; arr.destroy; // refcount drops to zero, arr.impl memory freed r++; // writes to unallocated memory }Here I think local refs must be prevented from initialization by return ref (-dip25). The rc DIP and RCArray would use return ref. It would be OK to initialize a local ref with a ref function result that is not return ref. Naturally, this would be safe: auto slice = [7]; ref r = slice[0]; slice.destroy; r++; // slice memory still allocated --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus
May 25 2016
On Wednesday, 25 May 2016 at 19:47:06 UTC, Nick Treleaven wrote:On 24/05/2016 14:48, Nick Treleaven wrote:To elaborate: neither `scope` nor reference counting can ever protect you against explicit premature destruction of a still-referenced owner. But there is a slightly different problematic scenario: RCArray!int arr = [7]; ref r = arr[0]; arr = [9]; // this releases the old array r++; // use after free But this issue exists even without locale `ref`s: void foo() { RCArray!int arr = [7]; bar(arr, arr[0]); } void bar(ref RCArray!int arr, ref int r) { arr = [9]; // this releases the old array r++; // use after free } It can be solved in one of two ways: Either by making the owner (`arr`) non-mutable during the existence of the references, thereby forbidding the call to `bar()` (I would prefer this one, as it's cleaner and can be used for many more things, e.g. the byLine problem), or by making the owner live longer by inserting the appropriate AddRef/Release pairs whenever such a situation arises.What about: safe unittest { RCArray!int arr;+ arr.length = 1;ref r = arr[0]; arr.destroy; // refcount drops to zero, arr.impl memory freed r++; // writes to unallocated memory }Here I think local refs must be prevented from initialization by return ref (-dip25). The rc DIP and RCArray would use return ref. It would be OK to initialize a local ref with a ref function result that is not return ref. Naturally, this would be safe: auto slice = [7]; ref r = slice[0]; slice.destroy; r++; // slice memory still allocated
May 26 2016
On Thursday, 26 May 2016 at 08:29:41 UTC, Marc Schütz wrote:On Wednesday, 25 May 2016 at 19:47:06 UTC, Nick Treleaven wrote:This is the same situation. There's nothing special about destroy, it just assigns arr = arr.init. destroy is safe so in fact it must work with safe RC. You seem to have ignored my suggestion (which maybe wasn't clear enough) to statically prevent the above from compiling using: RCArray(T) { ... ref opIndex(size_t) return; Local refs cannot be assigned from a function returning ref if that function has any parameters marked with the return attribute. If there is no attribute, local refs + function returning ref is OK.On 24/05/2016 14:48, Nick Treleaven wrote:To elaborate: neither `scope` nor reference counting can ever protect you against explicit premature destruction of a still-referenced owner. But there is a slightly different problematic scenario: RCArray!int arr = [7]; ref r = arr[0]; arr = [9]; // this releases the old array r++; // use after freesafe unittest { RCArray!int arr;+ arr.length = 1;ref r = arr[0]; arr.destroy; // refcount drops to zero, arr.impl memory freed r++; // writes to unallocated memory }Here I think local refs must be prevented from initialization by return ref (-dip25). The rc DIP and RCArray would use return ref. It would be OK to initialize a local ref with a ref function result that is not return ref. ...But this issue exists even without locale `ref`s: void foo() { RCArray!int arr = [7]; bar(arr, arr[0]); } void bar(ref RCArray!int arr, ref int r) { arr = [9]; // this releases the old array r++; // use after free }This is the reason for the rc DIP.It can be solved in one of two ways: Either by making the owner (`arr`) non-mutable during the existence of the references, thereby forbidding the call to `bar()` (I would prefer this one, as it's cleaner and can be used for many more things, e.g. the byLine problem)I don't see directly how this affects byLine.front, that does not return a reference.
May 27 2016
On Friday, 27 May 2016 at 10:04:14 UTC, Nick Treleaven wrote:On Thursday, 26 May 2016 at 08:29:41 UTC, Marc Schütz wrote:If this is what destroy does, then yes, that's ok. But it is a misleading name then, IMO.To elaborate: neither `scope` nor reference counting can ever protect you against explicit premature destruction of a still-referenced owner. But there is a slightly different problematic scenario: RCArray!int arr = [7]; ref r = arr[0]; arr = [9]; // this releases the old array r++; // use after freeThis is the same situation. There's nothing special about destroy, it just assigns arr = arr.init. destroy is safe so in fact it must work with safe RC.You seem to have ignored my suggestion (which maybe wasn't clear enough) to statically prevent the above from compiling using: RCArray(T) { ... ref opIndex(size_t) return; Local refs cannot be assigned from a function returning ref if that function has any parameters marked with the return attribute. If there is no attribute, local refs + function returning ref is OK.Huh? `return` means that the returned reference is owned by the RCArray struct and must therefore not outlive it. If the RCArray is a local variable (or parameter), the local ref is always declared after it (because it must be initialized immediately), and will have a shorter scope than the RCArray. Therefore, such an assignment is always accepted.It returns a reference in the wider sense, namely a slice to a private buffer that gets overwritten by each call to `byLine`. Currently, DIP25 only applies to `ref`s in the narrow sense, but I'm assuming it will be generalized to include pointer, slices, class references, AAs and hidden context pointers. Making the ByLine range constant as long as there's a reference to its buffer would prevent surprises like this: auto lines = stdin.byLine.array; // => all elements of `lines` are the same, should have used `byLineCopy` By the way, there's a theoretical third possibility: Force-invalidating the references after the owner has (potentially) been mutated. But this would require very advanced data flow tracking and is probably impossible to implement in the general case.But this issue exists even without locale `ref`s: void foo() { RCArray!int arr = [7]; bar(arr, arr[0]); } void bar(ref RCArray!int arr, ref int r) { arr = [9]; // this releases the old array r++; // use after free }This is the reason for the rc DIP.It can be solved in one of two ways: Either by making the owner (`arr`) non-mutable during the existence of the references, thereby forbidding the call to `bar()` (I would prefer this one, as it's cleaner and can be used for many more things, e.g. the byLine problem)I don't see directly how this affects byLine.front, that does not return a reference.
May 27 2016
On Friday, 27 May 2016 at 11:50:42 UTC, Marc Schütz wrote:On Friday, 27 May 2016 at 10:04:14 UTC, Nick Treleaven wrote:...On Thursday, 26 May 2016 at 08:29:41 UTC, Marc Schütz wrote:RCArray!int arr = [7]; ref r = arr[0]; arr = [9]; // this releases the old array r++; // use after freeWhat about if the RCArray (of ref count 1) is assigned to a different one after the local ref is initialised? That is what we're discussing -it's your example above(!)statically prevent the above from compiling using: RCArray(T) { ... ref opIndex(size_t) return; Local refs cannot be assigned from a function returning ref if that function has any parameters marked with the return attribute. If there is no attribute, local refs + function returning ref is OK.Huh? `return` means that the returned reference is owned by the RCArray struct and must therefore not outlive it. If the RCArray is a local variable (or parameter), the local ref is always declared after it (because it must be initialized immediately), and will have a shorter scope than the RCArray. Therefore, such an assignment is always accepted.So your solution would statically prevent popFront because front has escaped. I think we should just prevent front from escaping.It returns a reference in the wider sense, namely a slice to a private buffer that gets overwritten by each call to `byLine`. Currently, DIP25 only applies to `ref`s in the narrow sense, but I'm assuming it will be generalized to include pointer, slices, class references, AAs and hidden context pointers. Making the ByLine range constant as long as there's a reference to its buffer would prevent surprises like this: auto lines = stdin.byLine.array; // => all elements of `lines` are the same, should have used `byLineCopy`It can be solved in one of two ways: Either by making the owner (`arr`) non-mutable during the existence of the references, thereby forbidding the call to `bar()` (I would prefer this one, as it's cleaner and can be used for many more things, e.g. the byLine problem)I don't see directly how this affects byLine.front, that does not return a reference.
May 29 2016
On Sunday, 29 May 2016 at 14:27:51 UTC, Nick Treleaven wrote:What about if the RCArray (of ref count 1) is assigned to a different one after the local ref is initialised? That is what we're discussing -it's your example above(!)Exactly, and then one of the two suggested approaches will have to be used to prevent the use-after-free. But that's something that needs to happen on assignment, not when the reference is created.Yes.So your solution would statically prevent popFront because front has escaped.It returns a reference in the wider sense, namely a slice to a private buffer that gets overwritten by each call to `byLine`. Currently, DIP25 only applies to `ref`s in the narrow sense, but I'm assuming it will be generalized to include pointer, slices, class references, AAs and hidden context pointers. Making the ByLine range constant as long as there's a reference to its buffer would prevent surprises like this: auto lines = stdin.byLine.array; // => all elements of `lines` are the same, should have used `byLineCopy`It can be solved in one of two ways: Either by making the owner (`arr`) non-mutable during the existence of the references, thereby forbidding the call to `bar()` (I would prefer this one, as it's cleaner and can be used for many more things, e.g. the byLine problem)I don't see directly how this affects byLine.front, that does not return a reference.I think we should just prevent front from escaping.It doesn't necessarily need to escape for the problem to occur. Well, it does in this example, but it can be trivially rewritten: auto tmp = stdin.byLine; auto lines = tmp.array; Here, `lines` contains references to the buffer owned by `tmp`, but doesn't escape (assuming `array` takes its argument by `scope` or however the final solution will look like).
May 30 2016
On Monday, 30 May 2016 at 10:55:57 UTC, Marc Schütz wrote:On Sunday, 29 May 2016 at 14:27:51 UTC, Nick Treleaven wrote:Well my solution does work, disallowing the problematic local refs. But it wouldn't be great for generic code or consistency with the rc DIP. For the latter, we can add a temporary RC object to keep the referenced memory alive, but we only need to do this when the local ref is initialized from a function both (1) returning ref and (2) with a parameter marked return.What about if the RCArray (of ref count 1) is assigned to a different one after the local ref is initialised? That is what we're discussing -it's your example above(!)Exactly, and then one of the two suggested approaches will have to be used to prevent the use-after-free. But that's something that needs to happen on assignment, not when the reference is created.tmp and stdin.byLine are of type ByLine, whose front could be scope/return to prevent escaping. Above array() does escape ByLine.front so can't mark its argument with scope - the compiler would error.I think we should just prevent front from escaping.It doesn't necessarily need to escape for the problem to occur. Well, it does in this example, but it can be trivially rewritten: auto tmp = stdin.byLine; auto lines = tmp.array; Here, `lines` contains references to the buffer owned by `tmp`, but doesn't escape (assuming `array` takes its argument by `scope` or however the final solution will look like).
May 30 2016
On Monday, 30 May 2016 at 14:18:38 UTC, Nick Treleaven wrote:On Monday, 30 May 2016 at 10:55:57 UTC, Marc Schütz wrote:Sorry, I meant using array(scope R range) would not affect the compiler error when array attempts to escape range.front into its slice.auto tmp = stdin.byLine; auto lines = tmp.array; Here, `lines` contains references to the buffer owned by `tmp`, but doesn't escape (assuming `array` takes its argument by `scope` or however the final solution will look like).tmp and stdin.byLine are of type ByLine, whose front could be scope/return to prevent escaping. Above array() does escape ByLine.front so can't mark its argument with scope - the compiler would error.
May 30 2016
On Tuesday, 24 May 2016 at 13:48:41 UTC, Nick Treleaven wrote:What about: safe unittest { RCArray!int arr; ref r = arr[0]; arr.destroy; // refcount drops to zero, arr.impl memory freed r++; // writes to unallocated memory }You're calling `destroy` explicitly, what else would you expect to happen?
May 26 2016
On Monday, 23 May 2016 at 15:18:51 UTC, Nick Treleaven wrote:If we had local refs, we could use this instead: ref m = matrix.rawArr; The difference is m is already computed, it is not recomputed each time m is read, unlike M(). I think the reason D doesn't support local refs is because it would make it harder to design safe, particularly with the planned rc ref-counting. Because M() above is only a return reference, it can't live longer than the data it references. My ref m could persist longer than matrix.rawArr using (naive) reference counting. BTW local refs are possible in system code: http://forum.dlang.org/post/lmuokynffgljzvrpvkvd forum.dlang.orgJust to be clear: I'm not looking for anything touching the semantics, just a simple replacement mechanism/macro expansion. Recomputation or not, safe or system, should be no different from the expanded code. So within a scope, after aliasing 'm', 'm' should be replaced by 'matrix.rawArr'. Everything stays the same as is, even error messages. { // Some scope alias i = instance.targetIdx; alias m = matrix.rawArr; m[i] = m[j] + m[k]; // detected and conceptually replaced in one of the earlier compiler passes by // matrix.rawArr[instance.targetIdx] = matrix.rawArr[j] + matrix.rawArr[k] }
May 23 2016
On Monday, 23 May 2016 at 17:43:49 UTC, deed wrote:On Monday, 23 May 2016 at 15:18:51 UTC, Nick Treleaven wrote: Recomputation or not, safe or system, should be no different from the expanded code. So within a scope, after aliasing 'm', 'm' should be replaced by 'matrix.rawArr'. Everything stays the same as is, even error messages.I think it's more useful to support local refs than alias expressions. In fact apart from the rvalue support, I would rather use a local ref than the existing with statement (assuming the ref was memory safe). The only exception is when using `with (EnumType)`. Also, compile-time expressions use enum, not alias. It could be changed, but I think it's useful when reading meta-programming code to clearly see the difference. Currently aliased symbols are expanded by the compiler in error messages, but that's probably not ideal.
May 24 2016