www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - aliasing expressions and identifiers

reply deed <none none.none> writes:
As I've been finding myself constantly desiring to alias 
expressions and identifiers for many years, I'd like to ask again 
whether it would be possible to extend either the alias-statement 
or the with-statement. It would be great to relieve the constant 
tension between descriptiveness, (often outside your control), 
and brevity in a straightforward manner.

There is something missing when your fields and parameters start 
out with proper names but end up as cryptic abbreviations when 
you start using them. Sometimes you want to look at an identifier 
and understand its purpose, other times you want to look at a 
bunch of expressions and statements and understand their pattern. 
I think an expressive language should support both needs without 
tension. It would also isolate names and reduce editing when 
names change.

Some thoughts about extending the with-statement were brought up 
here earlier:
http://forum.dlang.org/post/txpifmwpmmhsvcpbcijw forum.dlang.org
I don't care much whether it would be with, alias or possibly 
something clever already existing, but a solution should be easy 
to use, easy to read and shouldn't introduce any possible 
overhead at runtime.

1) Are there any technical blockers for extending the alias- or 
with-statement?
2) Does it carry its own weight?
May 23 2016
parent reply Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On Monday, 23 May 2016 at 14:05:43 UTC, deed wrote:
 Some thoughts about extending the with-statement were brought 
 up here earlier:
 http://forum.dlang.org/post/txpifmwpmmhsvcpbcijw forum.dlang.org
 I don't care much whether it would be with, alias or possibly 
 something clever already existing, but a solution should be 
 easy to use, easy to read and shouldn't introduce any possible 
 overhead at runtime.
From the linked thread:
        ref M()   { return matrix.rawArr; }
        ref Ex1() { return e1.someProperties.someModulusX; }
If we had local refs, we could use this instead: ref m = matrix.rawArr; The difference is m is already computed, it is not recomputed each time m is read, unlike M(). I think the reason D doesn't support local refs is because it would make it harder to design safe, particularly with the planned rc ref-counting. Because M() above is only a return reference, it can't live longer than the data it references. My ref m could persist longer than matrix.rawArr using (naive) reference counting. BTW local refs are possible in system code: http://forum.dlang.org/post/lmuokynffgljzvrpvkvd forum.dlang.org
May 23 2016
next sibling parent reply Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:
On Monday, 23 May 2016 at 15:18:51 UTC, Nick Treleaven wrote:
 On Monday, 23 May 2016 at 14:05:43 UTC, deed wrote:
 Some thoughts about extending the with-statement were brought 
 up here earlier:
 http://forum.dlang.org/post/txpifmwpmmhsvcpbcijw forum.dlang.org
 I don't care much whether it would be with, alias or possibly 
 something clever already existing, but a solution should be 
 easy to use, easy to read and shouldn't introduce any possible 
 overhead at runtime.
From the linked thread:
        ref M()   { return matrix.rawArr; }
        ref Ex1() { return e1.someProperties.someModulusX; }
If we had local refs, we could use this instead: ref m = matrix.rawArr;
Note that this wouldn't work with rvalues, which `with` supports.
 The difference is m is already computed, it is not recomputed 
 each time m is read, unlike M(). I think the reason D doesn't 
 support local refs is because it would make it harder to design 
  safe, particularly with the planned  rc ref-counting. Because 
 M() above is only a return reference, it can't live longer than 
 the data it references. My ref m could persist longer than 
 matrix.rawArr using (naive) reference counting.
At some point during the `scope` discussion, Walter wanted to allow local `ref`s, so I guess he's not opposed to the idea. As far as I understand, the compiler already supports them internally, as they can result from lowering certain constructs, there's just no syntax for them. They wouldn't pose a problem for lifetime tracking, because they can never be assigned to, only initialized once.
May 23 2016
parent reply Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On Monday, 23 May 2016 at 17:03:32 UTC, Marc Schütz wrote:
 On Monday, 23 May 2016 at 15:18:51 UTC, Nick Treleaven wrote:
 If we had local refs, we could use this instead:

 ref m = matrix.rawArr;
Note that this wouldn't work with rvalues, which `with` supports.
OK. I suppose supporting local refs is a good reason not to allow rvalues to be passed as const ref arguments, if the function also returns by ref.
 I think the reason D doesn't support local refs is because it 
 would make it harder to design  safe, particularly with the 
 planned  rc ref-counting. Because M() above is only a return 
 reference, it can't live longer than the data it references. 
 My ref m could persist longer than matrix.rawArr using (naive) 
 reference counting.
At some point during the `scope` discussion, Walter wanted to allow local `ref`s, so I guess he's not opposed to the idea. As
Great :-)
 far as I understand, the compiler already supports them 
 internally, as they can result from lowering certain 
 constructs, there's just no syntax for them. They wouldn't pose 
 a problem for lifetime tracking, because they can never be 
 assigned to, only initialized once.
What about: safe unittest { RCArray!int arr; ref r = arr[0]; arr.destroy; // refcount drops to zero, arr.impl memory freed r++; // writes to unallocated memory }
May 24 2016
next sibling parent reply Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On 24/05/2016 14:48, Nick Treleaven wrote:
 On Monday, 23 May 2016 at 17:03:32 UTC, Marc Schütz wrote:
 On Monday, 23 May 2016 at 15:18:51 UTC, Nick Treleaven wrote:
 I think the reason D doesn't support local refs is because it would
 make it harder to design  safe, particularly with the planned  rc
 ref-counting.
They wouldn't pose a problem for lifetime tracking, because they can never be assigned to, only initialized once.
What about: safe unittest { RCArray!int arr;
+ arr.length = 1;
      ref r = arr[0];
      arr.destroy; // refcount drops to zero, arr.impl memory freed
      r++; // writes to unallocated memory
 }
Here I think local refs must be prevented from initialization by return ref (-dip25). The rc DIP and RCArray would use return ref. It would be OK to initialize a local ref with a ref function result that is not return ref. Naturally, this would be safe: auto slice = [7]; ref r = slice[0]; slice.destroy; r++; // slice memory still allocated --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus
May 25 2016
parent reply Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:
On Wednesday, 25 May 2016 at 19:47:06 UTC, Nick Treleaven wrote:
 On 24/05/2016 14:48, Nick Treleaven wrote:
 What about:

  safe unittest
 {
      RCArray!int arr;
+ arr.length = 1;
      ref r = arr[0];
      arr.destroy; // refcount drops to zero, arr.impl memory 
 freed
      r++; // writes to unallocated memory
 }
Here I think local refs must be prevented from initialization by return ref (-dip25). The rc DIP and RCArray would use return ref. It would be OK to initialize a local ref with a ref function result that is not return ref. Naturally, this would be safe: auto slice = [7]; ref r = slice[0]; slice.destroy; r++; // slice memory still allocated
To elaborate: neither `scope` nor reference counting can ever protect you against explicit premature destruction of a still-referenced owner. But there is a slightly different problematic scenario: RCArray!int arr = [7]; ref r = arr[0]; arr = [9]; // this releases the old array r++; // use after free But this issue exists even without locale `ref`s: void foo() { RCArray!int arr = [7]; bar(arr, arr[0]); } void bar(ref RCArray!int arr, ref int r) { arr = [9]; // this releases the old array r++; // use after free } It can be solved in one of two ways: Either by making the owner (`arr`) non-mutable during the existence of the references, thereby forbidding the call to `bar()` (I would prefer this one, as it's cleaner and can be used for many more things, e.g. the byLine problem), or by making the owner live longer by inserting the appropriate AddRef/Release pairs whenever such a situation arises.
May 26 2016
parent reply Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On Thursday, 26 May 2016 at 08:29:41 UTC, Marc Schütz wrote:
 On Wednesday, 25 May 2016 at 19:47:06 UTC, Nick Treleaven wrote:
 On 24/05/2016 14:48, Nick Treleaven wrote:
  safe unittest
 {
      RCArray!int arr;
+ arr.length = 1;
      ref r = arr[0];
      arr.destroy; // refcount drops to zero, arr.impl memory 
 freed
      r++; // writes to unallocated memory
 }
Here I think local refs must be prevented from initialization by return ref (-dip25). The rc DIP and RCArray would use return ref. It would be OK to initialize a local ref with a ref function result that is not return ref. ...
To elaborate: neither `scope` nor reference counting can ever protect you against explicit premature destruction of a still-referenced owner. But there is a slightly different problematic scenario: RCArray!int arr = [7]; ref r = arr[0]; arr = [9]; // this releases the old array r++; // use after free
This is the same situation. There's nothing special about destroy, it just assigns arr = arr.init. destroy is safe so in fact it must work with safe RC. You seem to have ignored my suggestion (which maybe wasn't clear enough) to statically prevent the above from compiling using: RCArray(T) { ... ref opIndex(size_t) return; Local refs cannot be assigned from a function returning ref if that function has any parameters marked with the return attribute. If there is no attribute, local refs + function returning ref is OK.
 But this issue exists even without locale `ref`s:

 void foo() {
     RCArray!int arr = [7];
     bar(arr, arr[0]);
 }

 void bar(ref RCArray!int arr, ref int r) {
     arr = [9];    // this releases the old array
     r++;          // use after free
 }
This is the reason for the rc DIP.
 It can be solved in one of two ways: Either by making the owner 
 (`arr`) non-mutable during the existence of the references, 
 thereby forbidding the call to `bar()` (I would prefer this 
 one, as it's cleaner and can be used for many more things, e.g. 
 the byLine problem)
I don't see directly how this affects byLine.front, that does not return a reference.
May 27 2016
parent reply Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:
On Friday, 27 May 2016 at 10:04:14 UTC, Nick Treleaven wrote:
 On Thursday, 26 May 2016 at 08:29:41 UTC, Marc Schütz wrote:
 To elaborate: neither `scope` nor reference counting can ever 
 protect you against explicit premature destruction of a 
 still-referenced owner. But there is a slightly different 
 problematic scenario:

 RCArray!int arr = [7];
 ref r = arr[0];
 arr = [9];        // this releases the old array
 r++;              // use after free
This is the same situation. There's nothing special about destroy, it just assigns arr = arr.init. destroy is safe so in fact it must work with safe RC.
If this is what destroy does, then yes, that's ok. But it is a misleading name then, IMO.
 You seem to have ignored my suggestion (which maybe wasn't 
 clear enough) to statically prevent the above from compiling 
 using:

 RCArray(T) {
     ...
     ref opIndex(size_t) return;

 Local refs cannot be assigned from a function returning ref if 
 that function has any parameters marked with the return 
 attribute. If there is no attribute, local refs + function 
 returning ref is OK.
Huh? `return` means that the returned reference is owned by the RCArray struct and must therefore not outlive it. If the RCArray is a local variable (or parameter), the local ref is always declared after it (because it must be initialized immediately), and will have a shorter scope than the RCArray. Therefore, such an assignment is always accepted.
 But this issue exists even without locale `ref`s:

 void foo() {
     RCArray!int arr = [7];
     bar(arr, arr[0]);
 }

 void bar(ref RCArray!int arr, ref int r) {
     arr = [9];    // this releases the old array
     r++;          // use after free
 }
This is the reason for the rc DIP.
 It can be solved in one of two ways: Either by making the 
 owner (`arr`) non-mutable during the existence of the 
 references, thereby forbidding the call to `bar()` (I would 
 prefer this one, as it's cleaner and can be used for many more 
 things, e.g. the byLine problem)
I don't see directly how this affects byLine.front, that does not return a reference.
It returns a reference in the wider sense, namely a slice to a private buffer that gets overwritten by each call to `byLine`. Currently, DIP25 only applies to `ref`s in the narrow sense, but I'm assuming it will be generalized to include pointer, slices, class references, AAs and hidden context pointers. Making the ByLine range constant as long as there's a reference to its buffer would prevent surprises like this: auto lines = stdin.byLine.array; // => all elements of `lines` are the same, should have used `byLineCopy` By the way, there's a theoretical third possibility: Force-invalidating the references after the owner has (potentially) been mutated. But this would require very advanced data flow tracking and is probably impossible to implement in the general case.
May 27 2016
parent reply Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On Friday, 27 May 2016 at 11:50:42 UTC, Marc Schütz wrote:
 On Friday, 27 May 2016 at 10:04:14 UTC, Nick Treleaven wrote:
 On Thursday, 26 May 2016 at 08:29:41 UTC, Marc Schütz wrote:
 RCArray!int arr = [7];
 ref r = arr[0];
 arr = [9];        // this releases the old array
 r++;              // use after free
...
 statically prevent the above from compiling using:

 RCArray(T) {
     ...
     ref opIndex(size_t) return;

 Local refs cannot be assigned from a function returning ref if 
 that function has any parameters marked with the return 
 attribute. If there is no attribute, local refs + function 
 returning ref is OK.
Huh? `return` means that the returned reference is owned by the RCArray struct and must therefore not outlive it. If the RCArray is a local variable (or parameter), the local ref is always declared after it (because it must be initialized immediately), and will have a shorter scope than the RCArray. Therefore, such an assignment is always accepted.
What about if the RCArray (of ref count 1) is assigned to a different one after the local ref is initialised? That is what we're discussing -it's your example above(!)
 It can be solved in one of two ways: Either by making the 
 owner (`arr`) non-mutable during the existence of the 
 references, thereby forbidding the call to `bar()` (I would 
 prefer this one, as it's cleaner and can be used for many 
 more things, e.g. the byLine problem)
I don't see directly how this affects byLine.front, that does not return a reference.
It returns a reference in the wider sense, namely a slice to a private buffer that gets overwritten by each call to `byLine`. Currently, DIP25 only applies to `ref`s in the narrow sense, but I'm assuming it will be generalized to include pointer, slices, class references, AAs and hidden context pointers. Making the ByLine range constant as long as there's a reference to its buffer would prevent surprises like this: auto lines = stdin.byLine.array; // => all elements of `lines` are the same, should have used `byLineCopy`
So your solution would statically prevent popFront because front has escaped. I think we should just prevent front from escaping.
May 29 2016
parent reply Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:
On Sunday, 29 May 2016 at 14:27:51 UTC, Nick Treleaven wrote:
 What about if the RCArray (of ref count 1) is assigned to a 
 different one after the local ref is initialised? That is what 
 we're discussing -it's your example above(!)
Exactly, and then one of the two suggested approaches will have to be used to prevent the use-after-free. But that's something that needs to happen on assignment, not when the reference is created.
 It can be solved in one of two ways: Either by making the 
 owner (`arr`) non-mutable during the existence of the 
 references, thereby forbidding the call to `bar()` (I would 
 prefer this one, as it's cleaner and can be used for many 
 more things, e.g. the byLine problem)
I don't see directly how this affects byLine.front, that does not return a reference.
It returns a reference in the wider sense, namely a slice to a private buffer that gets overwritten by each call to `byLine`. Currently, DIP25 only applies to `ref`s in the narrow sense, but I'm assuming it will be generalized to include pointer, slices, class references, AAs and hidden context pointers. Making the ByLine range constant as long as there's a reference to its buffer would prevent surprises like this: auto lines = stdin.byLine.array; // => all elements of `lines` are the same, should have used `byLineCopy`
So your solution would statically prevent popFront because front has escaped.
Yes.
 I think we should just prevent front from escaping.
It doesn't necessarily need to escape for the problem to occur. Well, it does in this example, but it can be trivially rewritten: auto tmp = stdin.byLine; auto lines = tmp.array; Here, `lines` contains references to the buffer owned by `tmp`, but doesn't escape (assuming `array` takes its argument by `scope` or however the final solution will look like).
May 30 2016
parent reply Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On Monday, 30 May 2016 at 10:55:57 UTC, Marc Schütz wrote:
 On Sunday, 29 May 2016 at 14:27:51 UTC, Nick Treleaven wrote:
 What about if the RCArray (of ref count 1) is assigned to a 
 different one after the local ref is initialised? That is what 
 we're discussing -it's your example above(!)
Exactly, and then one of the two suggested approaches will have to be used to prevent the use-after-free. But that's something that needs to happen on assignment, not when the reference is created.
Well my solution does work, disallowing the problematic local refs. But it wouldn't be great for generic code or consistency with the rc DIP. For the latter, we can add a temporary RC object to keep the referenced memory alive, but we only need to do this when the local ref is initialized from a function both (1) returning ref and (2) with a parameter marked return.
 I think we should just prevent front from escaping.
It doesn't necessarily need to escape for the problem to occur. Well, it does in this example, but it can be trivially rewritten: auto tmp = stdin.byLine; auto lines = tmp.array; Here, `lines` contains references to the buffer owned by `tmp`, but doesn't escape (assuming `array` takes its argument by `scope` or however the final solution will look like).
tmp and stdin.byLine are of type ByLine, whose front could be scope/return to prevent escaping. Above array() does escape ByLine.front so can't mark its argument with scope - the compiler would error.
May 30 2016
parent Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On Monday, 30 May 2016 at 14:18:38 UTC, Nick Treleaven wrote:
 On Monday, 30 May 2016 at 10:55:57 UTC, Marc Schütz wrote:
     auto tmp = stdin.byLine;
     auto lines = tmp.array;

 Here, `lines` contains references to the buffer owned by 
 `tmp`, but doesn't escape (assuming `array` takes its argument 
 by `scope` or however the final solution will look like).
tmp and stdin.byLine are of type ByLine, whose front could be scope/return to prevent escaping. Above array() does escape ByLine.front so can't mark its argument with scope - the compiler would error.
Sorry, I meant using array(scope R range) would not affect the compiler error when array attempts to escape range.front into its slice.
May 30 2016
prev sibling parent Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:
On Tuesday, 24 May 2016 at 13:48:41 UTC, Nick Treleaven wrote:
 What about:

  safe unittest
 {
 	RCArray!int arr;
 	ref r = arr[0];
 	arr.destroy; // refcount drops to zero, arr.impl memory freed
 	r++; // writes to unallocated memory
 }
You're calling `destroy` explicitly, what else would you expect to happen?
May 26 2016
prev sibling parent reply deed <none none.none> writes:
On Monday, 23 May 2016 at 15:18:51 UTC, Nick Treleaven wrote:
 If we had local refs, we could use this instead:

 ref m = matrix.rawArr;

 The difference is m is already computed, it is not recomputed 
 each time m is read, unlike M(). I think the reason D doesn't 
 support local refs is because it would make it harder to design 
  safe, particularly with the planned  rc ref-counting. Because 
 M() above is only a return reference, it can't live longer than 
 the data it references. My ref m could persist longer than 
 matrix.rawArr using (naive) reference counting.

 BTW local refs are possible in  system code:
 http://forum.dlang.org/post/lmuokynffgljzvrpvkvd forum.dlang.org
Just to be clear: I'm not looking for anything touching the semantics, just a simple replacement mechanism/macro expansion. Recomputation or not, safe or system, should be no different from the expanded code. So within a scope, after aliasing 'm', 'm' should be replaced by 'matrix.rawArr'. Everything stays the same as is, even error messages. { // Some scope alias i = instance.targetIdx; alias m = matrix.rawArr; m[i] = m[j] + m[k]; // detected and conceptually replaced in one of the earlier compiler passes by // matrix.rawArr[instance.targetIdx] = matrix.rawArr[j] + matrix.rawArr[k] }
May 23 2016
parent Nick Treleaven <ntrel-pub mybtinternet.com> writes:
On Monday, 23 May 2016 at 17:43:49 UTC, deed wrote:
 On Monday, 23 May 2016 at 15:18:51 UTC, Nick Treleaven wrote:
 Recomputation or not,  safe or  system, should be no different 
 from the expanded code. So within a scope, after aliasing 'm', 
 'm' should be replaced by 'matrix.rawArr'. Everything stays the 
 same as is, even error messages.
I think it's more useful to support local refs than alias expressions. In fact apart from the rvalue support, I would rather use a local ref than the existing with statement (assuming the ref was memory safe). The only exception is when using `with (EnumType)`. Also, compile-time expressions use enum, not alias. It could be changed, but I think it's useful when reading meta-programming code to clearly see the difference. Currently aliased symbols are expanded by the compiler in error messages, but that's probably not ideal.
May 24 2016