www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Helping with __mutable (which will be renamed to __metadata)

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Razvan Nitu is working on the DIP initiated by Timon Gehr, known 
colloquially as the one that introduces __mutable - i.e. a mechanism for 
allowing controlled changes to immutable data. Here's a draft:

https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/DIP1xxx-rn.md

We figured that we're dealing a misnomer - we don't want __mutable, but 
instead __metadata - information that is nominally part of the object 
but needs certain leeway from the type system. Typical use cases are:

* reference counting of immutable data structures
* caching
* lazy evaluation

We got stuck at the interaction of __mutable with const parent objects 
(unclear whether the parent object originated as immutable or 
unqualified), and how pure functions should deal with __mutable. The few 
solutions we are toying with are either incomplete or too complicated 
(or both).

The help of a few PL and compiler specialists would be very valuable 
here. I'm cc'ing a few, if anyone wants to help please let us know.
Apr 12 2019
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
I'm concerned that it will be used as an escape from the type system 
arbitrarily. Right now there is nothing to discourage the abuse of this 
storage class.

Is there something we can do to discourage abuse?
Apr 12 2019
parent reply Suleyman <sahmi.soulaimane gmail.com> writes:
On Saturday, 13 April 2019 at 01:17:55 UTC, rikki cattermole 
wrote:
 I'm concerned that it will be used as an escape from the type 
 system arbitrarily. Right now there is nothing to discourage 
 the abuse of this storage class.

 Is there something we can do to discourage abuse?
From skimping through the DIP it doesn't look too dangerous it seems like just a fancy 'shared' escape for immutable fields, the rest looks the same as casting immutable away in system code.
Apr 12 2019
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 13/04/2019 3:45 PM, Suleyman wrote:
 On Saturday, 13 April 2019 at 01:17:55 UTC, rikki cattermole wrote:
 I'm concerned that it will be used as an escape from the type system 
 arbitrarily. Right now there is nothing to discourage the abuse of 
 this storage class.

 Is there something we can do to discourage abuse?
From skimping through the DIP it doesn't look too dangerous it seems like just a fancy 'shared' escape for immutable fields, the rest looks the same as casting immutable away in system code.
What you described is a feature that escapes the type system which can lead to program crashes. Read only memory is what I'm concerned about.
Apr 12 2019
next sibling parent Suleyman <sahmi.soulaimane gmail.com> writes:
On Saturday, 13 April 2019 at 04:49:01 UTC, rikki cattermole 
wrote:
 ...
The DIP says you can edit a `__mutable` only in ` system` mode however if this is just a sugar around an unsafe cast I don't like it I like unsafe code to be ugly and scary.
Apr 12 2019
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 13.04.19 06:49, rikki cattermole wrote:
 On 13/04/2019 3:45 PM, Suleyman wrote:
 On Saturday, 13 April 2019 at 01:17:55 UTC, rikki cattermole wrote:
 I'm concerned that it will be used as an escape from the type system 
 arbitrarily. Right now there is nothing to discourage the abuse of 
 this storage class.

 Is there something we can do to discourage abuse?
 From skimping through the DIP it doesn't look too dangerous it seems like just a fancy 'shared' escape for immutable fields, the rest looks the same as casting immutable away in system code.
What you described is a feature that escapes the type system which can lead to program crashes. Read only memory is what I'm concerned about.
Won't happen, because data with __metadata annotations inside will not be put in read-only memory. (This can be known statically.)
Apr 13 2019
parent reply Suleyman <sahmi.soulaimane gmail.com> writes:
On Saturday, 13 April 2019 at 08:58:06 UTC, Timon Gehr wrote:
 Won't happen, because data with __metadata annotations inside 
 will not be put in read-only memory. (This can be known 
 statically.)
If it's adding these kinds of protections then I think it would be worth it and a step forward in making D more safe. you might also consider the 'shared' protection I mentioned earlier then maybe you can lift the restriction on modification in safe code.
Apr 13 2019
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 14.04.19 02:19, Suleyman wrote:
 On Saturday, 13 April 2019 at 08:58:06 UTC, Timon Gehr wrote:
 Won't happen, because data with __metadata annotations inside will not 
 be put in read-only memory. (This can be known statically.)
If it's adding these kinds of protections then I think it would be worth it and a step forward in making D more safe. you might also consider the 'shared' protection I mentioned earlier then maybe you can lift the restriction on modification in safe code.
I don't think it should be safe. Rather, `pure` and `immutable` should retain their meanings, which implies that there are wrong ways to use `__mutable` (hence unsafe), and there are still non-cosmetic reasons to use `immutable` even if there are `__mutable` fields.
Apr 13 2019
next sibling parent reply Suleyman <sahmi.soulaimane gmail.com> writes:
On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:
 I don't think it should be  safe. Rather, `pure` and 
 `immutable` should retain their meanings,
It cannot be strongly pure. Same as we have weakly pure this is weakly immutable we're dealing with here.
 which implies that there are wrong ways to use `__mutable` 
 (hence unsafe)
can you elaborate?
Apr 14 2019
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 14.04.19 22:55, Suleyman wrote:
 On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:
 I don't think it should be  safe. Rather, `pure` and `immutable` 
 should retain their meanings,
It cannot be strongly pure. Same as we have weakly pure this is weakly immutable we're dealing with here. ...
No, the point is that whatever high-level rewrites you can deduce from `immutable` and `pure` shouldn't be restricted by code that modifies `__metadata`.
 which implies that there are wrong ways to use `__mutable` (hence unsafe)
can you elaborate?
struct S{ private __metadata x; } void foo(immutable ref S s)pure{ s.x += 1; } void main(){ immutable S s; foo(s); // there is no reason for this call to happen assert(s.x==1); // can't rely on this, it might also be 0 } struct S{ private __mutable x; } int foo(immutable ref S s)pure{ s.x += 1; return s.x; } void main(){ immutable S s; int a=foo(s); int b=foo(s); // could just reuse the previous result assert(a!=b); // can't rely on this, might be the same }
Apr 15 2019
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/15/19 7:17 AM, Timon Gehr wrote:
 On 14.04.19 22:55, Suleyman wrote:
 On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:
 which implies that there are wrong ways to use `__mutable` (hence 
 unsafe)
can you elaborate?
struct S{     private __metadata x; } void foo(immutable ref S s)pure{     s.x += 1; } void main(){     immutable S s;     foo(s); // there is no reason for this call to happen     assert(s.x==1); // can't rely on this, it might also be 0 } struct S{     private __mutable x; } int foo(immutable ref S s)pure{     s.x += 1;     return s.x; } void main(){     immutable S s;     int a=foo(s);     int b=foo(s); // could just reuse the previous result     assert(a!=b); // can't rely on this, might be the same }
Note that this is exactly the use case of reference counting. Which is the main draw of having a mutable portion of an immutable. I would hazard to guess that if the above has to be the semantics, this is DOA. -Steve
Apr 15 2019
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 15.04.19 14:32, Steven Schveighoffer wrote:
 On 4/15/19 7:17 AM, Timon Gehr wrote:
 On 14.04.19 22:55, Suleyman wrote:
 On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:
 which implies that there are wrong ways to use `__mutable` (hence 
 unsafe)
can you elaborate?
struct S{      private __metadata x; } void foo(immutable ref S s)pure{      s.x += 1; } void main(){      immutable S s;      foo(s); // there is no reason for this call to happen      assert(s.x==1); // can't rely on this, it might also be 0 } struct S{      private __mutable x; } int foo(immutable ref S s)pure{      s.x += 1;      return s.x; } void main(){      immutable S s;      int a=foo(s);      int b=foo(s); // could just reuse the previous result      assert(a!=b); // can't rely on this, might be the same }
Note that this is exactly the use case of reference counting. Which is the main draw of having a mutable portion of an immutable. I would hazard to guess that if the above has to be the semantics, this is DOA. -Steve
No, this is clearly not the use case of reference counting. Where do you see references that are being counted? Why should the fact that the data structure is reference counted block optimizations such as eliding reference copies?
Apr 15 2019
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/15/19 8:49 AM, Timon Gehr wrote:
 On 15.04.19 14:32, Steven Schveighoffer wrote:
 On 4/15/19 7:17 AM, Timon Gehr wrote:
 On 14.04.19 22:55, Suleyman wrote:
 On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:
 which implies that there are wrong ways to use `__mutable` (hence 
 unsafe)
can you elaborate?
struct S{      private __metadata x; } void foo(immutable ref S s)pure{      s.x += 1; } void main(){      immutable S s;      foo(s); // there is no reason for this call to happen      assert(s.x==1); // can't rely on this, it might also be 0 } struct S{      private __mutable x; } int foo(immutable ref S s)pure{      s.x += 1;      return s.x; } void main(){      immutable S s;      int a=foo(s);      int b=foo(s); // could just reuse the previous result      assert(a!=b); // can't rely on this, might be the same }
Note that this is exactly the use case of reference counting. Which is the main draw of having a mutable portion of an immutable. I would hazard to guess that if the above has to be the semantics, this is DOA.
No, this is clearly not the use case of reference counting. Where do you see references that are being counted? Why should the fact that the data structure is reference counted block optimizations such as eliding reference copies?
I mean when you add/remove reference for an immutable, it's passing an immutable to what needs to be a pure function, which then increments/decrements a __metadata field (just like your examples above). If one of those 2 gets elided, the reference count is hosed. -Steve
Apr 15 2019
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 15.04.19 14:56, Steven Schveighoffer wrote:
 On 4/15/19 8:49 AM, Timon Gehr wrote:
 On 15.04.19 14:32, Steven Schveighoffer wrote:
 ...
No, this is clearly not the use case of reference counting. Where do you see references that are being counted? Why should the fact that the data structure is reference counted block optimizations such as eliding reference copies?
I mean when you add/remove reference for an immutable, it's passing an immutable to what needs to be a pure function, which then increments/decrements a __metadata field (just like your examples above). If one of those 2 gets elided, the reference count is hosed. -Steve
I know, and this is why my original DIP draft had the concept of a __mutable function.
Apr 15 2019
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/15/19 9:23 AM, Timon Gehr wrote:
 On 15.04.19 14:56, Steven Schveighoffer wrote:
 On 4/15/19 8:49 AM, Timon Gehr wrote:
 On 15.04.19 14:32, Steven Schveighoffer wrote:
 ...
No, this is clearly not the use case of reference counting. Where do you see references that are being counted? Why should the fact that the data structure is reference counted block optimizations such as eliding reference copies?
I mean when you add/remove reference for an immutable, it's passing an immutable to what needs to be a pure function, which then increments/decrements a __metadata field (just like your examples above). If one of those 2 gets elided, the reference count is hosed.
I know, and this is why my original DIP draft had the concept of a __mutable function.
Ah, ok. So a __mutable function is not strong-pure. That would work. -Steve
Apr 15 2019
prev sibling parent reply Suleyman <sahmi.soulaimane gmail.com> writes:
On Monday, 15 April 2019 at 11:17:55 UTC, Timon Gehr wrote:
 On 14.04.19 22:55, Suleyman wrote:
 ...
No, the point is that whatever high-level rewrites you can deduce from `immutable` and `pure` shouldn't be restricted by code that modifies `__metadata`. ...
Just making it system is not helping much instead of evading reality you rather treat as it really is an immutable ref with mutable fields hence the compiler must not elide function calls with these weakly immutable arguments.
Apr 15 2019
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 15.04.19 17:08, Suleyman wrote:
 On Monday, 15 April 2019 at 11:17:55 UTC, Timon Gehr wrote:
 On 14.04.19 22:55, Suleyman wrote:
 ...
No, the point is that whatever high-level rewrites you can deduce from `immutable` and `pure` shouldn't be restricted by code that modifies `__metadata`. ...
Just making it system is not helping much instead of evading reality you rather treat as it really is an immutable ref with mutable fields hence the compiler must not elide function calls with these weakly immutable arguments.
At that point, why do you need to slap `immutable` on your type in the first place? Why does your proposed semantics make any sense for reference counting or caching/lazy evaluation?
Apr 15 2019
next sibling parent reply Suleyman <sahmi.soulaimane gmail.com> writes:
On Monday, 15 April 2019 at 15:59:35 UTC, Timon Gehr wrote:
 At that point, why do you need to slap `immutable` on your type 
 in the first place? Why does your proposed semantics make any 
 sense for reference counting or caching/lazy evaluation?
I don't know but immutability has been breached for sure and hiding that with system doesn't help I would say disallowing __mutable inside immutable objects makes more sense than just concealing the crime with system unless someone has a safe solution.
Apr 15 2019
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 15.04.19 18:37, Suleyman wrote:
 On Monday, 15 April 2019 at 15:59:35 UTC, Timon Gehr wrote:
 At that point, why do you need to slap `immutable` on your type in the 
 first place? Why does your proposed semantics make any sense for 
 reference counting or caching/lazy evaluation?
I don't know but immutability has been breached for sure
No. It's like saying Haskell does not support immutable data because the only way to implement lazy evaluation on the machine is using mutation. (Or even worse, to claim that D does not support immutability because the garbage collector can reclaim memory that was previously typed immutable.) I think D needs to be able to implement its own runtime library and manual memory allocation schemes without compiler magic unavailable to user programs.
 and hiding that with  system doesn't help
Nothing is being hidden.
 I would say disallowing __mutable inside 
 immutable objects  makes more sense
Regarding sense: I have yet to find any in your "concealing" argument. I have tried and wasted too much time now.
 than just concealing the crime with  system
system means you are not protected against writing programs that do not have defined semantics. Seems like a good enough fit.
 unless someone has a safe solution.
No, that makes no sense at all. The only simple enough safe solution is to turn all __metadata use cases into compiler magic one-by-one. And you win nothing. The simplest way to implement that is to just add __metadata and simply add a compiler check that ensures user code cannot access it.
Apr 15 2019
prev sibling parent reply Suleyman <sahmi.soulaimane gmail.com> writes:
On Monday, 15 April 2019 at 15:59:35 UTC, Timon Gehr wrote:
 At that point, why do you need to slap `immutable` on your type 
 in the first place? Why does your proposed semantics make any 
 sense for reference counting or caching/lazy evaluation?
Otherwise the sole protection from being stored in readonly memory is not enough incentive for adding a keyword feature if this is the only contribution beyond an unsafe cast then a compiler directive should suffice such as pragma(noROM) or similar.
Apr 15 2019
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 15.04.19 19:24, Suleyman wrote:
 On Monday, 15 April 2019 at 15:59:35 UTC, Timon Gehr wrote:
 At that point, why do you need to slap `immutable` on your type in the 
 first place? Why does your proposed semantics make any sense for 
 reference counting or caching/lazy evaluation?
Otherwise the sole protection from being stored in readonly memory
This is not true and it is not even the right way to think about it. The language semantics is distinct from how it is implemented in a compiler. The reason why data with __mutable fields cannot be put in readonly memory is because crash on write is not acceptable behavior. It does not even need to be explicitly specified.
 is not enough incentive for adding a keyword feature
The cost of adding a __keyword is pretty close to zero. And anyway, that's a superficial syntactic concern, not even really worth debating over at this point, as a) the current version of the DIP gets the semantics wrong and b) the necessary changes to the language specification are of similar size whether you add a __keyword or a pragma. (And whether or not you add a tiny bit more type checking is independent of the annotation syntax.)
 if this is the only contribution beyond an unsafe cast
This is not what's being proposed. There are unsafe operations and there are invalid operations. The DIP's aim is to make some cases unsafe that were previously invalid. (And if you have to implement an annotation for fields anyway, adding the handful lines of frontend code that avoid additional casts, which are a blunt instrument that can hide what would otherwise be type errors, e.g. during a refactoring, does not seem bad enough to me to not even be counted as a contribution.)
 then a compiler directive should suffice such as pragma(noROM) or similar.
A pragma is sufficient (but not required) anyway, but noROM is a terrible name.
Apr 15 2019
prev sibling parent reply Doc Andrew <x x.com> writes:
On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:
 On 14.04.19 02:19, Suleyman wrote:
 On Saturday, 13 April 2019 at 08:58:06 UTC, Timon Gehr wrote:
 Won't happen, because data with __metadata annotations inside 
 will not be put in read-only memory. (This can be known 
 statically.)
If it's adding these kinds of protections then I think it would be worth it and a step forward in making D more safe. you might also consider the 'shared' protection I mentioned earlier then maybe you can lift the restriction on modification in safe code.
I don't think it should be safe. Rather, `pure` and `immutable` should retain their meanings, which implies that there are wrong ways to use `__mutable` (hence unsafe), and there are still non-cosmetic reasons to use `immutable` even if there are `__mutable` fields.
A few questions/thoughts (or at least food for thought for others more informed than me): 1. Maybe it's just me, but why the leading underscores? I'm wondering if this is because it's a scary/ugly-on-purpose feature like __gshared that is intended only sparingly? 1b. A lot of people have been clamoring for immutable by default, might keeping the DIP's original annotation "mutable" (with no underscores) make this a step in that direction? (with all the baggage and argument that carries with it, which is out of scope for this discussion). Later on, if the "mutable" qualifier is desired, we aren't carrying around both mutable and __mutable, or __mutable and mut or whatever ends up being the flavor of the day. 1c. Having said that, I think the "mutable" discussion and the "metadata" discussion are two separate things - which I think Andrei alluded to in the keyword change - this is potentially much bigger than just short-circuiting Immutability for a good cause. 2. If the __metadata qualifier can only be applied to private fields, and carries special semantics w.r.t. escaping the type system, is there any benefit to treating it more like a visibility attribute? struct something { metadata // maybe type qualifiers disallowed in here { int a; } private { int b; } public { int c; } } And... maybe metadata shouldn't be stored with the object at all? This way, it doesn't affect the layout of the object in memory, and doesn't require the need to short-circuit the type system if a later use of the class is marked Immutable. The object's metadata lives, not sorta-outside the type system, but outside the object itself? There are some challenges I can think of with this approach that I can expand on, but also some benefits. -Doc
Apr 14 2019
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 15.04.19 06:05, Doc Andrew wrote:
 On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:
 On 14.04.19 02:19, Suleyman wrote:
 On Saturday, 13 April 2019 at 08:58:06 UTC, Timon Gehr wrote:
 Won't happen, because data with __metadata annotations inside will 
 not be put in read-only memory. (This can be known statically.)
If it's adding these kinds of protections then I think it would be worth it and a step forward in making D more safe. you might also consider the 'shared' protection I mentioned earlier then maybe you can lift the restriction on modification in safe code.
I don't think it should be safe. Rather, `pure` and `immutable` should retain their meanings, which implies that there are wrong ways to use `__mutable` (hence unsafe), and there are still non-cosmetic reasons to use `immutable` even if there are `__mutable` fields.
A few questions/thoughts (or at least food for thought for others more informed than me): 1. Maybe it's just me, but why the leading underscores? I'm wondering if this is because it's a scary/ugly-on-purpose feature like __gshared that is intended only sparingly? ...
Yes. Also, it means we don't reserve a previously unreserved identifier.
 1b. A lot of people have been clamoring for immutable by default, might 
 keeping the DIP's original annotation "mutable" (with no underscores) 
 make this a step in that direction? (with all the baggage and argument 
 that carries with it, which is out of scope for this discussion).
 ...
It's not the same.
 
 2. If the __metadata qualifier can only be applied to private fields, 
 and carries special semantics w.r.t. escaping the type system, is there 
 any benefit to treating it more like a visibility attribute?
 
 struct something
 {
      metadata        // maybe type qualifiers disallowed in here
      {
          int a;
      }
      private
      {
          int b;
      }
      public
      {
          int c;
      }
 }
 ...
Is your suggestion just to make __metadata imply private, or is there anything else you are trying to say?
 And... maybe metadata shouldn't be stored with the object at all? This 
 way, it doesn't affect the layout of the object in memory, and doesn't 
 require the need to short-circuit the type system if a later use of the 
 class is marked Immutable. The object's metadata lives, not 
 sorta-outside the type system, but outside the object itself? There are 
 some challenges I can think of with this approach that I can expand on, 
 but also some benefits.
 
 -Doc
Storing the data within the object is the point. As the original post states, the motivation for the DIP is to allow memory allocation schemes such as reference counting, as well as lazy initialization to be implemented for `immutable`-qualified data structures. The mutability is supposed to be an implementation detail -- from the outside, the data will still appear to be `immutable`. (The current version of the DIP completely misses this point though, leaking __metadata into the type system.) Compiler optimizations can change the semantics of code that operates on __metadata. (e.g., if it is able to elide a read to a lazily-initialized field, that field will not be initialized, if it elides a reference copy, the reference count does not need to be updated, etc.) Therefore, modifications of __metadata need to be consistent with rewrites that are valid for strongly pure functions.
Apr 15 2019
parent Doc Andrew <x x.com> writes:
On Monday, 15 April 2019 at 11:09:23 UTC, Timon Gehr wrote:
 
 A few questions/thoughts (or at least food for thought for 
 others more informed than me):
 
 1. Maybe it's just me, but why the leading underscores? I'm 
 wondering if this is because it's a scary/ugly-on-purpose 
 feature like __gshared that is intended only sparingly?
 ...
Yes. Also, it means we don't reserve a previously unreserved identifier.
 1b. A lot of people have been clamoring for immutable by 
 default, might keeping the DIP's original annotation "mutable" 
 (with no underscores) make this a step in that direction? 
 (with all the baggage and argument that carries with it, which 
 is out of scope for this discussion).
 ...
It's not the same.
Oh no, I'm just suggesting that, under the original proposed DIP, the __mutable keyword might as well be made "mutable", which may be useful later on for other purposes. The metadata concept could be a lot more powerful, though, and I think it's basically orthogonal to the mutability concern:
 
 2. If the __metadata qualifier can only be applied to private 
 fields, and carries special semantics w.r.t. escaping the type 
 system, is there any benefit to treating it more like a 
 visibility attribute?
 
 struct something
 {
      metadata        // maybe type qualifiers disallowed in 
 here
      {
          int a;
      }
      private
      {
          int b;
      }
      public
      {
          int c;
      }
 }
 ...
Is your suggestion just to make __metadata imply private, or is there anything else you are trying to say?
Not just private, but it would be a way to annotate data that lives "outside" of the POD of the class. For instance, serializing an object with variables in the metadata{} block wouldn't (or couldn't?) include them. Another idea I had this morning is that the metadata{} block might be a nice place to put class/function attributes to avoid cluttering the signatures: class Foo { metadata { synchronized; // can move class attributes in here scope; (someUDA); // UDA's live here too int secretSauce; // Can always be accessed or modified by Foo, regardless of // mutability } } The concept fits nicely with contracts and functions, too. Just as metadata{} blocks in a immutable object allow modification of those member variables, maybe member variables in a function metadata{} block can be modified and persisted across calls for pure functions, too. It would act just like a static variable, available at both compile-time and run-time, maybe useful for things like debugging, profiling or benchmarking. Maybe code in the in & out contract blocks can access it, but we'd make it illegal for the function body to do it. I'd need to think through the side-effects a little more. int Bar(int a, int b) metadata { pure: // Not 100% sure what syntax would make sense nogc: // maybe just force for everything here safe: long startTicks; // Stuff for benchmarking long endTicks; // Not pertinent to the function itself static int numTimesCalled; // This is a pure function... } in { numTimesCalled++; // ...but it's OK, we're just modifying it's metadata ... } do { if(numTimesCalled > 4) // Illegal to do this here, breaks purity ... }
 And... maybe metadata shouldn't be stored with the object at 
 all? This way, it doesn't affect the layout of the object in 
 memory, and doesn't require the need to short-circuit the type 
 system if a later use of the class is marked Immutable. The 
 object's metadata lives, not sorta-outside the type system, 
 but outside the object itself? There are some challenges I can 
 think of with this approach that I can expand on, but also 
 some benefits.
 
 -Doc
Storing the data within the object is the point. As the original post states, the motivation for the DIP is to allow memory allocation schemes such as reference counting, as well as lazy initialization to be implemented for `immutable`-qualified data structures. The mutability is supposed to be an implementation detail -- from the outside, the data will still appear to be `immutable`. (The current version of the DIP completely misses this point though, leaking __metadata into the type system.) Compiler optimizations can change the semantics of code that operates on __metadata. (e.g., if it is able to elide a read to a lazily-initialized field, that field will not be initialized, if it elides a reference copy, the reference count does not need to be updated, etc.) Therefore, modifications of __metadata need to be consistent with rewrites that are valid for strongly pure functions.
Sure, I'm just suggesting that keeping that reference count or other metadata separately is a possible workaround - as long as there's an easy way to get the metadata for a given object (whether in memory alongside the rest of the object's data or not). The idea of storing metadata member variables separately is an implementation detail. To the class, it appears that the metadata members are no different than regular variables, but casting to void* or serializing it leaves out the metadata, so it's a safe place to keep things like reference counts. In this scheme, metadata can be added to classes or structs without changing their memory layout, and metadata vars have some other special powers as far as being treated differently by the type system, strictly because they are stored "outside" the object memory, not because of a special-case in the type system. -Doc
Apr 15 2019
prev sibling next sibling parent reply Suleyman <sahmi.soulaimane gmail.com> writes:
On Saturday, 13 April 2019 at 00:45:07 UTC, Andrei Alexandrescu 
wrote:
 ...
The problems you encountered are normal but the question is whether this feature is useful at all if the field is not explicitly shared? If so the a solution to the const case is to introduce an ' unshared' attribute instead which would protect against the worst cases of both being shared and being thread local. this attribute would disallow unsychronized read/write and at the same time prevent implicit conversion to 'shared' since it may just be thread local hence mixing it with normal 'shared' may not bring good results. example: ``` struct S { private shared int* p; // or unshared void inc() inout pure system { import core.atomic; atomicOp!"+="(*cast(shared(int*))p, 1); // or unshared } } void foo(ref immutable S a) pure { a.inc(); } void foo(ref const S a) pure { a.inc(); } int x; // thread local shared int y; void main() { auto i = immutable S(cast(immutable)&x); auto c = const S(&y); //auto d = const S(cast( unshared)&x); foo(i); foo(c); } ```
Apr 12 2019
parent reply Radu <void null.pt> writes:
On Saturday, 13 April 2019 at 03:45:35 UTC, Suleyman wrote:
 On Saturday, 13 April 2019 at 00:45:07 UTC, Andrei Alexandrescu 
 wrote:
 ...
The problems you encountered are normal but the question is whether this feature is useful at all if the field is not explicitly shared? If so the a solution to the const case is to introduce an ' unshared' attribute instead which would protect against the worst cases of both being shared and being thread local. this attribute would disallow unsychronized read/write and at the same time prevent implicit conversion to 'shared' since it may just be thread local hence mixing it with normal 'shared' may not bring good results. example: ``` struct S { private shared int* p; // or unshared void inc() inout pure system { import core.atomic; atomicOp!"+="(*cast(shared(int*))p, 1); // or unshared } } void foo(ref immutable S a) pure { a.inc(); } void foo(ref const S a) pure { a.inc(); } int x; // thread local shared int y; void main() { auto i = immutable S(cast(immutable)&x); auto c = const S(&y); //auto d = const S(cast( unshared)&x); foo(i); foo(c); } ```
I also have the impression that `shared` should be used here instead of `__mutable`. Even in the current incarnation shared offers some limited guarantees that atomicity is required for writing shared values. I think people are working on making stronger guarantees for shared. ATM uses like the one below are a no-op, I think the reuse of shared makes sense as it works nicely with the strong thread safety guarantees `immutable` has, and also offers the safety net for atomic changes. Not to mention that it is not ugly! immutable struct Foo { shared(int) x; }
Apr 15 2019
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/15/19 9:14 AM, Radu wrote:

 I also have the impression that `shared` should be used here instead of 
 `__mutable`. Even in the current incarnation shared offers some limited 
 guarantees that atomicity is required for writing shared values. I think 
 people are working on making stronger guarantees for shared.
 
Only immutable structs would have shared metadata. Normal mutable structs could have thread-local metadata. So that doesn't work. -Steve
Apr 15 2019
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 13.04.19 02:45, Andrei Alexandrescu wrote:
 Razvan Nitu is working on the DIP initiated by Timon Gehr,
(Disclaimer: Note that the current state of the DIP is significantly different from my original proposal, and I would not argue for acceptance of the current state of the DIP even though I am listed as an author.)
 known 
 colloquially as the one that introduces __mutable - i.e. a mechanism for 
 allowing controlled changes to immutable data. Here's a draft:
 
 https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/DIP1xxx-rn.md
 
 We figured that we're dealing a misnomer - we don't want __mutable, but 
 instead __metadata - information that is nominally part of the object 
 but needs certain leeway from the type system. Typical use cases are:
 
 * reference counting of immutable data structures
For that you need the more general (originally supported using __mutable functions): * manual allocation and deallocation of immutable data
 * caching
 * lazy evaluation
 
 We got stuck at the interaction of __mutable with const parent objects 
 (unclear whether the parent object originated as immutable or 
 unqualified),
I think we got that one sorted out.
 and how pure functions should deal with __mutable. The few 
 solutions we are toying with are either incomplete or too complicated 
 (or both).
 
 The help of a few PL and compiler specialists would be very valuable 
 here. I'm cc'ing a few, if anyone wants to help please let us know.
If the principled approach is too complicated, the feature is not worth it. My original proposal was to define a set of rewrites that is based on the function signature alone which is semantics-preserving if there is no __metadata (except possibly reference identity of immutable data), and then say that this set of rewrites is still permissible even if there are __metadata annotations. If complexity of the DIP is a concern, I guess we could simplify this by just saying that any rewrite based on the function signature alone that is sound without __metadata will remain permissible with it.
Apr 13 2019
prev sibling next sibling parent reply RazvanN <razvan.nitu1305 gmail.com> writes:
On Saturday, 13 April 2019 at 00:45:07 UTC, Andrei Alexandrescu 
wrote:
 Razvan Nitu is working on the DIP initiated by Timon Gehr, 
 known colloquially as the one that introduces __mutable - i.e. 
 a mechanism for allowing controlled changes to immutable data. 
 Here's a draft:

 https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/DIP1xxx-rn.md

 We figured that we're dealing a misnomer - we don't want 
 __mutable, but instead __metadata - information that is 
 nominally part of the object but needs certain leeway from the 
 type system. Typical use cases are:

 * reference counting of immutable data structures
 * caching
 * lazy evaluation

 We got stuck at the interaction of __mutable with const parent 
 objects (unclear whether the parent object originated as 
 immutable or unqualified), and how pure functions should deal 
 with __mutable. The few solutions we are toying with are either 
 incomplete or too complicated (or both).

 The help of a few PL and compiler specialists would be very 
 valuable here. I'm cc'ing a few, if anyone wants to help please 
 let us know.
This is a draft proposal and it is not finished. The initial draft that was done by Timon [1] considers __mutable functions that act as optimization blockers for the compiler with regards to `pure` functions. I feel that optimization blockers for `pure` are a totally different concept than __mutable/__metadata fields. Considering that the currently the compiler does not do any optimizations based on purity, I think it would be a lot easier to just get __mutable/__metadata fields in, because they are a needed feature; after that, we can think on what optimizations we want to implement and how they interact with __mutable/__mutable. Consider this code: int foo() pure { immutable(T)* x = allocate(); int y = bar(x); deallocate(x); return y; } Currently, the compiler does not do any optimizations regarding purity with it. If it did, it could have swapped the invocation of `deallocate` with the one of `bar` leading to use-after-free. Annotating `deallocate` with __mutable would solve the issue in the event of compiler optimizations. However, this example has nothing to do with __mutable/__metadata fields; this kind of optimizations should be discussed in a DIP of their own and not in a __mutable/__metadata DIP. Moreover, is this the direction we want to head in? Require the user to mentally trace the optimizations that the compiler might do? This is just too complicated. That is why, I think that we should focus on implementing __metadata with regards to fields and later on think about the optimizations that we can perform. As for __metadata fields: 1. If we decide that __metadata fields are not conceptually part of the object, why would accesses to them be unsafe? We could still make them `private`, but we can view them as normal accesses from a safety perspective. 2. I agree that `purity` should not be affected by __mutable/__metadata fields; the object passed as argument will not be conceptually modified. If we take out optimizations out of the picture, things become a lot more clearer. For me, the DIP is more of "how can we mutate fields in a non-mutable object", not "how do we implement optimizations regarding purity". I don't see why the addition of __metadata should be delayed by how optimizations based on purity are applied in the compiler, Cheers, RazvanN [1] https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/timon_dip.md
Apr 16 2019
next sibling parent reply Suleyman <sahmi.soulaimane gmail.com> writes:
On Tuesday, 16 April 2019 at 09:51:33 UTC, RazvanN wrote:
 ...

 Cheers,
 RazvanN
What it is that this DIP enables you to do that you can't already do in system code?
Apr 16 2019
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/16/19 4:15 PM, Suleyman wrote:
 On Tuesday, 16 April 2019 at 09:51:33 UTC, RazvanN wrote:
 ...

 Cheers,
 RazvanN
What it is that this DIP enables you to do that you can't already do in system code?
A simple answer is with the DIP you get defined behavior.
Apr 16 2019
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 16.04.19 11:51, RazvanN wrote:
 On Saturday, 13 April 2019 at 00:45:07 UTC, Andrei Alexandrescu wrote:
 Razvan Nitu is working on the DIP initiated by Timon Gehr, known 
 colloquially as the one that introduces __mutable - i.e. a mechanism 
 for allowing controlled changes to immutable data. Here's a draft:

 https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/DIP1xxx-rn.md

 We figured that we're dealing a misnomer - we don't want __mutable, 
 but instead __metadata - information that is nominally part of the 
 object but needs certain leeway from the type system. Typical use 
 cases are:

 * reference counting of immutable data structures
 * caching
 * lazy evaluation

 We got stuck at the interaction of __mutable with const parent objects 
 (unclear whether the parent object originated as immutable or 
 unqualified), and how pure functions should deal with __mutable. The 
 few solutions we are toying with are either incomplete or too 
 complicated (or both).

 The help of a few PL and compiler specialists would be very valuable 
 here. I'm cc'ing a few, if anyone wants to help please let us know.
This is a draft proposal and it is not finished. The initial draft that was done by Timon [1] considers __mutable functions that act as optimization blockers for the compiler with regards to `pure` functions. I feel that optimization blockers for `pure` are a totally different concept than __mutable/__metadata fields. ...
On the contrary, it's the type system information you _need_ in order to abstract over code that mutates __metadata. An assignment to a __metadata field is something you cannot ignore even though it is `pure` and does not return any interesting value. For any statement, you need to be able to assign a valid type to a function that contains this statement, and that captures what the type system knows about the statement.
 Considering that the currently the compiler does not do any 
 optimizations based on purity, I think it would be a lot easier to just 
 get __mutable/__metadata fields
 in, because they are a needed feature; after that, we can think on what 
 optimizations we want to implement and how they interact with 
 __mutable/__mutable.
 ...
You need to specify that there are optimizations that are allowed even though they change the set of mutations that the program performs. As I suggested earlier, if you want the semantics that's least restrictive for language evolution, you need to say something that implies that all optimizations based on the function signature are allowed, not "the compiler currently does not do any optimizations". That's precisely the wrong direction if you want to be conservative.
 Consider this code:
 
 int foo() pure
 {
       immutable(T)* x = allocate();
       int y = bar(x);
       deallocate(x);
       return y;
 }
 
 Currently, the compiler does not do any optimizations regarding purity 
 with it. If it did, it could have swapped the invocation of `deallocate` 
 with the one of `bar` leading to use-after-free.
What the compiler does and does not do has no bearing on the validity of the code per language semantics.
 Annotating `deallocate` 
 with __mutable would solve the issue in the event of compiler 
 optimizations. However, this example has nothing to do with 
 __mutable/__metadata fields; this kind of optimizations should be 
 discussed in a DIP of their own and not in a __mutable/__metadata DIP. 
That's simply nonsense. You define the language semantics, the compiler developer figures out optimizations that are consistent with that semantics. You don't need any DIPs for optimizations, you need DIPs to specify how programs may or may not behave. You can't really first come and say programs may not behave a certain way and then later relax that and say that actually those programs can also behave in this other way and at the same time claim that this is a conservative approach. It is not; it breaks code.
 Moreover,
 is this the direction we want to head in?
Absolutely. If you want to use __metadata you need to be aware that pure functions can be elided, and if the mutation of __metadata within a function is not self-contained such that it can be elided, you need to annotate that function.
 Require the user to mentally 
 trace the optimizations that the compiler might do?
The user of __metadata. _Some_ kinds of optimizations.
 This is just too complicated.
 ...
Then give up on __metadata and just implement your persistent data structures without the `immutable` qualifier slapped on top! `immutable` has no value if it cannot be used to justify equational reasoning on pure functions.
 That is why, I think that we should focus on implementing __metadata 
 with regards to fields and later on think about the optimizations that 
 we can perform.
 
 As for __metadata fields:
 
 1. If we decide that __metadata fields are not conceptually part of the
 object, why would accesses to them be unsafe? We could  still make them 
 `private`,
 but we can view them as normal accesses from a  safety perspective.
 ...
Then those accesses can't be pure because they are like accesses to global variables.
 2. I agree that `purity` should not be affected by __mutable/__metadata 
 fields; the object passed as argument will not be conceptually modified.
 ...
1 and 2 are fundamentally at odds with each other if you want __mutable fields to be accessible in pure functions, because you can't check 2 safely.
 If we take out optimizations out of the picture, things become a lot 
 more clearer.
You need to define the semantics!
 For me, the DIP is more of "how can we mutate fields in a non-mutable 
 object", not
You clearly can't mutate fields in a non-mutable object except non-mutable is some higher-level abstraction. That abstraction has to be defined. One way to define semantics for __metadata is to explicitly give the set of allowed rewrites, but you can also define it implicitly.
 "how do we implement optimizations regarding purity".
 
 I don't see why the addition of __metadata should be delayed by how 
 optimizations based on purity are applied in the compiler,
 
 ...
It shouldn't.
Apr 16 2019
parent RazvanN <razvan.nitu1305 gmail.com> writes:
On Wednesday, 17 April 2019 at 01:00:35 UTC, Timon Gehr wrote:
 On the contrary, it's the type system information you _need_ in 
 order to abstract over code that mutates __metadata. An 
 assignment to a __metadata field is something you cannot ignore 
 even though it is `pure` and does not return any interesting 
 value. For any statement, you need to be able to assign a valid 
 type to a function that contains this statement, and that 
 captures what the type system knows about the statement.

 You need to specify that there are optimizations that are 
 allowed even though they change the set of mutations that the 
 program performs. As I suggested earlier, if you want the 
 semantics that's least restrictive for language evolution, you 
 need to say something that implies that all optimizations based 
 on the function signature are allowed, not "the compiler 
 currently does not do any optimizations". That's precisely the 
 wrong direction if you want to be conservative.
I see, now I understand.
 What the compiler does and does not do has no bearing on the 
 validity of the code per language semantics.

 That's simply nonsense. You define the language semantics, the 
 compiler developer figures out optimizations that are 
 consistent with that semantics. You don't need any DIPs for 
 optimizations, you need DIPs to specify how programs may or may 
 not behave. You can't really first come and say programs may 
 not behave a certain way and then later relax that and say that 
 actually those programs can also behave in this other way and 
 at the same time claim that this is a conservative approach. It 
 is not; it breaks code.
I understand now. It was very confusing for me that `pure` was simply implemented as a set of compile time checks not backed by any optimizations. I figured that if none of them were implemented by now, it will certainly break a lot of code when we will have them, so there is a great chance that `pure` will simply remain in the current stage. If that would be the case, adding the cognitive load for __metadata functions to the user is not worth it.
 Moreover,
 is this the direction we want to head in?
Absolutely. If you want to use __metadata you need to be aware that pure functions can be elided, and if the mutation of __metadata within a function is not self-contained such that it can be elided, you need to annotate that function.
 Require the user to mentally trace the optimizations that the 
 compiler might do?
The user of __metadata. _Some_ kinds of optimizations.
But it's not just the user of __metadata. If you implement in a library a refcounted object then __metadata is going to be propagated to user code.
 This is just too complicated.
 ...
Then give up on __metadata and just implement your persistent data structures without the `immutable` qualifier slapped on top! `immutable` has no value if it cannot be used to justify equational reasoning on pure functions.
 That is why, I think that we should focus on implementing 
 __metadata with regards to fields and later on think about the 
 optimizations that we can perform.
 
 As for __metadata fields:
 
 1. If we decide that __metadata fields are not conceptually 
 part of the
 object, why would accesses to them be unsafe? We could  still 
 make them `private`,
 but we can view them as normal accesses from a  safety 
 perspective.
 ...
Then those accesses can't be pure because they are like accesses to global variables.
But they actually aren't because they are "physically" part of the object.
 2. I agree that `purity` should not be affected by 
 __mutable/__metadata fields; the object passed as argument 
 will not be conceptually modified.
 ...
1 and 2 are fundamentally at odds with each other if you want __mutable fields to be accessible in pure functions, because you can't check 2 safely.
 If we take out optimizations out of the picture, things become 
 a lot more clearer.
You need to define the semantics!
What I was trying to say is that it would be easier to define semantics if we wouldn't pretend that the compiler treats `pure` in a way that it actually doesn't. I see your point, however, even if we have __metadata/ __mutable functions, once the purity optimizations will be implemented, all the users will have to check their code to add __metadata/__mutable to their functions to be sure their calls to strongly pure functions are not elided.
 For me, the DIP is more of "how can we mutate fields in a 
 non-mutable object", not
You clearly can't mutate fields in a non-mutable object except non-mutable is some higher-level abstraction. That abstraction has to be defined. One way to define semantics for __metadata is to explicitly give the set of allowed rewrites, but you can also define it implicitly.
 "how do we implement optimizations regarding purity".
 
 I don't see why the addition of __metadata should be delayed 
 by how optimizations based on purity are applied in the 
 compiler,
 
 ...
It shouldn't.
From an academical perspective, I agree with everything you said and I think I finally understand your point of view and your DIP draft. I think that a small chat would have made it easier for both parties to reach an understanding. At this point, I think it all comes down to what we want to do with `pure`. It really doesn't make any sense to take all these optimizations into account and burden the user with __mutable functions if those optimizations will never be implemented. Cheers, RazvanN
Apr 17 2019
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/12/19 8:45 PM, Andrei Alexandrescu wrote:
 Razvan Nitu is working on the DIP initiated by Timon Gehr, known 
 colloquially as the one that introduces __mutable - i.e. a mechanism for 
 allowing controlled changes to immutable data. Here's a draft:
 
 https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/DIP1xxx-rn.md
 
 We figured that we're dealing a misnomer - we don't want __mutable, but 
 instead __metadata - information that is nominally part of the object 
 but needs certain leeway from the type system. Typical use cases are:
 
 * reference counting of immutable data structures
 * caching
 * lazy evaluation
 
 We got stuck at the interaction of __mutable with const parent objects 
 (unclear whether the parent object originated as immutable or 
 unqualified), and how pure functions should deal with __mutable. The few 
 solutions we are toying with are either incomplete or too complicated 
 (or both).
 
 The help of a few PL and compiler specialists would be very valuable 
 here. I'm cc'ing a few, if anyone wants to help please let us know.
There seems to be a good amount of interest in the forum. Thanks! To better move things forward, we'd need to have a more fleshed-out proposal to show the community for discussion. Until then, let's move to a special interest group. Those interested who can commit one hour a week to a meeting, please email me. We'll exchange information via email (mailing list if necessary) and meet every week to discuss progress.
Apr 16 2019
parent reply Suleyman <sahmi.soulaimane gmail.com> writes:
It looks to me that this proposal breaks everything that purity 
stands for and I don't have a real solution to it. I'm not 
against breaking the rules when necessary I just don't see why we 
need a DIP for it.

On Tuesday, 16 April 2019 at 20:40:58 UTC, Andrei Alexandrescu 
wrote:
 Those interested who can commit one hour a week to a meeting, 
 please email me. We'll exchange information via email (mailing 
 list if necessary) and meet every week to discuss progress.
If you take it somewhere else CC me I want to monitor this. My email is sahmi.soulaimane gmail.com.
Apr 17 2019
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/17/19 2:04 PM, Suleyman wrote:
 It looks to me that this proposal breaks everything that purity stands 
 for and I don't have a real solution to it. I'm not against breaking the 
 rules when necessary I just don't see why we need a DIP for it.
 
 On Tuesday, 16 April 2019 at 20:40:58 UTC, Andrei Alexandrescu wrote:
 Those interested who can commit one hour a week to a meeting, please 
 email me. We'll exchange information via email (mailing list if 
 necessary) and meet every week to discuss progress.
If you take it somewhere else CC me I want to monitor this. My email is sahmi.soulaimane gmail.com.
Thanks. You are the only who answered. So it's Timon (who is active but did not answer my request about weekly meetings), my student Razvan Nitu, Walter, and myself, with you as an observer. It seems to me that between the lot of us we don't have a passable solution to define __metadata in such a way that'd make it possible to implement the simplest case of reference counting. Of interest: https://github.com/dlang/dlang.org/pull/2627
Apr 19 2019