digitalmars.D - Helping with __mutable (which will be renamed to __metadata)
- Andrei Alexandrescu (17/17) Apr 12 2019 Razvan Nitu is working on the DIP initiated by Timon Gehr, known
- rikki cattermole (4/4) Apr 12 2019 I'm concerned that it will be used as an escape from the type system
- Suleyman (5/9) Apr 12 2019 From skimping through the DIP it doesn't look too dangerous it
- rikki cattermole (3/13) Apr 12 2019 What you described is a feature that escapes the type system which can
- Suleyman (5/6) Apr 12 2019 The DIP says you can edit a `__mutable` only in `@system` mode
- Timon Gehr (3/17) Apr 13 2019 Won't happen, because data with __metadata annotations inside will not
- Suleyman (5/8) Apr 13 2019 If it's adding these kinds of protections then I think it would
- Timon Gehr (5/13) Apr 13 2019 I don't think it should be @safe. Rather, `pure` and `immutable` should
- Suleyman (4/8) Apr 14 2019 It cannot be strongly pure. Same as we have weakly pure this is
- Timon Gehr (28/39) Apr 15 2019 No, the point is that whatever high-level rewrites you can deduce from
- Steven Schveighoffer (5/42) Apr 15 2019 Note that this is exactly the use case of reference counting. Which is
- Timon Gehr (5/49) Apr 15 2019 No, this is clearly not the use case of reference counting. Where do you...
- Steven Schveighoffer (6/54) Apr 15 2019 I mean when you add/remove reference for an immutable, it's passing an
- Timon Gehr (3/18) Apr 15 2019 I know, and this is why my original DIP draft had the concept of a
- Steven Schveighoffer (3/21) Apr 15 2019 Ah, ok. So a __mutable function is not strong-pure. That would work.
- Suleyman (5/11) Apr 15 2019 Just making it @system is not helping much instead of evading
- Timon Gehr (4/19) Apr 15 2019 At that point, why do you need to slap `immutable` on your type in the
- Suleyman (6/9) Apr 15 2019 I don't know but immutability has been breached for sure and
- Timon Gehr (19/30) Apr 15 2019 No. It's like saying Haskell does not support immutable data because the...
- Suleyman (6/9) Apr 15 2019 Otherwise the sole protection from being stored in readonly
- Timon Gehr (22/31) Apr 15 2019 This is not true and it is not even the right way to think about it. The...
- Doc Andrew (46/62) Apr 14 2019 A few questions/thoughts (or at least food for thought for others
- Timon Gehr (18/76) Apr 15 2019 It's not the same.
- Doc Andrew (76/144) Apr 15 2019 Oh no, I'm just suggesting that, under the original proposed DIP,
- Suleyman (41/42) Apr 12 2019 The problems you encountered are normal but the question is
- Radu (14/56) Apr 15 2019 I also have the impression that `shared` should be used here
- Steven Schveighoffer (4/9) Apr 15 2019 Only immutable structs would have shared metadata. Normal mutable
- Timon Gehr (18/42) Apr 13 2019 (Disclaimer: Note that the current state of the DIP is significantly
- RazvanN (58/78) Apr 16 2019 This is a draft proposal and it is not finished. The initial
- Suleyman (3/6) Apr 16 2019 What it is that this DIP enables you to do that you can't already
- Andrei Alexandrescu (2/10) Apr 16 2019 A simple answer is with the DIP you get defined behavior.
- Timon Gehr (43/131) Apr 16 2019 On the contrary, it's the type system information you _need_ in order to...
- RazvanN (43/124) Apr 17 2019 I understand now. It was very confusing for me that `pure` was
- Andrei Alexandrescu (8/30) Apr 16 2019 There seems to be a good amount of interest in the forum. Thanks! To
- Suleyman (8/11) Apr 17 2019 It looks to me that this proposal breaks everything that purity
- Andrei Alexandrescu (8/19) Apr 19 2019 Thanks. You are the only who answered. So it's Timon (who is active but
Razvan Nitu is working on the DIP initiated by Timon Gehr, known colloquially as the one that introduces __mutable - i.e. a mechanism for allowing controlled changes to immutable data. Here's a draft: https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/DIP1xxx-rn.md We figured that we're dealing a misnomer - we don't want __mutable, but instead __metadata - information that is nominally part of the object but needs certain leeway from the type system. Typical use cases are: * reference counting of immutable data structures * caching * lazy evaluation We got stuck at the interaction of __mutable with const parent objects (unclear whether the parent object originated as immutable or unqualified), and how pure functions should deal with __mutable. The few solutions we are toying with are either incomplete or too complicated (or both). The help of a few PL and compiler specialists would be very valuable here. I'm cc'ing a few, if anyone wants to help please let us know.
Apr 12 2019
I'm concerned that it will be used as an escape from the type system arbitrarily. Right now there is nothing to discourage the abuse of this storage class. Is there something we can do to discourage abuse?
Apr 12 2019
On Saturday, 13 April 2019 at 01:17:55 UTC, rikki cattermole wrote:I'm concerned that it will be used as an escape from the type system arbitrarily. Right now there is nothing to discourage the abuse of this storage class. Is there something we can do to discourage abuse?From skimping through the DIP it doesn't look too dangerous it seems like just a fancy 'shared' escape for immutable fields, the rest looks the same as casting immutable away in system code.
Apr 12 2019
On 13/04/2019 3:45 PM, Suleyman wrote:On Saturday, 13 April 2019 at 01:17:55 UTC, rikki cattermole wrote:What you described is a feature that escapes the type system which can lead to program crashes. Read only memory is what I'm concerned about.I'm concerned that it will be used as an escape from the type system arbitrarily. Right now there is nothing to discourage the abuse of this storage class. Is there something we can do to discourage abuse?From skimping through the DIP it doesn't look too dangerous it seems like just a fancy 'shared' escape for immutable fields, the rest looks the same as casting immutable away in system code.
Apr 12 2019
On Saturday, 13 April 2019 at 04:49:01 UTC, rikki cattermole wrote:...The DIP says you can edit a `__mutable` only in ` system` mode however if this is just a sugar around an unsafe cast I don't like it I like unsafe code to be ugly and scary.
Apr 12 2019
On 13.04.19 06:49, rikki cattermole wrote:On 13/04/2019 3:45 PM, Suleyman wrote:Won't happen, because data with __metadata annotations inside will not be put in read-only memory. (This can be known statically.)On Saturday, 13 April 2019 at 01:17:55 UTC, rikki cattermole wrote:What you described is a feature that escapes the type system which can lead to program crashes. Read only memory is what I'm concerned about.I'm concerned that it will be used as an escape from the type system arbitrarily. Right now there is nothing to discourage the abuse of this storage class. Is there something we can do to discourage abuse?From skimping through the DIP it doesn't look too dangerous it seems like just a fancy 'shared' escape for immutable fields, the rest looks the same as casting immutable away in system code.
Apr 13 2019
On Saturday, 13 April 2019 at 08:58:06 UTC, Timon Gehr wrote:Won't happen, because data with __metadata annotations inside will not be put in read-only memory. (This can be known statically.)If it's adding these kinds of protections then I think it would be worth it and a step forward in making D more safe. you might also consider the 'shared' protection I mentioned earlier then maybe you can lift the restriction on modification in safe code.
Apr 13 2019
On 14.04.19 02:19, Suleyman wrote:On Saturday, 13 April 2019 at 08:58:06 UTC, Timon Gehr wrote:I don't think it should be safe. Rather, `pure` and `immutable` should retain their meanings, which implies that there are wrong ways to use `__mutable` (hence unsafe), and there are still non-cosmetic reasons to use `immutable` even if there are `__mutable` fields.Won't happen, because data with __metadata annotations inside will not be put in read-only memory. (This can be known statically.)If it's adding these kinds of protections then I think it would be worth it and a step forward in making D more safe. you might also consider the 'shared' protection I mentioned earlier then maybe you can lift the restriction on modification in safe code.
Apr 13 2019
On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:I don't think it should be safe. Rather, `pure` and `immutable` should retain their meanings,It cannot be strongly pure. Same as we have weakly pure this is weakly immutable we're dealing with here.which implies that there are wrong ways to use `__mutable` (hence unsafe)can you elaborate?
Apr 14 2019
On 14.04.19 22:55, Suleyman wrote:On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:No, the point is that whatever high-level rewrites you can deduce from `immutable` and `pure` shouldn't be restricted by code that modifies `__metadata`.I don't think it should be safe. Rather, `pure` and `immutable` should retain their meanings,It cannot be strongly pure. Same as we have weakly pure this is weakly immutable we're dealing with here. ...struct S{ private __metadata x; } void foo(immutable ref S s)pure{ s.x += 1; } void main(){ immutable S s; foo(s); // there is no reason for this call to happen assert(s.x==1); // can't rely on this, it might also be 0 } struct S{ private __mutable x; } int foo(immutable ref S s)pure{ s.x += 1; return s.x; } void main(){ immutable S s; int a=foo(s); int b=foo(s); // could just reuse the previous result assert(a!=b); // can't rely on this, might be the same }which implies that there are wrong ways to use `__mutable` (hence unsafe)can you elaborate?
Apr 15 2019
On 4/15/19 7:17 AM, Timon Gehr wrote:On 14.04.19 22:55, Suleyman wrote:Note that this is exactly the use case of reference counting. Which is the main draw of having a mutable portion of an immutable. I would hazard to guess that if the above has to be the semantics, this is DOA. -SteveOn Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:struct S{ private __metadata x; } void foo(immutable ref S s)pure{ s.x += 1; } void main(){ immutable S s; foo(s); // there is no reason for this call to happen assert(s.x==1); // can't rely on this, it might also be 0 } struct S{ private __mutable x; } int foo(immutable ref S s)pure{ s.x += 1; return s.x; } void main(){ immutable S s; int a=foo(s); int b=foo(s); // could just reuse the previous result assert(a!=b); // can't rely on this, might be the same }which implies that there are wrong ways to use `__mutable` (hence unsafe)can you elaborate?
Apr 15 2019
On 15.04.19 14:32, Steven Schveighoffer wrote:On 4/15/19 7:17 AM, Timon Gehr wrote:No, this is clearly not the use case of reference counting. Where do you see references that are being counted? Why should the fact that the data structure is reference counted block optimizations such as eliding reference copies?On 14.04.19 22:55, Suleyman wrote:Note that this is exactly the use case of reference counting. Which is the main draw of having a mutable portion of an immutable. I would hazard to guess that if the above has to be the semantics, this is DOA. -SteveOn Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:struct S{ private __metadata x; } void foo(immutable ref S s)pure{ s.x += 1; } void main(){ immutable S s; foo(s); // there is no reason for this call to happen assert(s.x==1); // can't rely on this, it might also be 0 } struct S{ private __mutable x; } int foo(immutable ref S s)pure{ s.x += 1; return s.x; } void main(){ immutable S s; int a=foo(s); int b=foo(s); // could just reuse the previous result assert(a!=b); // can't rely on this, might be the same }which implies that there are wrong ways to use `__mutable` (hence unsafe)can you elaborate?
Apr 15 2019
On 4/15/19 8:49 AM, Timon Gehr wrote:On 15.04.19 14:32, Steven Schveighoffer wrote:I mean when you add/remove reference for an immutable, it's passing an immutable to what needs to be a pure function, which then increments/decrements a __metadata field (just like your examples above). If one of those 2 gets elided, the reference count is hosed. -SteveOn 4/15/19 7:17 AM, Timon Gehr wrote:No, this is clearly not the use case of reference counting. Where do you see references that are being counted? Why should the fact that the data structure is reference counted block optimizations such as eliding reference copies?On 14.04.19 22:55, Suleyman wrote:Note that this is exactly the use case of reference counting. Which is the main draw of having a mutable portion of an immutable. I would hazard to guess that if the above has to be the semantics, this is DOA.On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:struct S{ private __metadata x; } void foo(immutable ref S s)pure{ s.x += 1; } void main(){ immutable S s; foo(s); // there is no reason for this call to happen assert(s.x==1); // can't rely on this, it might also be 0 } struct S{ private __mutable x; } int foo(immutable ref S s)pure{ s.x += 1; return s.x; } void main(){ immutable S s; int a=foo(s); int b=foo(s); // could just reuse the previous result assert(a!=b); // can't rely on this, might be the same }which implies that there are wrong ways to use `__mutable` (hence unsafe)can you elaborate?
Apr 15 2019
On 15.04.19 14:56, Steven Schveighoffer wrote:On 4/15/19 8:49 AM, Timon Gehr wrote:I know, and this is why my original DIP draft had the concept of a __mutable function.On 15.04.19 14:32, Steven Schveighoffer wrote:I mean when you add/remove reference for an immutable, it's passing an immutable to what needs to be a pure function, which then increments/decrements a __metadata field (just like your examples above). If one of those 2 gets elided, the reference count is hosed. -Steve...No, this is clearly not the use case of reference counting. Where do you see references that are being counted? Why should the fact that the data structure is reference counted block optimizations such as eliding reference copies?
Apr 15 2019
On 4/15/19 9:23 AM, Timon Gehr wrote:On 15.04.19 14:56, Steven Schveighoffer wrote:Ah, ok. So a __mutable function is not strong-pure. That would work. -SteveOn 4/15/19 8:49 AM, Timon Gehr wrote:I know, and this is why my original DIP draft had the concept of a __mutable function.On 15.04.19 14:32, Steven Schveighoffer wrote:I mean when you add/remove reference for an immutable, it's passing an immutable to what needs to be a pure function, which then increments/decrements a __metadata field (just like your examples above). If one of those 2 gets elided, the reference count is hosed....No, this is clearly not the use case of reference counting. Where do you see references that are being counted? Why should the fact that the data structure is reference counted block optimizations such as eliding reference copies?
Apr 15 2019
On Monday, 15 April 2019 at 11:17:55 UTC, Timon Gehr wrote:On 14.04.19 22:55, Suleyman wrote:Just making it system is not helping much instead of evading reality you rather treat as it really is an immutable ref with mutable fields hence the compiler must not elide function calls with these weakly immutable arguments....No, the point is that whatever high-level rewrites you can deduce from `immutable` and `pure` shouldn't be restricted by code that modifies `__metadata`. ...
Apr 15 2019
On 15.04.19 17:08, Suleyman wrote:On Monday, 15 April 2019 at 11:17:55 UTC, Timon Gehr wrote:At that point, why do you need to slap `immutable` on your type in the first place? Why does your proposed semantics make any sense for reference counting or caching/lazy evaluation?On 14.04.19 22:55, Suleyman wrote:Just making it system is not helping much instead of evading reality you rather treat as it really is an immutable ref with mutable fields hence the compiler must not elide function calls with these weakly immutable arguments....No, the point is that whatever high-level rewrites you can deduce from `immutable` and `pure` shouldn't be restricted by code that modifies `__metadata`. ...
Apr 15 2019
On Monday, 15 April 2019 at 15:59:35 UTC, Timon Gehr wrote:At that point, why do you need to slap `immutable` on your type in the first place? Why does your proposed semantics make any sense for reference counting or caching/lazy evaluation?I don't know but immutability has been breached for sure and hiding that with system doesn't help I would say disallowing __mutable inside immutable objects makes more sense than just concealing the crime with system unless someone has a safe solution.
Apr 15 2019
On 15.04.19 18:37, Suleyman wrote:On Monday, 15 April 2019 at 15:59:35 UTC, Timon Gehr wrote:No. It's like saying Haskell does not support immutable data because the only way to implement lazy evaluation on the machine is using mutation. (Or even worse, to claim that D does not support immutability because the garbage collector can reclaim memory that was previously typed immutable.) I think D needs to be able to implement its own runtime library and manual memory allocation schemes without compiler magic unavailable to user programs.At that point, why do you need to slap `immutable` on your type in the first place? Why does your proposed semantics make any sense for reference counting or caching/lazy evaluation?I don't know but immutability has been breached for sureand hiding that with system doesn't helpNothing is being hidden.I would say disallowing __mutable inside immutable objects makes more senseRegarding sense: I have yet to find any in your "concealing" argument. I have tried and wasted too much time now.than just concealing the crime with systemsystem means you are not protected against writing programs that do not have defined semantics. Seems like a good enough fit.unless someone has a safe solution.No, that makes no sense at all. The only simple enough safe solution is to turn all __metadata use cases into compiler magic one-by-one. And you win nothing. The simplest way to implement that is to just add __metadata and simply add a compiler check that ensures user code cannot access it.
Apr 15 2019
On Monday, 15 April 2019 at 15:59:35 UTC, Timon Gehr wrote:At that point, why do you need to slap `immutable` on your type in the first place? Why does your proposed semantics make any sense for reference counting or caching/lazy evaluation?Otherwise the sole protection from being stored in readonly memory is not enough incentive for adding a keyword feature if this is the only contribution beyond an unsafe cast then a compiler directive should suffice such as pragma(noROM) or similar.
Apr 15 2019
On 15.04.19 19:24, Suleyman wrote:On Monday, 15 April 2019 at 15:59:35 UTC, Timon Gehr wrote:This is not true and it is not even the right way to think about it. The language semantics is distinct from how it is implemented in a compiler. The reason why data with __mutable fields cannot be put in readonly memory is because crash on write is not acceptable behavior. It does not even need to be explicitly specified.At that point, why do you need to slap `immutable` on your type in the first place? Why does your proposed semantics make any sense for reference counting or caching/lazy evaluation?Otherwise the sole protection from being stored in readonly memoryis not enough incentive for adding a keyword featureThe cost of adding a __keyword is pretty close to zero. And anyway, that's a superficial syntactic concern, not even really worth debating over at this point, as a) the current version of the DIP gets the semantics wrong and b) the necessary changes to the language specification are of similar size whether you add a __keyword or a pragma. (And whether or not you add a tiny bit more type checking is independent of the annotation syntax.)if this is the only contribution beyond an unsafe castThis is not what's being proposed. There are unsafe operations and there are invalid operations. The DIP's aim is to make some cases unsafe that were previously invalid. (And if you have to implement an annotation for fields anyway, adding the handful lines of frontend code that avoid additional casts, which are a blunt instrument that can hide what would otherwise be type errors, e.g. during a refactoring, does not seem bad enough to me to not even be counted as a contribution.)then a compiler directive should suffice such as pragma(noROM) or similar.A pragma is sufficient (but not required) anyway, but noROM is a terrible name.
Apr 15 2019
On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:On 14.04.19 02:19, Suleyman wrote:A few questions/thoughts (or at least food for thought for others more informed than me): 1. Maybe it's just me, but why the leading underscores? I'm wondering if this is because it's a scary/ugly-on-purpose feature like __gshared that is intended only sparingly? 1b. A lot of people have been clamoring for immutable by default, might keeping the DIP's original annotation "mutable" (with no underscores) make this a step in that direction? (with all the baggage and argument that carries with it, which is out of scope for this discussion). Later on, if the "mutable" qualifier is desired, we aren't carrying around both mutable and __mutable, or __mutable and mut or whatever ends up being the flavor of the day. 1c. Having said that, I think the "mutable" discussion and the "metadata" discussion are two separate things - which I think Andrei alluded to in the keyword change - this is potentially much bigger than just short-circuiting Immutability for a good cause. 2. If the __metadata qualifier can only be applied to private fields, and carries special semantics w.r.t. escaping the type system, is there any benefit to treating it more like a visibility attribute? struct something { metadata // maybe type qualifiers disallowed in here { int a; } private { int b; } public { int c; } } And... maybe metadata shouldn't be stored with the object at all? This way, it doesn't affect the layout of the object in memory, and doesn't require the need to short-circuit the type system if a later use of the class is marked Immutable. The object's metadata lives, not sorta-outside the type system, but outside the object itself? There are some challenges I can think of with this approach that I can expand on, but also some benefits. -DocOn Saturday, 13 April 2019 at 08:58:06 UTC, Timon Gehr wrote:I don't think it should be safe. Rather, `pure` and `immutable` should retain their meanings, which implies that there are wrong ways to use `__mutable` (hence unsafe), and there are still non-cosmetic reasons to use `immutable` even if there are `__mutable` fields.Won't happen, because data with __metadata annotations inside will not be put in read-only memory. (This can be known statically.)If it's adding these kinds of protections then I think it would be worth it and a step forward in making D more safe. you might also consider the 'shared' protection I mentioned earlier then maybe you can lift the restriction on modification in safe code.
Apr 14 2019
On 15.04.19 06:05, Doc Andrew wrote:On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:Yes. Also, it means we don't reserve a previously unreserved identifier.On 14.04.19 02:19, Suleyman wrote:A few questions/thoughts (or at least food for thought for others more informed than me): 1. Maybe it's just me, but why the leading underscores? I'm wondering if this is because it's a scary/ugly-on-purpose feature like __gshared that is intended only sparingly? ...On Saturday, 13 April 2019 at 08:58:06 UTC, Timon Gehr wrote:I don't think it should be safe. Rather, `pure` and `immutable` should retain their meanings, which implies that there are wrong ways to use `__mutable` (hence unsafe), and there are still non-cosmetic reasons to use `immutable` even if there are `__mutable` fields.Won't happen, because data with __metadata annotations inside will not be put in read-only memory. (This can be known statically.)If it's adding these kinds of protections then I think it would be worth it and a step forward in making D more safe. you might also consider the 'shared' protection I mentioned earlier then maybe you can lift the restriction on modification in safe code.1b. A lot of people have been clamoring for immutable by default, might keeping the DIP's original annotation "mutable" (with no underscores) make this a step in that direction? (with all the baggage and argument that carries with it, which is out of scope for this discussion). ...It's not the same.2. If the __metadata qualifier can only be applied to private fields, and carries special semantics w.r.t. escaping the type system, is there any benefit to treating it more like a visibility attribute? struct something { metadata // maybe type qualifiers disallowed in here { int a; } private { int b; } public { int c; } } ...Is your suggestion just to make __metadata imply private, or is there anything else you are trying to say?And... maybe metadata shouldn't be stored with the object at all? This way, it doesn't affect the layout of the object in memory, and doesn't require the need to short-circuit the type system if a later use of the class is marked Immutable. The object's metadata lives, not sorta-outside the type system, but outside the object itself? There are some challenges I can think of with this approach that I can expand on, but also some benefits. -DocStoring the data within the object is the point. As the original post states, the motivation for the DIP is to allow memory allocation schemes such as reference counting, as well as lazy initialization to be implemented for `immutable`-qualified data structures. The mutability is supposed to be an implementation detail -- from the outside, the data will still appear to be `immutable`. (The current version of the DIP completely misses this point though, leaking __metadata into the type system.) Compiler optimizations can change the semantics of code that operates on __metadata. (e.g., if it is able to elide a read to a lazily-initialized field, that field will not be initialized, if it elides a reference copy, the reference count does not need to be updated, etc.) Therefore, modifications of __metadata need to be consistent with rewrites that are valid for strongly pure functions.
Apr 15 2019
On Monday, 15 April 2019 at 11:09:23 UTC, Timon Gehr wrote:Oh no, I'm just suggesting that, under the original proposed DIP, the __mutable keyword might as well be made "mutable", which may be useful later on for other purposes. The metadata concept could be a lot more powerful, though, and I think it's basically orthogonal to the mutability concern:A few questions/thoughts (or at least food for thought for others more informed than me): 1. Maybe it's just me, but why the leading underscores? I'm wondering if this is because it's a scary/ugly-on-purpose feature like __gshared that is intended only sparingly? ...Yes. Also, it means we don't reserve a previously unreserved identifier.1b. A lot of people have been clamoring for immutable by default, might keeping the DIP's original annotation "mutable" (with no underscores) make this a step in that direction? (with all the baggage and argument that carries with it, which is out of scope for this discussion). ...It's not the same.Not just private, but it would be a way to annotate data that lives "outside" of the POD of the class. For instance, serializing an object with variables in the metadata{} block wouldn't (or couldn't?) include them. Another idea I had this morning is that the metadata{} block might be a nice place to put class/function attributes to avoid cluttering the signatures: class Foo { metadata { synchronized; // can move class attributes in here scope; (someUDA); // UDA's live here too int secretSauce; // Can always be accessed or modified by Foo, regardless of // mutability } } The concept fits nicely with contracts and functions, too. Just as metadata{} blocks in a immutable object allow modification of those member variables, maybe member variables in a function metadata{} block can be modified and persisted across calls for pure functions, too. It would act just like a static variable, available at both compile-time and run-time, maybe useful for things like debugging, profiling or benchmarking. Maybe code in the in & out contract blocks can access it, but we'd make it illegal for the function body to do it. I'd need to think through the side-effects a little more. int Bar(int a, int b) metadata { pure: // Not 100% sure what syntax would make sense nogc: // maybe just force for everything here safe: long startTicks; // Stuff for benchmarking long endTicks; // Not pertinent to the function itself static int numTimesCalled; // This is a pure function... } in { numTimesCalled++; // ...but it's OK, we're just modifying it's metadata ... } do { if(numTimesCalled > 4) // Illegal to do this here, breaks purity ... }2. If the __metadata qualifier can only be applied to private fields, and carries special semantics w.r.t. escaping the type system, is there any benefit to treating it more like a visibility attribute? struct something { metadata // maybe type qualifiers disallowed in here { int a; } private { int b; } public { int c; } } ...Is your suggestion just to make __metadata imply private, or is there anything else you are trying to say?Sure, I'm just suggesting that keeping that reference count or other metadata separately is a possible workaround - as long as there's an easy way to get the metadata for a given object (whether in memory alongside the rest of the object's data or not). The idea of storing metadata member variables separately is an implementation detail. To the class, it appears that the metadata members are no different than regular variables, but casting to void* or serializing it leaves out the metadata, so it's a safe place to keep things like reference counts. In this scheme, metadata can be added to classes or structs without changing their memory layout, and metadata vars have some other special powers as far as being treated differently by the type system, strictly because they are stored "outside" the object memory, not because of a special-case in the type system. -DocAnd... maybe metadata shouldn't be stored with the object at all? This way, it doesn't affect the layout of the object in memory, and doesn't require the need to short-circuit the type system if a later use of the class is marked Immutable. The object's metadata lives, not sorta-outside the type system, but outside the object itself? There are some challenges I can think of with this approach that I can expand on, but also some benefits. -DocStoring the data within the object is the point. As the original post states, the motivation for the DIP is to allow memory allocation schemes such as reference counting, as well as lazy initialization to be implemented for `immutable`-qualified data structures. The mutability is supposed to be an implementation detail -- from the outside, the data will still appear to be `immutable`. (The current version of the DIP completely misses this point though, leaking __metadata into the type system.) Compiler optimizations can change the semantics of code that operates on __metadata. (e.g., if it is able to elide a read to a lazily-initialized field, that field will not be initialized, if it elides a reference copy, the reference count does not need to be updated, etc.) Therefore, modifications of __metadata need to be consistent with rewrites that are valid for strongly pure functions.
Apr 15 2019
On Saturday, 13 April 2019 at 00:45:07 UTC, Andrei Alexandrescu wrote:...The problems you encountered are normal but the question is whether this feature is useful at all if the field is not explicitly shared? If so the a solution to the const case is to introduce an ' unshared' attribute instead which would protect against the worst cases of both being shared and being thread local. this attribute would disallow unsychronized read/write and at the same time prevent implicit conversion to 'shared' since it may just be thread local hence mixing it with normal 'shared' may not bring good results. example: ``` struct S { private shared int* p; // or unshared void inc() inout pure system { import core.atomic; atomicOp!"+="(*cast(shared(int*))p, 1); // or unshared } } void foo(ref immutable S a) pure { a.inc(); } void foo(ref const S a) pure { a.inc(); } int x; // thread local shared int y; void main() { auto i = immutable S(cast(immutable)&x); auto c = const S(&y); //auto d = const S(cast( unshared)&x); foo(i); foo(c); } ```
Apr 12 2019
On Saturday, 13 April 2019 at 03:45:35 UTC, Suleyman wrote:On Saturday, 13 April 2019 at 00:45:07 UTC, Andrei Alexandrescu wrote:I also have the impression that `shared` should be used here instead of `__mutable`. Even in the current incarnation shared offers some limited guarantees that atomicity is required for writing shared values. I think people are working on making stronger guarantees for shared. ATM uses like the one below are a no-op, I think the reuse of shared makes sense as it works nicely with the strong thread safety guarantees `immutable` has, and also offers the safety net for atomic changes. Not to mention that it is not ugly! immutable struct Foo { shared(int) x; }...The problems you encountered are normal but the question is whether this feature is useful at all if the field is not explicitly shared? If so the a solution to the const case is to introduce an ' unshared' attribute instead which would protect against the worst cases of both being shared and being thread local. this attribute would disallow unsychronized read/write and at the same time prevent implicit conversion to 'shared' since it may just be thread local hence mixing it with normal 'shared' may not bring good results. example: ``` struct S { private shared int* p; // or unshared void inc() inout pure system { import core.atomic; atomicOp!"+="(*cast(shared(int*))p, 1); // or unshared } } void foo(ref immutable S a) pure { a.inc(); } void foo(ref const S a) pure { a.inc(); } int x; // thread local shared int y; void main() { auto i = immutable S(cast(immutable)&x); auto c = const S(&y); //auto d = const S(cast( unshared)&x); foo(i); foo(c); } ```
Apr 15 2019
On 4/15/19 9:14 AM, Radu wrote:I also have the impression that `shared` should be used here instead of `__mutable`. Even in the current incarnation shared offers some limited guarantees that atomicity is required for writing shared values. I think people are working on making stronger guarantees for shared.Only immutable structs would have shared metadata. Normal mutable structs could have thread-local metadata. So that doesn't work. -Steve
Apr 15 2019
On 13.04.19 02:45, Andrei Alexandrescu wrote:Razvan Nitu is working on the DIP initiated by Timon Gehr,(Disclaimer: Note that the current state of the DIP is significantly different from my original proposal, and I would not argue for acceptance of the current state of the DIP even though I am listed as an author.)known colloquially as the one that introduces __mutable - i.e. a mechanism for allowing controlled changes to immutable data. Here's a draft: https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/DIP1xxx-rn.md We figured that we're dealing a misnomer - we don't want __mutable, but instead __metadata - information that is nominally part of the object but needs certain leeway from the type system. Typical use cases are: * reference counting of immutable data structuresFor that you need the more general (originally supported using __mutable functions): * manual allocation and deallocation of immutable data* caching * lazy evaluation We got stuck at the interaction of __mutable with const parent objects (unclear whether the parent object originated as immutable or unqualified),I think we got that one sorted out.and how pure functions should deal with __mutable. The few solutions we are toying with are either incomplete or too complicated (or both). The help of a few PL and compiler specialists would be very valuable here. I'm cc'ing a few, if anyone wants to help please let us know.If the principled approach is too complicated, the feature is not worth it. My original proposal was to define a set of rewrites that is based on the function signature alone which is semantics-preserving if there is no __metadata (except possibly reference identity of immutable data), and then say that this set of rewrites is still permissible even if there are __metadata annotations. If complexity of the DIP is a concern, I guess we could simplify this by just saying that any rewrite based on the function signature alone that is sound without __metadata will remain permissible with it.
Apr 13 2019
On Saturday, 13 April 2019 at 00:45:07 UTC, Andrei Alexandrescu wrote:Razvan Nitu is working on the DIP initiated by Timon Gehr, known colloquially as the one that introduces __mutable - i.e. a mechanism for allowing controlled changes to immutable data. Here's a draft: https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/DIP1xxx-rn.md We figured that we're dealing a misnomer - we don't want __mutable, but instead __metadata - information that is nominally part of the object but needs certain leeway from the type system. Typical use cases are: * reference counting of immutable data structures * caching * lazy evaluation We got stuck at the interaction of __mutable with const parent objects (unclear whether the parent object originated as immutable or unqualified), and how pure functions should deal with __mutable. The few solutions we are toying with are either incomplete or too complicated (or both). The help of a few PL and compiler specialists would be very valuable here. I'm cc'ing a few, if anyone wants to help please let us know.This is a draft proposal and it is not finished. The initial draft that was done by Timon [1] considers __mutable functions that act as optimization blockers for the compiler with regards to `pure` functions. I feel that optimization blockers for `pure` are a totally different concept than __mutable/__metadata fields. Considering that the currently the compiler does not do any optimizations based on purity, I think it would be a lot easier to just get __mutable/__metadata fields in, because they are a needed feature; after that, we can think on what optimizations we want to implement and how they interact with __mutable/__mutable. Consider this code: int foo() pure { immutable(T)* x = allocate(); int y = bar(x); deallocate(x); return y; } Currently, the compiler does not do any optimizations regarding purity with it. If it did, it could have swapped the invocation of `deallocate` with the one of `bar` leading to use-after-free. Annotating `deallocate` with __mutable would solve the issue in the event of compiler optimizations. However, this example has nothing to do with __mutable/__metadata fields; this kind of optimizations should be discussed in a DIP of their own and not in a __mutable/__metadata DIP. Moreover, is this the direction we want to head in? Require the user to mentally trace the optimizations that the compiler might do? This is just too complicated. That is why, I think that we should focus on implementing __metadata with regards to fields and later on think about the optimizations that we can perform. As for __metadata fields: 1. If we decide that __metadata fields are not conceptually part of the object, why would accesses to them be unsafe? We could still make them `private`, but we can view them as normal accesses from a safety perspective. 2. I agree that `purity` should not be affected by __mutable/__metadata fields; the object passed as argument will not be conceptually modified. If we take out optimizations out of the picture, things become a lot more clearer. For me, the DIP is more of "how can we mutate fields in a non-mutable object", not "how do we implement optimizations regarding purity". I don't see why the addition of __metadata should be delayed by how optimizations based on purity are applied in the compiler, Cheers, RazvanN [1] https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/timon_dip.md
Apr 16 2019
On Tuesday, 16 April 2019 at 09:51:33 UTC, RazvanN wrote:... Cheers, RazvanNWhat it is that this DIP enables you to do that you can't already do in system code?
Apr 16 2019
On 4/16/19 4:15 PM, Suleyman wrote:On Tuesday, 16 April 2019 at 09:51:33 UTC, RazvanN wrote:A simple answer is with the DIP you get defined behavior.... Cheers, RazvanNWhat it is that this DIP enables you to do that you can't already do in system code?
Apr 16 2019
On 16.04.19 11:51, RazvanN wrote:On Saturday, 13 April 2019 at 00:45:07 UTC, Andrei Alexandrescu wrote:On the contrary, it's the type system information you _need_ in order to abstract over code that mutates __metadata. An assignment to a __metadata field is something you cannot ignore even though it is `pure` and does not return any interesting value. For any statement, you need to be able to assign a valid type to a function that contains this statement, and that captures what the type system knows about the statement.Razvan Nitu is working on the DIP initiated by Timon Gehr, known colloquially as the one that introduces __mutable - i.e. a mechanism for allowing controlled changes to immutable data. Here's a draft: https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/DIP1xxx-rn.md We figured that we're dealing a misnomer - we don't want __mutable, but instead __metadata - information that is nominally part of the object but needs certain leeway from the type system. Typical use cases are: * reference counting of immutable data structures * caching * lazy evaluation We got stuck at the interaction of __mutable with const parent objects (unclear whether the parent object originated as immutable or unqualified), and how pure functions should deal with __mutable. The few solutions we are toying with are either incomplete or too complicated (or both). The help of a few PL and compiler specialists would be very valuable here. I'm cc'ing a few, if anyone wants to help please let us know.This is a draft proposal and it is not finished. The initial draft that was done by Timon [1] considers __mutable functions that act as optimization blockers for the compiler with regards to `pure` functions. I feel that optimization blockers for `pure` are a totally different concept than __mutable/__metadata fields. ...Considering that the currently the compiler does not do any optimizations based on purity, I think it would be a lot easier to just get __mutable/__metadata fields in, because they are a needed feature; after that, we can think on what optimizations we want to implement and how they interact with __mutable/__mutable. ...You need to specify that there are optimizations that are allowed even though they change the set of mutations that the program performs. As I suggested earlier, if you want the semantics that's least restrictive for language evolution, you need to say something that implies that all optimizations based on the function signature are allowed, not "the compiler currently does not do any optimizations". That's precisely the wrong direction if you want to be conservative.Consider this code: int foo() pure { immutable(T)* x = allocate(); int y = bar(x); deallocate(x); return y; } Currently, the compiler does not do any optimizations regarding purity with it. If it did, it could have swapped the invocation of `deallocate` with the one of `bar` leading to use-after-free.What the compiler does and does not do has no bearing on the validity of the code per language semantics.Annotating `deallocate` with __mutable would solve the issue in the event of compiler optimizations. However, this example has nothing to do with __mutable/__metadata fields; this kind of optimizations should be discussed in a DIP of their own and not in a __mutable/__metadata DIP.That's simply nonsense. You define the language semantics, the compiler developer figures out optimizations that are consistent with that semantics. You don't need any DIPs for optimizations, you need DIPs to specify how programs may or may not behave. You can't really first come and say programs may not behave a certain way and then later relax that and say that actually those programs can also behave in this other way and at the same time claim that this is a conservative approach. It is not; it breaks code.Moreover, is this the direction we want to head in?Absolutely. If you want to use __metadata you need to be aware that pure functions can be elided, and if the mutation of __metadata within a function is not self-contained such that it can be elided, you need to annotate that function.Require the user to mentally trace the optimizations that the compiler might do?The user of __metadata. _Some_ kinds of optimizations.This is just too complicated. ...Then give up on __metadata and just implement your persistent data structures without the `immutable` qualifier slapped on top! `immutable` has no value if it cannot be used to justify equational reasoning on pure functions.That is why, I think that we should focus on implementing __metadata with regards to fields and later on think about the optimizations that we can perform. As for __metadata fields: 1. If we decide that __metadata fields are not conceptually part of the object, why would accesses to them be unsafe? We could still make them `private`, but we can view them as normal accesses from a safety perspective. ...Then those accesses can't be pure because they are like accesses to global variables.2. I agree that `purity` should not be affected by __mutable/__metadata fields; the object passed as argument will not be conceptually modified. ...1 and 2 are fundamentally at odds with each other if you want __mutable fields to be accessible in pure functions, because you can't check 2 safely.If we take out optimizations out of the picture, things become a lot more clearer.You need to define the semantics!For me, the DIP is more of "how can we mutate fields in a non-mutable object", notYou clearly can't mutate fields in a non-mutable object except non-mutable is some higher-level abstraction. That abstraction has to be defined. One way to define semantics for __metadata is to explicitly give the set of allowed rewrites, but you can also define it implicitly."how do we implement optimizations regarding purity". I don't see why the addition of __metadata should be delayed by how optimizations based on purity are applied in the compiler, ...It shouldn't.
Apr 16 2019
On Wednesday, 17 April 2019 at 01:00:35 UTC, Timon Gehr wrote:On the contrary, it's the type system information you _need_ in order to abstract over code that mutates __metadata. An assignment to a __metadata field is something you cannot ignore even though it is `pure` and does not return any interesting value. For any statement, you need to be able to assign a valid type to a function that contains this statement, and that captures what the type system knows about the statement. You need to specify that there are optimizations that are allowed even though they change the set of mutations that the program performs. As I suggested earlier, if you want the semantics that's least restrictive for language evolution, you need to say something that implies that all optimizations based on the function signature are allowed, not "the compiler currently does not do any optimizations". That's precisely the wrong direction if you want to be conservative.I see, now I understand.What the compiler does and does not do has no bearing on the validity of the code per language semantics. That's simply nonsense. You define the language semantics, the compiler developer figures out optimizations that are consistent with that semantics. You don't need any DIPs for optimizations, you need DIPs to specify how programs may or may not behave. You can't really first come and say programs may not behave a certain way and then later relax that and say that actually those programs can also behave in this other way and at the same time claim that this is a conservative approach. It is not; it breaks code.I understand now. It was very confusing for me that `pure` was simply implemented as a set of compile time checks not backed by any optimizations. I figured that if none of them were implemented by now, it will certainly break a lot of code when we will have them, so there is a great chance that `pure` will simply remain in the current stage. If that would be the case, adding the cognitive load for __metadata functions to the user is not worth it.But it's not just the user of __metadata. If you implement in a library a refcounted object then __metadata is going to be propagated to user code.Moreover, is this the direction we want to head in?Absolutely. If you want to use __metadata you need to be aware that pure functions can be elided, and if the mutation of __metadata within a function is not self-contained such that it can be elided, you need to annotate that function.Require the user to mentally trace the optimizations that the compiler might do?The user of __metadata. _Some_ kinds of optimizations.But they actually aren't because they are "physically" part of the object.This is just too complicated. ...Then give up on __metadata and just implement your persistent data structures without the `immutable` qualifier slapped on top! `immutable` has no value if it cannot be used to justify equational reasoning on pure functions.That is why, I think that we should focus on implementing __metadata with regards to fields and later on think about the optimizations that we can perform. As for __metadata fields: 1. If we decide that __metadata fields are not conceptually part of the object, why would accesses to them be unsafe? We could still make them `private`, but we can view them as normal accesses from a safety perspective. ...Then those accesses can't be pure because they are like accesses to global variables.What I was trying to say is that it would be easier to define semantics if we wouldn't pretend that the compiler treats `pure` in a way that it actually doesn't. I see your point, however, even if we have __metadata/ __mutable functions, once the purity optimizations will be implemented, all the users will have to check their code to add __metadata/__mutable to their functions to be sure their calls to strongly pure functions are not elided.2. I agree that `purity` should not be affected by __mutable/__metadata fields; the object passed as argument will not be conceptually modified. ...1 and 2 are fundamentally at odds with each other if you want __mutable fields to be accessible in pure functions, because you can't check 2 safely.If we take out optimizations out of the picture, things become a lot more clearer.You need to define the semantics!From an academical perspective, I agree with everything you said and I think I finally understand your point of view and your DIP draft. I think that a small chat would have made it easier for both parties to reach an understanding. At this point, I think it all comes down to what we want to do with `pure`. It really doesn't make any sense to take all these optimizations into account and burden the user with __mutable functions if those optimizations will never be implemented. Cheers, RazvanNFor me, the DIP is more of "how can we mutate fields in a non-mutable object", notYou clearly can't mutate fields in a non-mutable object except non-mutable is some higher-level abstraction. That abstraction has to be defined. One way to define semantics for __metadata is to explicitly give the set of allowed rewrites, but you can also define it implicitly."how do we implement optimizations regarding purity". I don't see why the addition of __metadata should be delayed by how optimizations based on purity are applied in the compiler, ...It shouldn't.
Apr 17 2019
On 4/12/19 8:45 PM, Andrei Alexandrescu wrote:Razvan Nitu is working on the DIP initiated by Timon Gehr, known colloquially as the one that introduces __mutable - i.e. a mechanism for allowing controlled changes to immutable data. Here's a draft: https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/DIP1xxx-rn.md We figured that we're dealing a misnomer - we don't want __mutable, but instead __metadata - information that is nominally part of the object but needs certain leeway from the type system. Typical use cases are: * reference counting of immutable data structures * caching * lazy evaluation We got stuck at the interaction of __mutable with const parent objects (unclear whether the parent object originated as immutable or unqualified), and how pure functions should deal with __mutable. The few solutions we are toying with are either incomplete or too complicated (or both). The help of a few PL and compiler specialists would be very valuable here. I'm cc'ing a few, if anyone wants to help please let us know.There seems to be a good amount of interest in the forum. Thanks! To better move things forward, we'd need to have a more fleshed-out proposal to show the community for discussion. Until then, let's move to a special interest group. Those interested who can commit one hour a week to a meeting, please email me. We'll exchange information via email (mailing list if necessary) and meet every week to discuss progress.
Apr 16 2019
It looks to me that this proposal breaks everything that purity stands for and I don't have a real solution to it. I'm not against breaking the rules when necessary I just don't see why we need a DIP for it. On Tuesday, 16 April 2019 at 20:40:58 UTC, Andrei Alexandrescu wrote:Those interested who can commit one hour a week to a meeting, please email me. We'll exchange information via email (mailing list if necessary) and meet every week to discuss progress.If you take it somewhere else CC me I want to monitor this. My email is sahmi.soulaimane gmail.com.
Apr 17 2019
On 4/17/19 2:04 PM, Suleyman wrote:It looks to me that this proposal breaks everything that purity stands for and I don't have a real solution to it. I'm not against breaking the rules when necessary I just don't see why we need a DIP for it. On Tuesday, 16 April 2019 at 20:40:58 UTC, Andrei Alexandrescu wrote:Thanks. You are the only who answered. So it's Timon (who is active but did not answer my request about weekly meetings), my student Razvan Nitu, Walter, and myself, with you as an observer. It seems to me that between the lot of us we don't have a passable solution to define __metadata in such a way that'd make it possible to implement the simplest case of reference counting. Of interest: https://github.com/dlang/dlang.org/pull/2627Those interested who can commit one hour a week to a meeting, please email me. We'll exchange information via email (mailing list if necessary) and meet every week to discuss progress.If you take it somewhere else CC me I want to monitor this. My email is sahmi.soulaimane gmail.com.
Apr 19 2019