digitalmars.D - How mutable is immutable?
- Denis Shelomovskij (41/41) Jan 01 2012 So, I'm a function `f`, I have an `immutable(type)[]` argument and I
- Timon Gehr (5/46) Jan 01 2012 You are using unsafe language features to break the type system. That is...
- Don Clugston (9/58) Oct 17 2012 That's the point -- *which* checks are missing from @safe?
- Timon Gehr (16/76) Oct 17 2012 Escaping stack data and arbitrarily freeing memory are not operations
- Malte Skarupke (10/10) Oct 17 2012 The issue is that you're thinking as you would in Java.
- Don Clugston (29/110) Oct 18 2012 No it is not. Data on the stack *cannot* survive past the end of the
- Artur Skawina (33/139) Oct 18 2012 /How/ is not a problem (ignoring implementation costs), the /language de...
- Timon Gehr (26/61) Oct 18 2012 Static escape analysis. Use the 'scope' qualifier to designate
- Don Clugston (9/80) Oct 22 2012 Well, OK, but that involves changing the semantics of immutable. You
So, I'm a function `f`, I have an `immutable(type)[]` argument and I
want to store it for my friend `g` in an TLS variable `v`:
---
string v;
debug string sure;
void f(string s) { v = s; debug sure = s.idup; }
void g() { assert(v == sure); }
---
I also store a copy of `s` into `sure` for my friend to ensure immutable
date hasn't been mutated.
Can my friend's assertion ever fail without breaking a type-system?
Sure. Just consider this:
---
void main() {
auto s = "abba".idup;
f(s);
delete s;
g();
}
---
Is it by-design? Looks like deleting immutable (and const because of
implicit conversion) data should be prohibited.
OK. Let `delete` be fixed. Can we still fail?
---
void h() {
immutable(char)[4] s = "abba";
f(s);
}
void main() {
h();
g();
}
---
Damn! So, what can we do with it? Not sure, but I have a proposal.
Fix it in language:
* disallow `delete` of const/immutable data
* disallow immutable data on the stack
This makes data really immutable if I don't miss something. Anyway, I
want `immutable` qualified data to be immutable without breaking a
type-system (if one do it, its his own responsibility), so some changes
should be made (IMHO).
Jan 01 2012
On 01/01/2012 10:40 AM, Denis Shelomovskij wrote:
So, I'm a function `f`, I have an `immutable(type)[]` argument and I
want to store it for my friend `g` in an TLS variable `v`:
---
string v;
debug string sure;
void f(string s) { v = s; debug sure = s.idup; }
void g() { assert(v == sure); }
---
I also store a copy of `s` into `sure` for my friend to ensure immutable
date hasn't been mutated.
Can my friend's assertion ever fail without breaking a type-system?
Sure. Just consider this:
---
void main() {
auto s = "abba".idup;
f(s);
delete s;
g();
}
---
Is it by-design? Looks like deleting immutable (and const because of
implicit conversion) data should be prohibited.
OK. Let `delete` be fixed. Can we still fail?
---
void h() {
immutable(char)[4] s = "abba";
f(s);
}
void main() {
h();
g();
}
---
Damn! So, what can we do with it? Not sure, but I have a proposal.
Fix it in language:
* disallow `delete` of const/immutable data
* disallow immutable data on the stack
This makes data really immutable if I don't miss something. Anyway, I
want `immutable` qualified data to be immutable without breaking a
type-system (if one do it, its his own responsibility), so some changes
should be made (IMHO).
You are using unsafe language features to break the type system. That is
not the fault of the type system.
' safe:' at the top of the program should stop both examples from
working, it is a bug that it does not.
Jan 01 2012
On 01/01/12 13:50, Timon Gehr wrote:On 01/01/2012 10:40 AM, Denis Shelomovskij wrote:That's the point -- *which* checks are missing from safe? But I'm not sure that you're right, this looks broken to me, even without safe. What does it mean to create immutable data on the stack? The stack is intrinsically mutable! What does it mean to delete immutable data? I think it's reasonable for both of them to require a cast, even in system code.So, I'm a function `f`, I have an `immutable(type)[]` argument and I want to store it for my friend `g` in an TLS variable `v`: --- string v; debug string sure; void f(string s) { v = s; debug sure = s.idup; } void g() { assert(v == sure); } --- I also store a copy of `s` into `sure` for my friend to ensure immutable date hasn't been mutated. Can my friend's assertion ever fail without breaking a type-system? Sure. Just consider this: --- void main() { auto s = "abba".idup; f(s); delete s; g(); } --- Is it by-design? Looks like deleting immutable (and const because of implicit conversion) data should be prohibited. OK. Let `delete` be fixed. Can we still fail? --- void h() { immutable(char)[4] s = "abba"; f(s); } void main() { h(); g(); } --- Damn! So, what can we do with it? Not sure, but I have a proposal. Fix it in language: * disallow `delete` of const/immutable data * disallow immutable data on the stack This makes data really immutable if I don't miss something. Anyway, I want `immutable` qualified data to be immutable without breaking a type-system (if one do it, its his own responsibility), so some changes should be made (IMHO).You are using unsafe language features to break the type system. That is not the fault of the type system. ' safe:' at the top of the program should stop both examples from working, it is a bug that it does not.
Oct 17 2012
On 10/17/2012 01:49 PM, Don Clugston wrote:On 01/01/12 13:50, Timon Gehr wrote:Escaping stack data and arbitrarily freeing memory are not operations found in memory safe languages.On 01/01/2012 10:40 AM, Denis Shelomovskij wrote:That's the point -- *which* checks are missing from safe?So, I'm a function `f`, I have an `immutable(type)[]` argument and I want to store it for my friend `g` in an TLS variable `v`: --- string v; debug string sure; void f(string s) { v = s; debug sure = s.idup; } void g() { assert(v == sure); } --- I also store a copy of `s` into `sure` for my friend to ensure immutable date hasn't been mutated. Can my friend's assertion ever fail without breaking a type-system? Sure. Just consider this: --- void main() { auto s = "abba".idup; f(s); delete s; g(); } --- Is it by-design? Looks like deleting immutable (and const because of implicit conversion) data should be prohibited. OK. Let `delete` be fixed. Can we still fail? --- void h() { immutable(char)[4] s = "abba"; f(s); } void main() { h(); g(); } --- Damn! So, what can we do with it? Not sure, but I have a proposal. Fix it in language: * disallow `delete` of const/immutable data * disallow immutable data on the stack This makes data really immutable if I don't miss something. Anyway, I want `immutable` qualified data to be immutable without breaking a type-system (if one do it, its his own responsibility), so some changes should be made (IMHO).You are using unsafe language features to break the type system. That is not the fault of the type system. ' safe:' at the top of the program should stop both examples from working, it is a bug that it does not.But I'm not sure that you're right, this looks broken to me, even without safe. What does it mean to create immutable data on the stack? The stack is intrinsically mutable!So is the heap. What does it mean to garbage collect immutable data? What does it mean to allocate an 'int' on the stack?What does it mean to delete immutable data?Deallocate the storage for it and make it available for reuse. Accessing it afterwards leads to arbitrary behaviour. This is the same with mutable data. As the program may behave arbitrarily in this case, it is valid behaviour to act as if immutable data changed.I think it's reasonable for both of them to require a cast, even in system code.The implementation of the 'scope' storage class should be fixed. We could then require an unsafe cast(scope) to disable prevention of stack address escaping. Rust's borrowed pointers may give some hints on how to extend 'scope' to fields of structs. As to delete, delete is as unsafe when the involved data is immutable as when it is mutable. Why require an additional cast in one case?
Oct 17 2012
"Malte Skarupke" <malteskarupke web.de> The issue is that you're thinking as you would in Java. I guess the rule in D for immutable is this: Immutable data won't change as long as it exists. The last part of that sentence would be a stupid thing to say in Java because things don't just cease to exist while you're still doing something with them. That is not the case in D. That being said it's very unlikely that you will ever run into this situation. You have to end the lifetime of the object manually to run into these issues. And in that case it'll be very easy to figure out what's wrong.
Oct 17 2012
On 17/10/12 18:02, Timon Gehr wrote:On 10/17/2012 01:49 PM, Don Clugston wrote:HOW do you propose to check for escaping stack data?On 01/01/12 13:50, Timon Gehr wrote:Escaping stack data and arbitrarily freeing memory are not operations found in memory safe languages.On 01/01/2012 10:40 AM, Denis Shelomovskij wrote:That's the point -- *which* checks are missing from safe?So, I'm a function `f`, I have an `immutable(type)[]` argument and I want to store it for my friend `g` in an TLS variable `v`: --- string v; debug string sure; void f(string s) { v = s; debug sure = s.idup; } void g() { assert(v == sure); } --- I also store a copy of `s` into `sure` for my friend to ensure immutable date hasn't been mutated. Can my friend's assertion ever fail without breaking a type-system? Sure. Just consider this: --- void main() { auto s = "abba".idup; f(s); delete s; g(); } --- Is it by-design? Looks like deleting immutable (and const because of implicit conversion) data should be prohibited. OK. Let `delete` be fixed. Can we still fail? --- void h() { immutable(char)[4] s = "abba"; f(s); } void main() { h(); g(); } --- Damn! So, what can we do with it? Not sure, but I have a proposal. Fix it in language: * disallow `delete` of const/immutable data * disallow immutable data on the stack This makes data really immutable if I don't miss something. Anyway, I want `immutable` qualified data to be immutable without breaking a type-system (if one do it, its his own responsibility), so some changes should be made (IMHO).You are using unsafe language features to break the type system. That is not the fault of the type system. ' safe:' at the top of the program should stop both examples from working, it is a bug that it does not.No it is not. Data on the stack *cannot* survive past the end of the function call. Data on the heap can last forever.But I'm not sure that you're right, this looks broken to me, even without safe. What does it mean to create immutable data on the stack? The stack is intrinsically mutable!So is the heap.What does it mean to garbage collect immutable data?From the point of view of the application, it doesn't happen. There are no observable semantics. It's merely an implementation detail.What does it mean to allocate an 'int' on the stack?No, you've broken the type system if you've deleted immutable data. If I have a reference to an immutable variable, I have a guarantee that it will never change. delete will break that guarantee. With a mutable variable, I have no such guarantee. (It's not safe to allocate something different in the deleted location, but it's OK to run the finalizer and then wipe all the memory).What does it mean to delete immutable data?Deallocate the storage for it and make it available for reuse. Accessing it afterwards leads to arbitrary behaviour. This is the same with mutable data. As the program may behave arbitrarily in this case, it is valid behaviour to act as if immutable data changed.No we can't. f cannot know that the string it has been given is on the stack. So main() must prevent it from being given to f() in the first place. How can it do that? void foo(bool b, string y) { immutable (char)[4] x = "abba"; string s = b ? x : y; f(s); } Make it safe.I think it's reasonable for both of them to require a cast, even in system code.The implementation of the 'scope' storage class should be fixed. We could then require an unsafe cast(scope) to disable prevention of stack address escaping.Rust's borrowed pointers may give some hints on how to extend 'scope' to fields of structs.I think it is more fundamental than that.As to delete, delete is as unsafe when the involved data is immutable as when it is mutable. Why require an additional cast in one case?This is not about safety. Modifying immutable data breaks the type system. Deleting mutable data does not. AFAIK it is safe to implement delete as a call to the finalizer, followed by setting the memory to T.init. Only the GC can determine if it is safe to reuse the memory. Deleting immutable data just doesn't make sense.
Oct 18 2012
On 10/18/12 10:08, Don Clugston wrote:On 17/10/12 18:02, Timon Gehr wrote:/How/ is not a problem (ignoring implementation costs), the /language definition/ part is trickier - you need a very precise definition of what is allowed and what isn't; otherwise different compilers will make different decisions and every compiler will support only a vendor-specific non-std dialect... (eg storing a scoped-ref into some kind of container, passing that down to other functions could work, but what if you then need to let the container escape and want to do that by removing the scoped-ref? It might be possible for the compiler to prove that it's safe, but it's unlikely that every compiler will act the same)On 10/17/2012 01:49 PM, Don Clugston wrote:HOW do you propose to check for escaping stack data?On 01/01/12 13:50, Timon Gehr wrote:Escaping stack data and arbitrarily freeing memory are not operations found in memory safe languages.On 01/01/2012 10:40 AM, Denis Shelomovskij wrote:That's the point -- *which* checks are missing from safe?So, I'm a function `f`, I have an `immutable(type)[]` argument and I want to store it for my friend `g` in an TLS variable `v`: --- string v; debug string sure; void f(string s) { v = s; debug sure = s.idup; } void g() { assert(v == sure); } --- I also store a copy of `s` into `sure` for my friend to ensure immutable date hasn't been mutated. Can my friend's assertion ever fail without breaking a type-system? Sure. Just consider this: --- void main() { auto s = "abba".idup; f(s); delete s; g(); } --- Is it by-design? Looks like deleting immutable (and const because of implicit conversion) data should be prohibited. OK. Let `delete` be fixed. Can we still fail? --- void h() { immutable(char)[4] s = "abba"; f(s); } void main() { h(); g(); } --- Damn! So, what can we do with it? Not sure, but I have a proposal. Fix it in language: * disallow `delete` of const/immutable data * disallow immutable data on the stack This makes data really immutable if I don't miss something. Anyway, I want `immutable` qualified data to be immutable without breaking a type-system (if one do it, its his own responsibility), so some changes should be made (IMHO).You are using unsafe language features to break the type system. That is not the fault of the type system. ' safe:' at the top of the program should stop both examples from working, it is a bug that it does not.Lifetime and mutability are different things.No it is not. Data on the stack *cannot* survive past the end of the function call. Data on the heap can last forever.But I'm not sure that you're right, this looks broken to me, even without safe. What does it mean to create immutable data on the stack? The stack is intrinsically mutable!So is the heap.Yes. The alternative (to allow explicit delete on immutable data) would likely be too complicated to be worth implementing in the near future - you need to ensure the data is unique, there are no other refs to it, and forbid accessing it after the 'delete' op. I guess an easy way out would be to ask the GC to run a collect cycle and return back whether an object was successfully collected. But i can't really see a useful application for it, and you'd need a special convention, as the GC would have to given the last ref to the object.What does it mean to garbage collect immutable data?From the point of view of the application, it doesn't happen. There are no observable semantics. It's merely an implementation detail.What does it mean to allocate an 'int' on the stack?No, you've broken the type system if you've deleted immutable data. If I have a reference to an immutable variable, I have a guarantee that it will never change. delete will break that guarantee.What does it mean to delete immutable data?Deallocate the storage for it and make it available for reuse. Accessing it afterwards leads to arbitrary behaviour. This is the same with mutable data. As the program may behave arbitrarily in this case, it is valid behaviour to act as if immutable data changed.With a mutable variable, I have no such guarantee. (It's not safe to allocate something different in the deleted location, but it's OK to run the finalizer and then wipe all the memory).Trivial. Obviously, it would need a properly working 'scope'. /Then/ making it illegal to pass refs to local (or otherwise) scoped object in a way that would allow them to escape would work. Right now, when 'scope' is just decoration, trying to enforce such restrictions would only create more bugs and confusion, I'm afraid. The other reason why a working 'scope' is important is so that void g(string s) { f(s~"xyz"); } does not cause heap allocation unless necessary. Even when the allocation is necessary the new object can be reused and the given back to the allocator when it's really dead. Reducing the amount of garbage left behind is the best way to optimize the GC... [of course 'scope' should have been the default etc etc, but that's not really D2 material] arturNo we can't. f cannot know that the string it has been given is on the stack. So main() must prevent it from being given to f() in the first place. How can it do that? void foo(bool b, string y) { immutable (char)[4] x = "abba"; string s = b ? x : y; f(s); } Make it safe.I think it's reasonable for both of them to require a cast, even in system code.The implementation of the 'scope' storage class should be fixed. We could then require an unsafe cast(scope) to disable prevention of stack address escaping.
Oct 18 2012
On 10/18/2012 10:08 AM, Don Clugston wrote:On 17/10/12 18:02, Timon Gehr wrote:Static escape analysis. Use the 'scope' qualifier to designate data that is not allowed to be escaped in order to make it modular.On 10/17/2012 01:49 PM, Don Clugston wrote:HOW do you propose to check for escaping stack data?... That's the point -- *which* checks are missing from safe?Escaping stack data and arbitrarily freeing memory are not operations found in memory safe languages....f can know that it mustn't escape it, which is enough.The implementation of the 'scope' storage class should be fixed. We could then require an unsafe cast(scope) to disable prevention of stack address escaping.No we can't. f cannot know that the string it has been given is on the stack. So main() must prevent it from being given to f() in the first place. How can it do that?void foo(bool b, string y) { immutable (char)[4] x = "abba"; string s = b ? x : y; f(s); } Make it safe.It is safe if the parameter to f is marked with 'scope'. (and this in turn obliges f not to escape it.) Analyze scope on the expression level. The analysis would determine that x[] is 'scope'. It would conservatively propagate this fact to (b ? x[] : y). Then the local variable 's' will get the 'scope' storage class. In general, use a fixed-point iteration to determine all local variables that might refer to scope'd data and prevent that they get escaped.Now I see where you are coming from. This is indeed a safe approach for references to/arrays of fully mutable value types, but not for delete in general. Make sure to treat void* specially though. struct S{ immutable int x; this(int x){this.x=x;}} void main() safe{ void* s = new S(2); delete s; } Class instance memory does not have a T.init, because it is not assigned a T. And even if it was, how would you know at compile time if the bound instance has any immutable fields? Should that be a runtime exception?Rust's borrowed pointers may give some hints on how to extend 'scope' to fields of structs.I think it is more fundamental than that.As to delete, delete is as unsafe when the involved data is immutable as when it is mutable. Why require an additional cast in one case?This is not about safety. Modifying immutable data breaks the type system. Deleting mutable data does not. AFAIK it is safe to implement delete as a call to the finalizer, followed by setting the memory to T.init. ...
Oct 18 2012
On 18/10/12 19:43, Timon Gehr wrote:On 10/18/2012 10:08 AM, Don Clugston wrote:Well, OK, but that involves changing the semantics of immutable. You could not pass this kind of "local immutable" to _any_ existing code. It would render almost all existing code that uses immutable obsolete. And what do you get in exchange? Practically nothing!On 17/10/12 18:02, Timon Gehr wrote:Static escape analysis. Use the 'scope' qualifier to designate data that is not allowed to be escaped in order to make it modular.On 10/17/2012 01:49 PM, Don Clugston wrote:HOW do you propose to check for escaping stack data?... That's the point -- *which* checks are missing from safe?Escaping stack data and arbitrarily freeing memory are not operations found in memory safe languages....f can know that it mustn't escape it, which is enough.The implementation of the 'scope' storage class should be fixed. We could then require an unsafe cast(scope) to disable prevention of stack address escaping.No we can't. f cannot know that the string it has been given is on the stack. So main() must prevent it from being given to f() in the first place. How can it do that?void foo(bool b, string y) { immutable (char)[4] x = "abba"; string s = b ? x : y; f(s); } Make it safe.It is safe if the parameter to f is marked with 'scope'. (and this in turn obliges f not to escape it.)Analyze scope on the expression level. The analysis would determine that x[] is 'scope'. It would conservatively propagate this fact to (b ? x[] : y). Then the local variable 's' will get the 'scope' storage class. In general, use a fixed-point iteration to determine all local variables that might refer to scope'd data and prevent that they get escaped.Probably. Yeah, it's a bit hard if you have a base class, you can't statically check if it has immutable members in a derived class. Or you could be conservative and disallow delete of anything where you don't know the exact type at compile time.Now I see where you are coming from. This is indeed a safe approach for references to/arrays of fully mutable value types, but not for delete in general. Make sure to treat void* specially though. struct S{ immutable int x; this(int x){this.x=x;}} void main() safe{ void* s = new S(2); delete s; } Class instance memory does not have a T.init, because it is not assigned a T. And even if it was, how would you know at compile time if the bound instance has any immutable fields? Should that be a runtime exception?Rust's borrowed pointers may give some hints on how to extend 'scope' to fields of structs.I think it is more fundamental than that.As to delete, delete is as unsafe when the involved data is immutable as when it is mutable. Why require an additional cast in one case?This is not about safety. Modifying immutable data breaks the type system. Deleting mutable data does not. AFAIK it is safe to implement delete as a call to the finalizer, followed by setting the memory to T.init. ...
Oct 22 2012