digitalmars.D - Round V, Immutable arrays and pointers. Possible implementation.
- Andrew Fedoniouk (76/76) Jul 09 2005 For the purpose of this message let's assume that C++ 'const' model
- Uwe Salomon (98/108) Jul 10 2005 This struct is useless, as it is not possible to change its values. The ...
- Uwe Salomon (6/8) Jul 10 2005 This model would also combine the requirements of all three parties:
- Andrew Fedoniouk (29/138) Jul 10 2005 It is not so useless as you may think. Remeber how 'private' works in D?
- Uwe Salomon (4/8) Jul 10 2005 Did you read this other proposal? What do you think about it?
- David L. Davis (20/44) Jul 10 2005 The immutable contract method does seem like the best approach so far, a...
- Andrew Fedoniouk (24/80) Jul 10 2005 "somehow ensure" is main concern here.
- Uwe Salomon (125/138) Jul 11 2005 This immutable statement would be a contract like in, out, unittest and ...
- Andrew Fedoniouk (36/174) Jul 11 2005 Ok. I've got your idea.
- Uwe Salomon (4/24) Jul 11 2005 Nah, you cannot use local variable as template alias parameters. But you...
- Andrew Fedoniouk (19/19) Jul 10 2005 Possible syntax for immutable arrays and pointers.
For the purpose of this message let's assume that C++ 'const' model can be qualified as 'immutable reference model' - C++ does not guarantee immutability of resources/objects but guarantees that it is not possible to change state of the data/object through immutable reference (with defined exceptions). Seems like idea to have in D model of constness used by C++ is a) not quite popular (e.g. cluttering) and b) is not easy in implementation - will make compiler heavier and c) too complex for most cases. D has three reference types: arrays (includng assosiative arrays), pointers and object references. Let's narrow the task and agree that immutable references are highly desireable feature for arrays and pointers and nice to have feature (means not so critical) for object references. Rationale: Objects, to be precise - their classes, can be designed in the way to protect their "internal parts". Arrays and pointers are primitive types and cannot (?) be extended to implement protection of data they are referring to. Variable of array or pointer type may be considered as an observer or alias of some memory location - data. Arrays and pointers in D are read-write observers. Immutable observer then is a variable of type similar to base reference type but having only read-only set of methods/operators. Sidenote: In C++ type and const type are two distinct types: <blockquote> 3.9.3 CV-qualifiers .... The cv-qualified or cv-unqualified versions of a type are distinct types; </blockquote> And here popups that simple solution I've made before: To introduce two new explicit(sic!) types: read-only array (slice and assosiation) and read-only pointer with the following properties: read-only array/assosiation - no opIndexAssign and length(uint newlength) methods, ptr attribute has type of read-only pointer. read-only pointer - dereference cannot be lvalue. Variants of syntax of such types were already proposed. I believe that implementation of these new types in compiler will be simple as these types use only functionality already exists in compiler. One question remains: what to do with arrays containing structures or pointers to structures? Example: struct A { int m; int n; } a[0] = a[1]; // CT error, no opIndexAssign a[0].m = 1; // but this will compiled without error, undesireable! Possible solution: do exactly nothing as if such protection needed developer can declare struct as struct A { private int _m; private int _n; int m() { return _m; } int n() { return _n; } } ---- and in other module (consumer): a[0] = a[1]; // CT error, no opIndexAssign a[0].m = 1; // CT error, 'm(int)' - no such function a[0]._m = 1; // CT error, '_m' is not accessible ----------------------------------------------- I think this solution will satisfy major const cases and is quite D-ish - simple and effective. ----------------------------------------------- What do you guys think? Andrei Fedoniouk. http://terrainformatica.com
Jul 09 2005
Possible solution: do exactly nothing as if such protection needed developer can declare struct as struct A { private int _m; private int _n; int m() { return _m; } int n() { return _n; } }This struct is useless, as it is not possible to change its values. The trick behind “const” is that some part of the program creates some information and changes it, and then passes it on to another part in an immutable form.What do you guys think?Well, it is a simple solution, i must admit that. But in *simple* programs i will not need that (because i can overlook the whole structure). The “const” feature is most useful for large scale programs, to ensure that everything works according to the rules defined before in the specification. In large scale programs i use complex types, lots of classes and modules. Of what use is an immutable slice there: class ComplexClass { // Lots of members, functions that only read the // contents, and those that write, too. } void someFunc(ComplexClass[] complexList) { foreach (ComplexClass cl; complexList) { // Now ensure that i only call “reader” functions // on cl, as this is the requirement by the specs. } } That's what is really needed in larger programs, i think. I would vote for a solution that really covers this instead of a half-hearted attempt that will leave unsatisfied desires anyways, but still increases compiler&syntax complexity. And if that full solution is not feasible, then better forget about it as a whole, and write good specs before and adhere to it. I know that this is a very easygoing point of view ─ rejecting every good idea you come up with. Sorry. I'm quite impressed with the variety of your implementation ideas. This topic really seems to concern you a lot... Never did for me, i had only trouble with constness in the past, as there is nothing like an access control list that says “object A, B, C may only read object X, but D and E can also change them, except that E must not change member y.” I experienced this kind of program situation quite often, and there seems to be no hard-coded solution for it, only writing comments. ;o) By the way, what about some kind of “immutable code section” as a debugging statement: void someFunc() { SomeType someVar; someVar = someValue; // some comment :) immutable (&someVar) { someOtherFunc(someVar); } // do some more things with someVar. } The immutable statement would take a pointer (a list of pointers), and somehow ensure that the memory pointed to by it (them) is not modified in its block statement, for example by making a copy and comparing it to the original value at the end, or by blocking the memory pointed to by it. This would be a runtime checking, and compiled away in -release builds just like all other contracts. Use it like this: int* pInt; int[] intAry; SomeClass cl; * immutable(pInt) makes 4 bytes beginning at address pInt immutable. * immutable(intAry) makes 4*intAry.length bytes immutable (the whole array). * immutable(intAry[3..5]) makes only this section of memory immutable. * immutable(cl) makes cl.init.sizeof bytes immutable (the whole class). As you can see, all variants designate some defined part of the memory as immutable. Generalized to a void[] array this would mean copying this part of the memory at the beginning (regardless of the contents, just a simple & fast byte copy), and comparing the copy to the actual contents at the end of the immutable block. For classes/structs there could be an opImmutable(): class SomeClass { private: SomeOtherClass m_cl; int[] someArray; public: void opImmutable(void delegate() dg) { immutable (m_cl, someArray) { dg(); } } } This works like the foreach loop: The contents of an immutable section with class SomeClass involved are converted into an internal delegate, and opImmutable is called with this delegate. This would even work for derived classes, as opImmutable can be a virtual function!!! Ciao uwe
Jul 10 2005
By the way, what about some kind of “immutable code section” as a debugging statement:This model would also combine the requirements of all three parties: - “I need const to protect my code from myself!” - “Const should be useful for the compiler, not only syntactic ballast!” - “I do not want const to produce clutter!” Ciao uwe
Jul 10 2005
"Uwe Salomon" <post uwesalomon.de> wrote in message news:op.stosylai6yjbe6 sandmann.maerchenwald.net...It is not so useless as you may think. Remeber how 'private' works in D? In module where A defined you can access private fields. You can also set this fields in constructor or initializer method. This is the whole idea of such protection. If you want to create structure or class externally looks as readonly you can always solve this with access attributes. Yes it needs careful design in case of complex structures or classes but it is already possible and in most cases is just enough. Personally I've found this private/public solution pretty sufficient for the problem.Possible solution: do exactly nothing as if such protection needed developer can declare struct as struct A { private int _m; private int _n; int m() { return _m; } int n() { return _n; } }This struct is useless, as it is not possible to change its values. The trick behind "const" is that some part of the program creates some information and changes it, and then passes it on to another part in an immutable form.As you may see you already can protect such classes/structures But currently you cannot protect slices and pointers. At all. Just no way. Think about strings for example. The whole Idea of proposal is e.g. in to be able to say: class Recordset { private Field[] _fields; }What do you guys think?Well, it is a simple solution, i must admit that. But in *simple* programs i will not need that (because i can overlook the whole structure). The "const" feature is most useful for large scale programs, to ensure that everything works according to the rules defined before in the specification. In large scale programs i use complex types, lots of classes and modules.Of what use is an immutable slice there: class ComplexClass { // Lots of members, functions that only read the // contents, and those that write, too. } void someFunc(ComplexClass[] complexList) { foreach (ComplexClass cl; complexList) { // Now ensure that i only call "reader" functions // on cl, as this is the requirement by the specs. } }Again, declare sensitive methods as 'private' or 'package' and left readers public. This is simple and yet more flexible as by using different levels of access you can limit who can modify your objects and what.That's what is really needed in larger programs, i think. I would vote for a solution that really covers this instead of a half-hearted attempt that will leave unsatisfied desires anyways, but still increases compiler&syntax complexity. And if that full solution is not feasible, then better forget about it as a whole, and write good specs before and adhere to it.I think that you initially did not take private/protected/package/public in consideration. They *already* provide you such solution. Andrew.I know that this is a very easygoing point of view � rejecting every good idea you come up with. Sorry. I'm quite impressed with the variety of your implementation ideas. This topic really seems to concern you a lot... Never did for me, i had only trouble with constness in the past, as there is nothing like an access control list that says "object A, B, C may only read object X, but D and E can also change them, except that E must not change member y." I experienced this kind of program situation quite often, and there seems to be no hard-coded solution for it, only writing comments. ;o) By the way, what about some kind of "immutable code section" as a debugging statement: void someFunc() { SomeType someVar; someVar = someValue; // some comment :) immutable (&someVar) { someOtherFunc(someVar); } // do some more things with someVar. } The immutable statement would take a pointer (a list of pointers), and somehow ensure that the memory pointed to by it (them) is not modified in its block statement, for example by making a copy and comparing it to the original value at the end, or by blocking the memory pointed to by it. This would be a runtime checking, and compiled away in -release builds just like all other contracts. Use it like this: int* pInt; int[] intAry; SomeClass cl; * immutable(pInt) makes 4 bytes beginning at address pInt immutable. * immutable(intAry) makes 4*intAry.length bytes immutable (the whole array). * immutable(intAry[3..5]) makes only this section of memory immutable. * immutable(cl) makes cl.init.sizeof bytes immutable (the whole class). As you can see, all variants designate some defined part of the memory as immutable. Generalized to a void[] array this would mean copying this part of the memory at the beginning (regardless of the contents, just a simple & fast byte copy), and comparing the copy to the actual contents at the end of the immutable block. For classes/structs there could be an opImmutable(): class SomeClass { private: SomeOtherClass m_cl; int[] someArray; public: void opImmutable(void delegate() dg) { immutable (m_cl, someArray) { dg(); } } } This works like the foreach loop: The contents of an immutable section with class SomeClass involved are converted into an internal delegate, and opImmutable is called with this delegate. This would even work for derived classes, as opImmutable can be a virtual function!!! Ciao uwe
Jul 10 2005
I think that you initially did not take private/protected/package/public in consideration. They *already* provide you such solution.Yes, i forgot about that.Did you read this other proposal? What do you think about it? Ciao uweBy the way, what about some kind of "immutable code section" as a debugging statement:
Jul 10 2005
In article <op.stosylai6yjbe6 sandmann.maerchenwald.net>, Uwe Salomon says...... The immutable statement would take a pointer (a list of pointers), and somehow ensure that the memory pointed to by it (them) is not modified in its block statement, for example by making a copy and comparing it to the original value at the end, or by blocking the memory pointed to by it. This would be a runtime checking, and compiled away in -release builds just like all other contracts. Use it like this: int* pInt; int[] intAry; SomeClass cl; * immutable(pInt) makes 4 bytes beginning at address pInt immutable. * immutable(intAry) makes 4*intAry.length bytes immutable (the whole array). * immutable(intAry[3..5]) makes only this section of memory immutable. * immutable(cl) makes cl.init.sizeof bytes immutable (the whole class). As you can see, all variants designate some defined part of the memory as immutable. Generalized to a void[] array this would mean copying this part of the memory at the beginning (regardless of the contents, just a simple & fast byte copy), and comparing the copy to the actual contents at the end of the immutable block. ... Ciao uweThe immutable contract method does seem like the best approach so far, as long as the compiler itself applys a set of default contract rules to all the existing D code, leaving the developer to use the immutable {} block contracts only for any special cases (those that are the opposite of the default). Otherwise one may end up with a lot a code source clutter, much like C++'s const. I've found in my own code that the unittest {} contract can be very useful for testing code, but that it's only as useful for the test cases based the developer decides to tests for (which could be incomplete). Thus, there may be different values that when passed into a contract would normally produce an error could be missed during testing and debugging phase, which would end up in the -release version of the executable code. With that said, I think that a compiler applied immutable contract, could work hand in hand with a good unittest contract giving a great benefit to all D developers. David L. ------------------------------------------------------------------- "Dare to reach for the Stars...Dare to Dream, Build, and Achieve!" ------------------------------------------------------------------- MKoD: http://spottedtiger.tripod.com/D_Language/D_Main_XP.html
Jul 10 2005
[first part is skiped as I answered in previous post]By the way, what about some kind of "immutable code section" as a debugging statement: void someFunc() { SomeType someVar; someVar = someValue; // some comment :) immutable (&someVar) { someOtherFunc(someVar); } // do some more things with someVar. } The immutable statement would take a pointer (a list of pointers), and somehow ensure that the memory pointed to by it (them) is not modified in its block statement, for example by making a copy and comparing it to the original value at the end, or by blocking the memory pointed to by it. This would be a runtime checking, and compiled away in -release builds just like all other contracts."somehow ensure" is main concern here. Try to place yourself in compiler shoes. How you would ensure immutability for any given array, pointer, class? Practically you will need to reproduce the whole system of what C++ has now for 'const' or to implement readonly verification in runtime, which is highly undesireable.Use it like this: int* pInt; int[] intAry; SomeClass cl; * immutable(pInt) makes 4 bytes beginning at address pInt immutable.How to verify this? Remeber that address is not known at compile time.* immutable(intAry) makes 4*intAry.length bytes immutable (the whole array). * immutable(intAry[3..5]) makes only this section of memory immutable.How to verify this? Don't forget about slicing.* immutable(cl) makes cl.init.sizeof bytes immutable (the whole class).How to implement verification of immutability in this case? Is this about run time or compile time, btw?As you can see, all variants designate some defined part of the memory as immutable. Generalized to a void[] array this would mean copying this part of the memory at the beginning (regardless of the contents, just a simple & fast byte copy), and comparing the copy to the actual contents at the end of the immutable block.... and what will happen if it will not compare?For classes/structs there could be an opImmutable(): class SomeClass { private: SomeOtherClass m_cl; int[] someArray; public: void opImmutable(void delegate() dg) { immutable (m_cl, someArray) { dg(); } } } This works like the foreach loop: The contents of an immutable section with class SomeClass involved are converted into an internal delegate, and opImmutable is called with this delegate. This would even work for derived classes, as opImmutable can be a virtual function!!!I didn't get the idea of opImmutable... :( This "Matreshka" ( http://www.bestofrussia.ca/matreshka.html ) : void opImmutable(void delegate() dg) { immutable (m_cl, someArray) { dg(); } } looks suspicious for me. Could you shed some light here, what is the purpose? Andrew.
Jul 10 2005
Ok, we start over here:How to implement verification of immutability in this case? Is this about run time or compile time, btw?This immutable statement would be a contract like in, out, unittest and is therefore checked at run-time only. The difference is that the contents of immutable are always in the executable, whilst the code added before and behind it is dropped if compiled with -release. In the simplest case (the only downside to it is memory hunger) the added code would be something like this: int* somePointer; immutable (somePointer) { // Statements... } is replaced by: int* somePointer; int _copy = *somePointer; { // Statements... } if (*somePointer != _copy) throw ImmutableException(...); If somePointer were a dynamic array of ints instead of a pointer, it would be something like this: int[] somePointer; if (somePointer.length * int.sizeof >= SOME_MAXIMUM_AMOUNT) void[] _copy = somePointer.dup; else { // allocate _copy on the stack via SUB ESP, needed_size memcpy(_copy.ptr, somePointer.ptr, _copy.length); } { // Statements... } if (memcmp(_copy.ptr, somePointer.ptr, _copy.length) != 0) throw ImmutableException(...); if (_copy.length >= SOME_MAXIMUM_AMOUNT) delete _copy; else // take _copy from the stack. This example is a bit compiler-ish, and just an idea. If the needed space is less than a certain amount, the copy is put on the stack, otherwise on the heap. This could be changed to always use the heap. The principle is the same anyways: First take a copy, then execute the statements, then compare and assert if not identical. The downside to this implementation of the immutable() statement is: It costs memory. It can cost a lot of memory, especially when used wrong/everywhere. Imagine an Indigo container with 500.000 items that calls some helper function and wants to ensure immutability for its contents. $( I am not into assembler very much, thus i am not sure if the other possibility implementing immutable() exists: locking this part of the memory. Perhaps it is possible for whole pages?I didn't get the idea of opImmutable... :( This "Matreshka" ( http://www.bestofrussia.ca/matreshka.html ) :I know Matreshkas :) i am from former Eastern Germany...void opImmutable(void delegate() dg) { immutable (m_cl, someArray) { dg(); } } looks suspicious for me. Could you shed some light here, what is the purpose?Imagine a class SomeClass. If you put a reference to it into the immutable() statement like this: SomeClass cl; immutable (cl) { // Do something. } This would currently be expanded to something like this: void[] _copy = new void[SomeClass.init.sizeof]; memcpy(_copy, cl, _copy.sizeof); { // Do something. } if (memcmp(_copy, cl, _copy.sizeof) != 0) throw ImmutableException(...); delete _copy; Now this ensures immutability for the direct contents of the object. But what if the object has pointers as members that it wants to protect? class SomeClass { int[] myArray; // etc. } The above would ensure that the pointer is not changed, but it would not ensure that the contents of myArray are not changed. How to do that? This is where opImmutable() kicks in: If the compiler has to “immutabilize” a struct/class/union, and this struct/class/union provides a member function opImmutable(), it replaces the immutable() section like this: SomeClass cl; cl.opImmutable(void delegate() { // Do something. }); This is somehow similar to foreach loops, where there is also created an anonymous delegate which is then passed to opApply(). Now the opImmutable() member function of the class looks like this: void opImmutable(void delegate() dg) { immutable (myArray) { dg(); } } As you can see, it declares the arrays contents as immutable, and then calls the delegate. Thus, if you inlined all these functions, you would receive this code: SomeClass cl; immutable (cl.myArray) { // Do something. } And that is exactly what we want. Now the array's contents are checked for changes, not the pointer to them. Ciao uwe
Jul 11 2005
"Uwe Salomon" <post uwesalomon.de> wrote in message news:op.stqlqvaf6yjbe6 sandmann.maerchenwald.net...Ok, we start over here:Ok. I've got your idea. I'd classify it as "post-mortem immutability verification" :) or "immutability violation detector" and not as "declarative immutability" (compile time) As it all happens in runtime then I believe it is already possible to implement it in D now. Rough sketch: template ivDetector(alias V ) { auto IvStorage __ts = new IvStorage( V, (*V).sizeof ); } class IvStrorage { void[] _data; void* _p; this( void* p, uint sz ) { _p = p; _data = p[ 0..sz ]; } ~this( ) { compare data here and rise exception if... } // bad, but.... } And use it as: int* somePointer; //immutable (somePointer) { mixin ivDetector!(somePointer); // Statements... } Will it work for you?How to implement verification of immutability in this case? Is this about run time or compile time, btw?This immutable statement would be a contract like in, out, unittest and is therefore checked at run-time only. The difference is that the contents of immutable are always in the executable, whilst the code added before and behind it is dropped if compiled with -release. In the simplest case (the only downside to it is memory hunger) the added code would be something like this: int* somePointer; immutable (somePointer) { // Statements... } is replaced by: int* somePointer; int _copy = *somePointer; { // Statements... } if (*somePointer != _copy) throw ImmutableException(...); If somePointer were a dynamic array of ints instead of a pointer, it would be something like this: int[] somePointer; if (somePointer.length * int.sizeof >= SOME_MAXIMUM_AMOUNT) void[] _copy = somePointer.dup; else { // allocate _copy on the stack via SUB ESP, needed_size memcpy(_copy.ptr, somePointer.ptr, _copy.length); } { // Statements... } if (memcmp(_copy.ptr, somePointer.ptr, _copy.length) != 0) throw ImmutableException(...); if (_copy.length >= SOME_MAXIMUM_AMOUNT) delete _copy; else // take _copy from the stack. This example is a bit compiler-ish, and just an idea. If the needed space is less than a certain amount, the copy is put on the stack, otherwise on the heap. This could be changed to always use the heap. The principle is the same anyways: First take a copy, then execute the statements, then compare and assert if not identical. The downside to this implementation of the immutable() statement is: It costs memory. It can cost a lot of memory, especially when used wrong/everywhere. Imagine an Indigo container with 500.000 items that calls some helper function and wants to ensure immutability for its contents. $( I am not into assembler very much, thus i am not sure if the other possibility implementing immutable() exists: locking this part of the memory. Perhaps it is possible for whole pages?:) BTW, I was in Dresden month ago.I didn't get the idea of opImmutable... :( This "Matreshka" ( http://www.bestofrussia.ca/matreshka.html ) :I know Matreshkas :) i am from former Eastern Germany...Thanks, got it too. It is a "deep post-mortem immutability verification" :) I guess situations like this are pretty rare and when needed can be solved individually using approach above. In fact if you will have readonly referencing then violation detection will be detected at compile time in most cases eliminating need of creating massive memory snapshots. Andrew.void opImmutable(void delegate() dg) { immutable (m_cl, someArray) { dg(); } } looks suspicious for me. Could you shed some light here, what is the purpose?Imagine a class SomeClass. If you put a reference to it into the immutable() statement like this: SomeClass cl; immutable (cl) { // Do something. } This would currently be expanded to something like this: void[] _copy = new void[SomeClass.init.sizeof]; memcpy(_copy, cl, _copy.sizeof); { // Do something. } if (memcmp(_copy, cl, _copy.sizeof) != 0) throw ImmutableException(...); delete _copy; Now this ensures immutability for the direct contents of the object. But what if the object has pointers as members that it wants to protect? class SomeClass { int[] myArray; // etc. } The above would ensure that the pointer is not changed, but it would not ensure that the contents of myArray are not changed. How to do that? This is where opImmutable() kicks in: If the compiler has to "immutabilize" a struct/class/union, and this struct/class/union provides a member function opImmutable(), it replaces the immutable() section like this: SomeClass cl; cl.opImmutable(void delegate() { // Do something. }); This is somehow similar to foreach loops, where there is also created an anonymous delegate which is then passed to opApply(). Now the opImmutable() member function of the class looks like this: void opImmutable(void delegate() dg) { immutable (myArray) { dg(); } } As you can see, it declares the arrays contents as immutable, and then calls the delegate. Thus, if you inlined all these functions, you would receive this code: SomeClass cl; immutable (cl.myArray) { // Do something. } And that is exactly what we want. Now the array's contents are checked for changes, not the pointer to them.
Jul 11 2005
template ivDetector(alias V ) { auto IvStorage __ts = new IvStorage( V, (*V).sizeof ); } class IvStrorage { void[] _data; void* _p; this( void* p, uint sz ) { _p = p; _data = p[ 0..sz ]; } ~this( ) { compare data here and rise exception if... } // bad, but.... } And use it as: int* somePointer; //immutable (somePointer) { mixin ivDetector!(somePointer); // Statements... } Will it work for you?Nah, you cannot use local variable as template alias parameters. But you are right, this is possible with a little template magic and auto classes. Ciao uwe
Jul 11 2005
Possible syntax for immutable arrays and pointers. declarations: // arrays char a[]; // RW array/slice char a[ 100 ] ; // static RW array/slice. // pointers char * pa; // RW pointer // associative arrays/maps int[ char[] ] a; // RW map ---------------------------- Rationale: readonly attribute syntacticaly should be as close as possible to array/pointer designator ( [ ] ), ideally it should be part of it. Andrew. http://terrainformatica.com
Jul 10 2005