www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Temporary suspension of disbelief (invariant)

reply bearophile <bearophileHUGS lycos.com> writes:
I have not asked this in the D.learn newsgroup because I think it may be a bit
interesting for other general people too.

In D contract programming the invariant is not called before/after private
methods. I think that in some cases you may want to disable the invariant even
in some public methods. If your class/struct has a public method that
invalidates the state of the class instance, and one public method that fixes
the class instance state (I am thinking about certain data structures where in
a first phase you may add many items, and then you ask the data structure to
clean up itself and become coherent. This may avoid some computations), I have
found this way to implement it:



import std.stdio: writeln;

class Foo {
    private int someInstanceState = 1;

    // A ghost field, but it does't vanish in release mode
    //   as it is supposed to do:
    // http://d.puremagic.com/issues/show_bug.cgi?id=5027
    private int validState_ghostField = 0;

    private bool isInvariantEnabled() {
        return validState_ghostField >= 0;
    }

    private void incValidState()
        in {
            assert(validState_ghostField < validState_ghostField.max);
        } body {
            validState_ghostField++;
        }

    private void decValidState()
        in {
            assert(validState_ghostField > validState_ghostField.min);
        } body {
            validState_ghostField--;
        }

    invariant() {
        writeln("invariant validState_ghostField: ", validState_ghostField);
        if (isInvariantEnabled()) {
            writeln("Invariant is enabled");
            // tests the coherence of the instance state
            assert(someInstanceState == 1);
        } else {
            writeln("Invariant is disabled");
        }

    }

    public void fixState() {
        scope(exit) incValidState(); // scope(exit)?
        writeln("fixState validState_ghostField: ", validState_ghostField);
        someInstanceState = 1;
    }

    public void invalidateState() {
        scope(exit) decValidState(); // scope(exit)?
        writeln("invalidateState validState_ghostField: ",
validState_ghostField);
        someInstanceState = 0;
    }
}

void main() {
    auto f = new Foo;
    writeln("State 1: ", f.someInstanceState);
    writeln("isInvariantEnabled: ", f.isInvariantEnabled());
    f.invalidateState();
    writeln("State 2: ", f.someInstanceState);
    writeln("isInvariantEnabled: ", f.isInvariantEnabled());
    f.fixState();
    writeln("State 3: ", f.someInstanceState);
    writeln("isInvariantEnabled: ", f.isInvariantEnabled());
}


(Like the GC disable/enable validState_ghostField goes up and down, and allows
nesting too.)

All this looks bug-prone, and surely hairy, but it looks potentially useful. Is
it a good idea to design a class that uses such temporary suspension of the
invariant?

Bye,
bearophile
Oct 26 2010
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
bearophile wrote:

 All this looks bug-prone, and surely hairy, but it looks potentially useful.
 Is it a good idea to design a class that uses such temporary suspension of
 the invariant?


An invariant that is not invariant is a meaningless attribute. It's like 
"logical constness" where classes claim to be const but aren't.
Oct 26 2010
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, October 26, 2010 17:25:20 Walter Bright wrote:

 bearophile wrote:

 All this looks bug-prone, and surely hairy, but it looks potentially
 useful. Is it a good idea to design a class that uses such temporary
 suspension of the invariant?


 
 An invariant that is not invariant is a meaningless attribute. It's like
 "logical constness" where classes claim to be const but aren't.


The only case that I'm aware of where it makes sense for a public function to 
not run the invariant is for opAssign() to not run it before it itself is run ( 
http://d.puremagic.com/issues/show_bug.cgi?id=5058 ), but that's because 
opAssign() is replacing the state of the object rather than updating it, so you 
don't care what it's state was beforehand. That's a special case though.

In the general case, it would overly complicate things to try and make an 
invariant not really be an invariant. If you really wanted to do that, you
could 
use class variables which kept the state and made it so that the asserts in the 
invariant weren't run after a particular function call until another particular 
function call were made. But that strikes me as a colossally bad idea. Still,
if 
you wanted such functionality, you _can_ get it, so I see no reason to 
complicate invariants just to make it easier to write what is likely bad code 
(or at least which is at a higher risk of being bad code).

- Jonathan M Davis
Oct 26 2010
parent Walter Bright <newshound2 digitalmars.com> writes:
Jonathan M Davis wrote:

 In the general case, it would overly complicate things to try and make an 
 invariant not really be an invariant. If you really wanted to do that, you
could 
 use class variables which kept the state and made it so that the asserts in
the 
 invariant weren't run after a particular function call until another
particular 
 function call were made. But that strikes me as a colossally bad idea. Still,
if 
 you wanted such functionality, you _can_ get it, so I see no reason to 
 complicate invariants just to make it easier to write what is likely bad code 
 (or at least which is at a higher risk of being bad code).


Exactly. Having a class invariant that cheats is like painting over rust. It 
might look new, but the rust will shortly break through.
Oct 26 2010
prev sibling parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 10/26/2010 07:25 PM, Walter Bright wrote:

 An invariant that is not invariant is a meaningless attribute. It's like
 "logical constness" where classes claim to be const but aren't.


an invariant which isn't used because it is too strict isn't much better.

Example:

class with some sort of state and public property function which grabs a 
bit out of some field or something

inside method, state violates invariant and I can't call this property 
function

I needed to use the property function, but I also needed the invariant 
so I shoehorned it into a regular function and explicitly made the 
calls. Ugly.

Though I suppose it was a slightly different case than what bearophile 
is proposing
Oct 26 2010
prev sibling next sibling parent reply Rainer Deyke <rainerd eldwood.com> writes:
On 10/26/2010 18:00, bearophile wrote:

 All this looks bug-prone, and surely hairy, but it looks potentially
 useful. Is it a good idea to design a class that uses such temporary
 suspension of the invariant?


I think invariants should be checked whenever a public member function
is called from outside the object.  If setting the object to an invalid
state from outside is valid, then clearly the state isn't really invalid.

On the other hand, if the object itself calls it own public member
functions, then no invariants should be checked.  Not being able to call
public member functions while the object is temporarily in an invalid
state is too strict.  This is a problem that I actually ran into while
using D, and one of the reasons for why I stopped using invariants.


-- 
Rainer Deyke - rainerd eldwood.com
Oct 26 2010
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Rainer Deyke wrote:

 On the other hand, if the object itself calls it own public member
 functions, then no invariants should be checked.  Not being able to call
 public member functions while the object is temporarily in an invalid
 state is too strict.  This is a problem that I actually ran into while
 using D, and one of the reasons for why I stopped using invariants.


A solution is to redesign what the class considers public and private. A public 
member can be a shell around a private implementation, and other class members 
can call that private implementation without invoking the invariant.
Oct 26 2010
parent reply Rainer Deyke <rainerd eldwood.com> writes:
On 10/26/2010 20:16, Walter Bright wrote:

 Rainer Deyke wrote:

 On the other hand, if the object itself calls it own public member
 functions, then no invariants should be checked.  Not being able to call
 public member functions while the object is temporarily in an invalid
 state is too strict.  This is a problem that I actually ran into while
 using D, and one of the reasons for why I stopped using invariants.


 
 A solution is to redesign what the class considers public and private. A
 public member can be a shell around a private implementation, and other
 class members can call that private implementation without invoking the
 invariant.


Writing wrapper functions is a waste of my time.  Auto-generating
wrapper functions through some sort of meta-programming magic is still a
waste of my time, since the process cannot be completely automated.


-- 
Rainer Deyke - rainerd eldwood.com
Oct 26 2010
parent Walter Bright <newshound2 digitalmars.com> writes:
Rainer Deyke wrote:

 On 10/26/2010 20:16, Walter Bright wrote:

 Rainer Deyke wrote:

 On the other hand, if the object itself calls it own public member
 functions, then no invariants should be checked.  Not being able to call
 public member functions while the object is temporarily in an invalid
 state is too strict.  This is a problem that I actually ran into while
 using D, and one of the reasons for why I stopped using invariants.


 A solution is to redesign what the class considers public and private. A
 public member can be a shell around a private implementation, and other
 class members can call that private implementation without invoking the
 invariant.


 
 Writing wrapper functions is a waste of my time.  Auto-generating
 wrapper functions through some sort of meta-programming magic is still a
 waste of my time, since the process cannot be completely automated.


Then I agree that not using invariants at all is probably your best option.
Oct 26 2010
prev sibling next sibling parent reply Stanislav Blinov <blinov loniir.ru> writes:
  27.10.2010 4:00, bearophile wrote:

 I have not asked this in the D.learn newsgroup because I think it may be a bit
interesting for other general people too.

 In D contract programming the invariant is not called before/after private
methods. I think that in some cases you may want to disable the invariant even
in some public methods. If your class/struct has a public method that
invalidates the state of the class instance, and one public method that fixes
the class instance state (I am thinking about certain data structures where in
a first phase you may add many items, and then you ask the data structure to
clean up itself and become coherent. This may avoid some computations), I have
found this way to implement it:


If I get this right, then it is by design that your class may have 
several general logical states: e.g. "initializing" and "coherent". 
Given this, I don't see why you'd want to disable invariant checks 
rather than modify those checks instead to validate current logical 
state. In fact, that "ghost" field simply serves as a flag for invariant 
showing which logical state it should enforce. The fact that states are 
'logical', e.g. are different while represented by the same physical 
fields doesn't always rule them out as, uh, class states: you could as 
well have two separate inner classes that perform initialization and 
polishing, each with its own invariant. Then you could use those inner 
classes' private methods (without causing their invariant checks), but 
in main class' invariant perform an assert on them to ensure their state 
is valid.

IMHO, proper invariants should be strict and act by the numbers at all 
times, otherwise there's little to no gain in using them at all.
Oct 27 2010
parent bearophile <bearophileHUGS lycos.com> writes:
Walter:


 An invariant that is not invariant is a meaningless attribute. It's like
 "logical constness" where classes claim to be const but aren't.
 
 It's not an invariant if it only works some of the time.
 
 It's like C++ and the mutable keyword, which allows an escape from the const
 rules. Which makes C++ const fairly useless.
 
 A person should be able to see the invariant in a class declaration, and know
 that it offers a guarantee. He should not be required to read over everything
 else in the source code looking for the absence of a method that disables it.


In this thread I have read many answers and comments, but I am not experienced
enough yet about invariants and DbC to be able to answer them, I will need more
time and more experiments. D after all is the first language where I have used
DbC.

I agree that an invariant that you may disable sounds like an oxymoron, and
probably it leads to hairy code at best, or even to bug-prone code.

Regarding "logical constness" see below my comments about the good answer by
Stanislav Blinov.

----------------------

Stanislav Blinov:


 If I get this right, then it is by design that your class may have
 several general logical states: e.g. "initializing" and "coherent".


Right. An example from the Clojure language (it's a recent Lisp-like language
that runs on the JavaVM). In Clojure most data is immutable. People that use
Clojure have seen that this is nice and safe, but sometimes on the JavaVM this
leads to low efficiency code especially when you create a data structure. So
they have solved this problem inventing "transients", it's an initial phase
when a data structure is being built, and it's mutable. Then when it's done it
becomes immutable. This often regains lot of the lost performance introduced by
immutables on the JVM.

At the same way, I think that in some situations I may want to quickly build a
data structure, in this phase the data structure may be internally not
orderered (like an unbalanced tree), so in this phase the invariant may be
false and it's better to disable it. (I think this is an uncommon situation, in
most cases there is probably a way around this problem). It's like the
transients for the immutability in Clojure.



 Given this, I don't see why you'd want to disable invariant checks
 rather than modify those checks instead to validate current logical
 state. In fact, that "ghost" field simply serves as a flag for invariant
 showing which logical state it should enforce. The fact that states are
 'logical', e.g. are different while represented by the same physical
 fields doesn't always rule them out as, uh, class states:


I think you are right.



 you could as
 well have two separate inner classes that perform initialization and
 polishing, each with its own invariant. Then you could use those inner
 classes' private methods (without causing their invariant checks), but
 in main class' invariant perform an assert on them to ensure their state
 is valid.


I see. I have never used this compound solution for the invariants. I will try
it (as inner ones I may use structs instead, structs too support invariants in
D), thank you for the idea :-)

Thank you for all the interesting answers in this thread, I am learning.

Bye,
bearophile
Oct 27 2010
prev sibling next sibling parent reply Fawzi Mohamed <fawzi gmx.ch> writes:
An issue I encountered in D1 with invariant is using delete: I have a  
method that deletes the current object, and obviously then any  
invariant would fail.
It would be nice to have a way to disable invariant in that case.
In D2, as delete is not allowed anymore (if I got it correctly) this  
is not a problem.

Fawzi
On 27-ott-10, at 15:08, Stanislav Blinov wrote:


 27.10.2010 4:00, bearophile wrote:

 I have not asked this in the D.learn newsgroup because I think it  
 may be a bit interesting for other general people too.

 In D contract programming the invariant is not called before/after  
 private methods. I think that in some cases you may want to disable  
 the invariant even in some public methods. If your class/struct has  
 a public method that invalidates the state of the class instance,  
 and one public method that fixes the class instance state (I am  
 thinking about certain data structures where in a first phase you  
 may add many items, and then you ask the data structure to clean up  
 itself and become coherent. This may avoid some computations), I  
 have found this way to implement it:


 If I get this right, then it is by design that your class may have  
 several general logical states: e.g. "initializing" and "coherent".  
 Given this, I don't see why you'd want to disable invariant checks  
 rather than modify those checks instead to validate current logical  
 state. In fact, that "ghost" field simply serves as a flag for  
 invariant showing which logical state it should enforce. The fact  
 that states are 'logical', e.g. are different while represented by  
 the same physical fields doesn't always rule them out as, uh, class  
 states: you could as well have two separate inner classes that  
 perform initialization and polishing, each with its own invariant.  
 Then you could use those inner classes' private methods (without  
 causing their invariant checks), but in main class' invariant  
 perform an assert on them to ensure their state is valid.

 IMHO, proper invariants should be strict and act by the numbers at  
 all times, otherwise there's little to no gain in using them at all.
Oct 27 2010
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Fawzi Mohamed wrote:

 An issue I encountered in D1 with invariant is using delete: I have a 
 method that deletes the current object, and obviously then any invariant 
 would fail.
 It would be nice to have a way to disable invariant in that case.
 In D2, as delete is not allowed anymore (if I got it correctly) this is 
 not a problem.


It's not an invariant if it only works some of the time.

It's like C++ and the mutable keyword, which allows an escape from the const 
rules. Which makes C++ const fairly useless.

A person should be able to see the invariant in a class declaration, and know 
that it offers a guarantee. He should not be required to read over everything 
else in the source code looking for the absence of a method that disables it.
Oct 27 2010
parent reply Fawzi Mohamed <fawzi gmx.ch> writes:
On 27-ott-10, at 20:20, Walter Bright wrote:


 Fawzi Mohamed wrote:

 An issue I encountered in D1 with invariant is using delete: I have  
 a method that deletes the current object, and obviously then any  
 invariant would fail.
 It would be nice to have a way to disable invariant in that case.
 In D2, as delete is not allowed anymore (if I got it correctly)  
 this is not a problem.


 It's not an invariant if it only works some of the time.

 It's like C++ and the mutable keyword, which allows an escape from  
 the const rules. Which makes C++ const fairly useless.

 A person should be able to see the invariant in a class declaration,  
 and know that it offers a guarantee. He should not be required to  
 read over everything else in the source code looking for the absence  
 of a method that disables it.



well for methods that destroy the object I think that not checking the  
invariant is something reasonable, and not a hole that destroys the  
whole system. Being able to write methods that invalidate the object  
can be useful IMHO.
At the moment in those objects I simply have no invariant, and by the  
way, no I don't think that having the invalidating methods always  
external is really a good solution.
You can do without, but sometime an  invalidate property (or similar)  
has its uses.

Fawzi
Oct 27 2010
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Fawzi Mohamed wrote:

 well for methods that destroy the object I think that not checking the 
 invariant is something reasonable, and not a hole that destroys the 
 whole system. Being able to write methods that invalidate the object can 
 be useful IMHO.
 At the moment in those objects I simply have no invariant, and by the 
 way, no I don't think that having the invalidating methods always 
 external is really a good solution.
 You can do without, but sometime an  invalidate property (or similar) 
 has its uses.


I agree that having a method that destroys an object can be useful. Such
classes 
can't have an invariant, however.

The problem with an  invalidate property is the user has to go through every 
declaration in the class looking for such a property (grep isn't sufficient 
because it can pick up false positives (nested functions) and can miss things 
inserted by templates and mixins). An invariant should be a guarantee, not
"this 
is guaranteed unless someone somewhere violates it, which is perfectly legal".

It's the same problem that C++ "mutable" produces, and it's not worth it.
Oct 27 2010
parent reply Fawzi Mohamed <fawzi gmx.ch> writes:
On 27-ott-10, at 20:57, Walter Bright wrote:


 Fawzi Mohamed wrote:

 well for methods that destroy the object I think that not checking  
 the invariant is something reasonable, and not a hole that destroys  
 the whole system. Being able to write methods that invalidate the  
 object can be useful IMHO.
 At the moment in those objects I simply have no invariant, and by  
 the way, no I don't think that having the invalidating methods  
 always external is really a good solution.
 You can do without, but sometime an  invalidate property (or  
 similar) has its uses.


 I agree that having a method that destroys an object can be useful.  
 Such classes can't have an invariant, however.

 The problem with an  invalidate property is the user has to go  
 through every declaration in the class looking for such a property  
 (grep isn't sufficient because it can pick up false positives  
 (nested functions) and can miss things inserted by templates and  
 mixins). An invariant should be a guarantee, not "this is guaranteed  
 unless someone somewhere violates it, which is perfectly legal".

 It's the same problem that C++ "mutable" produces, and it's not  
 worth it.



the only thing is that the invariant is not checked *after* those  
method that destroy the object, it will be ok to require a special  
name for that method (for example destroy, dealloc or similar) if you  
want to avoid misusage.
The next method called on the object would fail (as it should), I  
often store invalid info on purpose before deleting to increase the  
probability of detecting such usages.
invariant is still valid (and checked) before and after all methods.
It is even impossible to call twice the destroy method (as the check  
at the beginning of it would fail.
So I think that everything would stay perfectly well defined.
I think it is a different situation than mutable
Oct 27 2010
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Fawzi Mohamed wrote:

 the only thing is that the invariant is not checked *after* those method 
 that destroy the object, it will be ok to require a special name for 
 that method (for example destroy, dealloc or similar) if you want to 
 avoid misusage.


There is a method that does that, it's called the destructor. An invariant is 
not checked after the destructor is run.
Oct 27 2010
parent reply Fawzi Mohamed <fawzi gmx.ch> writes:
On 27-ott-10, at 22:41, Walter Bright wrote:


 Fawzi Mohamed wrote:

 the only thing is that the invariant is not checked *after* those  
 method that destroy the object, it will be ok to require a special  
 name for that method (for example destroy, dealloc or similar) if  
 you want to avoid misusage.


 There is a method that does that, it's called the destructor. An  
 invariant is not checked after the destructor is run.


ok I don't think that this is a major feature, or anything like that,  
so I will stop arguing after this,
but I still believe that it would be useful.
The destructor is an example of a method that should not check the  
invariant, but it does not address my use case, because one cannot  
explicitly call a destructor, and there is an important difference  
between a destructor called by the GC and a destructor called  
explicitly: the one called explicitly can access the memory of the  
object.

Not only that it might be useful for some methods to delete the object  
itself.
A classical example is reference counting, the last release (the one  
that sets the refCount to 0) should be able to delete the object.
It is not natural to make this impossible, or to force al reference  
counted object to have no invariant.
Oct 27 2010
parent Walter Bright <newshound2 digitalmars.com> writes:
Fawzi Mohamed wrote:

 Not only that it might be useful for some methods to delete the object 
 itself.


The pimpl pattern should be used for such, if you want to use an invariant.

Note that the C++0x shared_ptr uses pimpl in that the ref count is held 
separately from the object.
Oct 27 2010
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, October 27, 2010 07:33:58 Fawzi Mohamed wrote:

 An issue I encountered in D1 with invariant is using delete: I have a
 method that deletes the current object, and obviously then any
 invariant would fail.
 It would be nice to have a way to disable invariant in that case.
 In D2, as delete is not allowed anymore (if I got it correctly) this
 is not a problem.


Except that calling a function on a deleted object would not be desirable, so 
having the invariant fail at that point would be a _good_ thing. If anything, 
you'd want _all_ function calls on a deleted object to fail invariant or no 
invariant, because none of them should be happening in the first place.

clear() in D2 does (or at least did - I don't know what it's current state is) 
put an object in its default state prior to any constructor call which would 
likely violate any invariant, which, on the whole, is a good thing as well. 
However, there was discussion of making a cleared object have a nuked vtbl
which 
would be even better, since then all function calls on it would fail period.

- Jonathan M Davis
Oct 27 2010
prev sibling parent reply Fawzi Mohamed <fawzi gmx.ch> writes:
On 27-ott-10, at 18:59, Jonathan M Davis wrote:


 On Wednesday, October 27, 2010 07:33:58 Fawzi Mohamed wrote:

 An issue I encountered in D1 with invariant is using delete: I have a
 method that deletes the current object, and obviously then any
 invariant would fail.
 It would be nice to have a way to disable invariant in that case.
 In D2, as delete is not allowed anymore (if I got it correctly) this
 is not a problem.


 Except that calling a function on a deleted object would not be  
 desirable, so
 having the invariant fail at that point would be a _good_ thing. If  
 anything,
 you'd want _all_ function calls on a deleted object to fail  
 invariant or no
 invariant, because none of them should be happening in the first  
 place.

 clear() in D2 does (or at least did - I don't know what it's current  
 state is)
 put an object in its default state prior to any constructor call  
 which would
 likely violate any invariant, which, on the whole, is a good thing  
 as well.
 However, there was discussion of making a cleared object have a  
 nuked vtbl which
 would be even better, since then all function calls on it would fail  
 period.

 - Jonathan M Davis


..except that clear (just like my dealloc methods) at their end will  
call the invariant...
*that* is what I was talking about,
:)
Fawzi
Oct 27 2010
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
Fawzi Mohamed wrote:

 
 On 27-ott-10, at 18:59, Jonathan M Davis wrote:
 

 On Wednesday, October 27, 2010 07:33:58 Fawzi Mohamed wrote:

 An issue I encountered in D1 with invariant is using delete: I have a
 method that deletes the current object, and obviously then any
 invariant would fail.
 It would be nice to have a way to disable invariant in that case.
 In D2, as delete is not allowed anymore (if I got it correctly) this
 is not a problem.


 Except that calling a function on a deleted object would not be 
 desirable, so
 having the invariant fail at that point would be a _good_ thing. If 
 anything,
 you'd want _all_ function calls on a deleted object to fail invariant 
 or no
 invariant, because none of them should be happening in the first place.

 clear() in D2 does (or at least did - I don't know what it's current 
 state is)
 put an object in its default state prior to any constructor call which 
 would
 likely violate any invariant, which, on the whole, is a good thing as 
 well.
 However, there was discussion of making a cleared object have a nuked 
 vtbl which
 would be even better, since then all function calls on it would fail 
 period.

 - Jonathan M Davis


 ...except that clear (just like my dealloc methods) at their end will 
 call the invariant...
 *that* is what I was talking about,
 :)
 Fawzi


What for? It doesn't now, and it shouldn't (and most probably will not) 
when it's design is solid and it's properly implemented.
The purpose of clear() is:
1) call dtor (which implies checking invariant *only before* 
destruction), if object has it.
2) put an object in an invalid (from the compiler/runtime point of view) 
state: this has been discussed at length, and latest discussion ended 
with what Jonathan M Davis already mentioned: nuking object's virtual 
call table.
3) Leave object's memory alone so GC will later decide what to do with it

If I haven't missed something, this is pretty much it. It's not a 
method, it's a function -> no invariant checks after call.
Oct 27 2010