www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - How do you test pre-/post-conditions and invariants?

reply Magnus Lie Hetland <magnus hetland.org> writes:
Or, more generally, how do you test asserts (which is what I'm using in 
my preconditions etc.)?

As far as I can see, collectException() won't collect errors, which is 
what assert() throws -- so what's the standard way of writing unit 
tests for preconditions that use assert? (I.e., test that they will, in 
fact, throw when you break them.)

-- 
Magnus Lie Hetland
http://hetland.org
Feb 25 2011
next sibling parent reply spir <denis.spir gmail.com> writes:
On 02/25/2011 04:30 PM, Magnus Lie Hetland wrote:

 Or, more generally, how do you test asserts (which is what I'm using in my
 preconditions etc.)?

 As far as I can see, collectException() won't collect errors, which is what
 assert() throws -- so what's the standard way of writing unit tests for
 preconditions that use assert? (I.e., test that they will, in fact, throw when
 you break them.)


See the page on DbC: http://www.digitalmars.com/d/2.0/dbc.html.

Denis
-- 
_________________
vita es estrany
spir.wikidot.com
Feb 25 2011
next sibling parent Ary Manzana <ary esperanto.org.ar> writes:
On 2/25/11 1:48 PM, spir wrote:

 On 02/25/2011 04:30 PM, Magnus Lie Hetland wrote:

 Or, more generally, how do you test asserts (which is what I'm using
 in my
 preconditions etc.)?

 As far as I can see, collectException() won't collect errors, which is
 what
 assert() throws -- so what's the standard way of writing unit tests for
 preconditions that use assert? (I.e., test that they will, in fact,
 throw when
 you break them.)


 See the page on DbC: http://www.digitalmars.com/d/2.0/dbc.html.

 Denis


By the way, the link to "Adding Contracts to Java" at the bottom of the 
page is broken.
Feb 25 2011
prev sibling parent reply Magnus Lie Hetland <magnus hetland.org> writes:
On 2011-02-25 17:48:54 +0100, spir said:


 On 02/25/2011 04:30 PM, Magnus Lie Hetland wrote:

 Or, more generally, how do you test asserts (which is what I'm using in my
 preconditions etc.)?
 
 As far as I can see, collectException() won't collect errors, which is what
 assert() throws -- so what's the standard way of writing unit tests for
 preconditions that use assert? (I.e., test that they will, in fact, throw when
 you break them.)


 
 See the page on DbC: http://www.digitalmars.com/d/2.0/dbc.html.
 
 Denis


Hm. I'm not sure how this answers my question. I know how 
pre/postconditions etc. work -- I was asking for how to test them in a 
unit test (i.e., ensure that they'll kick in if you provide faulty 
intput, for example). Maybe I'm missing something in the doc you 
referred me to?

-- 
Magnus Lie Hetland
http://hetland.org
Feb 25 2011
parent spir <denis.spir gmail.com> writes:
On 02/25/2011 11:20 PM, Magnus Lie Hetland wrote:

 On 2011-02-25 17:48:54 +0100, spir said:


 On 02/25/2011 04:30 PM, Magnus Lie Hetland wrote:

 Or, more generally, how do you test asserts (which is what I'm using in my
 preconditions etc.)?

 As far as I can see, collectException() won't collect errors, which is what
 assert() throws -- so what's the standard way of writing unit tests for
 preconditions that use assert? (I.e., test that they will, in fact, throw when
 you break them.)


 See the page on DbC: http://www.digitalmars.com/d/2.0/dbc.html.

 Denis


 Hm. I'm not sure how this answers my question. I know how pre/postconditions
 etc. work -- I was asking for how to test them in a unit test (i.e., ensure
 that they'll kick in if you provide faulty intput, for example). Maybe I'm
 missing something in the doc you referred me to?


No, sorry, /I/ read your post too superficially: Hadn't caught you wanted to 
"meta-check" the checks.

Denis
-- 
_________________
vita es estrany
spir.wikidot.com
Feb 25 2011
prev sibling next sibling parent reply Jesse Phillips <jessekphillips+D gmail.com> writes:
Magnus Lie Hetland Wrote:


 Or, more generally, how do you test asserts (which is what I'm using in 
 my preconditions etc.)?
 
 As far as I can see, collectException() won't collect errors, which is 
 what assert() throws -- so what's the standard way of writing unit 
 tests for preconditions that use assert? (I.e., test that they will, in 
 fact, throw when you break them.)


There has been talk of a std.unittest that would help with this, but for now:

unittest {
    try {
         callMeWrong(wrong);
         assert(0);
    catch(AssertError e) {
    }
}

}
Feb 25 2011
parent Magnus Lie Hetland <magnus hetland.org> writes:
On 2011-02-25 19:18:40 +0100, Jesse Phillips said:


 There has been talk of a std.unittest that would help with this, but for now:



 unittest {
     try {
          callMeWrong(wrong);
          assert(0);
     catch(AssertError e) {
     }
 }
 
 }


Ah. I used something like...

auto thrown = 0;
try foo() catch (AssertError) thrown++;
...
assert(thrown == k);

I guess I could wrap it up a bit, or something.

-- 
Magnus Lie Hetland
http://hetland.org
Feb 25 2011
prev sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, February 25, 2011 07:30:50 Magnus Lie Hetland wrote:

 Or, more generally, how do you test asserts (which is what I'm using in
 my preconditions etc.)?
 
 As far as I can see, collectException() won't collect errors, which is
 what assert() throws -- so what's the standard way of writing unit
 tests for preconditions that use assert? (I.e., test that they will, in
 fact, throw when you break them.)


I think that the reality of the matter is the most of the time people _don't_ 
check them. And on some level, it doesn't make sense to. It's kind of like 
asking how people test their unit tests. Unit tests are already testing code.
Do 
you want to be testing them on top of that? And if you do, do you test _that_ 
code? Where do you stop?

Pre-conditions, post-conditions, and invariants are testing the class or struct 
that they're on, and to some extent, the code that uses them. So, if you're 
looking to test them, you're looking to test test code.

And testing post-conditions and invariants in the manner that you're trying to 
do borders on impossible. What are you going to do, repeat the post-condition
or 
invariant test on the result of the function or on the state of the object that 
the function was called on after the function was called? That's just doing the 
test twice. You might as well just re-read the post-conditions and invariants
to 
make sure that you wrote them correctly.

I do see value in testing pre-conditions if you're using exceptions rather than 
assertions (which means that you're not use in blocks). In that case, you're 
testing the API to make sure that it does what it's supposed to do. But if 
you're dealing with assertions, then it's really test code as opposed to API 
code, and I don't see the same value in testing that. You'd just be testing
test 
code.

Now, assuming that you _do_ want to test that sort of thing (as you obviously 
want to), some new unit testing functions which would help were recently 
reviewed on the D newsgroup and voted for inclusion in Phobos. There's a pull 
request for them, but they haven't been merged in yet, and unless you use the 
development version of Phobos from git, you'll have to wait for the next
release 
regardless. Those changes _do_ make it so that you can use collectException to 
collect an Error (though it defaults to catching Exceptions only), but they
also 
include assertThrown and assertNotThrown which effectively assert that the 
Exception or Error that you expected to be thrown (or not) from a particular 
expression or function call was indeed thrown (or not). So, you _can_ use that 
with AssertError to verify your pre-conditions.

However, I would point out that catching Errors is generally a _bad_ idea. If 
you're careful in unit testing code, you should be okay, but _don't_ do it in 
normal code, and if you're not careful, you'll still get into trouble with unit 
testing code. Unlike with Exceptions, when an Error is thrown, scope statements 
and finally blocks do not run. I don't believe that destructors are run either. 
It's pretty much assumed that you can't recover from an Error, so no attempt to 
recover is made. Your program runs a high risk of being in an invalid state 
after an Error is thrown - AssertError included. In a unit test, if you're 
dealing with simple stuff, then you're probably okay, but if you have any code
in 
there that would be affected by scope statements, finally blocks, destructors, 
etc. not running, then you don't want to be catching any Errors and then trying 
to continue.

So, functions which will help you with such testing are on their way, but they 
weren't released with the latest release of dmd, and once they have been 
released, you're going to need to be careful if you use them to test 
AssertErrors or any other kind of Error.

- Jonathan M Davis
Feb 25 2011
parent reply Magnus Lie Hetland <magnus hetland.org> writes:
On 2011-02-25 20:04:10 +0100, Jonathan M Davis said:


 On Friday, February 25, 2011 07:30:50 Magnus Lie Hetland wrote:

 Or, more generally, how do you test asserts (which is what I'm using in
 my preconditions etc.)?
 
 As far as I can see, collectException() won't collect errors, which is
 what assert() throws -- so what's the standard way of writing unit
 tests for preconditions that use assert? (I.e., test that they will, in
 fact, throw when you break them.)


 
 I think that the reality of the matter is the most of the time people _don't_
 check them. And on some level, it doesn't make sense to. It's kind of like
 asking how people test their unit tests. Unit tests are already testing 
 code. Do
 you want to be testing them on top of that? And if you do, do you test _that_
 code? Where do you stop?


I guess so. But you could say the same thing about other cases where 
you throw an exception when you detect that something is wrong -- but 
those are normally tested, right? Also, the difference here is that the 
precondition is written as a general "test", whereas my actual test 
would have specific cases.

For example, I have a test that checks that I don't add the same object 
twice to some structure, and the check involves some traversal -- code 
that could potentially be wrong. I wanted to make sure that it wasn't 
by explicitly adding the same object twice -- code (i.e., my unit test) 
that most likely could not be wrong.

But I do see your point.

[snip]

 And testing post-conditions and invariants in the manner that you're trying to
 do borders on impossible. What are you going to do, repeat the 
 post-condition or
 invariant test on the result of the function or on the state of the object that
 the function was called on after the function was called? That's just doing the
 test twice.


Right.


 You might as well just re-read the post-conditions and invariants to
 make sure that you wrote them correctly.
 
 I do see value in testing pre-conditions if you're using exceptions rather than
 assertions (which means that you're not use in blocks). In that case, you're
 testing the API to make sure that it does what it's supposed to do. But if
 you're dealing with assertions, then it's really test code as opposed to API
 code, and I don't see the same value in testing that. You'd just be 
 testing test
 code.


OK. For the practical reason, I refer you to my explanation above. But 
I guess it's a style issue -- and I'm fine with not testing these 
things, by all means.

[snip]

 Those changes _do_ make it so that you can use collectException to
 collect an Error (though it defaults to catching Exceptions only), but 
 they also
 include assertThrown and assertNotThrown which effectively assert that the
 Exception or Error that you expected to be thrown (or not) from a particular
 expression or function call was indeed thrown (or not).
 So, you _can_ use that with AssertError to verify your pre-conditions.


OK, thanks.


 However, I would point out that catching Errors is generally a _bad_ idea.


[snip lots of useful stuff]

Thanks for educating me :D

I guess the conclusion will be that I'll focus on keeping my 
preconditions really simple. (And any utility functions I use in them 
can then get unit tests of their own instead ;)

-- 
Magnus Lie Hetland
http://hetland.org
Feb 25 2011
next sibling parent spir <denis.spir gmail.com> writes:
On 02/25/2011 11:33 PM, Magnus Lie Hetland wrote:

 On 2011-02-25 20:04:10 +0100, Jonathan M Davis said:


 On Friday, February 25, 2011 07:30:50 Magnus Lie Hetland wrote:

 Or, more generally, how do you test asserts (which is what I'm using in
 my preconditions etc.)?

 As far as I can see, collectException() won't collect errors, which is
 what assert() throws -- so what's the standard way of writing unit
 tests for preconditions that use assert? (I.e., test that they will, in
 fact, throw when you break them.)


 I think that the reality of the matter is the most of the time people _don't_
 check them. And on some level, it doesn't make sense to. It's kind of like
 asking how people test their unit tests. Unit tests are already testing code.
Do
 you want to be testing them on top of that? And if you do, do you test _that_
 code? Where do you stop?


 I guess so. But you could say the same thing about other cases where you throw
 an exception when you detect that something is wrong -- but those are normally
 tested, right? Also, the difference here is that the precondition is written as
 a general "test", whereas my actual test would have specific cases.

 For example, I have a test that checks that I don't add the same object twice
 to some structure, and the check involves some traversal -- code that could
 potentially be wrong. I wanted to make sure that it wasn't by explicitly adding
 the same object twice -- code (i.e., my unit test) that most likely could not
 be wrong.

 But I do see your point.



Now that you speak of it, I do agree with you. Just like for code paths or 
specific inputs that should throw an exception. I systematically check these 
bits in unittests. It's very easy to have them right, if only by distraction 
because they are not the points on which with we put most attention, and there 
often is a test invertion somewhere (and we don't watch them run). And 
actually, I often find an error somewhere in there.
What I do is trigger them in unitests, check all works as expected (ie an error 
is raised, and the message is sensible), then comment out but /keep/ the 
testing code.
Dunno if this makes sense for DbC checkings.


Denis
-- 
_________________
vita es estrany
spir.wikidot.com
Feb 25 2011
prev sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, February 25, 2011 14:33:20 Magnus Lie Hetland wrote:

 On 2011-02-25 20:04:10 +0100, Jonathan M Davis said:

 On Friday, February 25, 2011 07:30:50 Magnus Lie Hetland wrote:

 Or, more generally, how do you test asserts (which is what I'm using in
 my preconditions etc.)?
 
 As far as I can see, collectException() won't collect errors, which is
 what assert() throws -- so what's the standard way of writing unit
 tests for preconditions that use assert? (I.e., test that they will, in
 fact, throw when you break them.)


 
 I think that the reality of the matter is the most of the time people
 _don't_ check them. And on some level, it doesn't make sense to. It's
 kind of like asking how people test their unit tests. Unit tests are
 already testing code. Do
 you want to be testing them on top of that? And if you do, do you test
 _that_ code? Where do you stop?


 
 I guess so. But you could say the same thing about other cases where
 you throw an exception when you detect that something is wrong -- but
 those are normally tested, right? Also, the difference here is that the
 precondition is written as a general "test", whereas my actual test
 would have specific cases.
 
 For example, I have a test that checks that I don't add the same object
 twice to some structure, and the check involves some traversal -- code
 that could potentially be wrong. I wanted to make sure that it wasn't
 by explicitly adding the same object twice -- code (i.e., my unit test)
 that most likely could not be wrong.
 
 But I do see your point.
 
 [snip]
 

 And testing post-conditions and invariants in the manner that you're
 trying to do borders on impossible. What are you going to do, repeat the
 post-condition or
 invariant test on the result of the function or on the state of the
 object that the function was called on after the function was called?
 That's just doing the test twice.


 
 Right.
 

 You might as well just re-read the post-conditions and invariants to
 make sure that you wrote them correctly.
 
 I do see value in testing pre-conditions if you're using exceptions
 rather than assertions (which means that you're not use in blocks). In
 that case, you're testing the API to make sure that it does what it's
 supposed to do. But if you're dealing with assertions, then it's really
 test code as opposed to API code, and I don't see the same value in
 testing that. You'd just be testing test
 code.


 
 OK. For the practical reason, I refer you to my explanation above. But
 I guess it's a style issue -- and I'm fine with not testing these
 things, by all means.


When using assertions, you're checking the logic of your program and they
should 
_always_ be true. When using exceptions, it's something that can conceivably 
fail at runtime. And if you view your function or class/struct as being part of 
an API, then you don't know who or what code will be using it, so it generally 
makes sense to use exceptions. If you use a pre-condition and assert that input 
is valid, then you're really testing the code that's calling your function, not 
the function itself. So, using such assertions makes good sense when you
control 
both the caller and the callee and it's something that should never happen. 
However, if you don't necessarily control the caller or if it's something that 
_could_ happen at runtime (even if it shouldn't), then an exception makes a lot 
more sense.

I tend to view exceptions as part of the API and think that they should be 
tested. Assertions, on the other hand, aren't really part of the API, since
they 
go away in release mode, and I therefore view them as test code. They're 
verifying that your logic is correct.

So, on some level, it is indeed a stylistic thing, but where you choose to use 
exceptions and where you choose to use assertions can have a big effect on code 
that uses your code.


 [snip]
 

 Those changes _do_ make it so that you can use collectException to
 collect an Error (though it defaults to catching Exceptions only), but
 they also
 include assertThrown and assertNotThrown which effectively assert that
 the Exception or Error that you expected to be thrown (or not) from a
 particular expression or function call was indeed thrown (or not).
 So, you _can_ use that with AssertError to verify your pre-conditions.


 
 OK, thanks.
 

 However, I would point out that catching Errors is generally a _bad_
 idea.


 
 [snip lots of useful stuff]
 
 Thanks for educating me :D
 
 I guess the conclusion will be that I'll focus on keeping my
 preconditions really simple. (And any utility functions I use in them
 can then get unit tests of their own instead ;)


That's probably a good way to handle it. I find that I often tend to have
helper 
functions like that simply because I end up testing the same thing in a variety 
of places, and I don't want to duplicate the code. It also has the advantage of 
making it so that you can therefore explicitly test that that function works 
correctly rather than having to worry about how it's used - like dealing with 
AssertErrors.

Personally, the only place that I've caught AssertErrors is in testing
functions 
like assertThrown, because in that case, the fact that the function throws an 
AssertError is an integral part of its behavior and API.

- Jonathan M Davis
Feb 25 2011
parent reply Magnus Lie Hetland <magnus hetland.org> writes:
On 2011-02-26 01:20:49 +0100, Jonathan M Davis said:


 So, using such assertions makes good sense when you control
 both the caller and the callee and it's something that should never happen.


Yeah, in my case that's what's going on. I'll only be using the 
contracts during testing anyway, and remove them with -release in the 
code that's actually to be used. (The code is part of some benchmarking 
experiments, so I'd rather not have any kind of checking like that when 
running those.)


 However, if you don't necessarily control the caller or if it's something that
 _could_ happen at runtime (even if it shouldn't), then an exception makes a lot
 more sense.


OK. I had the impression that using assert() in contracts was standard, 
also for API functions. I thought contracts fulfilled a similar sort of 
function to assert(), in that they're removed in release code -- as 
opposed to enforce(), for example...? I'm guessing that if I released a 
binary version of a library, I wouldn't leave the contracts in? Or 
perhaps I would (but, as you say, with exceptions)? Depends on the 
situation, perhaps?

What kind of exceptions would be most relevant to indicate a contract 
failure (if the contracts are made part of the API)?


 I tend to view exceptions as part of the API and think that they should be
 tested. Assertions, on the other hand, aren't really part of the API, 
 since they
 go away in release mode, and I therefore view them as test code. They're
 verifying that your logic is correct.


Exactly. The same, of course, applies to contracts -- which is why I'm 
a bit confused by your suggestion to use exceptions in them.

Or perhaps I'm misreading you completely, and you're only suggesting 
that I use code paths that throw exceptions in the function body 
itself, e.g., with enforce(foo, exception) (which would make sense to 
me)?


 So, on some level, it is indeed a stylistic thing, but where you choose to use
 exceptions and where you choose to use assertions can have a big effect on code
 that uses your code.


Sure thing. It just seems to me that contracts and assertions go well 
together, and have the same function, of testing your program logic?

I guess the driving force of my original query was the old "first, see 
your test fail" idea of test-driven programming. If I just slap a 
precondition on some code, it won't fail because things aren't 
implemented properly yet (as a normal unit test would) -- I'd actively 
have to implement a call to it *improperly*. It just seemed naturally 
to me to do that as part of the test code, rather than a one-off thing 
in the main code.

However, I could always add a call to my unit test, run it, and see it 
crash -- and then comment it out. Doesn't seem like the prettiest way 
to handle things, but it's OK, I guess together with the idea of making 
the contracts super-simple (and to test any functionality they use 
separately).

[snip]

 I guess the conclusion will be that I'll focus on keeping my
 preconditions really simple. (And any utility functions I use in them
 can then get unit tests of their own instead ;)


 That's probably a good way to handle it .


OK, good :)

-- 
Magnus Lie Hetland
http://hetland.org
Feb 26 2011
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday 26 February 2011 03:24:15 Magnus Lie Hetland wrote:

 On 2011-02-26 01:20:49 +0100, Jonathan M Davis said:

 So, using such assertions makes good sense when you control
 both the caller and the callee and it's something that should never
 happen.


 
 Yeah, in my case that's what's going on. I'll only be using the
 contracts during testing anyway, and remove them with -release in the
 code that's actually to be used. (The code is part of some benchmarking
 experiments, so I'd rather not have any kind of checking like that when
 running those.)
 

 However, if you don't necessarily control the caller or if it's something
 that _could_ happen at runtime (even if it shouldn't), then an exception
 makes a lot more sense.


 
 OK. I had the impression that using assert() in contracts was standard,
 also for API functions. I thought contracts fulfilled a similar sort of
 function to assert(), in that they're removed in release code -- as
 opposed to enforce(), for example...? I'm guessing that if I released a
 binary version of a library, I wouldn't leave the contracts in? Or
 perhaps I would (but, as you say, with exceptions)? Depends on the
 situation, perhaps?
 
 What kind of exceptions would be most relevant to indicate a contract
 failure (if the contracts are made part of the API)?


Well, the biggest problem with using assertions to verify input to a function
is 
that if you distribute your code as a library, odds are it will be in release 
mode, and then there won't be any assertions in it. In that case, I believe
that 
template functions will still end up with the assertions in it when the user of 
your library compiles with assertions enabled, since template code is not 
generated until it's instantiated, but none of the other assertions will work. 
So, assertions for public APIs really don't work very well. On top of that,
even 
if assertions _are_ enabled (either because it's a templated function or
they're 
actually using a non-release version of your library), then an assertion
failure 
makes it look like _your_ code is wrong rather than theirs.

Regardless, if you're using an assertion, what you're doing is requiring that 
the input meet some sort of pre-conditions and you're testing that the caller's 
code to verify that it meets those conditions. If you use an exception, then 
it's perfectly legitimate for a caller to give input which violates your pre-
conditions, but then the caller has to deal with it. In some cases, you
actually 
_need_ to do it that way simply because the input could reasonably be invalid
at 
runtime, and which point it _needs_ to be checked at runtime and have the error 
reported without killing the program - which means that you need an exception.

Personally, I only ever use assertions for pre-conditions if I'm controlling 
both the caller and the callee and I really expect that they will _never_ fail. 
It's test code plain and simple. In pretty much all of the cases, I use 
exceptions. Assertions are purely for catching logic errors in code.

Now, if you're throwing an exception from a function due to a pre-condition 
failure, then the type that you throw depends entirely on what you're doing. In 
Phobos, it tends to depend on what module it's in. std.datetime throws 
DateTimeExceptions. std.file throws FileExceptions. In other cases, it's more 
specific. e.g. std.typecons.conv throws ConvExceptions. What type of Exception
a 
function throws is entirely up to you. It could be a plain old Exception if 
that's what you want. It's just that if it's a specific type of Exception than 
code can catch that specific type and handle it differently than it might
handle a 
generic Exception.


 I tend to view exceptions as part of the API and think that they should
 be tested. Assertions, on the other hand, aren't really part of the API,
 since they
 go away in release mode, and I therefore view them as test code. They're
 verifying that your logic is correct.


 
 Exactly. The same, of course, applies to contracts -- which is why I'm
 a bit confused by your suggestion to use exceptions in them.
 
 Or perhaps I'm misreading you completely, and you're only suggesting
 that I use code paths that throw exceptions in the function body
 itself, e.g., with enforce(foo, exception) (which would make sense to
 me)?


Never throw exceptions form in, out, or invariant blocks. They'll just go away 
in release mode. Only use assertions in there. So, if you're going to throw an 
Exception, throw it from the function body or from some other function that
gets 
called by that function.

Regardless of that, however, assertions should only be used when testing the 
internal logic of your program. If code from other libraries or any other code 
which you wouldn't be looking to test calls your function, then don't use an 
assertion to verify pre-conditions. If you're using assertions, you're testing 
that the caller is correct. You're verifying that the caller is not violating 
your contract, but you're _not_ guaranteeing that the function will fail if
they 
violate the contract (since assertions can go away). The test for the contract 
is therefore _not_ part of the API. With Exceptions it _is_. So, what it really 
comes down to is whether you looking to test the code which calls your function 
and are therefore willing to have that code give you bad input and let your 
function process it anyway (when assertions aren't compiled in) and you 
therefore use assertions, _or_ you're looking to guarantee that your function 
does _not_ continue if the contract is violated, and you want to _always_ error 
out - in which case you use Exceptions. If you're dealing with any kind of 
public API, Exceptions are going to tend to be the correct way to go. If it's 
just your code and you control the caller and want to test it, then assertions 
may be the correct way to go.

In any case, don't throw exceptions from in, out, or invariant blocks. They 
definitely go in the function bodies.


 So, on some level, it is indeed a stylistic thing, but where you choose
 to use exceptions and where you choose to use assertions can have a big
 effect on code that uses your code.


 
 Sure thing. It just seems to me that contracts and assertions go well
 together, and have the same function, of testing your program logic?
 
 I guess the driving force of my original query was the old "first, see
 your test fail" idea of test-driven programming. If I just slap a
 precondition on some code, it won't fail because things aren't
 implemented properly yet (as a normal unit test would) -- I'd actively
 have to implement a call to it *improperly*. It just seemed naturally
 to me to do that as part of the test code, rather than a one-off thing
 in the main code.
 
 However, I could always add a call to my unit test, run it, and see it
 crash -- and then comment it out. Doesn't seem like the prettiest way
 to handle things, but it's OK, I guess together with the idea of making
 the contracts super-simple (and to test any functionality they use
 separately).


That makes sense except that what you're really testing when you assert pre-
conditions is the caller code, not the function that they're in. And when you
do 
unit tests, you're normally testing the functions that you call. You throw the 
assertions in there to make sure that the caller code doesn't violate the pre-
conditions, so it's test code. And if you're using unit tests to test those, 
you're testing test code. There's nothing stopping you from doing it, but it's 
already undefined behavior per DbC when a pre-condition is violated, and if 
you're then testing those asserted pre-conditions with unit tests, you're 
testing to make sure that the behavior _is_ defined (that it throws an 
AssertError).

I suppose that while from the standpoint of principle it's just plain weird if 
not outright wrong to test that your assertions which test your pre-conditions 
actually throw AssertErrors when they're supposed to and don't when they're
not, 
it _does_ have some practical benefit. Still, if you start testing test code,
at 
what point does it make sense to stop?

Regardless, it's totally up to you when and where you use assertions or 
exceptions. But in general, assertions are really test code and shouldn't be 
considered part of the API (so testing them as part of the API like you're 
looking to do is just plain weird), whereas Exceptions most definitely _are_
part 
of the API. And from a perfectly practical standpoint, as soon as your code
ends 
up in a library, assertions are generally useless anyway, because they likely 
won't be enabled, and anyone using your library won't ever see them even if
they 
violate your pre-conditions on every call that they make.


 [snip]
 

 I guess the conclusion will be that I'll focus on keeping my
 preconditions really simple. (And any utility functions I use in them
 can then get unit tests of their own instead ;)


 
 That's probably a good way to handle it .


 
 OK, good :)


Complicated tests of _any_ kind are a bit dangerous. If your test code (be it 
assertions in DbC or in unit tests or wherever) should be simple enough that 
you're unlikely to screw it up. You don't want to have much risk of getting
your 
test code wrong. It makes it far too likely to miss bugs, and it makes it much 
harder to determine whether a test failure is due to the code being tested
being 
bad or due to the test being bad.

- Jonathan M Davis
Feb 26 2011
next sibling parent reply David Nadlinger <see klickverbot.at> writes:
On 2/26/11 1:15 PM, Jonathan M Davis wrote:

 […]  And from a perfectly practical standpoint, as soon as your code ends
 up in a library, assertions are generally useless anyway,[…]


I don't quite think asserts are useless in libraries. If you need to 
care about performance in a library, you hit cases quite frequently 
where sanity-checking the input would be too expensive to be done in 
release mode, and thus you can't specify behavior on invalid input as 
part of your API using exceptions. Nevertheless, it is still useful to 
people using your library to get notified when they are messing 
something up as early as possible in debug mode, which is precisely what 
asserts are made for, at least in my opinion.

You can find this use of asserts in many libraries out there, LLVM being 
the first example that comes to my mind.

David
Feb 26 2011
parent reply Magnus Lie Hetland <magnus hetland.org> writes:
On 2011-02-26 15:20:19 +0100, David Nadlinger said:


 On 2/26/11 1:15 PM, Jonathan M Davis wrote:

 [...]  And from a perfectly practical standpoint, as soon as your code ends
 up in a library, assertions are generally useless anyway,[...]


 
 I don't quite think asserts are useless in libraries. If you need to 
 care about performance in a library, you hit cases quite frequently 
 where sanity-checking the input would be too expensive to be done in 
 release mode, and thus you can't specify behavior on invalid input as 
 part of your API using exceptions. Nevertheless, it is still useful to 
 people using your library to get notified when they are messing 
 something up as early as possible in debug mode, which is precisely 
 what asserts are made for, at least in my opinion.


But that would only work if they had access to the source, or a version 
not compiled in release mode, right?

Hmm.

This is also related to what Jonathan said about programming by 
contract -- and only using in-clauses (for example) when you also 
control the calling code. I guess what you're saying could be an 
argument in the other direction: that even though they certainly 
shouldn't be considered part of the public API (beyond documenting what 
would be input causing undefined behavior), they *could* be useful in a 
library that a client could use in debug mode, because it gives them 
some extra tests for their own code, "for free". They can test that 
their own code is using your code correctly.

That sounds quite in line with programming by contract to me ... but 
then, again, I'm a reall n00b on the subject :)

-- 
Magnus Lie Hetland
http://hetland.org
Feb 26 2011
parent reply David Nadlinger <see klickverbot.at> writes:
On 2/26/11 4:08 PM, Magnus Lie Hetland wrote:

 On 2011-02-26 15:20:19 +0100, David Nadlinger said:


 On 2/26/11 1:15 PM, Jonathan M Davis wrote:

 [...] And from a perfectly practical standpoint, as soon as your code
 ends
 up in a library, assertions are generally useless anyway,[...]


 I don't quite think asserts are useless in libraries. If you need to
 care about performance in a library, you hit cases quite frequently
 where sanity-checking the input would be too expensive to be done in
 release mode, and thus you can't specify behavior on invalid input as
 part of your API using exceptions. Nevertheless, it is still useful to
 people using your library to get notified when they are messing
 something up as early as possible in debug mode, which is precisely
 what asserts are made for, at least in my opinion.


 But that would only work if they had access to the source, or a version
 not compiled in release mode, right?


True, but as shipping debug libraries and headers is precisely what SDKs 
are for, I don't see much of a problem there. Heck, even Microsoft's C 
Runtime comes with an extra debug version…

David
Feb 26 2011
parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday 26 February 2011 08:23:41 David Nadlinger wrote:

 On 2/26/11 4:08 PM, Magnus Lie Hetland wrote:

 On 2011-02-26 15:20:19 +0100, David Nadlinger said:

 On 2/26/11 1:15 PM, Jonathan M Davis wrote:

 [...] And from a perfectly practical standpoint, as soon as your code
 ends
 up in a library, assertions are generally useless anyway,[...]


=20
 I don't quite think asserts are useless in libraries. If you need to
 care about performance in a library, you hit cases quite frequently
 where sanity-checking the input would be too expensive to be done in
 release mode, and thus you can't specify behavior on invalid input as
 part of your API using exceptions. Nevertheless, it is still useful to
 people using your library to get notified when they are messing
 something up as early as possible in debug mode, which is precisely
 what asserts are made for, at least in my opinion.


=20
 But that would only work if they had access to the source, or a version
 not compiled in release mode, right?


=20
 True, but as shipping debug libraries and headers is precisely what SDKs
 are for, I don't see much of a problem there. Heck, even Microsoft's C
 Runtime comes with an extra debug version=E2=80=A6


Sure, you _can_ use assertions in public APIs, but you _can't_ rely on them=
=20
being there, because the programmer using the API could be using a release=
=20
version.

If you really want to have an error for giving bad input, then it should be=
 an=20
Exception of some kind. Assertions are there for testing code. They go away=
=2E=20
Also, it just plain looks bad when your library throws an AssertError. Sinc=
e=20
assertions are used to test the internal logic of your code, it makes it lo=
ok=20
like _your_ code is wrong rather than the code which is using your code.

You can't rely on assertions being in a library, so if you want those check=
s to=20
be guaranteed to take place, you need to use exceptions. If you want the ch=
eck=20
to always take place, you need to use exceptions. If you want to report an =
input=20
error as opposed to reporting a code logic error, then you should be using=
=20
exceptions. So, assertions make great sense for testing that your code is=20
correct, but when you hand it off to someone else to use, it shouldn't gene=
rally=20
be throwing AssertErrors. There are times where that makes sense (particula=
rly=20
in code that _needs_ to be highly efficient and can't afford the extra chec=
ks in=20
release mode), but at that point, you're saying that the check is _not_ par=
t of=20
the API (just an extra service to the programmer using it), and you're sayi=
ng=20
that you're willing to deal with programmers thinking that your code is fau=
lty,=20
because it's throwing AssertErrors.

What I've been saying is essentially how Phobos goes about dealing with=20
assertions and exceptions in its functions.

=2D Jonathan M Davis
Feb 26 2011
prev sibling parent reply Magnus Lie Hetland <magnus hetland.org> writes:
On 2011-02-26 13:15:58 +0100, Jonathan M Davis said:


 On Saturday 26 February 2011 03:24:15 Magnus Lie Hetland wrote:

 OK. I had the impression that using assert() in contracts was standard,
 also for API functions. I thought contracts fulfilled a similar sort of
 function to assert(), in that they're removed in release code -- as
 opposed to enforce(), for example...? I'm guessing that if I released a
 binary version of a library, I wouldn't leave the contracts in? Or
 perhaps I would (but, as you say, with exceptions)? Depends on the
 situation, perhaps?
 
 What kind of exceptions would be most relevant to indicate a contract
 failure (if the contracts are made part of the API)?


 
 Well, the biggest problem with using assertions to verify input to a 
 function is
 that if you distribute your code as a library, odds are it will be in release
 mode, and then there won't be any assertions in it.


[snip lots of stuff]

After reading your response, I first made lots of comments, but it's 
all a bit redundant. My summary is:

- You're (at times) talking about preconditions as a general concept, 
and that for public APIs, they should be enforced using exceptions.
- I've only been talking about the *language feature* of preconditions, 
i.e., in-clauses.
- We're both clear on that preconditions and asserts disappear in 
release mode, and that the two belong together, as part of your test 
scaffolding (and not as part of your public API).

Sound about right?

[snip]

 Regardless of that, however, assertions should only be used when testing the
 internal logic of your program. If code from other libraries or any other code
 which you wouldn't be looking to test calls your function, then don't use an
 assertion to verify pre-conditions. If you're using assertions, you're testing
 that the caller is correct. You're verifying that the caller is not violating
 your contract, but you're _not_ guaranteeing that the function will 
 fail if they
 violate the contract (since assertions can go away).


A very clarifying way of putting it, indeed.

As for my "testing the test code" intention, I guess (as I said) I 
actually *did* want to test the test. Not, perhaps, that it was correct 
(as discussed, it should be really simple), but to see it fail at least 
once -- a basic principle of test-driven programming. But I'll find 
other ways of doing that -- for example deliberately making the 
precondition slightly wrong at first :)


 The test for the contract  is therefore _not_ part of the API. With 
 Exceptions it _is_.


Right.


 So, what it really comes down to is whether you looking to test the 
 code which calls your function and are therefore willing to have that 
 code give you bad input and let your function process it anyway (when 
 assertions aren't compiled in) and you therefore use assertions, _or_ 
 you're looking to guarantee that your function does _not_ continue if 
 the contract is violated, and you want to _always_ error out - in which 
 case you use Exceptions.


Yep. All in all, a very useful clarification for me.

As a side note: Why isn't there a release-version of the contract 
mechanisms? I would've thought that contracts would be even more useful 
between different programmers, than just between you and yourself...?-)

That is, wouldn't the same kind of mechanism be useful for *exactly* 
the kind of exception-based input checking that you're describing as 
the alternative to contracts+asserts?

I mean, the reason to remove preconditions and asserts is primarily 
performance and not semantics (although it certainly affects semantics, 
as you've pointed out)? We have enforce() as the alternative to 
assert(); why no alternative to in/out and invariants?

[snip]

 And if you're using unit tests to test those, you're testing test code.


Sure. I've already accepted this :)

[snip]

 Still, if you start testing test code, at what point does it make sense 
 to stop?


Hm. Maybe I should write a test that tests itself?-)

More seriously: your points are well taken.

I still have a vague feeling that in-clauses are a bit different from 
out-closes, invariants and plain unit tests when it comes to the "fail 
first" approach to test-driven programming. A precondition won't fail 
because your code isn't yet functional -- it will only fail if you've 
actively written *wrong* code. But I guess that's just how it is :)


 Complicated tests of _any_ kind are a bit dangerous.


[snip]

Hm. True.

Thanks for lots of useful input!

(Still curious about the hypothetical "public API contract" 
functionality, though, and why it's non-existent.)

-- 
Magnus Lie Hetland
http://hetland.org
Feb 26 2011
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday 26 February 2011 07:03:29 Magnus Lie Hetland wrote:

 On 2011-02-26 13:15:58 +0100, Jonathan M Davis said:

 On Saturday 26 February 2011 03:24:15 Magnus Lie Hetland wrote:

 OK. I had the impression that using assert() in contracts was standard,
 also for API functions. I thought contracts fulfilled a similar sort of
 function to assert(), in that they're removed in release code -- as
 opposed to enforce(), for example...? I'm guessing that if I released a
 binary version of a library, I wouldn't leave the contracts in? Or
 perhaps I would (but, as you say, with exceptions)? Depends on the
 situation, perhaps?
 
 What kind of exceptions would be most relevant to indicate a contract
 failure (if the contracts are made part of the API)?


 
 Well, the biggest problem with using assertions to verify input to a
 function is
 that if you distribute your code as a library, odds are it will be in
 release mode, and then there won't be any assertions in it.


 
 [snip lots of stuff]
 
 After reading your response, I first made lots of comments, but it's
 all a bit redundant. My summary is:
 
 - You're (at times) talking about preconditions as a general concept,
 and that for public APIs, they should be enforced using exceptions.
 - I've only been talking about the *language feature* of preconditions,
 i.e., in-clauses.
 - We're both clear on that preconditions and asserts disappear in
 release mode, and that the two belong together, as part of your test
 scaffolding (and not as part of your public API).
 
 Sound about right?


Yeah.


 [snip]
 

 Regardless of that, however, assertions should only be used when testing
 the internal logic of your program. If code from other libraries or any
 other code which you wouldn't be looking to test calls your function,
 then don't use an assertion to verify pre-conditions. If you're using
 assertions, you're testing that the caller is correct. You're verifying
 that the caller is not violating your contract, but you're _not_
 guaranteeing that the function will fail if they
 violate the contract (since assertions can go away).


 
 A very clarifying way of putting it, indeed.
 
 As for my "testing the test code" intention, I guess (as I said) I
 actually *did* want to test the test. Not, perhaps, that it was correct
 (as discussed, it should be really simple), but to see it fail at least
 once -- a basic principle of test-driven programming. But I'll find
 other ways of doing that -- for example deliberately making the
 precondition slightly wrong at first :)


 The test for the contract  is therefore _not_ part of the API. With
 Exceptions it _is_.


 
 Right.


If you really want to test the test code, then test the test code. But even in 
test driven development, you _wouldn't_ be testing that the function fails when 
given values which violate its pre-conditions. That is _undefined_ behavior and 
arguably doesn't matter. The caller violated the pre-condition. They get what 
they get. The behavior is completely undefined at that point. You just put in 
assertions to test pre-conditions so that you can find bugs in the calling
code. 
As soon as you're testing that an AssertError is thrown, you're testing
behavior 
that DbC considers undefined. DbC says that if you give a function input that 
does not violate its pre-conditions, the function will give you output which 
does not violate its post-conditions. It says nothing about what happens when 
you give it input which violates your pre-conditions. All bets are off at that 
point. You violated the contract.

So, testing your pre-conditions with assertions is simply testing that the 
caller code is correct. The function itself doesn't care whether the input was 
correct or not. It's only obligated to return valid values when the contract is 
kept. If the caller violates the contract, then tough luck for it.

The difference with exceptions is that you're _requiring_ that the caller give 
you correct input and erroring out if it doesn't.

Regardless, if you really want to test that your pre-condition checks are 
correct, then just test them. I'd argue against it because you're testing test 
code, and that shouldn't be necessary, but if you're going to test that your 
assertions work correctly, then test them right.


 So, what it really comes down to is whether you looking to test the
 code which calls your function and are therefore willing to have that
 code give you bad input and let your function process it anyway (when
 assertions aren't compiled in) and you therefore use assertions, _or_
 you're looking to guarantee that your function does _not_ continue if
 the contract is violated, and you want to _always_ error out - in which
 case you use Exceptions.


 
 Yep. All in all, a very useful clarification for me.
 
 As a side note: Why isn't there a release-version of the contract
 mechanisms? I would've thought that contracts would be even more useful
 between different programmers, than just between you and yourself...?-)
 
 That is, wouldn't the same kind of mechanism be useful for *exactly*
 the kind of exception-based input checking that you're describing as
 the alternative to contracts+asserts?
 
 I mean, the reason to remove preconditions and asserts is primarily
 performance and not semantics (although it certainly affects semantics,
 as you've pointed out)? We have enforce() as the alternative to
 assert(); why no alternative to in/out and invariants?


Contracts are meant for testing _your_ code. Yes, in blocks are testing the 
caller's code, but you do all of that because you want to verify that _your_ 
code is correct. If other libraries or programs are using your code, then they 
should be able to assume that your code is correct and not need to have all of 
its internal checks running, slowing it down.

If a caller violates your pre-conditions, then they're violating the contract, 
and your code has no obligation at that point to give them anything in the way 
of error messages or correct output. DbC is specifically saying that they're 
_required_ to give you correct input. The fact that they didn't is their fault 
and their problem. Assertions test that your code follows the contract. Whether 
other code does or not is its problem.

Now, if you want to actually have your code error out when given bad input,
then 
you use exceptions (enforce is one way to do that). But all you'd be using them 
for is input. You're not going to throw an exception on bad output, because 
that's a bug in _your_ code, not theirs (the same goes for invariants). And 
since exceptions are then part of the function's normal operation rather than 
testing, it makes no sense to put them in an in block. What does an in block
buy 
you at that point, even if it does stick around? Just put the checks at the top 
of your function and throw if they fail. And honestly, there are plenty of
times 
when such checks actually need to go deeper in the function anyway, because you 
have to process some of the input before you know that it's invalid.


 [snip]
 

 And if you're using unit tests to test those, you're testing test code.


 
 Sure. I've already accepted this :)
 
 [snip]
 

 Still, if you start testing test code, at what point does it make sense
 to stop?


 
 Hm. Maybe I should write a test that tests itself?-)
 
 More seriously: your points are well taken.
 
 I still have a vague feeling that in-clauses are a bit different from
 out-closes, invariants and plain unit tests when it comes to the "fail
 first" approach to test-driven programming. A precondition won't fail
 because your code isn't yet functional -- it will only fail if you've
 actively written *wrong* code. But I guess that's just how it is :)


in blocks are definitely different in that they test the caller's code, but if
all 
of the code involved is _your_ code, then that does make some sense. It's when 
it's part of an API that it doesn't.


 Complicated tests of _any_ kind are a bit dangerous.


 
 [snip]
 
 Hm. True.
 
 Thanks for lots of useful input!
 
 (Still curious about the hypothetical "public API contract"
 functionality, though, and why it's non-existent.)


The contracts are there. They're what is agreed upon. What's not there is 
_testing_ that those contracts aren't violated. And since assertions are really 
for testing the internal logic of your own code, it really doesn't make sense
to 
be throw AssertErrors at 3rd party code that calls yours. It just slows your 
code down having to do the checks makes it look like your code is wrong when 
they give you bad input.

- Jonathan M Davis
Feb 26 2011
parent Magnus Lie Hetland <magnus hetland.org> writes:
On 2011-02-27 02:33:46 +0100, Jonathan M Davis said:


 


[snip lots of useful stuff]

Thanks for your patience, and more useful clarifications. I think I get 
it now (but I may very well be wrong ;)

- M

-- 
Magnus Lie Hetland
http://hetland.org
Feb 28 2011
prev sibling parent spir <denis.spir gmail.com> writes:
On 02/26/2011 12:24 PM, Magnus Lie Hetland wrote:

 However, I could always add a call to my unit test, run it, and see it crash --
 and then comment it out. Doesn't seem like the prettiest way to handle things,
 but it's OK, I guess together with the idea of making the contracts
 super-simple (and to test any functionality they use separately).


With named unittests, you could also have one of them check failure cases, and 
just comment out its call.

Denis
-- 
_________________
vita es estrany
spir.wikidot.com
Feb 26 2011