www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - SQLite testing procedures

reply Robert Clipsham <robert octarineparrot.com> writes:
This is a fairly old document now, but it has some interesting ideas 
that could have a place in D (either the language, tools, or phobos):

http://www.sqlite.org/testing.html

Some notable things other than the range of testing methods used:
  * 100% branch test coverage (7.1)
     - It would be nice if dmd's -cov offered a way to give the coverage 
for statements and branches rather than just per line
  * Coverage testing of defensive code (7.2)
    - Some branches are always (or never) executed, which can mean 100% 
branch coverage is not possible unless you remove the "defensive" code.
    - SQLite solves this with the ALWAYS() and NEVER() macros
    - This would only become relevant if -cov was improved
  * Forcing coverage of boundary values and boolean vector tests (7.3)
    - SQLite has a testcase() macro which adds additional branches 
during coverage analysis to test boundary conditions and similar.
    - D currently has no way of doing this, it would be a simple (and 
useful) addition to phobos though.
  * SQLite makes extensive use of assert().
  * The SQLite developers have found static analysis to be completely 
useless compared to having extensive (100%) coverage tests, as well as 
help from tools such as valgrind (particularly praised in the above). 
They've even found themselves introducing bugs while trying to silence 
compiler warnings!

-- 
Robert
http://octarineparrot.com/
Aug 05 2011
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday 05 August 2011 21:13:17 Robert Clipsham wrote:

 This is a fairly old document now, but it has some interesting ideas
 that could have a place in D (either the language, tools, or phobos):
 
 http://www.sqlite.org/testing.html
 
 Some notable things other than the range of testing methods used:
   * 100% branch test coverage (7.1)
      - It would be nice if dmd's -cov offered a way to give the coverage
 for statements and branches rather than just per line


But each branch of code is on a particular branch, so if every branch is hit, 
every line is hit. Do you mean cases such as where you have 5 if statements in 
a row, and every one of them gets hit at one time or another, but in some 
cases the execution path goes through 3 of them or 2 of them or whatever, but 
not every single combination is tested? That would be difficult to do given
that 
there's a good chance that certain combinations of branches should never 
actually occur. It would also complicate the implementation of -cov 
considerably I would think.


   * Coverage testing of defensive code (7.2)
     - Some branches are always (or never) executed, which can mean 100%
 branch coverage is not possible unless you remove the "defensive" code.
     - SQLite solves this with the ALWAYS() and NEVER() macros
     - This would only become relevant if -cov was improved


It already happens. If you use assert(0) in your code anywhere (prime examples 
of that being in the default case of a switch statement which should never 
happen or in a catch block which should never be hit but is required to make 
the function nothrow because a function in the try block theoretically _could_ 
throw but never actually will with the arguments that it's given in that 
case). std.datetime has very high code coverage (100% aside from the assert(0) 
lines, I believe), but the assert(0) lines are naturally never hit (it would 
be a bug if they were), so -cov does not actually give it 100% (it gets 
something more like 98%).

- Jonathan M Davis
Aug 05 2011
parent Robert Clipsham <robert octarineparrot.com> writes:
On 05/08/2011 21:22, Jonathan M Davis wrote:

 On Friday 05 August 2011 21:13:17 Robert Clipsham wrote:

 This is a fairly old document now, but it has some interesting ideas
 that could have a place in D (either the language, tools, or phobos):

 http://www.sqlite.org/testing.html

 Some notable things other than the range of testing methods used:
    * 100% branch test coverage (7.1)
       - It would be nice if dmd's -cov offered a way to give the coverage
 for statements and branches rather than just per line


 But each branch of code is on a particular branch, so if every branch is hit,
 every line is hit. Do you mean cases such as where you have 5 if statements in
 a row, and every one of them gets hit at one time or another, but in some
 cases the execution path goes through 3 of them or 2 of them or whatever, but
 not every single combination is tested? That would be difficult to do given
that
 there's a good chance that certain combinations of branches should never
 actually occur. It would also complicate the implementation of -cov
 considerably I would think.


// This gets 100% coverage currently if a == 1. No testing of the
// second branch is necessary. Or vice-versa.
if (a == 1 || b > 2)
{
}

With branch based coverage, both branches must get tested for 100% 
coverage - SQLite's testing is incredibly thorough, and it shows in 
their rock solid product.


    * Coverage testing of defensive code (7.2)
      - Some branches are always (or never) executed, which can mean 100%
 branch coverage is not possible unless you remove the "defensive" code.
      - SQLite solves this with the ALWAYS() and NEVER() macros
      - This would only become relevant if -cov was improved


 It already happens. If you use assert(0) in your code anywhere (prime examples
 of that being in the default case of a switch statement which should never
 happen or in a catch block which should never be hit but is required to make
 the function nothrow because a function in the try block theoretically _could_
 throw but never actually will with the arguments that it's given in that
 case). std.datetime has very high code coverage (100% aside from the assert(0)
 lines, I believe), but the assert(0) lines are naturally never hit (it would
 be a bug if they were), so -cov does not actually give it 100% (it gets
 something more like 98%).


It might be worth reading the article about this, I can't summarize it 
sufficiently.


 - Jonathan M Davis



-- 
Robert
http://octarineparrot.com/
Aug 05 2011
prev sibling next sibling parent Robert Clipsham <robert octarineparrot.com> writes:
On 05/08/2011 21:13, Robert Clipsham wrote:

 * Forcing coverage of boundary values and boolean vector tests (7.3)
 - SQLite has a testcase() macro which adds additional branches during
 coverage analysis to test boundary conditions and similar.
 - D currently has no way of doing this, it would be a simple (and
 useful) addition to phobos though.


I came up with a quick template to do this - should I make a pull 
request, and if so, where to?

----
void coverageTest(size_t line)
{
     // Make sure this doesn't get optimized away
     static size_t count = 0;
     count += line;
}
string testcase(string test, size_t line=__LINE__)()
{
     version(D_Coverage)
     {
         // This has to be on one line so coverage
         // line numbers don't get messed up.
         return `if (`~test.stringof~`) { 
coverageTest(`~line.stringof~`); }`;
     }
     else
     {
         return ``;
     }
}
----
Usage: mixin(testcase!(`a < b`));

-- 
Robert
http://octarineparrot.com/
Aug 05 2011
prev sibling parent bearophile <bearophileHUGS lycos.com> writes:
Robert Clipsham Wrote:


 This is a fairly old document now, but it has some interesting ideas 
 that could have a place in D (either the language, tools, or phobos):


There is a part about integer overflow checks:


8.6 Signed-Integer Overflow Checks


The various C language standards say that the signed-integer overflow behavior
is undefined. In other words, when you add a value to a signed integer such
that the result is too large to fit in that integer, the value does not
necessarily wrap around to a negative number, as most programmers expect. It
might do that. But it might do something completely different. See, for
example, here and here. Even the same compiler might do something different
with signed integer overflow in different places in the code or at different
optimizations settings.

SQLite never overflows a signed integer. To verify this, the test suites are
run at least once when compiled with the -ftrapv option to GCC. The -ftrapv
option causes GCC to generate code that will panic() on a signed integer
overflow. In addition, there are many test cases the strive to provoke integer
overflows using boundary value calculations such as "SELECT
-1*(-9223372036854775808);".<

Bye,
bearophile
Aug 06 2011