www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - unittest behaviour

reply DLearner <bmqazwsx123 gmail.com> writes:
Please consider:

```
size_t foo() {
    static size_t var1 = 1;

    var1 = var1 + 1;
    return var1;
}

unittest {

    assert(foo() == 2);
    assert(foo() == 3);
}

```

which works as expected.

But

```
size_t foo1() {
    static size_t var1 = 1;

    var1 = var1 + 1;
    return var1;
}

unittest {
    assert(foo1() == 2);
}

unittest {
    assert(foo1() == 2);
}

```

Fails on the second unittest.

I appreciate this behaviour matches the docs (so not a bug), but 
is it desirable?

To me, as a test harness, a umittest block should be a completely 
fresh-from-scratch invocation of the code inside the block, and 
thus not depend on the result/effects of any previous unittest.
Dec 15 2024
next sibling parent "H. S. Teoh" <hsteoh qfbox.info> writes:
On Sun, Dec 15, 2024 at 08:45:22AM +0000, DLearner via Digitalmars-d-learn
wrote:
[...]

 I appreciate this behaviour matches the docs (so not a bug), but is it
 desirable?
 
 To me, as a test harness, a umittest block should be a completely
 fresh-from-scratch invocation of the code inside the block, and thus
 not depend on the result/effects of any previous unittest.


That's for you, the programmer, to ensure.  Using a static variable
breaks this assumption.  As does a whole bunch of other things you could
do that have side-effects, such as file I/O or network traffic.  So if
you want your code to be unittest-able in an encapsulated way, refactor
it so that it doesn't have side-effects of this kind.


T

-- 
Caffeine underflow. Brain dumped.
Dec 15 2024
prev sibling next sibling parent reply monkyyy <crazymonkyyy gmail.com> writes:
On Sunday, 15 December 2024 at 08:45:22 UTC, DLearner wrote:

 I appreciate this behaviour matches the docs (so not a bug), 
 but is it desirable?


yes, the alternative would be that unittests attempt to undo 
themselves, and that would make bugs horrible horrible bugs or 
executable clear global scope and stack effectively restarting 
the program, this could be incredibly slow if you have big arrays 
in global scope and then hundards of small unrelated 
unittests(which btw you do, the std has plenty and I think the 
run time also injects some)
Dec 15 2024
parent reply DLearner <bmqazwsx123 gmail.com> writes:
On Sunday, 15 December 2024 at 20:30:21 UTC, monkyyy wrote:

 On Sunday, 15 December 2024 at 08:45:22 UTC, DLearner wrote:

 I appreciate this behaviour matches the docs (so not a bug), 
 but is it desirable?


 yes, the alternative would be that unittests attempt to undo 
 themselves, and that would make bugs horrible horrible bugs or 
 executable clear global scope and stack effectively restarting 
 the program, this could be incredibly slow if you have big 
 arrays in global scope and then hundards of small unrelated 
 unittests(which btw you do, the std has plenty and I think the 
 run time also injects some)


What is wrong with changing the specification of unittest so that 
it recompiles/reexecutes the associated source on every unittest 
{} block?

That way, there would be no alteration in behaviour if the source 
was 'pure', but you would have flexibility if you wanted to test 
source that was (intentionally) 'stateful'.
Dec 17 2024
next sibling parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Tue, Dec 17, 2024 at 07:16:55PM +0000, DLearner via Digitalmars-d-learn
wrote:
[...]

 What is wrong with changing the specification of unittest so that it
 recompiles/reexecutes the associated source on every unittest {}
 block?


That means the compiler will have to rerun your program once per
unittest.  That means your OS has to create a new process per unittest.
If you have a lot of unittests, that adds a huge amount of overhead.



 That way, there would be no alteration in behaviour if the source was
 'pure', but you would have flexibility if you wanted to test source
 that was (intentionally) 'stateful'.


[...]

It still does not solve the problem of unittests with side-effects, like
file or network I/O.


T

-- 
Did you hear about the kidnapping at school today?  Everything's OK now -- he
woke up.
Dec 17 2024
parent reply DLearner <bmqazwsx123 gmail.com> writes:
On Tuesday, 17 December 2024 at 19:56:32 UTC, H. S. Teoh wrote:
[...]

 That means the compiler will have to rerun your program once 
 per unittest.  That means your OS has to create a new process 
 per unittest. If you have a lot of unittests, that adds a huge 
 amount of overhead.


Agreed, but only an issue when testing, not in production.

[...]


 It still does not solve the problem of unittests with 
 side-effects, like file or network I/O.


To me, it is unreasonable to expect unittest (as a part of the 
compiler suite) to establish/reset those parts of the environment 
outside of the program source code (like creating/restoring test 
files etc).
I regard that as something that is the programmer's 
responsibility [perhaps by writing a script that performs those 
tasks, and calling that script within the unittest].
Dec 17 2024
next sibling parent monkyyy <crazymonkyyy gmail.com> writes:
On Tuesday, 17 December 2024 at 21:31:13 UTC, DLearner wrote:

 On Tuesday, 17 December 2024 at 19:56:32 UTC, H. S. Teoh wrote:
 [...]

 That means the compiler will have to rerun your program once 
 per unittest.  That means your OS has to create a new process 
 per unittest. If you have a lot of unittests, that adds a huge 
 amount of overhead.


 Agreed, but only an issue when testing, not in production.


testing is where compile times matter

Only need to ship code once, ideally I compile code every few 
seconds
Dec 17 2024
prev sibling next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, December 17, 2024 2:31:13 PM MST DLearner via Digitalmars-d-learn
wrote:

 On Tuesday, 17 December 2024 at 19:56:32 UTC, H. S. Teoh wrote:
 [...]

 That means the compiler will have to rerun your program once
 per unittest.  That means your OS has to create a new process
 per unittest. If you have a lot of unittests, that adds a huge
 amount of overhead.


 Agreed, but only an issue when testing, not in production.

 [...]


 It still does not solve the problem of unittests with
 side-effects, like file or network I/O.


 To me, it is unreasonable to expect unittest (as a part of the
 compiler suite) to establish/reset those parts of the environment
 outside of the program source code (like creating/restoring test
 files etc).
 I regard that as something that is the programmer's
 responsibility [perhaps by writing a script that performs those
 tasks, and calling that script within the unittest].


Well, making sure that your tests don't rely on static state also ends up
falling under the programmer.

I can understand being frustrated in this particular situation, but to run a
separate process for every single unit test would slow down unit tests
_considerably_, and for many programmers, the common workflow involves
writing some code, running the tests, and then fixing the code all in quick
succession, and having that be slow can slow down development significantly.
So, running unittest blocks needs to be pretty fast so long as the code
being tested isn't slow.

And running the unittest blocks in separate processes would be for solving
an issue that most programs don't have, because most code does not have
static variables in the fashion that your example does. The vast majority of
code is written in a way that if you pass the same arguments to a function,
you'll get the same result. There are definitely counter-examples (e.g.
getting the time or using a random number generator), so obviously, it's not
always the case that a function returns the same result for the same
arguments, but when a program has a situation where it really does need to
have a function whose result is not going to always be the same for the same
arguments, then it's simply up to the programmer to figure out how best to
test it. It would be far too disruptive to tests in general to do otherwise.

Ultimately, D's unittest blocks are designed for the common case to make it
easy to add tests to your code. They don't have fancy features, and there
are going to be situations where they don't fit for on reason or another,
but they work quite well for the common case. You've just found an uncommon
case where they don't work as well, and for cases like that, you're
potentially going to need to do something else. What that something else is
will presumably depend on what exactly your actual code is doing. Often,
it's matter of refactoring your code so that the core logic can be tested
separately, but that really depends on what your code is doing.

- Jonathan M Davis
Dec 17 2024
parent DLearner <bmqazwsx123 gmail.com> writes:
On Wednesday, 18 December 2024 at 07:43:01 UTC, Jonathan M Davis 
wrote:
[...]

 - Jonathan M Davis


Thinking about this a little more:

Suggestion:

Leave the current behaviour as default, but introduce an option 
(-r ?) so that
```
unittest -r {


}
```
forces a complete rebuild
(for clarity, if the function under test has updated data files, 
etc., it is the programmers responsibility that the unittest 
contains a script that resets those files).

The motive for all this is to address the situation where the 
function under test is not small, but is the top (or near top) of 
a complex system wih lots of state,
and the task is to create a test suite that clearly shows that 
multiple situations, involving state, have indeed been tested.
Jan 08
prev sibling parent Manfred Nowak <svv1999 hotmail.com> writes:
On Tuesday, 17 December 2024 at 21:31:13 UTC, DLearner wrote:

[... environmental requirements]

 I regard that as something that is the programmer's 
 responsibility [perhaps by writing a script that performs those 
 tasks, and calling that script within the unittest].


This holds for the requirments inside the code of the unit too: 
by adding a routine for the preferred re-initialization of the 
unit---wrapped by `version( unittest)'.

For the example presented the line of code
|  version( unittest) var1= 1;
would suffice, if correctly placed into the function.

-manfred
Dec 18 2024
prev sibling parent monkyyy <crazymonkyyy gmail.com> writes:
On Tuesday, 17 December 2024 at 19:16:55 UTC, DLearner wrote:

 On Sunday, 15 December 2024 at 20:30:21 UTC, monkyyy wrote:

 On Sunday, 15 December 2024 at 08:45:22 UTC, DLearner wrote:

 I appreciate this behaviour matches the docs (so not a bug), 
 but is it desirable?


 yes, the alternative would be that unittests attempt to undo 
 themselves, and that would make bugs horrible horrible bugs or 
 executable clear global scope and stack effectively restarting 
 the program, this could be incredibly slow if you have big 
 arrays in global scope and then hundards of small unrelated 
 unittests(which btw you do, the std has plenty and I think the 
 run time also injects some)


 What is wrong with changing the specification of unittest so 
 that it recompiles/reexecutes the associated source on every 
 unittest {} block?


Sounds also slow
Dec 17 2024
prev sibling parent user1234 <user1234 12.de> writes:
On Sunday, 15 December 2024 at 08:45:22 UTC, DLearner wrote:

 Please consider:

 ```
 size_t foo() {
    static size_t var1 = 1;

    var1 = var1 + 1;
    return var1;
 }

 unittest {

    assert(foo() == 2);
    assert(foo() == 3);
 }

 ```

 which works as expected.

 But

 ```
 size_t foo1() {
    static size_t var1 = 1;

    var1 = var1 + 1;
    return var1;
 }

 unittest {
    assert(foo1() == 2);
 }

 unittest {
    assert(foo1() == 2);
 }

 ```

 Fails on the second unittest.

 I appreciate this behaviour matches the docs (so not a bug), 
 but is it desirable?


Yes. Remember that you have the function attribute `pure` [[1]]. 
It would have avoided the problem.. for instance:

```d
size_t foo1() {
    static size_t var1 = 1;

    var1 = var1 + 1;
    return var1;
}

pure unittest {
    assert(foo1() == 2);
}

pure unittest {
    assert(foo1() == 2);
}
```
refuses to compile with the following errors


 test.d(9,15): Error: `pure` function 
 `temp_7F58D0140210.__unittest_L8_C6` cannot call impure 
 function `temp_7F58D0140210.foo1`
 test.d(13,15): Error: `pure` function 
 `temp_7F58D0140210.__unittest_L12_C6` cannot call impure 
 function `temp_7F58D0140210.foo1`


With `pure` that you would have seen the problem, that is "oh, 
the global state".

[1]: https://dlang.org/spec/function.html#pure-functions
Dec 16 2024