www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - unittests for inhereted classes

reply John R <John_member pathlink.com> writes:
Hello, I, like every good little java programmer, was raised on contracts, rep
invariants, and abstraction barriers, so naturally I like D. I've been playing
with it a bit and one of my unittests seems to be doing strange things. It is of
the form:

module FooClass;

public class FooClass {

unittest {
assert(FooClass.tick() == 0);
assert(FooClass.tick() == 1);
}

public static int count = 0;

public static int tick() {
return(FooClass.count++);
}

}

public class BarClass : FooClass{

unittest {
assert(BarClass.tick() == 0);		
}

}

The FooClass unittest runs fine, of course, but the BarClass test fails because
the variable has alread been incremented and is 2 when it gets to the Bar
unittest. If you move BarClass to the top of the file its unittest runs ok and
FooClass' fails.

I can only conclude that this implementation of unittest does not fully capture
the concept behind unit testing. My understanding has always been that a unit
test must not be dependent on other unittests. If it were, then a unittest of a
child class would not, in fact, be testing the child class, but the child class
with some arbitrary test code already run upon it. Then the child test may pass,
while giving no guarantee that the test code will work by itself in actual
program code.

I looked for other posts on this but I couldn't find any. Is this meant to be
this way? If this was a design choice, I might suggest that when class
hierarchies get large (I only had 4 levels of classes and I banged my head on
this bug for a while before I realized what was happening) independent unittests
get more important.

I haven't looked at the implementation (I'm not quite that brave), but couldn't
the program just sort of start with a clean slate for each unittest and not run
into this issue? Just a thought.

Thanks,
John R.
Feb 27 2005
next sibling parent Manfred Nowak <svv1999 hotmail.com> writes:
John R <John_member pathlink.com> wrote:

[...]

 My understanding has always been that a unit test must not be
 dependent on other unittests. 


[...]

`unittest' is meant to contain code that tests the unit they are 
contained in. But I do not know of any semantically constraint that 
hinders the coder to test something completely different.

In your example the code of the unittests depends on a public static 
variable, which can be changed at runtime from everywhere, and your 
testcase in the `unittest' is choosen in a way that points this out.

So you can be lucky, that you had such golden fingers to choose your 
testcase right to convince you, that your code is vulnerable. Only 
your requirement that `unitest' should not point out dangerous code 
seems to be wrong.

-manfred
Feb 27 2005
prev sibling parent reply John Demme <me teqdruid.com> writes:
Since you're using a static class variable, I'm relatively sure that a 
test like this would fail in Java using JUnit.  This is not a behavior 
that I think the compiler can help.  It would have to keep track of all 
the changes that happen as a result of the unittests, then reset them 
afterall.  I don't see it happening.

John (the other John)

John R wrote:

 Hello, I, like every good little java programmer, was raised on contracts, rep
 invariants, and abstraction barriers, so naturally I like D. I've been playing
 with it a bit and one of my unittests seems to be doing strange things. It is
of
 the form:
 
 module FooClass;
 
 public class FooClass {
 
 unittest {
 assert(FooClass.tick() == 0);
 assert(FooClass.tick() == 1);
 }
 
 public static int count = 0;
 
 public static int tick() {
 return(FooClass.count++);
 }
 
 }
 
 public class BarClass : FooClass{
 
 unittest {
 assert(BarClass.tick() == 0);		
 }
 
 }
 
 The FooClass unittest runs fine, of course, but the BarClass test fails because
 the variable has alread been incremented and is 2 when it gets to the Bar
 unittest. If you move BarClass to the top of the file its unittest runs ok and
 FooClass' fails.
 
 I can only conclude that this implementation of unittest does not fully capture
 the concept behind unit testing. My understanding has always been that a unit
 test must not be dependent on other unittests. If it were, then a unittest of a
 child class would not, in fact, be testing the child class, but the child class
 with some arbitrary test code already run upon it. Then the child test may
pass,
 while giving no guarantee that the test code will work by itself in actual
 program code.
 
 I looked for other posts on this but I couldn't find any. Is this meant to be
 this way? If this was a design choice, I might suggest that when class
 hierarchies get large (I only had 4 levels of classes and I banged my head on
 this bug for a while before I realized what was happening) independent
unittests
 get more important.
 
 I haven't looked at the implementation (I'm not quite that brave), but couldn't
 the program just sort of start with a clean slate for each unittest and not run
 into this issue? Just a thought.
 
 Thanks,
 John R.
Feb 27 2005
next sibling parent reply John Reimer <brk_6502 yahoo.com> writes:
You know what?  That post wasn't by me!

Darn it.  This name is just too common. :-(

- John R. (the other John R.!)


John Demme wrote:

 Since you're using a static class variable, I'm relatively sure that a 
 test like this would fail in Java using JUnit.  This is not a behavior 
 that I think the compiler can help.  It would have to keep track of all 
 the changes that happen as a result of the unittests, then reset them 
 afterall.  I don't see it happening.
 
 John (the other John)
 
 John R wrote:
 

 Hello, I, like every good little java programmer, was raised on 
 contracts, rep
 invariants, and abstraction barriers, so naturally I like D. I've been 
 playing
 with it a bit and one of my unittests seems to be doing strange 
 things. It is of
 the form:

 module FooClass;

 public class FooClass {

 unittest {
 assert(FooClass.tick() == 0);
 assert(FooClass.tick() == 1);
 }

 public static int count = 0;

 public static int tick() {
 return(FooClass.count++);
 }

 }

 public class BarClass : FooClass{

 unittest {
 assert(BarClass.tick() == 0);       
 }

 }

 The FooClass unittest runs fine, of course, but the BarClass test 
 fails because
 the variable has alread been incremented and is 2 when it gets to the Bar
 unittest. If you move BarClass to the top of the file its unittest 
 runs ok and
 FooClass' fails.

 I can only conclude that this implementation of unittest does not 
 fully capture
 the concept behind unit testing. My understanding has always been that 
 a unit
 test must not be dependent on other unittests. If it were, then a 
 unittest of a
 child class would not, in fact, be testing the child class, but the 
 child class
 with some arbitrary test code already run upon it. Then the child test 
 may pass,
 while giving no guarantee that the test code will work by itself in 
 actual
 program code.

 I looked for other posts on this but I couldn't find any. Is this 
 meant to be
 this way? If this was a design choice, I might suggest that when class
 hierarchies get large (I only had 4 levels of classes and I banged my 
 head on
 this bug for a while before I realized what was happening) independent 
 unittests
 get more important.

 I haven't looked at the implementation (I'm not quite that brave), but 
 couldn't
 the program just sort of start with a clean slate for each unittest 
 and not run
 into this issue? Just a thought.

 Thanks,
 John R.
Feb 27 2005
parent reply John Demme <me teqdruid.com> writes:
Heh.  OK... so from now on, to prevent confusion, we all must sign our 
messages as so:

Sincerely,
<Last Name>, <First Name> <Middle Name>
<Social Security Number, or non-USA equivalent>
<PGP Key Fingerprint>
<Email Address>

Or maybe it would be easier to just sign our messages via PGP.

John Demme

John Reimer wrote:

 You know what?  That post wasn't by me!
 
 Darn it.  This name is just too common. :-(
 
 - John R. (the other John R.!)
 
 
 John Demme wrote:
 

 Since you're using a static class variable, I'm relatively sure that a 
 test like this would fail in Java using JUnit.  This is not a behavior 
 that I think the compiler can help.  It would have to keep track of 
 all the changes that happen as a result of the unittests, then reset 
 them afterall.  I don't see it happening.

 John (the other John)

 John R wrote:


 Hello, I, like every good little java programmer, was raised on 
 contracts, rep
 invariants, and abstraction barriers, so naturally I like D. I've 
 been playing
 with it a bit and one of my unittests seems to be doing strange 
 things. It is of
 the form:

 module FooClass;

 public class FooClass {

 unittest {
 assert(FooClass.tick() == 0);
 assert(FooClass.tick() == 1);
 }

 public static int count = 0;

 public static int tick() {
 return(FooClass.count++);
 }

 }

 public class BarClass : FooClass{

 unittest {
 assert(BarClass.tick() == 0);       }

 }

 The FooClass unittest runs fine, of course, but the BarClass test 
 fails because
 the variable has alread been incremented and is 2 when it gets to the 
 Bar
 unittest. If you move BarClass to the top of the file its unittest 
 runs ok and
 FooClass' fails.

 I can only conclude that this implementation of unittest does not 
 fully capture
 the concept behind unit testing. My understanding has always been 
 that a unit
 test must not be dependent on other unittests. If it were, then a 
 unittest of a
 child class would not, in fact, be testing the child class, but the 
 child class
 with some arbitrary test code already run upon it. Then the child 
 test may pass,
 while giving no guarantee that the test code will work by itself in 
 actual
 program code.

 I looked for other posts on this but I couldn't find any. Is this 
 meant to be
 this way? If this was a design choice, I might suggest that when class
 hierarchies get large (I only had 4 levels of classes and I banged my 
 head on
 this bug for a while before I realized what was happening) 
 independent unittests
 get more important.

 I haven't looked at the implementation (I'm not quite that brave), 
 but couldn't
 the program just sort of start with a clean slate for each unittest 
 and not run
 into this issue? Just a thought.

 Thanks,
 John R.
Feb 27 2005
parent reply John Reimer <brk_6502 yahoo.com> writes:
He he... no SSN nor SIN (Canadian) numbers going on this list! ... 
although I have both.

I haven't used PGP before... probably a good idea.  For now, I think I 
might just use my initials or my full name :-P. I'll experiment.

It's kinda funny.  At my station, there's two of us with this name, 
although the other guy goes by the "Jon" variant (I'm always careful to 
emphasize that I'm "John" with an "h" :-) ), but when we do calls 
together he usually introduces himself to the patient as "little" Jon 
and refers to me as "big" John.  The naming convention has stuck even in 
the hospital. That's what you get for being 6'7". :-D

-JJR

John Demme wrote:

 Heh.  OK... so from now on, to prevent confusion, we all must sign our 
 messages as so:
 
 Sincerely,
 <Last Name>, <First Name> <Middle Name>
 <Social Security Number, or non-USA equivalent>
 <PGP Key Fingerprint>
 <Email Address>
 
 Or maybe it would be easier to just sign our messages via PGP.
 
 John Demme
 
 John Reimer wrote:
 

 You know what?  That post wasn't by me!

 Darn it.  This name is just too common. :-(

 - John R. (the other John R.!)


 John Demme wrote:


 Since you're using a static class variable, I'm relatively sure that 
 a test like this would fail in Java using JUnit.  This is not a 
 behavior that I think the compiler can help.  It would have to keep 
 track of all the changes that happen as a result of the unittests, 
 then reset them afterall.  I don't see it happening.

 John (the other John)
Feb 27 2005
parent reply John Demme <me teqdruid.com> writes:
I'm in a Rocky Horror cast with two Johns.  The other one is Good John, 
and I'm Bad John.  It's fun being bad.

Bad John

John Reimer wrote:

 He he... no SSN nor SIN (Canadian) numbers going on this list! ... 
 although I have both.
 
 I haven't used PGP before... probably a good idea.  For now, I think I 
 might just use my initials or my full name :-P. I'll experiment.
 
 It's kinda funny.  At my station, there's two of us with this name, 
 although the other guy goes by the "Jon" variant (I'm always careful to 
 emphasize that I'm "John" with an "h" :-) ), but when we do calls 
 together he usually introduces himself to the patient as "little" Jon 
 and refers to me as "big" John.  The naming convention has stuck even in 
 the hospital. That's what you get for being 6'7". :-D
 
 -JJR
 
 John Demme wrote:
 

 Heh.  OK... so from now on, to prevent confusion, we all must sign our 
 messages as so:

 Sincerely,
 <Last Name>, <First Name> <Middle Name>
 <Social Security Number, or non-USA equivalent>
 <PGP Key Fingerprint>
 <Email Address>

 Or maybe it would be easier to just sign our messages via PGP.

 John Demme

 John Reimer wrote:


 You know what?  That post wasn't by me!

 Darn it.  This name is just too common. :-(

 - John R. (the other John R.!)


 John Demme wrote:


 Since you're using a static class variable, I'm relatively sure that 
 a test like this would fail in Java using JUnit.  This is not a 
 behavior that I think the compiler can help.  It would have to keep 
 track of all the changes that happen as a result of the unittests, 
 then reset them afterall.  I don't see it happening.

 John (the other John)
Feb 27 2005
parent John Reimer <brk_6502 yahoo.com> writes:
John Demme wrote:

 I'm in a Rocky Horror cast with two Johns.  The other one is Good John, 
 and I'm Bad John.  It's fun being bad.
 
 Bad John


Ha! That paints a ...um... picture.  Now I'm reminded of the classic 
song "big bad John".  I like that song.  The combination is striking. :-)

Big John
Feb 28 2005
prev sibling parent John R <John_member pathlink.com> writes:
In article <cvt9is$nnh$1 digitaldaemon.com>, John Demme says...

Since you're using a static class variable, I'm relatively sure that a 
test like this would fail in Java using JUnit.  This is not a behavior 
that I think the compiler can help.  It would have to keep track of all 
the changes that happen as a result of the unittests, then reset them 
afterall.  I don't see it happening.

John (the other John)


I did a JUnit test of this:

public class ClassFoo {

public static int count = 0;

public static int tick() {
return(ClassFoo.count++);
}

}

public class ClassBar extends ClassFoo {

}

public class FooTest extends TestCase {

public static Test suite() {
return new TestSuite(FooTest.class);
}	

protected void setUp() {
//ClassFoo.count = 0;	
}

public void testTick() {
assertTrue(ClassFoo.tick() == 0);
assertTrue(ClassFoo.tick() == 1);
}

public void testTickAgain() {
assertTrue(ClassFoo.tick() == 0);
assertTrue(ClassFoo.tick() == 1);
}

}

public class BarExam extends TestCase {

public static Test suite() {
return new TestSuite(BarExam.class);
}

public void testInheretedTick() {
assertTrue(ClassBar.tick() == 0);	
}
}


and found that the problem with the d unit test (that the inhereted 
FooClass' unittest inhereted a modified object) did not occur. That is,
the BarExam unit test passes after having run FooTest. That being said, 
the tests were run as seperate junit tests, which means the two executions
were independent. 

Now, the FooTest exam fails as is, you have to uncomment
the line in the initializing function setUp() to get it to work. I realize 
that you could implement this setUp() function in D and then call it at the
end of the unittest, and so be more or less assured that the unittest does not
affect the subclass' unittest, but I think this might be a less than ideal 
solution. It still leaves all the subclass' unittests dependent on you having 
correctly reinitialized every class in the chain. There is also no test for
correct reinitialization, short of the framework keeping track of all changes to
the class (which I agree is an unreasonable expectation). This could lead to
problems like:
Adding in a new static variable to a base class, unittesting it, and then
forgetting to reinitialize at the end of the unittest. This could lead to
bizarre behavior in later unittests which would be hard to track down.
Or you might want to extend a class that you wrote a long time ago or someone
else wrote. If you run the unittest for it you have no guarantee that it has
been properly reinitialized and it could mess up the subclass' and all
subsequent unittests.

Just as an idea for the future of unittests, perhaps they could be somewhat like
the JUnit tests in the sense that each module has its unittest run seperately.
I've seen suggestions that the unittests be run automaticaly after compiling and
linking. If the tests were run then a seperate (temporary) executable could be
linked and run for each module's unittests. Therefore you would not have the
problem of subclass unittest dependence because each execution would be clean.

Walter, I believe you run this show, any thoughts in this direction?

John-Matt
(the poster formerly known as John R.)



John R wrote:

 Hello, I, like every good little java programmer, was raised on contracts, rep
 invariants, and abstraction barriers, so naturally I like D. I've been playing
 with it a bit and one of my unittests seems to be doing strange things. It is
of
 the form:
 
 module FooClass;
 
 public class FooClass {
 
 unittest {
 assert(FooClass.tick() == 0);
 assert(FooClass.tick() == 1);
 }
 
 public static int count = 0;
 
 public static int tick() {
 return(FooClass.count++);
 }
 
 }
 
 public class BarClass : FooClass{
 
 unittest {
 assert(BarClass.tick() == 0);		
 }
 
 }
 
 The FooClass unittest runs fine, of course, but the BarClass test fails because
 the variable has alread been incremented and is 2 when it gets to the Bar
 unittest. If you move BarClass to the top of the file its unittest runs ok and
 FooClass' fails.
 
 I can only conclude that this implementation of unittest does not fully capture
 the concept behind unit testing. My understanding has always been that a unit
 test must not be dependent on other unittests. If it were, then a unittest of a
 child class would not, in fact, be testing the child class, but the child class
 with some arbitrary test code already run upon it. Then the child test may
pass,
 while giving no guarantee that the test code will work by itself in actual
 program code.
 
 I looked for other posts on this but I couldn't find any. Is this meant to be
 this way? If this was a design choice, I might suggest that when class
 hierarchies get large (I only had 4 levels of classes and I banged my head on
 this bug for a while before I realized what was happening) independent
unittests
 get more important.
 
 I haven't looked at the implementation (I'm not quite that brave), but couldn't
 the program just sort of start with a clean slate for each unittest and not run
 into this issue? Just a thought.
 
 Thanks,
 John R.
Feb 28 2005