www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Hack to name unit tests?

reply Robert Fraser <fraserofthenight gmail.com> writes:
I'm creating a Flectioned-based way to run unit tests individually 
(it'll be hooked into a front-end, so you can run test suites, repeat 
only failed tests after a code change, etc.)

However, I have run into a problem: unit tests can't be named. Right now 
I have a signature system, but I was wondering if there's any hackish 
way to predictably insert a string into the generated object file in a 
way that users can name unittests. That is to say, given a function 
pointer, is there any way I can scan the code to see if a name has been 
given to the unit test, and if so extract that name at runtime?

Ideally, it'd be as easy for the user as something like:

unittest
{
    assert(":testName:");
    // Rest of test goes here...
}

Right now, I have the associations being done in the front-end I'm 
working on, that scans the code for comments in a particular format and 
associates those with the unit tests. However, that ties the unit test 
executor back-end to the code analysis front-end, so I was hoping 
there's a better way.

Thanks!
Nov 28 2007
next sibling parent reply Dan <murpsoft hotmail.com> writes:
Robert Fraser Wrote:


 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)


I would have recommended you approach Walter and his team (whomever they may be
nowadays) with the idea.
 

 Ideally, it'd be as easy for the user as something like:
 
 unittest
 {
     assert(":testName:");
     // Rest of test goes here...
 }


I'd like to see:

unittest testName
{
  // rest of text
}

I'd then like to be able to specifically execute only specific unit test(s) so
that for instance I can debug the relationship between my x and y modules
without triggering off a test of the whole freakin' alphabet.

I would then like an improvement of unittests so that one can do more than
merely assert().  Complete self-examination code should be possible - which
means being able to examine program flow and trace variables.  This would
ultimately lead to *being able to* generate proofs on the correctness of the
code.

Someone could then just write a library.
Nov 28 2007
next sibling parent Leandro Lucarella <llucax gmail.com> writes:
Dan, el 28 de noviembre a las 18:20 me escribiste:

 Robert Fraser Wrote:
 

 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)


 
 I would have recommended you approach Walter and his team (whomever they may
be nowadays) with the idea.
  

 Ideally, it'd be as easy for the user as something like:
 
 unittest
 {
     assert(":testName:");
     // Rest of test goes here...
 }


 
 I'd like to see:
 
 unittest testName
 {
   // rest of text
 }


Maybe something like

unittest "test name"
{
  // rest of the test
}

could be more flexible about naming =)

-- 
Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/
----------------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------------
You should've seen her face. It was the exact same look my father gave me when
I told him I wanted to be a ventriloquist.
	-- George Constanza
Nov 28 2007
prev sibling parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Dan wrote:

 I'd then like to be able to specifically execute only specific unit test(s) so
that for instance I can debug the relationship between my x and y modules
without triggering off a test of the whole freakin' alphabet.
 


Besides the naming, all that is possible right now, which is what my 
tool is doing.


 I would then like an improvement of unittests so that one can do more than
merely assert().  Complete self-examination code should be possible - which
means being able to examine program flow and trace variables.  This would
ultimately lead to *being able to* generate proofs on the correctness of the
code.
 
 Someone could then just write a library.


Hmmm... that sounds a bit complex for an average tester. Might be a good 
idea in a pure-functional language, but I can't see it being much use in D.
Nov 28 2007
parent =?iso-8859-15?Q?Jari-Matti_M=E4kel=E4?= <jmjmak utu.fi.invalid> writes:
On Wed, 28 Nov 2007, Robert Fraser wrote:


 Dan wrote:

 I'd then like to be able to specifically execute only specific unit test(s) 
 so that for instance I can debug the relationship between my x and y 
 modules without triggering off a test of the whole freakin' alphabet.
 


 Besides the naming, all that is possible right now, which is what my tool is 
 doing.


 I would then like an improvement of unittests so that one can do more than 
 merely assert().  Complete self-examination code should be possible - which 
 means being able to examine program flow and trace variables.  This would 
 ultimately lead to *being able to* generate proofs on the correctness of 
 the code.
 
 Someone could then just write a library.


 Hmmm... that sounds a bit complex for an average tester. Might be a good idea 
 in a pure-functional language, but I can't see it being much use in D.


Yep, it's a bit difficult to prove the correctness with unittests because 
of side effects. Also 100% test coverage won't mean it works perfectly.

Hmm, what about unittest as an attribute / annotation for functions. This 
works pretty well in Java and is more flexible than the one in D.
Nov 29 2007
prev sibling next sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
Robert Fraser wrote:

 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)
 
 However, I have run into a problem: unit tests can't be named. Right now 
 I have a signature system, but I was wondering if there's any hackish 
 way to predictably insert a string into the generated object file in a 
 way that users can name unittests. That is to say, given a function 
 pointer, is there any way I can scan the code to see if a name has been 
 given to the unit test, and if so extract that name at runtime?
 
 Ideally, it'd be as easy for the user as something like:
 
 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }
 
 Right now, I have the associations being done in the front-end I'm 
 working on, that scans the code for comments in a particular format and 
 associates those with the unit tests. However, that ties the unit test 
 executor back-end to the code analysis front-end, so I was hoping 
 there's a better way.
 
 Thanks!


class Unittest (string _name, alias _dg) : IUnittest {
    static string name = _name;
    static void delegate() test = _dg;
}

Usage:
Unittest!("my test for wossname", {
    assert (false, "haven't gotten around to implementing this yet");
});

Not guaranteed to work.

Another tactic would be something like:

class UnittestManager {
    static void registerCurrentTest (string name) {}
    static void endTest (bool success) {}
    static bool performTest () {}
}

template Unittest (string _name, alias _dg) {
    unittest {
       UnittestManager.registerCurrentTest(_name);

       // This lets us skip the test if we're just trying
       // to find out what tests there are currently.
       if (UnittestManager.performTest) {
          scope(success) UnittestManager.endTest(true);
          scope(failure) UnittestManager.endTest(false);
          _dg();
       }
    }
}

mixin Unittest!("my test name", { assert (false, "not yet implemented"); });
Nov 28 2007
parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Christopher Wright wrote:

 Robert Fraser wrote:

 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named. Right 
 now I have a signature system, but I was wondering if there's any 
 hackish way to predictably insert a string into the generated object 
 file in a way that users can name unittests. That is to say, given a 
 function pointer, is there any way I can scan the code to see if a 
 name has been given to the unit test, and if so extract that name at 
 runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

 Right now, I have the associations being done in the front-end I'm 
 working on, that scans the code for comments in a particular format 
 and associates those with the unit tests. However, that ties the unit 
 test executor back-end to the code analysis front-end, so I was hoping 
 there's a better way.

 Thanks!


 
 class Unittest (string _name, alias _dg) : IUnittest {
    static string name = _name;
    static void delegate() test = _dg;
 }
 
 Usage:
 Unittest!("my test for wossname", {
    assert (false, "haven't gotten around to implementing this yet");
 });
 
 Not guaranteed to work.
 
 Another tactic would be something like:
 
 class UnittestManager {
    static void registerCurrentTest (string name) {}
    static void endTest (bool success) {}
    static bool performTest () {}
 }
 
 template Unittest (string _name, alias _dg) {
    unittest {
       UnittestManager.registerCurrentTest(_name);
 
       // This lets us skip the test if we're just trying
       // to find out what tests there are currently.
       if (UnittestManager.performTest) {
          scope(success) UnittestManager.endTest(true);
          scope(failure) UnittestManager.endTest(false);
          _dg();
       }
    }
 }
 
 mixin Unittest!("my test name", { assert (false, "not yet implemented"); 
 });


Thanks! That's a good idea, but I want something compatible with current 
unittest {} declarations.
Nov 28 2007
parent reply Christopher Wright <dhasenan gmail.com> writes:
Robert Fraser wrote:

 Christopher Wright wrote:

 Robert Fraser wrote:

 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named. Right 
 now I have a signature system, but I was wondering if there's any 
 hackish way to predictably insert a string into the generated object 
 file in a way that users can name unittests. That is to say, given a 
 function pointer, is there any way I can scan the code to see if a 
 name has been given to the unit test, and if so extract that name at 
 runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

 Right now, I have the associations being done in the front-end I'm 
 working on, that scans the code for comments in a particular format 
 and associates those with the unit tests. However, that ties the unit 
 test executor back-end to the code analysis front-end, so I was 
 hoping there's a better way.

 Thanks!


 class Unittest (string _name, alias _dg) : IUnittest {
    static string name = _name;
    static void delegate() test = _dg;
 }

 Usage:
 Unittest!("my test for wossname", {
    assert (false, "haven't gotten around to implementing this yet");
 });

 Not guaranteed to work.

 Another tactic would be something like:

 class UnittestManager {
    static void registerCurrentTest (string name) {}
    static void endTest (bool success) {}
    static bool performTest () {}
 }

 template Unittest (string _name, alias _dg) {
    unittest {
       UnittestManager.registerCurrentTest(_name);

       // This lets us skip the test if we're just trying
       // to find out what tests there are currently.
       if (UnittestManager.performTest) {
          scope(success) UnittestManager.endTest(true);
          scope(failure) UnittestManager.endTest(false);
          _dg();
       }
    }
 }

 mixin Unittest!("my test name", { assert (false, "not yet 
 implemented"); });


 
 Thanks! That's a good idea, but I want something compatible with current 
 unittest {} declarations.


You already have to modify the tests to insert the name, but you want 
something that works as a decorator inside the unittest. Can your 
unittest runner do a try/catch on a unittest? Then you could have 
something like:

void testName(string name) {
    if (collectingNames)
       throw new UnittestNameException(name);
}

struct NamedTest {
    Unittest test;
    string name;
}

...
void main () {
    NamedTest[] tests;
    collectingNames = true;
    foreach (test; getUnittests) {
        try {
            test();
        } catch (UnittestNameException e) {
            tests ~= NamedTest(test, e.name);
        }
    }
    collectingNames = false;
}

unittest {
    testName("blargh");
    assert (false, "still working on this");
}
Nov 29 2007
next sibling parent reply Dan <murpsoft hotmail.com> writes:
Christopher Wright Wrote:


 Robert Fraser wrote:

 Christopher Wright wrote:

 Robert Fraser wrote:

 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named. Right 
 now I have a signature system, but I was wondering if there's any 
 hackish way to predictably insert a string into the generated object 
 file in a way that users can name unittests. That is to say, given a 
 function pointer, is there any way I can scan the code to see if a 
 name has been given to the unit test, and if so extract that name at 
 runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

 Right now, I have the associations being done in the front-end I'm 
 working on, that scans the code for comments in a particular format 
 and associates those with the unit tests. However, that ties the unit 
 test executor back-end to the code analysis front-end, so I was 
 hoping there's a better way.

 Thanks!


 class Unittest (string _name, alias _dg) : IUnittest {
    static string name = _name;
    static void delegate() test = _dg;
 }

 Usage:
 Unittest!("my test for wossname", {
    assert (false, "haven't gotten around to implementing this yet");
 });

 Not guaranteed to work.

 Another tactic would be something like:

 class UnittestManager {
    static void registerCurrentTest (string name) {}
    static void endTest (bool success) {}
    static bool performTest () {}
 }

 template Unittest (string _name, alias _dg) {
    unittest {
       UnittestManager.registerCurrentTest(_name);

       // This lets us skip the test if we're just trying
       // to find out what tests there are currently.
       if (UnittestManager.performTest) {
          scope(success) UnittestManager.endTest(true);
          scope(failure) UnittestManager.endTest(false);
          _dg();
       }
    }
 }

 mixin Unittest!("my test name", { assert (false, "not yet 
 implemented"); });


 
 Thanks! That's a good idea, but I want something compatible with current 
 unittest {} declarations.


 
 You already have to modify the tests to insert the name, but you want 
 something that works as a decorator inside the unittest. Can your 
 unittest runner do a try/catch on a unittest? Then you could have 
 something like:
 
 void testName(string name) {
     if (collectingNames)
        throw new UnittestNameException(name);
 }
 
 struct NamedTest {
     Unittest test;
     string name;
 }
 
 ...
 void main () {
     NamedTest[] tests;
     collectingNames = true;
     foreach (test; getUnittests) {
         try {
             test();
         } catch (UnittestNameException e) {
             tests ~= NamedTest(test, e.name);
         }
     }
     collectingNames = false;
 }
 
 unittest {
     testName("blargh");
     assert (false, "still working on this");
 }


Eww... no.  You guys are suggesting that code provability is useless in D
because it's "too complicated" and then go on to suggest it can be used as a
heavily over-classed debug statement.

unittest is currently used more or less as a place to put a barrage of random
sample tests.  I was suggesting (beyond the obvious improvement of naming and
only executing select unittests) that we should be able to put hooks to a
provability program here.

The provability program would be a separate utility that would go through and
show things like bit overflow potential in numbers, potential infinities or
potential bad memory accesses.  It would do these things by following variables
throughout their lifetimes to show all the different values they could possibly
be at each time they're used.

Combined with contract programming (in/out and invariant) this kind of thing
exposes code in a way that running it with test values simply can't.

For example:

In libc, in qsort(), there was a bug because:

median = (low + high) >> 1;

The problem here is that in a very large array, low + high will eventually be >
MAX_INT.  This kind of bug isn't detected by simply doing a dumb random test on
an array of 10.  You need to be aware of the full range of possible values and
use inductive logic to predict the bug without testing every last case.

Regards,
Dan
Nov 29 2007
next sibling parent Christopher Wright <dhasenan gmail.com> writes:
Dan wrote:

 Eww... no.  You guys are suggesting that code provability is useless in D
because it's "too complicated" and then go on to suggest it can be used as a
heavily over-classed debug statement.


<snip>

I don't know what thread you meant to reply to. This is only about
naming tests somehow, not about proving anything about test coverage. Or 
am I confused about something?
Nov 29 2007
prev sibling parent Robert Fraser <fraserofthenight gmail.com> writes:
Dan wrote:

 Christopher Wright Wrote:
 

 Robert Fraser wrote:

 Christopher Wright wrote:

 Robert Fraser wrote:

 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named. Right 
 now I have a signature system, but I was wondering if there's any 
 hackish way to predictably insert a string into the generated object 
 file in a way that users can name unittests. That is to say, given a 
 function pointer, is there any way I can scan the code to see if a 
 name has been given to the unit test, and if so extract that name at 
 runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

 Right now, I have the associations being done in the front-end I'm 
 working on, that scans the code for comments in a particular format 
 and associates those with the unit tests. However, that ties the unit 
 test executor back-end to the code analysis front-end, so I was 
 hoping there's a better way.

 Thanks!


 class Unittest (string _name, alias _dg) : IUnittest {
    static string name = _name;
    static void delegate() test = _dg;
 }

 Usage:
 Unittest!("my test for wossname", {
    assert (false, "haven't gotten around to implementing this yet");
 });

 Not guaranteed to work.

 Another tactic would be something like:

 class UnittestManager {
    static void registerCurrentTest (string name) {}
    static void endTest (bool success) {}
    static bool performTest () {}
 }

 template Unittest (string _name, alias _dg) {
    unittest {
       UnittestManager.registerCurrentTest(_name);

       // This lets us skip the test if we're just trying
       // to find out what tests there are currently.
       if (UnittestManager.performTest) {
          scope(success) UnittestManager.endTest(true);
          scope(failure) UnittestManager.endTest(false);
          _dg();
       }
    }
 }

 mixin Unittest!("my test name", { assert (false, "not yet 
 implemented"); });


 Thanks! That's a good idea, but I want something compatible with current 
 unittest {} declarations.


 You already have to modify the tests to insert the name, but you want 
 something that works as a decorator inside the unittest. Can your 
 unittest runner do a try/catch on a unittest? Then you could have 
 something like:

 void testName(string name) {
     if (collectingNames)
        throw new UnittestNameException(name);
 }

 struct NamedTest {
     Unittest test;
     string name;
 }

 ...
 void main () {
     NamedTest[] tests;
     collectingNames = true;
     foreach (test; getUnittests) {
         try {
             test();
         } catch (UnittestNameException e) {
             tests ~= NamedTest(test, e.name);
         }
     }
     collectingNames = false;
 }

 unittest {
     testName("blargh");
     assert (false, "still working on this");
 }


 
 Eww... no.  You guys are suggesting that code provability is useless in D
because it's "too complicated" and then go on to suggest it can be used as a
heavily over-classed debug statement.
 
 unittest is currently used more or less as a place to put a barrage of random
sample tests.  I was suggesting (beyond the obvious improvement of naming and
only executing select unittests) that we should be able to put hooks to a
provability program here.
 
 The provability program would be a separate utility that would go through and
show things like bit overflow potential in numbers, potential infinities or
potential bad memory accesses.  It would do these things by following variables
throughout their lifetimes to show all the different values they could possibly
be at each time they're used.
 
 Combined with contract programming (in/out and invariant) this kind of thing
exposes code in a way that running it with test values simply can't.
 
 For example:
 
 In libc, in qsort(), there was a bug because:
 
 median = (low + high) >> 1;
 
 The problem here is that in a very large array, low + high will eventually be
> MAX_INT.  This kind of bug isn't detected by simply doing a dumb random test
on an array of 10.  You need to be aware of the full range of possible values
and use inductive logic to predict the bug without testing every last case.
 
 Regards,
 Dan


It's not a bad idea, especially for smaller algorithms, but from my 
experience, using Coverity on a ~18 million LOC Java project fixed about 
10 bugs, while the unit tests were constantly auditing check-ins, and 
prevented me from checking in at least that many of my own.
Nov 29 2007
prev sibling parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Christopher Wright wrote:

 Robert Fraser wrote:

 Christopher Wright wrote:

 Robert Fraser wrote:

 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, 
 repeat only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named. Right 
 now I have a signature system, but I was wondering if there's any 
 hackish way to predictably insert a string into the generated object 
 file in a way that users can name unittests. That is to say, given a 
 function pointer, is there any way I can scan the code to see if a 
 name has been given to the unit test, and if so extract that name at 
 runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

 Right now, I have the associations being done in the front-end I'm 
 working on, that scans the code for comments in a particular format 
 and associates those with the unit tests. However, that ties the 
 unit test executor back-end to the code analysis front-end, so I was 
 hoping there's a better way.

 Thanks!


 class Unittest (string _name, alias _dg) : IUnittest {
    static string name = _name;
    static void delegate() test = _dg;
 }

 Usage:
 Unittest!("my test for wossname", {
    assert (false, "haven't gotten around to implementing this yet");
 });

 Not guaranteed to work.

 Another tactic would be something like:

 class UnittestManager {
    static void registerCurrentTest (string name) {}
    static void endTest (bool success) {}
    static bool performTest () {}
 }

 template Unittest (string _name, alias _dg) {
    unittest {
       UnittestManager.registerCurrentTest(_name);

       // This lets us skip the test if we're just trying
       // to find out what tests there are currently.
       if (UnittestManager.performTest) {
          scope(success) UnittestManager.endTest(true);
          scope(failure) UnittestManager.endTest(false);
          _dg();
       }
    }
 }

 mixin Unittest!("my test name", { assert (false, "not yet 
 implemented"); });


 Thanks! That's a good idea, but I want something compatible with 
 current unittest {} declarations.


 
 You already have to modify the tests to insert the name, but you want 
 something that works as a decorator inside the unittest. Can your 
 unittest runner do a try/catch on a unittest? Then you could have 
 something like:
 
 void testName(string name) {
    if (collectingNames)
       throw new UnittestNameException(name);
 }
 
 struct NamedTest {
    Unittest test;
    string name;
 }
 
 ...
 void main () {
    NamedTest[] tests;
    collectingNames = true;
    foreach (test; getUnittests) {
        try {
            test();
        } catch (UnittestNameException e) {
            tests ~= NamedTest(test, e.name);
        }
    }
    collectingNames = false;
 }
 
 unittest {
    testName("blargh");
    assert (false, "still working on this");
 }


Heh; alright; that's what I was looking for! Thanks!
Nov 29 2007
parent reply Christopher Wright <dhasenan gmail.com> writes:
Robert Fraser wrote:

 Christopher Wright wrote:

 Robert Fraser wrote:

 Christopher Wright wrote:

 Robert Fraser wrote:

 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, 
 repeat only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named. 
 Right now I have a signature system, but I was wondering if there's 
 any hackish way to predictably insert a string into the generated 
 object file in a way that users can name unittests. That is to say, 
 given a function pointer, is there any way I can scan the code to 
 see if a name has been given to the unit test, and if so extract 
 that name at runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

 Right now, I have the associations being done in the front-end I'm 
 working on, that scans the code for comments in a particular format 
 and associates those with the unit tests. However, that ties the 
 unit test executor back-end to the code analysis front-end, so I 
 was hoping there's a better way.

 Thanks!


 class Unittest (string _name, alias _dg) : IUnittest {
    static string name = _name;
    static void delegate() test = _dg;
 }

 Usage:
 Unittest!("my test for wossname", {
    assert (false, "haven't gotten around to implementing this yet");
 });

 Not guaranteed to work.

 Another tactic would be something like:

 class UnittestManager {
    static void registerCurrentTest (string name) {}
    static void endTest (bool success) {}
    static bool performTest () {}
 }

 template Unittest (string _name, alias _dg) {
    unittest {
       UnittestManager.registerCurrentTest(_name);

       // This lets us skip the test if we're just trying
       // to find out what tests there are currently.
       if (UnittestManager.performTest) {
          scope(success) UnittestManager.endTest(true);
          scope(failure) UnittestManager.endTest(false);
          _dg();
       }
    }
 }

 mixin Unittest!("my test name", { assert (false, "not yet 
 implemented"); });


 Thanks! That's a good idea, but I want something compatible with 
 current unittest {} declarations.


 You already have to modify the tests to insert the name, but you want 
 something that works as a decorator inside the unittest. Can your 
 unittest runner do a try/catch on a unittest? Then you could have 
 something like:

 void testName(string name) {
    if (collectingNames)
       throw new UnittestNameException(name);
 }

 struct NamedTest {
    Unittest test;
    string name;
 }

 ...
 void main () {
    NamedTest[] tests;
    collectingNames = true;
    foreach (test; getUnittests) {
        try {
            test();
        } catch (UnittestNameException e) {
            tests ~= NamedTest(test, e.name);
        }
    }
    collectingNames = false;
 }

 unittest {
    testName("blargh");
    assert (false, "still working on this");
 }


 Heh; alright; that's what I was looking for! Thanks!


No worries. It's pretty much identical in principle to mocked methods in 
dmocks, except for using exceptions to pass the name around and abort 
the method.
Nov 29 2007
parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Christopher Wright wrote:

 Robert Fraser wrote:

 Christopher Wright wrote:

 Robert Fraser wrote:

 Christopher Wright wrote:

 Robert Fraser wrote:

 I'm creating a Flectioned-based way to run unit tests individually
 (it'll be hooked into a front-end, so you can run test suites,
 repeat only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named.
 Right now I have a signature system, but I was wondering if
 there's any hackish way to predictably insert a string into the
 generated object file in a way that users can name unittests. That
 is to say, given a function pointer, is there any way I can scan
 the code to see if a name has been given to the unit test, and if
 so extract that name at runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

 Right now, I have the associations being done in the front-end I'm
 working on, that scans the code for comments in a particular
 format and associates those with the unit tests. However, that
 ties the unit test executor back-end to the code analysis
 front-end, so I was hoping there's a better way.

 Thanks!


 class Unittest (string _name, alias _dg) : IUnittest {
    static string name = _name;
    static void delegate() test = _dg;
 }

 Usage:
 Unittest!("my test for wossname", {
    assert (false, "haven't gotten around to implementing this yet");
 });

 Not guaranteed to work.

 Another tactic would be something like:

 class UnittestManager {
    static void registerCurrentTest (string name) {}
    static void endTest (bool success) {}
    static bool performTest () {}
 }

 template Unittest (string _name, alias _dg) {
    unittest {
       UnittestManager.registerCurrentTest(_name);

       // This lets us skip the test if we're just trying
       // to find out what tests there are currently.
       if (UnittestManager.performTest) {
          scope(success) UnittestManager.endTest(true);
          scope(failure) UnittestManager.endTest(false);
          _dg();
       }
    }
 }

 mixin Unittest!("my test name", { assert (false, "not yet
 implemented"); });


 Thanks! That's a good idea, but I want something compatible with
 current unittest {} declarations.


 You already have to modify the tests to insert the name, but you want
 something that works as a decorator inside the unittest. Can your
 unittest runner do a try/catch on a unittest? Then you could have
 something like:

 void testName(string name) {
    if (collectingNames)
       throw new UnittestNameException(name);
 }

 struct NamedTest {
    Unittest test;
    string name;
 }

 ...
 void main () {
    NamedTest[] tests;
    collectingNames = true;
    foreach (test; getUnittests) {
        try {
            test();
        } catch (UnittestNameException e) {
            tests ~= NamedTest(test, e.name);
        }
    }
    collectingNames = false;
 }

 unittest {
    testName("blargh");
    assert (false, "still working on this");
 }


 Heh; alright; that's what I was looking for! Thanks!


 
 No worries. It's pretty much identical in principle to mocked methods in
 dmocks, except for using exceptions to pass the name around and abort
 the method.


Wouldn't it be simpler to provide an exern method that unittests could
call into?

extern(C) void testName(char[] name);

unittest
{
    testName("blargh");
    assert(false, "too lazy to write a real unit test...");
}

You then provide testName somewhere in your runner program.  If you want
to compile without the unittest frontend, just create a stub object file
with this in it:

extern(C) void testName(char[] name) {}

Should be a hell of a lot faster than tossing exceptions willy-nilly.

	-- Daniel
Nov 29 2007
parent reply Christopher Wright <dhasenan gmail.com> writes:
Daniel Keep wrote:

 
 Christopher Wright wrote:

 Robert Fraser wrote:

 Christopher Wright wrote:

 Robert Fraser wrote:

 Christopher Wright wrote:

 Robert Fraser wrote:

 I'm creating a Flectioned-based way to run unit tests individually
 (it'll be hooked into a front-end, so you can run test suites,
 repeat only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named.
 Right now I have a signature system, but I was wondering if
 there's any hackish way to predictably insert a string into the
 generated object file in a way that users can name unittests. That
 is to say, given a function pointer, is there any way I can scan
 the code to see if a name has been given to the unit test, and if
 so extract that name at runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

 Right now, I have the associations being done in the front-end I'm
 working on, that scans the code for comments in a particular
 format and associates those with the unit tests. However, that
 ties the unit test executor back-end to the code analysis
 front-end, so I was hoping there's a better way.

 Thanks!


 class Unittest (string _name, alias _dg) : IUnittest {
    static string name = _name;
    static void delegate() test = _dg;
 }

 Usage:
 Unittest!("my test for wossname", {
    assert (false, "haven't gotten around to implementing this yet");
 });

 Not guaranteed to work.

 Another tactic would be something like:

 class UnittestManager {
    static void registerCurrentTest (string name) {}
    static void endTest (bool success) {}
    static bool performTest () {}
 }

 template Unittest (string _name, alias _dg) {
    unittest {
       UnittestManager.registerCurrentTest(_name);

       // This lets us skip the test if we're just trying
       // to find out what tests there are currently.
       if (UnittestManager.performTest) {
          scope(success) UnittestManager.endTest(true);
          scope(failure) UnittestManager.endTest(false);
          _dg();
       }
    }
 }

 mixin Unittest!("my test name", { assert (false, "not yet
 implemented"); });


 Thanks! That's a good idea, but I want something compatible with
 current unittest {} declarations.


 You already have to modify the tests to insert the name, but you want
 something that works as a decorator inside the unittest. Can your
 unittest runner do a try/catch on a unittest? Then you could have
 something like:

 void testName(string name) {
    if (collectingNames)
       throw new UnittestNameException(name);
 }

 struct NamedTest {
    Unittest test;
    string name;
 }

 ...
 void main () {
    NamedTest[] tests;
    collectingNames = true;
    foreach (test; getUnittests) {
        try {
            test();
        } catch (UnittestNameException e) {
            tests ~= NamedTest(test, e.name);
        }
    }
    collectingNames = false;
 }

 unittest {
    testName("blargh");
    assert (false, "still working on this");
 }


 Heh; alright; that's what I was looking for! Thanks!


 No worries. It's pretty much identical in principle to mocked methods in
 dmocks, except for using exceptions to pass the name around and abort
 the method.


 
 Wouldn't it be simpler to provide an exern method that unittests could
 call into?
 
 extern(C) void testName(char[] name);
 
 unittest
 {
     testName("blargh");
     assert(false, "too lazy to write a real unit test...");
 }
 
 You then provide testName somewhere in your runner program.  If you want
 to compile without the unittest frontend, just create a stub object file
 with this in it:
 
 extern(C) void testName(char[] name) {}
 
 Should be a hell of a lot faster than tossing exceptions willy-nilly.
 
 	-- Daniel


There are advantages to throwing an exception when registering the name:

unittest {
    testName("this is the test that doesn't end");
    while (true) {}
}

I was thinking that AssertionErrors couldn't be caught, but that was an 
error on my part.
Nov 29 2007
parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Christopher Wright wrote:

 ...
 There are advantages to throwing an exception when registering the name:
 
 unittest {
    testName("this is the test that doesn't end");
    while (true) {}
 }
 
 I was thinking that AssertionErrors couldn't be caught, but that was an
 error on my part.


How is an exception an advantage in that case?  When you run the test,
it's still going to spinlock, no matter how you've obtained the name.

If what you're trying to get at is that you can build a list of all the
names without having to run the actual test itself, then yes, I can see
that being an advantage.

	-- Daniel
Nov 29 2007
parent reply Christopher Wright <dhasenan gmail.com> writes:
Daniel Keep wrote:

 
 Christopher Wright wrote:

 ...
 There are advantages to throwing an exception when registering the name:

 unittest {
    testName("this is the test that doesn't end");
    while (true) {}
 }

 I was thinking that AssertionErrors couldn't be caught, but that was an
 error on my part.


 
 How is an exception an advantage in that case?  When you run the test,
 it's still going to spinlock, no matter how you've obtained the name.
 
 If what you're trying to get at is that you can build a list of all the
 names without having to run the actual test itself, then yes, I can see
 that being an advantage.


Exactly, exactly! Precisely what I told Miss Chow. Tissue augmentation, 
it's not a matter of twiddling bits.

On a more serious note, tools like the XUnit test runners display a list 
of names without running the tests. That's convenient, I think. The 
NUnit gui takes about twenty seconds to find all the unit tests we have 
at work, and it takes two or three minutes to run the tests. If I want 
to check a single test, two or three minutes is a long time.


 	-- Daniel
Nov 30 2007
parent Robert Fraser <fraserofthenight gmail.com> writes:
Christopher Wright wrote:

 Daniel Keep wrote:

 Christopher Wright wrote:

 ...
 There are advantages to throwing an exception when registering the name:

 unittest {
    testName("this is the test that doesn't end");
    while (true) {}
 }

 I was thinking that AssertionErrors couldn't be caught, but that was an
 error on my part.


 How is an exception an advantage in that case?  When you run the test,
 it's still going to spinlock, no matter how you've obtained the name.

 If what you're trying to get at is that you can build a list of all the
 names without having to run the actual test itself, then yes, I can see
 that being an advantage.


 
 Exactly, exactly! Precisely what I told Miss Chow. Tissue augmentation, 
 it's not a matter of twiddling bits.
 
 On a more serious note, tools like the XUnit test runners display a list 
 of names without running the tests. That's convenient, I think. The 
 NUnit gui takes about twenty seconds to find all the unit tests we have 
 at work, and it takes two or three minutes to run the tests. If I want 
 to check a single test, two or three minutes is a long time.
 

     -- Daniel




Check Don's post in this topic for a way to get test names w/o runnning 
them at all (and depending on them throwing the exception; that is, 
unnamed tests will still work correctly). I'm going to be implementing 
that... assuming I can get the free time & sleep required for that.
Nov 30 2007
prev sibling parent reply Don Clugston <dac nospam.com.au> writes:
Robert Fraser wrote:

 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)
 
 However, I have run into a problem: unit tests can't be named. Right now 
 I have a signature system, but I was wondering if there's any hackish 
 way to predictably insert a string into the generated object file in a 
 way that users can name unittests. That is to say, given a function 
 pointer, is there any way I can scan the code to see if a name has been 
 given to the unit test, and if so extract that name at runtime?
 
 Ideally, it'd be as easy for the user as something like:
 
 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }


I think this is possible, but tricky. If your unit test has a local variable, 
and you instantiate a template using that local variable as an alias parameter, 
the module and the number of the unit test gets included into the mangled name.
This means that it's possible for code to know which unit test it is inside.
By making the template a class, you could to walk the RTTI list and make a 
lookup table to associate test numbers with their names. If you can get the
list 
  of unittest numbers from flectioned, you can join it

BUT... that's all pretty hairy. Maybe there's a less convoluted way of doing it.
Nov 29 2007
parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Don Clugston wrote:

 Robert Fraser wrote:

 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named. Right 
 now I have a signature system, but I was wondering if there's any 
 hackish way to predictably insert a string into the generated object 
 file in a way that users can name unittests. That is to say, given a 
 function pointer, is there any way I can scan the code to see if a 
 name has been given to the unit test, and if so extract that name at 
 runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }


 
 I think this is possible, but tricky. If your unit test has a local 
 variable, and you instantiate a template using that local variable as an 
 alias parameter, the module and the number of the unit test gets 
 included into the mangled name.
 This means that it's possible for code to know which unit test it is 
 inside.
 By making the template a class, you could to walk the RTTI list and make 
 a lookup table to associate test numbers with their names. If you can 
 get the list  of unittest numbers from flectioned, you can join it
 
 BUT... that's all pretty hairy. Maybe there's a less convoluted way of 
 doing it.


OK; _that's_ what I was looking for, since the test doesn't need to be 
run that way... time to get hacking!
Nov 30 2007
parent Don Clugston <dac nospam.com.au> writes:
Robert Fraser wrote:

 Don Clugston wrote:

 Robert Fraser wrote:

 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named. Right 
 now I have a signature system, but I was wondering if there's any 
 hackish way to predictably insert a string into the generated object 
 file in a way that users can name unittests. That is to say, given a 
 function pointer, is there any way I can scan the code to see if a 
 name has been given to the unit test, and if so extract that name at 
 runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }


 I think this is possible, but tricky. If your unit test has a local 
 variable, and you instantiate a template using that local variable as 
 an alias parameter, the module and the number of the unit test gets 
 included into the mangled name.
 This means that it's possible for code to know which unit test it is 
 inside.
 By making the template a class, you could to walk the RTTI list and 
 make a lookup table to associate test numbers with their names. If you 
 can get the list  of unittest numbers from flectioned, you can join it

 BUT... that's all pretty hairy. Maybe there's a less convoluted way of 
 doing it.


 OK; _that's_ what I was looking for, since the test doesn't need to be 
 run that way... time to get hacking!


Unfortunately, templated class don't show up in the normal class list
accessable 
through moduleinit.d, so it's you'll need to use flectioned for that too.

Here's the simplest example I came up with which displays the module and unit 
test number (taken from my now-defunct NameOf module -- hmmm, might be worth 
rescuscitating a couple of ideas from it; it's much easier now we have CTFE).
It only works if you've defined a local variable to use as an alias, which is a 
bit annoying.

---
template testname(alias F) { enum testname { ignore }  }
template TESTNAME(alias K) {
    typedef void function(testname!(K)) TESTNAME;
}

unittest {
     int foo;
     pragma(msg, TESTNAME!(foo).mangleof[3..$-13]);
}
----------
Dec 01 2007