www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - Stress Suite codes

reply Derek Parnell <derek psych.ward> writes:
 I still don't understand xpass, fail, xfail etc. What does the 'x' mean?


 
 pass	test case was expected to pass, and it did
 xpass	test case was expected to fail, but passed
 fail	test case was expected to pass, but failed
 xfail	test case was expected to fail, and it did


It is probably too late but could you use something based on this idea ...

goodpass  test case was expected to pass, and it did
badpass   test case was expected to fail, but passed
badfail   test case was expected to pass, but failed
goodfail  test case was expected to fail, and it did


-- 
Derek
Melbourne, Australia
14/02/2005 10:39:07 AM
Feb 13 2005
next sibling parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
I think that's my confusion as well.  To me it looks like this, by the 
way I reckon it:

        pass      fail
no x   goodpass  badfail
x      badpass   goodfail

It would make more sense to me if it was simply:


        pass      fail
no x   goodpass  goodfail
x      badpass   badfail

In other words, pass and fail would mean it did as expected, and xpass 
and xfail would mean it did what it says but shouldn't have (thus the x 
would be a modifier meaning "bad".)

Doesn't really matter, though, and I think the tests are wonderful. 
Thanks for maintaining them, Thomas Kühne.

(let's just hope Thunderbird properly preserves my whitespace...)

-[Unknown]



I still don't understand xpass, fail, xfail etc. What does the 'x' mean?


pass	test case was expected to pass, and it did
xpass	test case was expected to fail, but passed
fail	test case was expected to pass, but failed
xfail	test case was expected to fail, and it did


 
 
 It is probably too late but could you use something based on this idea ...
 
 goodpass  test case was expected to pass, and it did
 badpass   test case was expected to fail, but passed
 badfail   test case was expected to pass, but failed
 goodfail  test case was expected to fail, and it did
Feb 13 2005
parent reply =?ISO-8859-1?Q?Thomas_K=FChne?= <thomas-dloop kuehne.THISISSPAM.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

|>>> I still don't understand xpass, fail, xfail etc. What does the 'x'
|>>> mean?
|>>
|>>
|>> pass    test case was expected to pass, and it did
|>> xpass    test case was expected to fail, but passed
|>> fail    test case was expected to pass, but failed
|>> xfail    test case was expected to fail, and it did
|>
|>
|>
|> It is probably too late but could you use something based on this
|> idea ...
|>
|> goodpass  test case was expected to pass, and it did
|> badpass   test case was expected to fail, but passed
|> badfail   test case was expected to pass, but failed
|> goodfail  test case was expected to fail, and it did

| I think that's my confusion as well.  To me it looks like this, by the
| way I reckon it:
|
|        pass      fail
| no x   goodpass  badfail
| x      badpass   goodfail
|
| It would make more sense to me if it was simply:
|
|
|        pass      fail
| no x   goodpass  goodfail
| x      badpass   badfail
|
| In other words, pass and fail would mean it did as expected, and xpass
| and xfail would mean it did what it says but shouldn't have (thus the
| x would be a modifier meaning "bad".)

I'm not trying to create my own language but rather to use the general
testframe words and meanings.
http://gcc.gnu.org/install/test.html#TOC3
http://www.gnu.org/software/greg/greg.html
http://www.gnu.org/software/dejagnu/dejagnu.html

I agree that "xpass" and "xfail" used as "unexpected pass" and "expected
failure" pose some linguistic difficulties.

In your example above "fail" would be an expected - good - behavior.
That's somehow troublesome too.

| Doesn't really matter, though, and I think the tests are wonderful.
Thanks

Thomas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (MingW32)

iD8DBQFCEAXE3w+/yD4P9tIRAnIoAJ9wzFkXWOcAGqu0rFp5P5oAueG/ZgCeJ9kM
7NpcbjNJKRp2hv76PaaF/Hs=
=+Ot/
-----END PGP SIGNATURE-----
Feb 13 2005
next sibling parent "Regan Heath" <regan netwin.co.nz> writes:
On Mon, 14 Feb 2005 02:58:28 +0100, Thomas Kühne  
<thomas-dloop kuehne.THISISSPAM.cn> wrote:

 I'm not trying to create my own language but rather to use the general
 testframe words and meanings.
 http://gcc.gnu.org/install/test.html#TOC3
 http://www.gnu.org/software/greg/greg.html
 http://www.gnu.org/software/dejagnu/dejagnu.html

 I agree that "xpass" and "xfail" used as "unexpected pass" and "expected
 failure" pose some linguistic difficulties.


It does, not only is one good, and one bad, but the 'x' stands for  
something different in each case.


 In your example above "fail" would be an expected - good - behavior.
 That's somehow troublesome too.


I agree.

Ok, I think the difficulty comes because some tests are expected to  
compile and some aren't, and further, some are also expected to produce  
some sort of result.

I suggest a table like so:

Test Name | Compile? | Result? |
--------------------------------
foo       | yes      | NA      |
bar       | no       | pass    |
baz       | yes      | NA      |
bob       | no       | pass    |

We then colour the compile column entries green if they match the expected  
outcome, and red otherwise.


 | Doesn't really matter, though, and I think the tests are wonderful.
 Thanks


I agree, you're doing a great job.

Regan
Feb 13 2005
prev sibling next sibling parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
Ick.  Well, sorry for assuming you came up with it.... but I suppose if 
it's used for gcc, etc. it's a good idea to use it for D as well.

If you want my personal opinion, though, I wouldn't use the "x" at all. 
  Probably more like "+pass", "+fail", "-pass", "-fail", and "fault".... 
which imho are more logical (+ means good, - means bad) and get the 
added bonus of all being the same length.

-[Unknown]



 I'm not trying to create my own language but rather to use the general
 testframe words and meanings.
 http://gcc.gnu.org/install/test.html#TOC3
 http://www.gnu.org/software/greg/greg.html
 http://www.gnu.org/software/dejagnu/dejagnu.html
Feb 13 2005
parent zwang <nehzgnaw gmail.com> writes:
Unknown W. Brackets wrote:

 Ick.  Well, sorry for assuming you came up with it.... but I suppose if 
 it's used for gcc, etc. it's a good idea to use it for D as well.
 
 If you want my personal opinion, though, I wouldn't use the "x" at all. 
  Probably more like "+pass", "+fail", "-pass", "-fail", and "fault".... 
 which imho are more logical (+ means good, - means bad) and get the 
 added bonus of all being the same length.
 
 -[Unknown]
 
 

 I'm not trying to create my own language but rather to use the general
 testframe words and meanings.
 http://gcc.gnu.org/install/test.html#TOC3
 http://www.gnu.org/software/greg/greg.html
 http://www.gnu.org/software/dejagnu/dejagnu.html




IMO, "fail"/"xfail" are as clear as "+fail"/"-fail", if not clearer.
Anyway, this is trivial. Get used to the codes:)

BTW, I have to say Thomas is doing a really great job.  Kudos!
Feb 13 2005
prev sibling parent reply Dave <Dave_member pathlink.com> writes:
In article <cup0kv$2cjj$1 digitaldaemon.com>, =?ISO-8859-1?Q?Thomas_K=FChne?=
says...


<snip>

| Doesn't really matter, though, and I think the tests are wonderful.
Thanks

Thomas


Yes, they are great Thomas - Thanks!


maybe even a ratio next to the raw number for each result, i.e.:

--------------
687 (51.8%) |
--------------

- Dave



 Feb 14 2005



  parent  =?UTF-8?B?VGhvbWFzIEvDvGhuZQ==?=  writes:


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dave wrote:


| summary and maybe even a ratio next to the raw number for each
| result, i.e.:
|
| --------------
|  687 (51.8%) |
| --------------

Presenting absolute and relative numbers in on cell will blast the table.

Maybe I'll take Graf, GnuPlot or co. to create some diagrams.

Just in case: Please keep in mind that there is no relation like:
n test cases -> m bugs
2m bugs -> 2n test cases

Thomas

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (MingW32)

iD8DBQFCFnQ/3w+/yD4P9tIRAovGAJ4lo6iABenwb38RP3PiINF7y854tQCgkmXR
z9NU86eChOtQgnZ5CBTxHLM=
=ZUTD
-----END PGP SIGNATURE-----



 Feb 18 2005



prev sibling  parent reply Stewart Gordon <smjg_1998 yahoo.com>  writes:


Derek Parnell wrote:

 I still don't understand xpass, fail, xfail etc. What does the 
 'x' mean?


 
 pass test case was expected to pass, and it did xpass test case was 
 expected to fail, but passed fail test case was expected to pass, 
 but failed xfail test case was expected to fail, and it did


 
 It is probably too late but could you use something based on this idea ...
 
 goodpass  test case was expected to pass, and it did
 badpass   test case was expected to fail, but passed
 badfail   test case was expected to pass, but failed
 goodfail  test case was expected to fail, and it did


Guess you're right, but I also guess that those names would make the 
results table too wide.  But I think many of us now know what the 
current names mean - can we really change it now without confusing more 
people?

A few more of my thoughts:

1. What about issues where the spec is unclear about whether something 
should work or not?  Maybe the results could be called "?pass" and 
"?fail".  Or are these classified by some opinion on whether it should 
be valid?

2. Problems with the abstract keyword are somewhat underrepresented in 
DStress:

http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D.bugs/1940

3. Two other things that don't currently seem to fit into DStress, but 
could do with tracking somehow:

(a) issues where code correctly fails to compile, but the error message 
is meaningless or hopelessly inadequate (but doesn't constitute an error 
result)

(b) certain problems that depend on I/O operations, such as

http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D.bugs/2001

Stewart.

-- 
My e-mail is valid but not my primary mailbox.  Please keep replies on 
the 'group where everyone may benefit.



 Feb 14 2005



  parent reply =?ISO-8859-1?Q?Thomas_K=FChne?= <thomas-dloop kuehne.THISISSPAM.cn>  writes:


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stewart Gordon wrote:

| A few more of my thoughts:
|
| 1. What about issues where the spec is unclear about whether
| something should work or not?  Maybe the results could be called
| "?pass" and "?fail".  Or are these classified by some opinion on
| whether it should be valid?

Example?

| 2. Problems with the abstract keyword are somewhat underrepresented
| in DStress:
|
| http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D.bugs/1940

I'll have a look.

| 3. Two other things that don't currently seem to fit into DStress,
| but could do with tracking somehow:
|
| (a) issues where code correctly fails to compile, but the error
| message is meaningless or hopelessly inadequate (but doesn't
| constitute an error result)

That's a tricky one. Inadequate error messages could be black listed for
a specific test case, but I'm not sure how to properly white list error
messages.

I plan to add a check for correct line numbers in the error messages.
This might detect some of the botched error messages.

| (b) certain problems that depend on I/O operations, such as
|
| http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D.bugs/2001

This isn't a bug, it is an unconventional style.




The documentation simply states "displays" without any limitations on
the howto, could be a pop-up, stderr, stdout, the keyboard leds...

A simple fix.
1) Open internal/dmain2.d

2) Change from:



3) Change to:


You'll notice that SomeError.toString()!=SomeError.print() arg...

Thomas


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (MingW32)

iD8DBQFCFnIp3w+/yD4P9tIRAsyOAJ9bNOAQfuFaVLdbpn4v4FmCt3BwjACfSx4M
LF5Me9fgTA5e1np+bl0+LjI=
=WArs
-----END PGP SIGNATURE-----



 Feb 18 2005



  parent  Stewart Gordon <smjg_1998 yahoo.com>  writes:


Thomas Kühne wrote:

 Stewart Gordon wrote:
 

 A few more of my thoughts:
 
 1. What about issues where the spec is unclear about whether 
 something should work or not?  Maybe the results could be called 
 "?pass" and "?fail".  Or are these classified by some opinion on 
 whether it should be valid?


 
 Example?


Any test of the layout of bits within a struct.  Theoretically,
anything related to an undocumented language feature (such as the
abstract keyword, though deciding how this should work is fairly
straightforward).  No doubt others.

<snip>

 I plan to add a check for correct line numbers in the error 
 messages. This might detect some of the botched error messages.


For that matter, should error messages with no line number at all be
considered an error result, considering that such things aren't
supposed to be part of DMD?

<snip>



 
 The documentation simply states "displays" without any limitations 
 on the howto, could be a pop-up, stderr, stdout, the keyboard 
 leds...


And as I said, the documentation states that the program "gracefully
exits".


 A simple fix.
 1) Open internal/dmain2.d
 
 2) Change from:


 
 3) Change to:



You mean

     fprintf(stderr,"Error: %.*s\n", o.toString());

?


 You'll notice that SomeError.toString()!=SomeError.print() arg...


Indeed.  Is Object.print supposed to be pure syntactic sugar, or is
there some other hidden motive for its existence?

Stewart.

-- 
My e-mail is valid but not my primary mailbox.  Please keep replies on 
the 'group where everyone may benefit.



 Feb 21 2005