www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - DMD-0.122 regressions

reply Thomas Kuehne <thomas-dloop kuehne.thisisspam.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

PASS -> ERROR
http://dstress.kuehne.cn/compile/interface_20.d
http://dstress.kuehne.cn/compile/interface_21.d

XFAIL -> ERROR
http://dstress.kuehne.cn/nocompile/alias_17.d
http://dstress.kuehne.cn/nocompile/bug_dsymbol_611_A.d
http://dstress.kuehne.cn/nocompile/bug_dsymbol_611_D.d
http://dstress.kuehne.cn/nocompile/bug_mtype_507_A.d
http://dstress.kuehne.cn/nocompile/bug_mtype_507_C.d
http://dstress.kuehne.cn/nocompile/bug_mtype_507_D.d
http://dstress.kuehne.cn/nocompile/overload_12.d
http://dstress.kuehne.cn/nocompile/overload_14.d
http://dstress.kuehne.cn/nocompile/overload_16.d
http://dstress.kuehne.cn/nocompile/super_06.d

unexpected results:
http://dstress.kuehne.cn/www/dstress.html

all resuts: (~1.9MB)
http://dstress.kuehne.cn/www/results.html

todo:
https://developer.berlios.de/pm/?group_id=2732

Thomas


-----BEGIN PGP SIGNATURE-----

iD4DBQFCe53d3w+/yD4P9tIRAszsAJj0322RKCmhAsAgr5AAfGccYrvsAJ0WlxhX
rNMNW83c/oCoJxZRldfUQg==
=pMac
-----END PGP SIGNATURE-----
May 06 2005
parent reply "Lionello Lunesu" <lio lunesu.removethis.com> writes:
Thomas,

I love watching your dstress site. I wish we had something like that for the 
software I'm working on.

But, there's something I don't understand.

A test case should either pass or fail. So any test case should only have 
results { pass, fail, error } or only {xpass, xfail, error }.

But some tests have "xpass" for some compiler and "fail" for another? Is it 
expected to fail or expected to pass? Did the definition change perhaps? If 
so, wouldn't it be better to erase old results (since they are meaningless).

For example:

 debug info 01 : xfail for 0.122, but has fail for 0.121 and again xpass for 
0.119? Did the definition change twice??
 private 05 : both xpass and fail?
 try 05 : xfail and fail
 union 11 : xfail and fail

etc..

Lionello.
May 11 2005
parent reply Thomas Kuehne <thomas-dloop kuehne.thisisspam.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Lionello Lunesu schrieb am Wed, 11 May 2005 18:49:10 +0300:

 Thomas,

 I love watching your dstress site. I wish we had something like that for the 
 software I'm working on.


How about hireing me? :) 


 But, there's something I don't understand.

 A test case should either pass or fail. So any test case should only have 
 results { pass, fail, error } or only {xpass, xfail, error }.

 But some tests have "xpass" for some compiler and "fail" for another? Is it 
 expected to fail or expected to pass?


Some of the test cases are expected to fail and report the specific source
file/line location.

XFAIL: failed and reported correct source location
FAIL: failed, but reported no/incorrect source location


 Did the definition change perhaps? If 
 so, wouldn't it be better to erase old results (since they are meaningless).


The current test framework ensures that only the current definitions are
reported - even if results for the old definitions are still present in
the raw data.

Thomas


-----BEGIN PGP SIGNATURE-----

iD8DBQFCgk+03w+/yD4P9tIRAm8WAJ9VAYgOuHIO1Q4Lm1kPW9TGP9NmzwCfSXts
EZN28fJ5arNYSqaCybLIE9Q=
=owkx
-----END PGP SIGNATURE-----
May 11 2005
next sibling parent reply "Lionello Lunesu" <lio lunesu.removethis.com> writes:
Hi,


 I love watching your dstress site. I wish we had something like that for 
 the
 software I'm working on.


 How about hireing me? :)


We could use some automated test framework to report regressions. Too many 
times new fixes break the software in some rare cases..

Unfortunately, I'm not the one that hires people :-/ (Besides, we've stopped 
hiring people for some time now.. Waiting for the cash flow to pick up)


 But some tests have "xpass" for some compiler and "fail" for another? Is 
 it
 expected to fail or expected to pass?


 Some of the test cases are expected to fail and report the specific source
 file/line location.

 XFAIL: failed and reported correct source location
 FAIL: failed, but reported no/incorrect source location


So there's no difference between "expected to pass but failed" (FAIL) and 
"expected to fail but wrong error" (FAIL). Maybe the latter should be 
reported as ERROR, or a seperate category?


 Did the definition change perhaps? If
 so, wouldn't it be better to erase old results (since they are 
 meaningless).


 The current test framework ensures that only the current definitions are
 reported - even if results for the old definitions are still present in
 the raw data.


Does this mean that old definitions don't appear in the table? Or do you 
adapt the entire table (history) to the new definition?

Lionello.
May 12 2005
parent Thomas Kuehne <thomas-dloop kuehne.thisisspam.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lionello Lunesu schrieb am Thu, 12 May 2005 12:36:01 +0300:

 Did the definition change perhaps? If
 so, wouldn't it be better to erase old results (since they are 
 meaningless).


 The current test framework ensures that only the current definitions are
 reported - even if results for the old definitions are still present in
 the raw data.


 Does this mean that old definitions don't appear in the table? Or do you 
 adapt the entire table (history) to the new definition?


They are listed in the table as "untested / outdated".

Thomas


-----BEGIN PGP SIGNATURE-----

iD8DBQFChTiS3w+/yD4P9tIRAmYtAJwKAjmaOP82P06V+oEtnP8cJLFWTQCeMsTO
oYB76T3Fwh7gr3BDih8Ocro=
=7ouI
-----END PGP SIGNATURE-----
May 13 2005
prev sibling parent reply "Lionello Lunesu" <lio lunesu.removethis.com> writes:
Ah, wait, I think I got it :-)

It's the 'overlined' "fail" that's used for wrong/missing information.

Does this mean that tests that are expected to fail (xfail/xpass) should 
only have the 'overlined' fail (in case the error information is 
wrong/missing) ?

A quick FART gives me 31 tests that have the normal fail together with 
xpass/xfail. It seems only for GDC 0.10 and GDC 0.10 Mac. (I can send you 
the html with only these cases, if you want)

Please know that I'm just doing this for my understanding of this wonderful 
test suite :-)

Lionello.
May 12 2005
parent Thomas Kuehne <thomas-dloop kuehne.thisisspam.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lionello Lunesu schrieb am Thu, 12 May 2005 13:09:02 +0300:

 Ah, wait, I think I got it :-)

 It's the 'overlined' "fail" that's used for wrong/missing information.

 Does this mean that tests that are expected to fail (xfail/xpass) should 
 only have the 'overlined' fail (in case the error information is 
 wrong/missing) ?


yes


 A quick FART gives me 31 tests that have the normal fail together with 
 xpass/xfail. It seems only for GDC 0.10 and GDC 0.10 Mac. 


GDC-0.10 was tested with the current logic but didn't yet provide the
"overline" marker to the table generator.

Thomas


-----BEGIN PGP SIGNATURE-----

iD8DBQFChTgz3w+/yD4P9tIRAtylAKCUy5/agnZ6dFHnf6ZPmjVcRWgV6gCfWeV1
RiAMkO/hda5ZS1hxRrOzB+Q=
=TYNC
-----END PGP SIGNATURE-----
May 13 2005