www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - DMD-0.123 regressions

reply Thomas Kuehne <thomas-dloop kuehne.thisisspam.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

PASS -> FAIL
http://dstress.kuehne.cn/run/line_token_03.d
http://dstress.kuehne.cn/run/line_token_04.d

XFAIL -> ERROR
http://dstress.kuehne.cn/nocompile/bug_constfold_575_A.d
http://dstress.kuehne.cn/nocompile/bug_constfold_575_B.d
http://dstress.kuehne.cn/nocompile/bug_constfold_575_C.d

XFAIL -> XPASS
http://dstress.kuehne.cn/nocompile/bug_constfold_575_J.d
http://dstress.kuehne.cn/nocompile/bug_constfold_575_O.d
http://dstress.kuehne.cn/nocompile/bug_constfold_575_R.d

I'm not sure why, but repeating the following test cases with DMD-0.123 
sometimes results in XFAIL and sometimes ERROR.
http://dstress.kuehne.cn/nocompile/array_initialization_09.d
http://dstress.kuehne.cn/nocompile/array_initialization_11.d

Summary: (370KB)
http://dstress.kuehne.cn/www/dstress.html

Snapshot:
http://developer.berlios.de/project/showfiles.php?group_id=2732

Todo:
https://developer.berlios.de/pm/?group_id=2732

Thomas


-----BEGIN PGP SIGNATURE-----

iD8DBQFChS3O3w+/yD4P9tIRAvnrAKCREXIy4r52gdSNCpkWiRIInEENcACgw1u3
ufGCBBhf1BS/Ygyq2ix/1nc=
=MKpT
-----END PGP SIGNATURE-----
May 13 2005
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Success rate up from 63.1% through 66.3% to 66.4%
Stability rate up from 92.5% through 97.9% to 99.4%

http://smjg.port5.com/pr/d/dstress.xls

I see DStress has grown considerably over the last few versions.  Always 
a good sign.  Now, considering that we're at 0.123 with a 66.4 success 
rate at the moment, can we guess that we'll reach DMD 0.185 before we 
get to 1.0?  :-)

Stewart.

-- 
My e-mail is valid but not my primary mailbox.  Please keep replies on 
the 'group where everyone may benefit.
May 16 2005
next sibling parent reply =?ISO-8859-1?Q?Thomas_K=FChne?= <thomas-dloop kuehne.THISISSPAM.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stewart Gordon wrote:

| Success rate up from 63.1% through 66.3% to 66.4%

The XFAIL and FAIL data in your table seems to be interchanged.

Thomas

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (MingW32)

iD8DBQFCiI7u3w+/yD4P9tIRApICAJ4gh05aNJ7x5AJtR12gvFUj8x/NTgCgz7ei
/OxCOvjXSqT/8C09k+Zaz10=
=sexr
-----END PGP SIGNATURE-----
May 16 2005
next sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Thomas Kühne wrote:

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 Stewart Gordon wrote:
 
 | Success rate up from 63.1% through 66.3% to 66.4%
 
 The XFAIL and FAIL data in your table seems to be interchanged.


I copied and pasted the table as I normally do.  So they must have got 
interchanged on your page (which seems to be down at the mo).

Stewart.

-- 
My e-mail is valid but not my primary mailbox.  Please keep replies on 
the 'group where everyone may benefit.
May 16 2005
parent =?ISO-8859-1?Q?Thomas_K=FChne?= <thomas-dloop kuehne.THISISSPAM.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stewart Gordon wrote:

| Thomas Kühne wrote:
|
|> Stewart Gordon wrote:
|>
|> | Success rate up from 63.1% through 66.3% to 66.4%
|>
|> The XFAIL and FAIL data in your table seems to be interchanged.
|
|
| I copied and pasted the table as I normally do.  So they must have got
| interchanged on your page (which seems to be down at the mo).

Seems like I have to change the provider.

backup:
http://svn.berlios.de/viewcvs/*checkout*/dstress/www/dstress.html

Thomas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (MingW32)

iD8DBQFCiKcx3w+/yD4P9tIRAiunAJ9eLIiVCZsMAco2YOWz4ASf1uRMUgCeMyab
su3bn8UBcudUsGuqg2NAOes=
=clEu
-----END PGP SIGNATURE-----
May 16 2005
prev sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Thomas Kühne wrote:

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 Stewart Gordon wrote:
 

 Success rate up from 63.1% through 66.3% to 66.4%


 
 The XFAIL and FAIL data in your table seems to be interchanged.


Fixed now.

Success rate up from 82.4% through 86.9% to 91.0%.

And is it me, or have the numbers of untested cases with respect to 
version gone rather random?

Stewart.

-- 
My e-mail is valid but not my primary mailbox.  Please keep replies on 
the 'group where everyone may benefit.
May 16 2005
parent =?ISO-8859-1?Q?Thomas_K=FChne?= <thomas-dloop kuehne.THISISSPAM.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stewart Gordon wrote:
| And is it me, or have the numbers of untested cases with respect to
| version gone rather random?

Older DMD versions are tested once in a while, however updating all of
them at once would result in approx. 27*2000 tests  ;)
Maybe I am going to store the test in a SQL db so that only
incremental updates of the changed/new tests are required.

Older GDC versions aren't tested due to some hard coded pathes in GDC
that would have to be fixed before installing several different GDC
versions side by side.

Thomas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (MingW32)

iD8DBQFCiM9l3w+/yD4P9tIRAkWwAKCPQ+MtCr8vGlGVFU024iYehfhaZgCfSdzX
03m/5OFZu8oi5bFtbO3PakU=
=073u
-----END PGP SIGNATURE-----
May 16 2005
prev sibling parent reply "Walter" <newshound digitalmars.com> writes:
"Stewart Gordon" <smjg_1998 yahoo.com> wrote in message
news:d69pkm$1f26$1 digitaldaemon.com...

 Success rate up from 63.1% through 66.3% to 66.4%
 Stability rate up from 92.5% through 97.9% to 99.4%

 http://smjg.port5.com/pr/d/dstress.xls

 I see DStress has grown considerably over the last few versions.  Always
 a good sign.  Now, considering that we're at 0.123 with a 66.4 success
 rate at the moment, can we guess that we'll reach DMD 0.185 before we
 get to 1.0?  :-)


I keep fixing bugs, and Thomas keeps adding more test cases, makes me feel
like I'm on a treadmill <g>. Furthermore, the same problem tends to be
spread across many test cases. This means that some care must be taken in
projecting trends.
May 18 2005
next sibling parent Thomas Kuehne <thomas-dloop kuehne.this-is.spam.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Walter schrieb am Wed, 18 May 2005 15:30:35 -0700:

 "Stewart Gordon" <smjg_1998 yahoo.com> wrote in message
 news:d69pkm$1f26$1 digitaldaemon.com...

 Success rate up from 63.1% through 66.3% to 66.4%
 Stability rate up from 92.5% through 97.9% to 99.4%

 http://smjg.port5.com/pr/d/dstress.xls

 I see DStress has grown considerably over the last few versions.  Always
 a good sign.  Now, considering that we're at 0.123 with a 66.4 success
 rate at the moment, can we guess that we'll reach DMD 0.185 before we
 get to 1.0?  :-)


 I keep fixing bugs, and Thomas keeps adding more test cases, makes me feel
 like I'm on a treadmill <g>. Furthermore, the same problem tends to be
 spread across many test cases. This means that some care must be taken in
 projecting trends.


Exactly. Sometimes one bug report results in some 25+ test cases to cover all
native types and sometimes only 1-3 test cases.

One would have to look at the  uri  and  date  fields in the test cases
in order to get a rough idea of the bug statistics.

Thomas


-----BEGIN PGP SIGNATURE-----

iD8DBQFCi+L33w+/yD4P9tIRAkehAKDJeECxrBL7ij+1p2KiLHztq9yRyQCcCIFt
B1WkkENBZjEAgmdTyftNqSQ=
=oGzw
-----END PGP SIGNATURE-----
May 18 2005
prev sibling parent reply "Uwe Salomon" <post uwesalomon.de> writes:
 I see DStress has grown considerably over the last few versions.  Always
 a good sign.  Now, considering that we're at 0.123 with a 66.4 success
 rate at the moment, can we guess that we'll reach DMD 0.185 before we
 get to 1.0?  :-)


 I keep fixing bugs, and Thomas keeps adding more test cases, makes me  
 feel
 like I'm on a treadmill <g>. Furthermore, the same problem tends to be
 spread across many test cases. This means that some care must be taken in
 projecting trends.


Hmm, the question is: When will you have finished? But i would need the  
statistics of DStress over time to calculate that (bad description... i  
mean the numbers "count of test cases" and "count of successfull test  
cases", but from the past, i.e. "how many test cases did DStress have when  
dmd 0.99 (for example) was released"). This way we could estimate the  
future growth?

Anyways, the earlier compilers already had most of the bugs the current  
DMD has, thus you are definitely making progress. :)  Very good work...

Ciao
uwe
May 18 2005
parent reply Thomas Kuehne <thomas-dloop kuehne.this-is.spam.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Uwe Salomon schrieb am Thu, 19 May 2005 07:30:50 +0200:
<snip>

 Hmm, the question is: When will you have finished? But i would need the  
 statistics of DStress over time to calculate that (bad description... i  
 mean the numbers "count of test cases" and "count of successfull test  
 cases", but from the past, i.e. "how many test cases did DStress have when  
 dmd 0.99 (for example) was released"). This way we could estimate the  
 future growth?


sample:
svn checkout -r "{2005-04-28}" http://dstress.kuehne.cn

Thomas


-----BEGIN PGP SIGNATURE-----

iD8DBQFCjEfA3w+/yD4P9tIRAmHvAKDD/AxXSvlU81M42bqvSpLIBqPGjgCfZ2o+
JKVVjxripp+bdjB22Dqdcys=
=gBHY
-----END PGP SIGNATURE-----
May 19 2005
parent reply "Uwe Salomon" <post uwesalomon.de> writes:
 Hmm, the question is: When will you have finished? But i would need the
 statistics of DStress over time to calculate that (bad description... i
 mean the numbers "count of test cases" and "count of successfull test
 cases", but from the past, i.e. "how many test cases did DStress have  
 when
 dmd 0.99 (for example) was released"). This way we could estimate the
 future growth?


 sample:
 svn checkout -r "{2005-04-28}" http://dstress.kuehne.cn


Ok thanks i'll try to pull something out of it, especially regarding the  
"multiplication" of the test cases. You said one would have to examine the  
URI, because all test cases with the same URI belong to the same bug?

Thanks
uwe
May 18 2005
parent reply Thomas Kuehne <thomas-dloop kuehne.this-is.spam.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Uwe Salomon schrieb am Thu, 19 May 2005 08:16:03 +0200:

 Hmm, the question is: When will you have finished? But i would need the
 statistics of DStress over time to calculate that (bad description... i
 mean the numbers "count of test cases" and "count of successfull test
 cases", but from the past, i.e. "how many test cases did DStress have  
 when
 dmd 0.99 (for example) was released"). This way we could estimate the
 future growth?


 sample:
 svn checkout -r "{2005-04-28}" http://dstress.kuehne.cn


 Ok thanks i'll try to pull something out of it, especially regarding the  
 "multiplication" of the test cases. You said one would have to examine the  
 URI, because all test cases with the same URI belong to the same bug?


All test cases with the same URI belong to the same bug report and often 
describe - or are caused by - 1 to 3 bugs.

Thomas


-----BEGIN PGP SIGNATURE-----

iD8DBQFCjE3m3w+/yD4P9tIRAtQ/AKCityozjS5JuzD4PpI43H+9oU7wNgCfWwQq
PKxLrC9m+3nLGiYhtVh8Or4=
=sKoT
-----END PGP SIGNATURE-----
May 19 2005
parent reply "Uwe Salomon" <post uwesalomon.de> writes:
 Hmm, the question is: When will you have finished? But i would need  
 the
 statistics of DStress over time to calculate that (bad description...  
 i
 mean the numbers "count of test cases" and "count of successfull test
 cases", but from the past, i.e. "how many test cases did DStress have
 when
 dmd 0.99 (for example) was released"). This way we could estimate the
 future growth?


 sample:
 svn checkout -r "{2005-04-28}" http://dstress.kuehne.cn


 Ok thanks i'll try to pull something out of it, especially regarding the
 "multiplication" of the test cases. You said one would have to examine  
 the
 URI, because all test cases with the same URI belong to the same bug?


 All test cases with the same URI belong to the same bug report and often
 describe - or are caused by - 1 to 3 bugs.


Hmm. After some work i made a statistic showing the evolution of  
DStress... The PDF is 200k so i put it on my homepage:

http://www.uwesalomon.de/code/dstresseval/eval.pdf

In the directory there are also the files i used to gather that input (a  
dstresseval.d which reads the input of a grep on the dstress files, all  
commands are in auswerten.sh), just if you are interested... Now to the  
table in the PDF:

- 1st column is the date of the subversion checkout
- 2nd column is the number of *.d files in DStress
- 3rd column is the number of *.d files tested by DStressEval. Only the  
files with a  uri  or  date  could be tested, as you can see there are a  
lot without this designation.
- 4th column is the number of files which were determined to be  
independent. They have a different  uri , or if they did not contain an  
 uri  a different  date/author  (but those are only 6 files, thus that  
won't break the statistic).
- 6th column is an extrapolation of the rate independent/total of the  
DStressEval files to the test cases total. This is somehow vague, because  
only 20-45% of the total test cases could be tested by DStressEval. Not  
sure what to do about that.

The diagram below depicts the number of total and independent testcases.  
As you can see, the independent test cases are stagnating (but note that  
the last column is not a full month yet). And at least for this month the  
total testcases seem to be stagnating as well? Hmm, well, everyone may  
interpret into this what he wants. I think there are a lot of redundant  
test cases that cover different variants of the same bug (or 2/3 bugs).  
Mostly they are all fixed together, still it is very useful to have them  
in DStress.

Ciao
uwe
May 19 2005
next sibling parent "Uwe Salomon" <post uwesalomon.de> writes:
 And at least for this month the total testcases seem to be stagnating as  
 well?


I have to add here that the number of test cases is "stagnating", yes, but  
on a very high niveau. Impressive work, Thomas!

Ciao
uwe
May 19 2005
prev sibling parent reply Thomas Kuehne <thomas-dloop kuehne.THISISSPAM.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Uwe Salomon wrote:
| Hmm. After some work i made a statistic showing the evolution of
| DStress... The PDF is 200k so i put it on my homepage:
|
| http://www.uwesalomon.de/code/dstresseval/eval.pdf
|

Link seems to be broken

Thomas
-----BEGIN PGP SIGNATURE-----

iD8DBQFCjmcc3w+/yD4P9tIRAn3CAKCSMua3qHq7qwcyQycJekXeVcAONACfemMc
36s/g90p1wJfujtw+VOVoxQ=
=bsiN
-----END PGP SIGNATURE-----
May 20 2005
parent "Uwe Salomon" <post uwesalomon.de> writes:
 | Hmm. After some work i made a statistic showing the evolution of
 | DStress... The PDF is 200k so i put it on my homepage:
 |
 | http://www.uwesalomon.de/code/dstresseval/eval.pdf
 |

 Link seems to be broken


Oops sorry, i somehow copied it wrong.

Ciao
uwe
May 20 2005