www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Csmith - C compiler stress test

reply Robert Clipsham <robert octarineparrot.com> writes:
I've just come across Csmith:

http://embed.cs.utah.edu/csmith/

The tool randomly generates valid C99 code to stress test C compilers 
and has found 350+ bugs in GCC/LLVM alone over the past couple of years. 
It might be interesting to run dmc through this, and maybe adapt the 
tool to generate D test cases?

-- 
Robert
http://octarineparrot.com/
Apr 13 2011
next sibling parent reply Trass3r <un known.com> writes:
Am 13.04.2011, 16:13 Uhr, schrieb Robert Clipsham  =

<robert octarineparrot.com>:

 I've just come across Csmith:

 http://embed.cs.utah.edu/csmith/

 The tool randomly generates valid C99 code to stress test C compilers =


 =


 and has found 350+ bugs in GCC/LLVM alone over the past couple of year=


s.  =


 It might be interesting to run dmc through this, and maybe adapt the  =



 tool to generate D test cases?


I think I also came across such a tool but I'm not sure if it was this o=
ne.
Since it uses a grammar (subset) to generate the programs it should be  =

perfectly possible to adapt it for D.
Though it "is mainly intended to =EF=AC=81nd bugs in the parts of a comp=
iler that  =

perform transformations on an intermediate representation".
Apr 13 2011
parent reply David Nadlinger <see klickverbot.at> writes:
On 4/13/11 5:16 PM, Trass3r wrote:

 I think I also came across such a tool but I'm not sure if it was this one.
 Since it uses a grammar (subset) to generate the programs it should be
 perfectly possible to adapt it for D.
 Though it "is mainly intended to find bugs in the parts of a compiler
 that perform transformations on an intermediate representation".


 From what I recall from a quick look at the Csmith sources and Xuejun 
Yang's talk at the LLVM developer meeting, Csmith is tailored rather 
specifically to C, and isn't based on a grammar representation that 
could be easily swapped out…

David
Apr 13 2011
parent Emil Madsen <sovende gmail.com> writes:
On 13 April 2011 17:27, David Nadlinger <see klickverbot.at> wrote:


 On 4/13/11 5:16 PM, Trass3r wrote:


 I think I also came across such a tool but I'm not sure if it was this
 one.
 Since it uses a grammar (subset) to generate the programs it should be
 perfectly possible to adapt it for D.
 Though it "is mainly intended to =EF=AC=81nd bugs in the parts of a comp=




iler

 that perform transformations on an intermediate representation".


 From what I recall from a quick look at the Csmith sources and Xuejun
 Yang's talk at the LLVM developer meeting, Csmith is tailored rather
 specifically to C, and isn't based on a grammar representation that could=


 be

 easily swapped out=E2=80=A6

 David


*Can Csmith be altered to emit programs in a language other than C or C++?*=
 Not
easily.
http://embed.cs.utah.edu/csmith/using.html

--=20
// Yours sincerely
// Emil 'Skeen' Madsen
Apr 13 2011
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/13/2011 7:13 AM, Robert Clipsham wrote:

 I've just come across Csmith:

 http://embed.cs.utah.edu/csmith/

 The tool randomly generates valid C99 code to stress test C compilers and has
 found 350+ bugs in GCC/LLVM alone over the past couple of years. It might be
 interesting to run dmc through this, and maybe adapt the tool to generate D
test
 cases?


I talked to one of the others, he didn't think it would be that hard to adapt
it 
to D, but we'd have to do it ourselves.

I think this could be fairly valuable for D compilers.

Anyone looking for a project?
Apr 13 2011
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 13 Apr 2011 14:45:17 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:


 On 4/13/2011 7:13 AM, Robert Clipsham wrote:

 I've just come across Csmith:

 http://embed.cs.utah.edu/csmith/

 The tool randomly generates valid C99 code to stress test C compilers  
 and has
 found 350+ bugs in GCC/LLVM alone over the past couple of years. It  
 might be
 interesting to run dmc through this, and maybe adapt the tool to  
 generate D test
 cases?


 I talked to one of the others, he didn't think it would be that hard to  
 adapt it to D, but we'd have to do it ourselves.

 I think this could be fairly valuable for D compilers.

 Anyone looking for a project?


As much as this might be a fun project, and might find bugs for D, I have  
to point out that there are *plenty* of bugs in dmd that need fixing.  Any  
would be takers for this project should also consider lending a hand at  
fixing existing bugs.  Then again, fixing dmd bugs might be less fun, so I  
could understand if someone preferred not to.

I think this tool would be great once dmd is fully functional.

-Steve
Apr 13 2011
next sibling parent Daniel Gibson <metalcaedes gmail.com> writes:
Am 13.04.2011 20:57, schrieb Steven Schveighoffer:

 On Wed, 13 Apr 2011 14:45:17 -0400, Walter Bright
 <newshound2 digitalmars.com> wrote:
 

 On 4/13/2011 7:13 AM, Robert Clipsham wrote:

 I've just come across Csmith:

 http://embed.cs.utah.edu/csmith/

 The tool randomly generates valid C99 code to stress test C compilers
 and has
 found 350+ bugs in GCC/LLVM alone over the past couple of years. It
 might be
 interesting to run dmc through this, and maybe adapt the tool to
 generate D test
 cases?


 I talked to one of the others, he didn't think it would be that hard
 to adapt it to D, but we'd have to do it ourselves.

 I think this could be fairly valuable for D compilers.

 Anyone looking for a project?


 
 As much as this might be a fun project, and might find bugs for D, I
 have to point out that there are *plenty* of bugs in dmd that need
 fixing.  Any would be takers for this project should also consider
 lending a hand at fixing existing bugs.  Then again, fixing dmd bugs
 might be less fun, so I could understand if someone preferred not to.
 
 I think this tool would be great once dmd is fully functional.
 
 -Steve


As developing it probably needs some time anyway it may make sense to
start developing it before dmd is mostly bugfree.

Cheers,
- Daniel
Apr 13 2011
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/13/2011 11:57 AM, Steven Schveighoffer wrote:

 As much as this might be a fun project, and might find bugs for D, I have to
 point out that there are *plenty* of bugs in dmd that need fixing. Any would be
 takers for this project should also consider lending a hand at fixing existing
 bugs. Then again, fixing dmd bugs might be less fun, so I could understand if
 someone preferred not to.

 I think this tool would be great once dmd is fully functional.


I think it's a mistake to avoid looking for bugs since we already have bugs to 
fix. We have bugzilla to keep track of them, and sure it might take time to fix 
them, but at least they will be there.

Also, many people submit bug fix patches. Different people are interested in 
different kinds of bugs, and having some integer math bugs (which is what
csmith 
finds) may appeal to a particular fixer.
Apr 13 2011
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 13 Apr 2011 16:00:46 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:


 On 4/13/2011 11:57 AM, Steven Schveighoffer wrote:

 As much as this might be a fun project, and might find bugs for D, I  
 have to
 point out that there are *plenty* of bugs in dmd that need fixing. Any  
 would be
 takers for this project should also consider lending a hand at fixing  
 existing
 bugs. Then again, fixing dmd bugs might be less fun, so I could  
 understand if
 someone preferred not to.

 I think this tool would be great once dmd is fully functional.


 I think it's a mistake to avoid looking for bugs since we already have  
 bugs to fix. We have bugzilla to keep track of them, and sure it might  
 take time to fix them, but at least they will be there.


I'm not saying, don't write a program to find bugs.  I'm saying, if you  
want to help D, a better option than trying to find bugs is to fix the  
bugs we have.  If all you want to do is find bugs, then I don't have any  
issue with that, and welcome the additional tools.  But we have a wealth  
of bug finders, and few fixers.  Someone who is good enough to write a  
D-code-generating program might have the same skillset as it takes to work  
on the compiler.  I'd love for that person to spend time fixing bugs  
rather than building a tool to find them :)

In other words, D doesn't *need* more bugs (though we will happily accept  
more bugs in bugzilla), it needs more fixes.

I didn't mean to sound like I don't think we should promote such a tool or  
that I don't like someone working on it.  I just want to also bring up  
that we need people to fix bugs too, and if you are talented enough and  
like doing either, I think it would be more appreciated by the community  
if bugs were fixed rather than generated.


 Also, many people submit bug fix patches. Different people are  
 interested in different kinds of bugs, and having some integer math bugs  
 (which is what csmith finds) may appeal to a particular fixer.


I hope that a csmith-like tool will be written and that it helps more  
people enjoy fixing bugs in dmd.  I hope *more* that the existing const,  
inout, alias this, template overload,  property, etc. bugs are fixed.

-Steve
Apr 13 2011
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/13/2011 1:22 PM, Steven Schveighoffer wrote:

 I'm not saying, don't write a program to find bugs. I'm saying, if you want to
 help D, a better option than trying to find bugs is to fix the bugs we have.


Often, people come to me with "what can I do for D?" I give a list of 
suggestions, and never once has the person done any of them.

I can't tell people what to work on.

People work on what they want to. They're self-starters and are self-motivated. 
If someone finds adapting Csmith to D a fun project, I'm going to encourage
that :-)
Apr 13 2011
prev sibling next sibling parent Iain Buclaw <ibuclaw ubuntu.com> writes:
== Quote from Walter Bright (newshound2 digitalmars.com)'s article

 On 4/13/2011 7:13 AM, Robert Clipsham wrote:

 I've just come across Csmith:

 http://embed.cs.utah.edu/csmith/

 The tool randomly generates valid C99 code to stress test C compilers and has
 found 350+ bugs in GCC/LLVM alone over the past couple of years. It might be
 interesting to run dmc through this, and maybe adapt the tool to generate D
test
 cases?


 I talked to one of the others, he didn't think it would be that hard to adapt
it
 to D, but we'd have to do it ourselves.
 I think this could be fairly valuable for D compilers.
 Anyone looking for a project?


As coincidences goes, I was considering about writing a fuzz testing tool for D
just the other day (whilst in the middle of brooding over several bugs found in
GDC that were not covered in the D2 testsuite).

Will bookmark this for later I think. ;)
Apr 13 2011
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/13/2011 11:45 AM, Walter Bright wrote:

 I talked to one of the others, he didn't think it would be that hard to adapt
it
 to D, but we'd have to do it ourselves.


s/others/authors/
Apr 13 2011