www.digitalmars.com         C & C++   DMDScript  

D.gnu - Defuzzed: a fuzzer for D compilers

reply Sebastien Alaiwan <ace17 free.fr> writes:
Hi guys,

I just started a D fuzzer. The goal is to randomly generate input 
source files in an attempt to crash the compiler. It's in very 
early stage at this point (less than 300 loc), but it already can 
crash gdc.

https://github.com/Ace17/defuzzed

Please let me know if you think this could be a valuable tool.
( I already know about dsmith (a fork of csmith), but I wasn't 
able to crash neither gdc nor dmd with it. I also aim for a 
simpler tool, and for a one written in D )
Apr 02 2016
next sibling parent "Iain Buclaw via D.gnu" <d.gnu puremagic.com> writes:
On 2 Apr 2016 7:30 pm, "Sebastien Alaiwan via D.gnu" <d.gnu puremagic.com>
wrote:
 Hi guys,

 I just started a D fuzzer. The goal is to randomly generate input source
files in an attempt to crash the compiler. It's in very early stage at this point (less than 300 loc), but it already can crash gdc.
 https://github.com/Ace17/defuzzed

 Please let me know if you think this could be a valuable tool.
 ( I already know about dsmith (a fork of csmith), but I wasn't able to
crash neither gdc nor dmd with it. I also aim for a simpler tool, and for a one written in D )

I expect many bug reports.  :-)
Apr 02 2016
prev sibling next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 2 April 2016 at 17:29:53 UTC, Sebastien Alaiwan 
wrote:
 Hi guys,

 I just started a D fuzzer. The goal is to randomly generate 
 input source files in an attempt to crash the compiler. It's in 
 very early stage at this point (less than 300 loc), but it 
 already can crash gdc.

 https://github.com/Ace17/defuzzed

 Please let me know if you think this could be a valuable tool.
 ( I already know about dsmith (a fork of csmith), but I wasn't 
 able to crash neither gdc nor dmd with it. I also aim for a 
 simpler tool, and for a one written in D )
there is also generated from Brain Schott aka. Hackerpilot. It may be useful to take a peek there.
Apr 03 2016
parent reply Sebastien Alaiwan <ace17 free.fr> writes:
On Sunday, 3 April 2016 at 12:08:32 UTC, Stefan Koch wrote:
 there is also generated from Brain Schott aka. Hackerpilot.
 It may be useful to take a peek there.
Thanks, this definitely is going to be a source of inspiration! (I'm also looking at AFL (American Fuzzy Lop) - lots of great stuff in this tool). Unlike 'generated', I aim to produce semantically valid random programs. I might want to allow some degree of invalidity, as the same error could manifest itself as both 'ice-on-valid' and 'ice-on-invalid', but might be more easily revealed through 'ice-on-invalid'. My long-term goal is the complete eradication of "ice-on-valid" errors from gdc :-) (I plan to keep the tool compiler-agnostic). Your ideas/contributions are welcome!
Apr 03 2016
parent "Brad Roberts via D.gnu" <d.gnu puremagic.com> writes:
On 4/3/2016 5:48 AM, Sebastien Alaiwan via D.gnu wrote:
 On Sunday, 3 April 2016 at 12:08:32 UTC, Stefan Koch wrote:
 there is also generated from Brain Schott aka. Hackerpilot.
 It may be useful to take a peek there.
Thanks, this definitely is going to be a source of inspiration! (I'm also looking at AFL (American Fuzzy Lop) - lots of great stuff in this tool). Unlike 'generated', I aim to produce semantically valid random programs. I might want to allow some degree of invalidity, as the same error could manifest itself as both 'ice-on-valid' and 'ice-on-invalid', but might be more easily revealed through 'ice-on-invalid'. My long-term goal is the complete eradication of "ice-on-valid" errors from gdc :-) (I plan to keep the tool compiler-agnostic). Your ideas/contributions are welcome!
Also, given the shared frontend among dmd, gdc, and ldc, most of the bugs will likely be in that shared code and should result in bugs going to issues.dlang.org. I wouldn't worry too much about valid vs invalid. ICE's in general tend to be fairly easy to fix and have a pretty big negative impact on user experience. So, both are very worth discovering and reporting. Later, Brad
Apr 03 2016
prev sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Saturday, 2 April 2016 at 17:29:53 UTC, Sebastien Alaiwan 
wrote:
 Please let me know if you think this could be a valuable tool.
I am really thrilled to see effort being put towards more testing tools, and I'm sure that most other LDC devs are as well. There are so many opportunities which remain unexplored simply because nobody took the time to actively try and break the current D compilers yet. ICE fuzz-testing like this is a first step, and I'm pretty sure that csmith-style comparison of executable results between the three compilers will yield lots of low-hanging fruit as well. Please do keep the LDC community posted about your progress too! — David
Apr 03 2016
parent Johan Engelen <j j.nl> writes:
On Sunday, 3 April 2016 at 17:05:56 UTC, David Nadlinger wrote:
 On Saturday, 2 April 2016 at 17:29:53 UTC, Sebastien Alaiwan 
 wrote:
 Please let me know if you think this could be a valuable tool.
I am really thrilled to see effort being put towards more testing tools, and I'm sure that most other LDC devs are as well. There are so many opportunities which remain unexplored simply because nobody took the time to actively try and break the current D compilers yet. ICE fuzz-testing like this is a first step, and I'm pretty sure that csmith-style comparison of executable results between the three compilers will yield lots of low-hanging fruit as well. Please do keep the LDC community posted about your progress too!
+1 ! There is a "dsmith" tool out there?! Didn't know about it, that's great news!
Apr 03 2016