www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Article: Fuzzing D code with LDC

reply Johan Engelen <j j.nl> writes:
It's been a work-in-progress for half a year, but finished now:

http://johanengelen.github.io/ldc/2018/01/14/Fuzzing-with-LDC.html

"A not-so-well-written article about the fuzzing capability 
recently added to LDC, using LLVM’s libFuzzer. Compiling code 
with -fsanitize=fuzzer adds control-flow instrumentation used to 
guide the fuzzing and links-in the libFuzzer library that drives 
the fuzz testing (same as Clang). -fsanitize=fuzzer is available 
from LDC 1.4.0, not on Windows. LDC 1.6.0 was used for the 
examples in this article."

Hope the article gives you enough information to start fuzz 
testing your own projects.

cheers,
   Johan

(per Ali's request, the article has a nice fuzzy font for some of 
you :P)
Jan 13 2018
next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Saturday, 13 January 2018 at 23:59:52 UTC, Johan Engelen wrote:

 It's been a work-in-progress for half a year, but finished now:

 http://johanengelen.github.io/ldc/2018/01/14/Fuzzing-with-LDC.html

 "A not-so-well-written article about the fuzzing capability 
 recently added to LDC, using LLVM’s libFuzzer. Compiling code 
 with -fsanitize=fuzzer adds control-flow instrumentation used 
 to guide the fuzzing and links-in the libFuzzer library that 
 drives the fuzz testing (same as Clang). -fsanitize=fuzzer is 
 available from LDC 1.4.0, not on Windows. LDC 1.6.0 was used 
 for the examples in this article."

 Hope the article gives you enough information to start fuzz 
 testing your own projects.

 cheers,
   Johan

 (per Ali's request, the article has a nice fuzzy font for some 
 of you :P)


Nice post, enjoyed reading it.  I need to look into trying out 
the shift sanitizer sometime.
Jan 14 2018
parent Johan Engelen <j j.nl> writes:
On Sunday, 14 January 2018 at 17:33:45 UTC, Joakim wrote:

 Nice post, enjoyed reading it.  I need to look into trying out 
 the shift sanitizer sometime.


Definitely would be *great* to have `-fsanitize=undefined` !

- Johan
Jan 14 2018
prev sibling next sibling parent Johan Engelen <j j.nl> writes:
On Saturday, 13 January 2018 at 23:59:52 UTC, Johan Engelen wrote:

 It's been a work-in-progress for half a year, but finished now:

 http://johanengelen.github.io/ldc/2018/01/14/Fuzzing-with-LDC.html

 "A not-so-well-written article about the fuzzing capability 
 recently added to LDC


Argh, and just now I find out about:
- https://www.youtube.com/watch?v=k-Cv8Q3zWNQ
- https://github.com/google/oss-fuzz

Anybody interested in looking at OSS Fuzz?
"OSS-Fuzz - Continuous Fuzzing for Open Source Software
Status: Beta. We are now accepting applications from widely-used 
open source projects."

Cheers,
   Johan
Jan 14 2018
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/13/2018 3:59 PM, Johan Engelen wrote:

 It's been a work-in-progress for half a year, but finished now:



It's now on Reddit and Hackernews:

   https://www.reddit.com/r/programming/comments/7qbpa3/fuzzing_d_code_with_ldc/

   https://news.ycombinator.com/item?id=16144860

(Don't click on the specific ycombinator link, or your upvotes will not be 
counted. Click on:

   https://news.ycombinator.com/news

instead and look for it.)

Also, such postings do a LOT better when they:

1. Include a synopsis of what the article is about. (I've added them.)

2. Include an "Ask me anything" posting from the author.

Not doing these things means the impact of the article is severely blunted from 
what it could be.
Jan 14 2018
prev sibling next sibling parent reply Temtaime <temtaime gmail.com> writes:
On Saturday, 13 January 2018 at 23:59:52 UTC, Johan Engelen wrote:

 It's been a work-in-progress for half a year, but finished now:

 http://johanengelen.github.io/ldc/2018/01/14/Fuzzing-with-LDC.html

 "A not-so-well-written article about the fuzzing capability 
 recently added to LDC, using LLVM’s libFuzzer. Compiling code 
 with -fsanitize=fuzzer adds control-flow instrumentation used 
 to guide the fuzzing and links-in the libFuzzer library that 
 drives the fuzz testing (same as Clang). -fsanitize=fuzzer is 
 available from LDC 1.4.0, not on Windows. LDC 1.6.0 was used 
 for the examples in this article."

 Hope the article gives you enough information to start fuzz 
 testing your own projects.

 cheers,
   Johan

 (per Ali's request, the article has a nice fuzzy font for some 
 of you :P)


// `Throwable`s thrown are not bugs (in contrast to `Errors`).

They _can_ be bugs.
class Error : Throwable
class Exception : Throwable
Jan 15 2018
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, January 15, 2018 09:49:24 Temtaime via Digitalmars-d-announce 
wrote:

 On Saturday, 13 January 2018 at 23:59:52 UTC, Johan Engelen wrote:

 It's been a work-in-progress for half a year, but finished now:

 http://johanengelen.github.io/ldc/2018/01/14/Fuzzing-with-LDC.html

 "A not-so-well-written article about the fuzzing capability
 recently added to LDC, using LLVM’s libFuzzer. Compiling code
 with -fsanitize=fuzzer adds control-flow instrumentation used
 to guide the fuzzing and links-in the libFuzzer library that
 drives the fuzz testing (same as Clang). -fsanitize=fuzzer is
 available from LDC 1.4.0, not on Windows. LDC 1.6.0 was used
 for the examples in this article."

 Hope the article gives you enough information to start fuzz
 testing your own projects.

 cheers,

   Johan

 (per Ali's request, the article has a nice fuzzy font for some
 of you :P)


 // `Throwable`s thrown are not bugs (in contrast to `Errors`).

 They _can_ be bugs.
 class Error : Throwable
 class Exception : Throwable


Talking about Throwables always gets a bit weird. Exceptions aren't bugs,
whereas Errors are, and we usually talk about Exceptions and Errors rather
than Throwables, and talking about Throwables implies both, but you _can_
also throw Throwables that aren't derived from Exception or Error (be it a
naked Throwable or a class derived from Throwable). You're asking for it if
you do, since there really isn't much difference between a Throwable and an
Error in terms of how the compiler and runtime deal with clean-up code and
nothrow, so trying to use a Throwable that isn't an Exception or Error can
work, but it's basically going to be treated like an Error. Really, as far
as the language and compiler are concerned, you have Exceptions, and then
you have all other Throwables. nothrow has to do with Exceptions, and
whether clean-up code is guaranteed to run or not has to do with Exceptions.
Error really doesn't come into it except that we derive from Error rather
than Throwable to more easily distinguish them from Exception. Ultimately,
Errors and other non-Exception Throwables are really treated the same.

So, all-in-all, things just get weird when you try and talk about
Throwables, and I'd generally advise against anyone trying to use any
Throwables that aren't Exceptions or Errors. Certainly, if you do, you had
better know what you're doing and be _very_ careful about it.

- Jonathan M Davis
Jan 15 2018
prev sibling parent Johan Engelen <j j.nl> writes:
On Monday, 15 January 2018 at 09:49:24 UTC, Temtaime wrote:

 // `Throwable`s thrown are not bugs (in contrast to `Errors`).

 They _can_ be bugs.
 class Error : Throwable
 class Exception : Throwable


Good catch ( ;-) ) thanks.

-Johan
Jan 15 2018
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Jan 13, 2018 at 11:59:52PM +0000, Johan Engelen via
Digitalmars-d-announce wrote:

 It's been a work-in-progress for half a year, but finished now:
 
 http://johanengelen.github.io/ldc/2018/01/14/Fuzzing-with-LDC.html


[...]

It would be good to program custom fuzzers for Phobos modules,
especially the ones with generic code like std.algorithm and std.range,
a lot of which I suspect have unhandled edge cases and other such
overlooked flaws.  Fuzzing at this level will be harder than merely
fuzzing with random data; as input we'd need higher-level objects like
ranges of various types and by-reference / by-value semantics.  And
detecting problem cases may not be so easy (how to detect if, say,
reduce() returned the wrong answer when the input is randomized?).


T

-- 
Beware of bugs in the above code; I have only proved it correct, not tried it.
-- Donald Knuth
Jan 15 2018