www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - if(false) vs if(isNaN) benchmark

reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
Hi all,

A trivial little benchmark for comparing the performance of a function using
if(!booleanValue) as a conditional, versus one using if(isNan(x)).

Run with dmd -O -release -inline -noboundscheck on my system, I find the two are
virtually equivalent, with it taking about 30ms for 1 million if(!boolean)
versus about 31 for a million if(isNaN(floatingpoint)).

With ldmd2 and the same optimizations, I get about 25 compared to 27-28 ms for
the two cases, again with a million realizations of each.

Finally, with gdmd I get 30ms for a million if(!boolean) versus about 150ms for
a million if(isNaN(floatingpoint)).

Code attached, feel free to critique.  Generally I think these results show that
the difference between checking a boolean and checking isNan are negligible, but
I'm surprised at the relatively bad performance with gdmd and I wonder if
there's some particular reason for this.

The relevance is to a fix I'm preparing for this bug:
http://d.puremagic.com/issues/show_bug.cgi?id=10322

See also: http://forum.dlang.org/post/lcrwybqszclitzravrqo forum.dlang.org

Thanks in advance for any feedback :-)

Best wishes,

    -- Joe
Jul 04 2013
parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 4 July 2013 at 12:53:48 UTC, Joseph Rushton Wakeling 
wrote:

 Hi all,

 A trivial little benchmark for comparing the performance of a 
 function using
 if(!booleanValue) as a conditional, versus one using 
 if(isNan(x)).

 Run with dmd -O -release -inline -noboundscheck on my system, I 
 find the two are
 virtually equivalent, with it taking about 30ms for 1 million 
 if(!boolean)
 versus about 31 for a million if(isNaN(floatingpoint)).

 With ldmd2 and the same optimizations, I get about 25 compared 
 to 27-28 ms for
 the two cases, again with a million realizations of each.

 Finally, with gdmd I get 30ms for a million if(!boolean) versus 
 about 150ms for
 a million if(isNaN(floatingpoint)).

 Code attached, feel free to critique.  Generally I think these 
 results show that
 the difference between checking a boolean and checking isNan 
 are negligible, but
 I'm surprised at the relatively bad performance with gdmd and I 
 wonder if
 there's some particular reason for this.

 The relevance is to a fix I'm preparing for this bug:
 http://d.puremagic.com/issues/show_bug.cgi?id=10322

 See also: 
 http://forum.dlang.org/post/lcrwybqszclitzravrqo forum.dlang.org

 Thanks in advance for any feedback :-)

 Best wishes,

     -- Joe


Me thinks the test is biased with answering true to isNan. You 
should first initialize your arrays with random [true/false] | 
[nan/nonan] value. In particular, it is more important for isNan 
to answer "No" as fast as possible, rather than actually find 
nan's. (common, use case is checking that a number is *not*) nan. 
I'd say a mix of 90% rands + 10% Nan's would be representative?

Also, there might be optimizations in bool iteration over double 
iteration given their size, as well as the fact that the "first" 
test is typically faster.

I think you should create an array of S{bool; double;} So that 
the iteration is not biased either.

I'd add a dummy "warmup" loop of 10% of the iterations too, just 
'cause.

Gotta go.
Jul 04 2013
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 07/04/2013 03:05 PM, monarch_dodra wrote:

 Me thinks the test is biased with answering true to isNan. You should first
 initialize your arrays with random [true/false] | [nan/nonan] value. In
 particular, it is more important for isNan to answer "No" as fast as possible,
 rather than actually find nan's. (common, use case is checking that a number is
 *not*) nan. I'd say a mix of 90% rands + 10% Nan's would be representative?


Quite right -- the case of isNaN being false is the case that needs to be really
quick.


 Also, there might be optimizations in bool iteration over double iteration
given
 their size, as well as the fact that the "first" test is typically faster.


I'd assumed that putting the start/stop of the stopwatch inside the loop would
address that, but apparently not ...


 I think you should create an array of S{bool; double;} So that the iteration is
 not biased either.
 
 I'd add a dummy "warmup" loop of 10% of the iterations too, just 'cause.


See attached.  I haven't put in the dummy warmup loop, but as is, there's
consistently less than 2ms difference between the times reported for all
compilers.  If I use rdmd without optimizations, the difference is slightly
larger at 3-5ms.

So, on the basis of that I think I feel happy about using isNaN as a test,
especially as it lets me remove an otherwise superfluous variable from
RandomSample.  Thanks for the advice! :-)
Jul 04 2013
parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 4 July 2013 at 13:43:13 UTC, Joseph Rushton Wakeling 
wrote:

 On 07/04/2013 03:05 PM, monarch_dodra wrote:

 Me thinks the test is biased with answering true to isNan. You 
 should first
 initialize your arrays with random [true/false] | [nan/nonan] 
 value. In
 particular, it is more important for isNan to answer "No" as 
 fast as possible,
 rather than actually find nan's. (common, use case is checking 
 that a number is
 *not*) nan. I'd say a mix of 90% rands + 10% Nan's would be 
 representative?


 Quite right -- the case of isNaN being false is the case that 
 needs to be really
 quick.


 Also, there might be optimizations in bool iteration over 
 double iteration given
 their size, as well as the fact that the "first" test is 
 typically faster.


 I'd assumed that putting the start/stop of the stopwatch inside 
 the loop would
 address that, but apparently not ...


Holly crap, I didn't see that. Don't do that. The problem if you 
do that is that on a lot of machines, the time it takes to do 1 
iteration is shorter than the granularity of the clock. For 
example, on windows, it is typically 15 ms, which is HUGE! I 
think *nix is more precise, but not *that* precise either. If you 
are unlucky, the system clock may increment between iterations, 
meaning you'll miss out on that time.

In particular, if your tests last 30ms, it can make a big 
difference. I usually try to make sure my tests last at least a 
whole second. I also try to add an entire second of warm up, the 
time it takes for the OS to get its caching done.

I also try to run the tests in such a way that each case is 
tested several times alternatively. Sometimes, you'd be surprised 
with the result.


 I think you should create an array of S{bool; double;} So that 
 the iteration is
 not biased either.
 
 I'd add a dummy "warmup" loop of 10% of the iterations too, 
 just 'cause.


 See attached.  I haven't put in the dummy warmup loop, but as 
 is, there's
 consistently less than 2ms difference between the times 
 reported for all
 compilers.  If I use rdmd without optimizations, the difference 
 is slightly
 larger at 3-5ms.


Cool. Even gdmd? Your first trial showed 150 vs 30 ms.


 So, on the basis of that I think I feel happy about using isNaN 
 as a test,
 especially as it lets me remove an otherwise superfluous 
 variable from
 RandomSample.  Thanks for the advice! :-)


No problem. Seeing people make tests is always interesting. You 
could also throw in a few "inf" in there, as inf is *really* 
close binary wise to NaN: It will pass the first "test", and not 
the second. But I think inf's are even rarer then NaN's, so...

Also, those tests are with inline? Could you report the times 
without please (for my own curiosity)?
Jul 04 2013
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 07/04/2013 04:05 PM, monarch_dodra wrote:

 Holly crap, I didn't see that. Don't do that. The problem if you do that is
that
 on a lot of machines, the time it takes to do 1 iteration is shorter than the
 granularity of the clock. For example, on windows, it is typically 15 ms, which
 is HUGE! I think *nix is more precise, but not *that* precise either. If you
are
 unlucky, the system clock may increment between iterations, meaning you'll miss
 out on that time.
 
 In particular, if your tests last 30ms, it can make a big difference. I usually
 try to make sure my tests last at least a whole second. I also try to add an
 entire second of warm up, the time it takes for the OS to get its caching done.
 
 I also try to run the tests in such a way that each case is tested several
times
 alternatively. Sometimes, you'd be surprised with the result.


You're absolutely right.  With the watch stop/start placed outside the foreach
loop, the times measured are cut dramatically.  The difference is still in the
range of about 2-5 ms.

To avoid the problem of the stopwatch increments being too small relative to
test time, I've put the watch stop/start outside the foreach loops, and made the
foreach loops iterate over a cycle of 100 times the underlying array.

Typical results with dmd -O -release -inline -noboundscheck:

    Time for 100000000 if(boolean)'s with 90000000 counts of true:
        1.3926e+06 microseconds [0.013926 per test].
    Time for 100000000 if(isNaN)'s with 90000000 counts of true:
        1.53494e+06 microseconds [0.0153494 per test].
    Total difference: -142345 [-0.00142345 per test].

Typical results with gdmd -O -release -inline -noboundscheck:

    Time for 100000000 if(boolean)'s with 90000000 counts of true:
        899075 microseconds [0.00899075 per test].
    Time for 100000000 if(isNaN)'s with 90000000 counts of true:
        1.33906e+06 microseconds [0.0133906 per test].

    Total difference: -439986 [-0.00439986 per test].

Typical results with ldmd2 -O -release -inline -noboundscheck:

    Time for 100000000 if(boolean)'s with 90000000 counts of true:
        912115 microseconds [0.00912115 per test].
    Time for 100000000 if(isNaN)'s with 90000000 counts of true:
        1.27287e+06 microseconds [0.0127287 per test].

    Total difference: -360753 [-0.00360753 per test].

Now if we miss off -inline:

dmd -O -release -noboundscheck:
    Time for 100000000 if(boolean)'s with 90000000 counts of true:
        1.95245e+06 microseconds [0.0195245 per test].
    Time for 100000000 if(isNaN)'s with 90000000 counts of true:
        2.31204e+06 microseconds [0.0231204 per test].

    Total difference: -359596 [-0.00359596 per test].

gdmd -O -release -noboundscheck:
    Time for 100000000 if(boolean)'s with 90000000 counts of true:
        905912 microseconds [0.00905912 per test].
    Time for 100000000 if(isNaN)'s with 90000000 counts of true:
        1.3392e+06 microseconds [0.013392 per test].

    Total difference: -433283 [-0.00433283 per test].

ldmd2 -O -release -noboundscheck:
    Time for 100000000 if(boolean)'s with 90000000 counts of true:
        900240 microseconds [0.0090024 per test].
    Time for 100000000 if(isNaN)'s with 90000000 counts of true:
        1.2728e+06 microseconds [0.012728 per test].

    Total difference: -372557 [-0.00372557 per test].

Missing off -noboundscheck seems to make no difference with gdmd or ldmd2 but
creates a bit of slowdown for dmd:

dmd -O -release:
    Time for 100000000 if(boolean)'s with 90000000 counts of true:
        2.68152e+06 microseconds [0.0268152 per test].
    Time for 100000000 if(isNaN)'s with 90000000 counts of true:
        3.00835e+06 microseconds [0.0300835 per test].

    Total difference: -326823 [-0.00326823 per test].

Lastly, I tried replacing the non-NaN double value with double.infinity instead
of 1.0.  It makes no appreciable difference to the above results, at least with
all the optimization flags enabled (I didn't test it without).

Code attached again. :-)

Anyway, in summary, it looks like the extra time cost of using an if(isNaN) as
opposed to an if(boolean) is at most about 5e-9 seconds, and this is consistent
pretty much regardless of the implementation flags used (even rdmd with no
optimization flags reports a difference per test of about 0.003 microseconds).
Jul 04 2013