www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Precision Error?

reply WhatMeWorry` <kheaser gmail.com> writes:
     float computeToFitInWindow(uint rows, uint cols)
     {
         // code removed ...
         writeln(hexWidth);
         return hexWidth;
     }


     unittest
     {
         assert(computeToFitInWindow(1, 1) == 2.0f, "This assert 
failed");
         assert(computeToFitInWindow(3, 3) == 0.8f, "This assert 
failed");
         assert(computeToFitInWindow(2, 2) == 1.14286f, "This 
assert failed");   // Line 74
     }


2
0.8
1.14286
main.d(74): [unittest] This assert failed

Can someone tell me why line 74 fails?   Aren't they both 
1.14286?  I presume it has to do with precision.  Is there a 
technique to address this issue.
Oct 06
parent Sergey <kornburn yandex.ru> writes:
On Sunday, 6 October 2024 at 21:09:22 UTC, WhatMeWorry` wrote:

     float computeToFitInWindow(uint rows, uint cols)
     {
         // code removed ...
         writeln(hexWidth);
         return hexWidth;
     }


     unittest
     {
         assert(computeToFitInWindow(1, 1) == 2.0f, "This assert 
 failed");
         assert(computeToFitInWindow(3, 3) == 0.8f, "This assert 
 failed");
         assert(computeToFitInWindow(2, 2) == 1.14286f, "This 
 assert failed");   // Line 74
     }


 2
 0.8
 1.14286
 main.d(74): [unittest] This assert failed

 Can someone tell me why line 74 fails?   Aren't they both 
 1.14286?  I presume it has to do with precision.  Is there a 
 technique to address this issue.


https://dlang.org/library/std/math/operations/is_close.html
Oct 06