I'm working on a piece of software in Flash Builder that has to perform calculations at some point and I've noticed the following:-
If I calculate the following I end up with these results:
0.045 * 4333 = 194.98499999999999
0.035 * 4333 = 151.655
Why does the first calculation provide a result with so many decimal places? Is there any way to modify this behaviour? Note that if I perform the first calculation on a regular calculator, the result is 194.985.
If I calculate the following with the following decimal place modification, I end up with the following:
(0.045 * 4333).toFixed(2) = 194.98
(0.035 * 4333).toFixed(2) = 151.66
The first calculation ends up being incorrect as it's off by 0.01.
Is there any explanation for the behaviour of the first calculation?
These are floating point precision errors; search for "flash floating point calculations" to read up on these, e.g.
http://joshblog.net/2007/01/30/flash-floating-point-number-errors/
http://www.dreaminginflash.com/2011/05/09/flash-floating-point-precision/
I think rather than using Number.toFixed(2), you want to use rounding, e.g.
Math.round((0.045 * 4333) * 100) / 100 = 194.99
HTH,
-- Tom