This content has been marked as final. Show 5 replies
"ed_skinner" <email@example.com> wrote in message news:firstname.lastname@example.org...
> I am adding two float numbers that return an incorrect result. Here is
> what I am doing...
> var simpleMath:Number = Number(parseFloat("110.50")) -
> alert("Simple Math=" + String(simpleMath));
> The result I get is... Simple Math=3.40000000000001
> How is this possible?
It's the same old problem we have always had with floating point stored in computer memory.. not just Flash. For dealing with
dollars and cents you can make it easy on yourself by multiplying both sides by 100 and rounding to eliminate decimals, then in
final calculation divide by 100.
You can also multiply/divide by 1000 if you need higher precision in the pennies.
That is why most commercial programs store currency as whole numbers of
cents, rather than as floating point numbers (which by their very nature
cannot be exact)
You need to change your program to take the nature of numbers in computers
into account to avoid errors. The bug is really in how you are doing your
calculations, not in Flash, and the solution also lies with you.
whoa, there's jedi in the jeckyl. who knew?
Hey Thanks guys. I used the technique of *100, calculate, round and then /100 to get it to work right. One might think that in all the advancement in computers that floating point numbers would be handled correctly by now!
Oh well, past another problem with the help of fellow programmers.
Folks... and may the force be with you!
It doesn't matter how advanced a computer it is .. 0.1 (for example) can not
be expressed exactly in binary (which computers use) .. no matter what you
do, it is always only an approximation (a bit like 1/3 =
0.3333333333333333.... in decimal). Its just the mathematics of it.
Fixed point arithmetic (where you implicitly multiply by some number.. like
multiply by 100 for currency amounts to give you cents) is often done. For
instance, the x and y coordinates in Flash are internally fixed point
numbers (multiples of 1/20 = 0.05)