7 Replies Latest reply on Aug 30, 2007 5:38 PM by SymTsb

# Simple Math Calculation is incorrect

I have seen this topic here before but cant find it. When I do the following script it returns an incorrect value;

var myNumber = 999.43
trace(myNumber-999) //returns: 0.42999999999995

So how can I retrieve the value .43? I suppose I can do this;

var myNumber = 999.43
var hundred = myNumber*100;
var whole = hundred-99900;
trace(whole*.01) // returns: .43

But is there a better way? And why is it that Flash is calculating like this?
• ###### 1. Re: Simple Math Calculation is incorrect
(Assuming AS3)

use the toFixed method of the Number class. Keep in mind that this returns a string value so you'll have to type cast it back to a number.

var myNumber:Number = new Number(999.43);
var result:Number = myNumber - 999;
var precise:String = result.toFixed(2);
result = Number(precise);
• ###### 2. Re: Simple Math Calculation is incorrect
Ah thanks, that is nice to see for AS3, but I am using AS2 here still. Just out of curiosity, would you know why it is doing this?
• ###### 3. Re: Simple Math Calculation is incorrect
it's an issue of precision. I believe you can still do something similar in AS2. I believe the Number class in AS2 allows you to convert a number to a string which you could then concatenate with the Sring class methods and convert back to a number.
• ###### 4. Re: Simple Math Calculation is incorrect
the reason for the apparent error is because flash is performing arithmetic on a binary computer. within the limits of flash'es accuracy (16 digits),

0.42999999999995 = .43

to eliminate those errors, you can use integer arithmetic or the flash string methods. integer arithmetic is probably preferable if you're doing arithmetic operations.
• ###### 5. Re: Simple Math Calculation is incorrect
ah...I understand now, thanks to the both of you.
• ###### 6. Re: Simple Math Calculation is incorrect
you're welcome.
• ###### 7. Re: Simple Math Calculation is incorrect
you're welcome.