0 Replies Latest reply on Nov 23, 2010 2:09 PM by jexer

# What's the best way to deal with floating point errors?

What's the best way to deal with the decimal number errors in Flex?

Lets say I have:

var number1:Number = 1.1;

var number2:Number = 1;

var result:Number = number1 - number2;

trace(result);

I get "0.10000000000000009".

What's the best way to deal with this so that i get the right result?

Using the trick: result = Math.round( result * Math.pow(10,13) ) / Math.pow(10,13); is not useful as such when using big numbers.

For example, If number1 = 100000000001.1, and number2 = 0.2, I get "100000000000.90001" as result. The previous rounding fixes rounding errors after 13th decimal, but here the first rounding errors come after 4 digits.

The toPrecision method in Number and NumberFormatter dont seem be of use by themselves because of the same reason.

So I'd have to check how big the number is, and then round the number based on that.

So far I've constructed a method that does that with logarithms like this:

public function floatFix(number:Number):Number{

var precision:int = 14-Math.floor(Math.log(Math.abs(number))*Math.LOG10E);
var precisionFactor:Number = Math.pow(10, precision);
return Math.round(Number(number*precisionFactor))/precisionFactor;
}

It just seems rather odd that one would have to create and use something like that to just count 1.1-1; There's a lot of calculating in that method, and i'm guessing that having to use that in a loop could slow down a program quite a bit.

I think there really should be a pre-built method in flex for something like this, but I can't find any.

Anyone know any better/faster ways?