2 Replies Latest reply on Dec 26, 2006 6:17 PM by Newsgroup_User

# wierd math

I'm having some wierd math results with Flash 8. Anybody know what could be
causing this?

n1 = 30.1
n2 = 30

trace(n1-n2) // traces 0.100000000000001

• ###### 1. Re: wierd math
That is expected .. because they use binary number, computers cannot
represent all decimal numbers exactly (ie there is no exact representation
of 30.1 in a computer)

That is why there is a very slight error. It is to be expected. You should
always round your results to a given reasonable number of decimal places, or
compare numbers to be within a known range. Never expect exact results with
floating point (decimal place) numbers .. that just can't be relied on in a
computer.
--
Jeckyl

• ###### 2. Re: wierd math
Thanks. I did not know that. Here it is in a tech note:

In the example below, I've rounded the result to the nearest hundreds, and
then compare it to literal .1, and it comes out true. Can that comparison
be relied on, or does the division by 100 introduce the error again?

n1 = 30.1
n2 = 30
n3 = Math.round((n1-n2)*100) /100;
trace(n3 ) // traces 0.1
trace(n3 == .1 ) // traces true

"Jeckyl" <jeckyl@hyde.com> wrote in message
news:emsgo2\$p5a\$1@forums.macromedia.com...
> That is expected .. because they use binary number, computers cannot
> represent all decimal numbers exactly (ie there is no exact representation
> of 30.1 in a computer)
>
> That is why there is a very slight error. It is to be expected. You
> should always round your results to a given reasonable number of decimal
> places, or compare numbers to be within a known range. Never expect exact
> results with floating point (decimal place) numbers .. that just can't be
> relied on in a computer.
> --
> Jeckyl
>
>