5 Replies Latest reply on Jul 21, 2006 6:12 PM by Newsgroup_User

math.floor sucks?

copy this code in your layer as code:

var tt:Number = Number("1002.36");
trace("tt: "+Number(tt));

trace("\n***********\n");

trace(Math.floor(Number(1002.36)*Number(100)));
trace(Math.floor(Number(tt)*Number(100)));

and you will see:

tt: 1002.36

***********

100236
100235

I spend a lot of time to findout this problem in my project, this is just a resume of it.
• 1. Re: math.floor sucks?
that's not a problem with Math.floor. it's a problem with converting a string to a number and the fact that your computer uses binary arithmetic. so, what looks like 1002.36 to you, looks like 1002.3599999999 to your computer.

use Math.round(), if you are able.
• 2. Re: math.floor sucks?
but it's not , i just trace it before i used in math {look this line trace("tt: "+Number(tt));}

you can try this:

// cool numbers

for(var i:Number = 0; i<1500; i++){
for(var j:Number = 0; j<100; j++){
var tt:Number = Number(String(i)+"."+String(j));
//trace("tt: "+Number(tt));
if((Number(tt)*Number(100)) != (Math.floor(Number(tt)*Number(100)))){
trace((Number(tt)*Number(100))+" != "+(Math.floor(Number(tt)*Number(100)))+" original tt: "+tt);
}
}
}

the out put should be empty but it's not
• 3. Re: math.floor sucks?
paullu wrote:
>
> [...]
>
> I spend a lot of time to findout this problem in my project, this is just a
> resume of it.

This is not a problem with Math.floor. Instead you hit the known issue
that numbers are not converted in the same way from string to binary
representation between the flash authoring tool and the flash player.

Typically:
==================
// conversion from string to binary in the authoring tool (compilation):
trace(1002.36 == 1002.36); // true

// conversion from string to binary in the flash player (run time):
trace(Number("1002.36") == Number("1002.36")); // true

// comparison of the 2 conversions:
trace(1002.36 == Number("1002.36")); // false
==================

I've been told it isn't a bug but it's still a problem in my book, and
one that's definitely worth knowing.

Tim.
• 4. Re: math.floor sucks?
"but it's not...", what?
• 5. Re: math.floor sucks?
I've simplified the code a little .. no need for a lot of those 'Number'
calls .. and the full range generates a lot of output .. so try this

for (var i:Number = 1000; i<1100; i++) {
for (var j:Number = 10; j<100; j++) {
var tt:Number = Number(String(i)+"."+String(j));
if (tt*100 != Math.floor(tt*100)) {
trace(tt*100+" != "+Math.floor(tt*100));
}
}
}

The results are consistent with what kglad says .. converting something like
"1042.13" to a number does not give you an exact results .. sometimes it
will be slightly over, and sometimes slightly under and VERY rarely will it
be exact.

That is the nature of numbers and how they are stored in a computer.

So you've proved what kglad has said.
--
Jeckyl