2 Replies Latest reply on Dec 26, 2006 6:17 PM by Newsgroup_User

    wierd math

    Level 7
      I'm having some wierd math results with Flash 8. Anybody know what could be
      causing this?

      n1 = 30.1
      n2 = 30

      trace(n1-n2) // traces 0.100000000000001


        • 1. Re: wierd math
          Level 7
          That is expected .. because they use binary number, computers cannot
          represent all decimal numbers exactly (ie there is no exact representation
          of 30.1 in a computer)

          That is why there is a very slight error. It is to be expected. You should
          always round your results to a given reasonable number of decimal places, or
          compare numbers to be within a known range. Never expect exact results with
          floating point (decimal place) numbers .. that just can't be relied on in a
          computer.
          --
          Jeckyl


          • 2. Re: wierd math
            Level 7
            Thanks. I did not know that. Here it is in a tech note:
            http://www.adobe.com/go/tn_13989

            In the example below, I've rounded the result to the nearest hundreds, and
            then compare it to literal .1, and it comes out true. Can that comparison
            be relied on, or does the division by 100 introduce the error again?

            n1 = 30.1
            n2 = 30
            n3 = Math.round((n1-n2)*100) /100;
            trace(n3 ) // traces 0.1
            trace(n3 == .1 ) // traces true


            "Jeckyl" <jeckyl@hyde.com> wrote in message
            news:emsgo2$p5a$1@forums.macromedia.com...
            > That is expected .. because they use binary number, computers cannot
            > represent all decimal numbers exactly (ie there is no exact representation
            > of 30.1 in a computer)
            >
            > That is why there is a very slight error. It is to be expected. You
            > should always round your results to a given reasonable number of decimal
            > places, or compare numbers to be within a known range. Never expect exact
            > results with floating point (decimal place) numbers .. that just can't be
            > relied on in a computer.
            > --
            > Jeckyl
            >
            >