14 Replies Latest reply on Dec 9, 2008 10:57 AM by Boxy_999

    Integer math just doesn't add up

    Boxy_999
      I'm a Flex 3 newbie so apologies if I'm being stupid but I'm seeing discrepancies in the debugger between the integer value of a calculation and what an integer variable is reported to contain. See example code.

      In the debugger, when I highlight ((71/9)/3)*3 a popup displays the result 6. However, when I highlight numRegion the popup box displays 7. The Variables window shows the value of numRegion to be 7 also.

      I tried using int(((71/9)/3)*3) and Math.floor too but I haven't yet found a way to get the correct result of 6.

      Can anyone help?

      Thanks in advance
        • 1. Re: Integer math just doesn't add up
          Ansury Level 3
          Not 100% sure but I'd bet that the math is being done by default as a Number type, which is like a float in C/Java - so no dropping of the remainders. If you don't drop the remainders you get almost 8 (7.888...), which then gets rounded down (drop the entire remainder for "cpu style" integer rounding).... to 7.

          I haven't done a heck of alot of math in Flex/Flash so I can't say what the default type it uses if none is specified, but I think it's Number.
          • 2. Re: Integer math just doesn't add up
            Ansury Level 3
            Wait wait, the debugger is saying 6? Heh. That seems strange, I'll have to check into that. I'd think you should get 7 there too..
            • 3. Integer math just doesn't add up
              Ansury Level 3
              My debugger says 7, of course I'm on a Flex 2 workstation now, not FB3. Can't test 3 atm, maybe someone else can. Maybe it's a bug in the debugger? Kinda surprised that's possible, though. It's as if the debugger is using "int" instead of Number by default to do that math, or at least rounding down after each operation.
              • 4. Re: Integer math just doesn't add up
                rtalton Level 4
                Hi,
                I'm no math major, but I get a result of 7.8888...
                How are you arriving at an answer of 6?
                • 5. Re: Integer math just doesn't add up
                  Boxy_999 Level 1
                  Follow up:

                  I've broken the computation down into seperate elements. See example code below.

                  Using this approach produces the correct result : 6

                  It's not an elegant solution but it works. This suggests to me that the underlying code isn't evaluating a complex computation even with the aid of brackets.

                  • 6. Re: Integer math just doesn't add up
                    Boxy_999 Level 1
                    Hi rtalton,

                    Using integer math:

                    71 / 9 = 7
                    7 / 3 = 2
                    2 * 3 = 6

                    Regards
                    • 7. Re: Integer math just doesn't add up
                      Ansury Level 3
                      Well, doing it that way, you're converting your answer into a uint each time instead of leaving it a Number for the next computation. That effectively rounds down the remainder, and there you go.

                      This doesn't show up in other languages typically because an integer type is usually the default type.
                      • 8. Re: Integer math just doesn't add up
                        Ansury Level 3
                        quote:

                        Originally posted by: rtalton
                        Hi,
                        I'm no math major, but I get a result of 7.8888...
                        How are you arriving at an answer of 6?


                        Rounding down after EACH operation (computers like to always round down when doing integer math instead of using the .5 rule).
                        • 9. Re: Integer math just doesn't add up
                          rtalton Level 4
                          I did a little testing, and what Flex is doing is rounding the result of each part of your formula, resulting in rounding errors.
                          71/9=7.888 (rounded to 7)
                          7/3=2.333 (rounded to 2)
                          2*3=6
                          If you want the result of six, you need to round each of your results, like this:
                          var x:int = 71/9;
                          var y:int = x/3;
                          var z:int = y*3; (z = 6)

                          Don't know why you'd want to do that, though.
                          • 10. Re: Integer math just doesn't add up
                            Boxy_999 Level 1
                            Hi Ansury,

                            I agree with you that it would seem that the underlying code is using a Number type rather than Integer math. I've substituted Number for int/uint and seen the 7.888 result mentioned in other replies. I suspect this should be regarded as a bug simply because the difference between the desired result: 6 and 7.888 could in some situations be enormous. Out of interest, I have checked the computation in Python and it returns the correct result.

                            Regards
                            • 11. Re: Integer math just doesn't add up
                              Boxy_999 Level 1
                              Hi rtalton,

                              This is one element of a computation being used in a games program where based on a known position of a cell in a grid (71 in this example) it's possible to calculate the grid position of another key cell. Integer math is required and other languages cope with it - certainly C, C++, Python, SQL but not Flex or should I be referring to it as ActionScript. It's not a criticism, it's just a little awkward but I can work around it.

                              Thanks for your responses.
                              All the best,
                              Simon
                              • 12. Integer math just doesn't add up
                                rtalton Level 4
                                Actionscript handles it as long as the vars are declared first and it knows that you want integer results. AS is using strong typing, and all your vars should be typed. So, you just have to declare three vars to get it typed right. That may be different from other languages, I don't know. All I know is that I would want the exact number, not rounded, in most computations I'd be doing. Think payroll: hours, wages, etc. We sure wouldn't our numbers rounded down there now, would we? ;)
                                • 13. Re: Integer math just doesn't add up
                                  Boxy_999 Level 1
                                  Hi rtalton,

                                  I agree with everything you say but to get familiar with Flex I'm adapting a Suduko solver program that I've written in another language and the math is all integer based.

                                  For example:
                                  From a known cell position (cellPos) in a 9x9 cell grid...
                                  To find x,y coords:
                                  y=int(cellPos/9)
                                  x=int(cellpos%9)

                                  To find out which 3x3 cell region contains the known cell is:
                                  int((((cellPos / 9) /3 ) * 3)) + int(((cellPos % 9) / 3))

                                  The last equation should resolve to a number between between 0 and 8 (and it does in other languages) but I think the last time I tried the equation in a single statement in Flex it returned 10!!!

                                  But now that I have a workaround using Flex everything is OK.

                                  I really like to thank you and Ansury for your assistance in resolving this issue.

                                  Best Regards.
                                  • 14. Re: Integer math just doesn't add up
                                    Boxy_999 Level 1
                                    I've got it working!!!!
                                    This really is the last post.
                                    I've been experimenting with brackets and I've been far too liberal with them (in some respects).
                                    The following syntax works properly:-