Skip navigation
Peter Kahrel
Peter Kahrel
Currently Being Moderated

JS: toPrecision()

May 12, 2012 7:55 AM

I used to think that to Number.toPrecision(d) set the number of decimal places (and the OMV help confirms that) so that, for instance,

 

n = 12.34567

n.toPrecision (2)

 

would return 123.45. But it doesn't: it returns 12. n.toPrecision(4) returns 12.34. In fact, it looks as if toPrecision now works like a kind of slice():

 

n = 12.34567

n.toPrecision (4)

 

returns 12.35: it removes the decimal point, takes the first four digits, rounds up.

 

Clearly there's something I miss. But what?

 

Thanks,

 

Peter

883 Views   7 Replies   Latest reply: Peter Kahrel, May 13, 2012 3:05 AM
 
Replies
  • Pickory
    Currently Being Moderated
    1. Pickory,
    May 12, 2012 8:13 AM   in reply to Peter Kahrel
    Report

    I just gooled this.

     

    http://www.w3schools.com/jsref/jsref_toprecision.asp

     
    |
    Mark as:
  • Peter Kahrel
    Currently Being Moderated
    2. Peter Kahrel,
    May 12, 2012 8:21 AM   in reply to Pickory
    Report

    Thanks. The description in your linked web page made me look at the OMV again, and the help there appears to be completely wrong. It says

    Number.toPrecision (decimals: number ): number

    with an additional comment that decimals is the number of decimals. In addition, it returns a string, not a number.

     

    Anyway, another look showed that what I was after was Number.toFixed(d), which does clip decimals the way I thought toPrecision would.

     

    Peter

     
    |
    Mark as:
  • John Hawkinson
    5,527 posts
    Jun 25, 2009
    Currently Being Moderated
    3. John Hawkinson,
    May 12, 2012 8:52 PM   in reply to Pickory
    Report

    Pickory wrote:

     

    I just gooled this.

    http://www.w3schools.com/jsref/jsref_toprecision.asp

    w3schools is a very problematic reference. Read http://w3fools.com/ for some info on why.

    Instead, Mozilla has a much better reference: https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/N umber/toPrecision

     
    |
    Mark as:
  • Marc Autret
    Currently Being Moderated
    4. Marc Autret,
    May 13, 2012 2:44 AM   in reply to Peter Kahrel
    Report

    Hi Peter,

     

    Be very careful when you manipulate JS rounding methods such as Number.toFixed()—especially if you treat currency values that require correct results! Since JS floating point numbers are considered "only as reliable as possible and no more," rounding methods are highly non-predictable. For example,

     

    (1.255).toFixed(2) may return 1.25

    whereas

    (1.355).toFixed(2) may return 1.36

     

    I wrote 'may' because it also depends on the OS:

     

    (19.125).toFixed(2) returns 19.12 on Mac Lion (64bits) while the same code returns 19.13 on Win XP (32bits), all tested with ID CS5.5.

     

    I learned the hard way that if you critically need to preserve correct rounded values, the best is to process your data as integers (which are reliable to 15 digits in JS).

     

    @+

    Marc

     
    |
    Mark as:
  • Peter Kahrel
    Currently Being Moderated
    5. Peter Kahrel,
    May 13, 2012 2:52 AM   in reply to Marc Autret
    Report

    Thanks, Marc, for pointing out those inconsistencies.

     

    > the best is to process your data as integers (which are reliable to 15 digits in JS)

     

    You did mean to write 'reals', did you not

     

    Peter

     
    |
    Mark as:
  • Marc Autret
    Currently Being Moderated
    6. Marc Autret,
    May 13, 2012 3:01 AM   in reply to Peter Kahrel
    Report

    No, I really meant integers. E.g.: store 123.45 as 12345 and keep somewhere else the E-2.

     

    @+

    Marc

     
    |
    Mark as:
  • Peter Kahrel
    Currently Being Moderated
    7. Peter Kahrel,
    May 13, 2012 3:05 AM   in reply to Marc Autret
    Report

    Aha, in that way. Thanks.

     

    P.

     
    |
    Mark as:
Actions

More Like This

  • Retrieving data ...

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points