I used to think that to Number.toPrecision(d) set the number of decimal places (and the OMV help confirms that) so that, for instance,
n = 12.34567
would return 123.45. But it doesn't: it returns 12. n.toPrecision(4) returns 12.34. In fact, it looks as if toPrecision now works like a kind of slice():
n = 12.34567
returns 12.35: it removes the decimal point, takes the first four digits, rounds up.
Clearly there's something I miss. But what?
Thanks. The description in your linked web page made me look at the OMV again, and the help there appears to be completely wrong. It says
Number.toPrecision (decimals: number ): number
with an additional comment that decimals is the number of decimals. In addition, it returns a string, not a number.
Anyway, another look showed that what I was after was Number.toFixed(d), which does clip decimals the way I thought toPrecision would.
I just gooled this.
w3schools is a very problematic reference. Read http://w3fools.com/ for some info on why.
Be very careful when you manipulate JS rounding methods such as Number.toFixed()—especially if you treat currency values that require correct results! Since JS floating point numbers are considered "only as reliable as possible and no more," rounding methods are highly non-predictable. For example,
(1.255).toFixed(2) may return 1.25
(1.355).toFixed(2) may return 1.36
I wrote 'may' because it also depends on the OS:
(19.125).toFixed(2) returns 19.12 on Mac Lion (64bits) while the same code returns 19.13 on Win XP (32bits), all tested with ID CS5.5.
I learned the hard way that if you critically need to preserve correct rounded values, the best is to process your data as integers (which are reliable to 15 digits in JS).
Europe, Middle East and Africa