Thanks. The description in your linked web page made me look at the OMV again, and the help there appears to be completely wrong. It says
Number.toPrecision (decimals: number ): number
with an additional comment that decimals is the number of decimals. In addition, it returns a string, not a number.
Anyway, another look showed that what I was after was Number.toFixed(d), which does clip decimals the way I thought toPrecision would.
I just gooled this.
w3schools is a very problematic reference. Read http://w3fools.com/ for some info on why.
Be very careful when you manipulate JS rounding methods such as Number.toFixed()—especially if you treat currency values that require correct results! Since JS floating point numbers are considered "only as reliable as possible and no more," rounding methods are highly non-predictable. For example,
(1.255).toFixed(2) may return 1.25
(1.355).toFixed(2) may return 1.36
I wrote 'may' because it also depends on the OS:
(19.125).toFixed(2) returns 19.12 on Mac Lion (64bits) while the same code returns 19.13 on Win XP (32bits), all tested with ID CS5.5.
I learned the hard way that if you critically need to preserve correct rounded values, the best is to process your data as integers (which are reliable to 15 digits in JS).
Thanks, Marc, for pointing out those inconsistencies.
> the best is to process your data as integers (which are reliable to 15 digits in JS)
You did mean to write 'reals', did you not
No, I really meant integers. E.g.: store 123.45 as 12345 and keep somewhere else the E-2.
Aha, in that way. Thanks.