2 Replies Latest reply on Aug 14, 2008 9:52 AM by gkaiseril

# simple calculation - subtraction

Why is it when I subtract 7,068.38 from 104,175.37 do I get 97,106.989999..
I even tried to force it to work properly and it still does it. I tried writing
Math.round(100*this.getField("Text#").value)/100 -
Math.round(100*this.getField("Text#").value)/100;
Same answer?? This isn't that crucial but I'm just curious if anyone knows why it's calulating a simple subtraction formula that way? The rest of my from us calculating perfectly. Strange!
• ###### 1. Re: simple calculation - subtraction
Seems fairly typical rounding inaccuracies with floating point
arithmetic. If working with money it's vital to use only whole
numbers (e.g. work in cents): no accountant would accept money treated
as floating point.

Aandi Inston
• ###### 2. Re: simple calculation - subtraction
Are the numbers you are subtracting calculated?

Because if I enter the values into form fields and perform the calculation, I come up with a different answer. The format changes the display of the value not the actual value. And you should also round the computed value. Remember the computer uses binary values for math computations and decimal values do not convert to finite binary decimals.

Unrounded values:
Text0 = 104175.38
Text1 = 7068.37
Text0 - Text1 = 97107.01000000001

Rounded Values:
Text0 = 104175.38
Text1 = 7068.37
Text.0 - Text.1 = 97107.01000000001

Rounded Difference:
Text0 - Text1 = 97107.01

Since you will be using the rounding routine repeatedly, I would create a document level funciton to round a value to a given number of decimal places.

function Round(fValue, fPrecision) {
/*
Purpose: round a number to the requested number of decimal places

Inputs:
fValue = floating point number to be rounded
fPrecison = number of decimal places to be rounded to

Returns: rounded number
*/

// compute power of 10 for the number of decimal places
// returned the value rounded to the requested number of decimal places
}

// Custom Calculation script using above function to demonstrate rounding
console.show();
console.clear();
var Text0 = this.getField("Text.0").value;
var Text1 = this.getField("Text.1").value;
console.println("Unrounded values:");
console.println("Text0 = " + Text0);
console.println("Text1 = " + Text1);
var Diff = Text0 - Text1;
console.println("Text0 - Text1 = " + Diff);
console.println("\nRounded Values:");
var Text0 = Round(Text0, 2)
var Text1 = Round(Text1, 2)
console.println("Text0 = " + Text0);
console.println("Text1 = " + Text1);
Diff = Text0 - Text1;
console.println("Text0 - Text1 = " + Diff);
console.println("\nRounded Difference:");
Diff = Round(Diff, 2);
console.println("Text0 - Text1 = " + Diff);

event.value = this.getField("Text.0").value - this.getField("Text.1").value;