Click here to Skip to main content
16,016,067 members
Home / Discussions / C#
   

C#

 
GeneralRe: float calculation issue Pin
George_George28-May-08 0:23
George_George28-May-08 0:23 
GeneralRe: float calculation issue Pin
DaveyM6927-May-08 12:05
professionalDaveyM6927-May-08 12:05 
GeneralRe: float calculation issue Pin
George_George28-May-08 0:26
George_George28-May-08 0:26 
GeneralRe: float calculation issue Pin
DaveyM6928-May-08 1:42
professionalDaveyM6928-May-08 1:42 
GeneralRe: float calculation issue Pin
George_George28-May-08 18:54
George_George28-May-08 18:54 
AnswerRe: float calculation issue Pin
DaveyM6926-May-08 1:40
professionalDaveyM6926-May-08 1:40 
GeneralRe: float calculation issue Pin
George_George26-May-08 15:16
George_George26-May-08 15:16 
AnswerRe: float calculation issue Pin
The Nightcoder28-May-08 22:52
The Nightcoder28-May-08 22:52 
This is actually the subject of a book or two... and you haven't yet noticed that when using binary floating-point (as in float or double), numbers that look fairly simple (like 0.1) using decimal notation look more like PI using binary notation (0.1 converted to binary has an infinite number of binary decimals and thus cannot be represented exactly using a binary floating-point type).

In any case - here are some rules of thumb (coming from years of building business software doing monetary calculations using binary floating-point types):

- After each step in a calculation, round the result to the desired number of decimals (depends on currency - different currencies have different number of decimals - and in some cases such as invoice totals, you have no decimals depending on local traditions).

- When doing comparisons, never compare for equality (0.1 stored as a double will NOT equal 0.1 stored as a float, for instance). Instead, make sure the difference is within an acceptable range.

Examples:

// calculate a sum of order rows:
double total = 0;
foreach (row in rows)
{
    row.Amount = Math.Round(row.Quantity * row.UnitPrice, 2);
    total = Math.Round(total + row.Amount, 2);
    // note: total += Math.Round(...) can cause very small
    // roundoff errors (caused by binary encoding) to build
    // up and become visible.
}
 
// check for a = b:
if (Math.Round(Math.Abs(a - b), 2) < 0.001)
{
    // consider a and b sufficiently equal,
    // assuming two significant decimals.
}
 
// check for a < b:
if (Math.Round(b - a, 2) > 0.001)
{
    // consider a (sufficiently) less than b
}


And so on... Note that my using 0.001 is an assumption that any round-off errors (after rounding to two decimals, which is the norm for Swedish currency) will be less than that. Using double (and not working on extremely large amounts), this is normally true.

To a "computer math expert" (I used to be one a long time ago) these rules are somewhat naïve and over-simplified, but they do work well for simple financial calculation (such as totaling an invoice and so on). The goal in such circumstances isn't to be as mathematically accurate as possible but to get the same result a person using a financial calculator would (you don't want customers calling in telling you that you can't add properly).

The examples all fulfill that requirement.

If you're interested in more about how binary floating-point works, try looking for a good book on it - it's not a small subject(actually, it's a career for some). I take it you're a CS student? If so, ask your teacher - he or she should be able to point you in the right direction (I no longer know what's current). It's a very interesting field - at least I thought so when I studied (and worked as an assistant teacher in) numerical analysis almost 30 years ago... Smile | :)

Peter the small turnip

(1) It Has To Work. --RFC 1925[^]

GeneralRe: float calculation issue Pin
George_George31-May-08 3:15
George_George31-May-08 3:15 
GeneralRe: float calculation issue Pin
The Nightcoder2-Jun-08 2:35
The Nightcoder2-Jun-08 2:35 
GeneralRe: float calculation issue Pin
George_George3-Jun-08 1:59
George_George3-Jun-08 1:59 
GeneralRe: float calculation issue Pin
The Nightcoder3-Jun-08 2:39
The Nightcoder3-Jun-08 2:39 
GeneralRe: float calculation issue Pin
George_George3-Jun-08 2:51
George_George3-Jun-08 2:51 
GeneralRe: float calculation issue Pin
The Nightcoder3-Jun-08 3:18
The Nightcoder3-Jun-08 3:18 
GeneralRe: float calculation issue Pin
George_George3-Jun-08 21:57
George_George3-Jun-08 21:57 
GeneralRe: float calculation issue Pin
The Nightcoder4-Jun-08 2:55
The Nightcoder4-Jun-08 2:55 
GeneralRe: float calculation issue Pin
George_George4-Jun-08 22:31
George_George4-Jun-08 22:31 
GeneralRe: float calculation issue Pin
The Nightcoder4-Jun-08 23:08
The Nightcoder4-Jun-08 23:08 
GeneralRe: float calculation issue Pin
George_George4-Jun-08 23:28
George_George4-Jun-08 23:28 
AnswerRe: float calculation issue (decimal is no solution) [modified] Pin
The Nightcoder28-May-08 23:12
The Nightcoder28-May-08 23:12 
GeneralRe: float calculation issue (decimal is no solution) Pin
George_George31-May-08 3:28
George_George31-May-08 3:28 
GeneralRe: float calculation issue (decimal is no solution) Pin
The Nightcoder2-Jun-08 3:02
The Nightcoder2-Jun-08 3:02 
GeneralRe: float calculation issue (decimal is no solution) Pin
George_George3-Jun-08 2:09
George_George3-Jun-08 2:09 
GeneralRe: float calculation issue (decimal is no solution) Pin
The Nightcoder3-Jun-08 2:48
The Nightcoder3-Jun-08 2:48 
GeneralRe: float calculation issue (decimal is no solution) Pin
George_George3-Jun-08 3:00
George_George3-Jun-08 3:00 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.