Click here to Skip to main content
16,016,088 members
Home / Discussions / C#
   

C#

 
GeneralRe: float calculation issue Pin
George_George28-May-08 18:54
George_George28-May-08 18:54 
AnswerRe: float calculation issue Pin
DaveyM6926-May-08 1:40
professionalDaveyM6926-May-08 1:40 
GeneralRe: float calculation issue Pin
George_George26-May-08 15:16
George_George26-May-08 15:16 
AnswerRe: float calculation issue Pin
The Nightcoder28-May-08 22:52
The Nightcoder28-May-08 22:52 
GeneralRe: float calculation issue Pin
George_George31-May-08 3:15
George_George31-May-08 3:15 
GeneralRe: float calculation issue Pin
The Nightcoder2-Jun-08 2:35
The Nightcoder2-Jun-08 2:35 
GeneralRe: float calculation issue Pin
George_George3-Jun-08 1:59
George_George3-Jun-08 1:59 
GeneralRe: float calculation issue Pin
The Nightcoder3-Jun-08 2:39
The Nightcoder3-Jun-08 2:39 
The reason for the 7-digit precision in ToString() is that the ULP for float is 2^-23 (approximately 10^-7). Showing more digits is generally misleading. However, the internal precision can sometimes be larger, up to a maximum of approximately nine digits, which is probably (wild guess) the reason the debugger chooses to show that many.

This is only briefly explained in the help for System.Single, but for a reasonably good definition of ULP and floating point arithmetics, look here[^].

Then again: Never use float. Use double. Except possibly at gunpoint. Consider float to be obsolete. And remember that in SQL Server, float is double and real is single (go figure).

Peter the small turnip

(1) It Has To Work. --RFC 1925[^]

GeneralRe: float calculation issue Pin
George_George3-Jun-08 2:51
George_George3-Jun-08 2:51 
GeneralRe: float calculation issue Pin
The Nightcoder3-Jun-08 3:18
The Nightcoder3-Jun-08 3:18 
GeneralRe: float calculation issue Pin
George_George3-Jun-08 21:57
George_George3-Jun-08 21:57 
GeneralRe: float calculation issue Pin
The Nightcoder4-Jun-08 2:55
The Nightcoder4-Jun-08 2:55 
GeneralRe: float calculation issue Pin
George_George4-Jun-08 22:31
George_George4-Jun-08 22:31 
GeneralRe: float calculation issue Pin
The Nightcoder4-Jun-08 23:08
The Nightcoder4-Jun-08 23:08 
GeneralRe: float calculation issue Pin
George_George4-Jun-08 23:28
George_George4-Jun-08 23:28 
AnswerRe: float calculation issue (decimal is no solution) [modified] Pin
The Nightcoder28-May-08 23:12
The Nightcoder28-May-08 23:12 
GeneralRe: float calculation issue (decimal is no solution) Pin
George_George31-May-08 3:28
George_George31-May-08 3:28 
GeneralRe: float calculation issue (decimal is no solution) Pin
The Nightcoder2-Jun-08 3:02
The Nightcoder2-Jun-08 3:02 
GeneralRe: float calculation issue (decimal is no solution) Pin
George_George3-Jun-08 2:09
George_George3-Jun-08 2:09 
GeneralRe: float calculation issue (decimal is no solution) Pin
The Nightcoder3-Jun-08 2:48
The Nightcoder3-Jun-08 2:48 
GeneralRe: float calculation issue (decimal is no solution) Pin
George_George3-Jun-08 3:00
George_George3-Jun-08 3:00 
GeneralRe: float calculation issue (decimal is no solution) Pin
The Nightcoder3-Jun-08 3:37
The Nightcoder3-Jun-08 3:37 
GeneralRe: float calculation issue (decimal is no solution) Pin
George_George3-Jun-08 21:53
George_George3-Jun-08 21:53 
GeneralRe: float calculation issue (decimal is no solution) Pin
The Nightcoder4-Jun-08 3:04
The Nightcoder4-Jun-08 3:04 
GeneralRe: float calculation issue (decimal is no solution) Pin
George_George4-Jun-08 22:22
George_George4-Jun-08 22:22 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.