Click here to Skip to main content
16,016,067 members
Home / Discussions / C#
   

C#

 
GeneralRe: float calculation issue Pin
George_George26-May-08 15:15
George_George26-May-08 15:15 
GeneralRe: float calculation issue Pin
GuyThiebaut26-May-08 22:28
professionalGuyThiebaut26-May-08 22:28 
GeneralRe: float calculation issue Pin
George_George27-May-08 3:13
George_George27-May-08 3:13 
GeneralRe: float calculation issue Pin
GuyThiebaut27-May-08 4:36
professionalGuyThiebaut27-May-08 4:36 
GeneralRe: float calculation issue Pin
George_George28-May-08 0:23
George_George28-May-08 0:23 
GeneralRe: float calculation issue Pin
DaveyM6927-May-08 12:05
professionalDaveyM6927-May-08 12:05 
GeneralRe: float calculation issue Pin
George_George28-May-08 0:26
George_George28-May-08 0:26 
GeneralRe: float calculation issue Pin
DaveyM6928-May-08 1:42
professionalDaveyM6928-May-08 1:42 
I think you meant 10 divided by 3?

If so - try this:
float a = 10f;
float b = 3f;
float c = a / b;
decimal d = 10m;
decimal e = 3m;
decimal f = d / e;
Console.WriteLine("Float Result: " + c);
Console.WriteLine("DecimalResult: " + f);


There is a finite limit to the precision a computer can provide due to the way the data is stored and manipulated. After all, we want to deal with large numbers with large numbers of decimal places that we think about in base 10. A processor can in reality only deal with two numbers - 0 and 1. Putting many of these bits together gives it the apparent capability to do so much, but it's basically still just crunching 0s and 1s - which means until we stop having fractions, start thinking in binary (base 2), octal, hex ...., and have unlimited memory and processor address busses then we can never get total precision.

Dave

GeneralRe: float calculation issue Pin
George_George28-May-08 18:54
George_George28-May-08 18:54 
AnswerRe: float calculation issue Pin
DaveyM6926-May-08 1:40
professionalDaveyM6926-May-08 1:40 
GeneralRe: float calculation issue Pin
George_George26-May-08 15:16
George_George26-May-08 15:16 
AnswerRe: float calculation issue Pin
The Nightcoder28-May-08 22:52
The Nightcoder28-May-08 22:52 
GeneralRe: float calculation issue Pin
George_George31-May-08 3:15
George_George31-May-08 3:15 
GeneralRe: float calculation issue Pin
The Nightcoder2-Jun-08 2:35
The Nightcoder2-Jun-08 2:35 
GeneralRe: float calculation issue Pin
George_George3-Jun-08 1:59
George_George3-Jun-08 1:59 
GeneralRe: float calculation issue Pin
The Nightcoder3-Jun-08 2:39
The Nightcoder3-Jun-08 2:39 
GeneralRe: float calculation issue Pin
George_George3-Jun-08 2:51
George_George3-Jun-08 2:51 
GeneralRe: float calculation issue Pin
The Nightcoder3-Jun-08 3:18
The Nightcoder3-Jun-08 3:18 
GeneralRe: float calculation issue Pin
George_George3-Jun-08 21:57
George_George3-Jun-08 21:57 
GeneralRe: float calculation issue Pin
The Nightcoder4-Jun-08 2:55
The Nightcoder4-Jun-08 2:55 
GeneralRe: float calculation issue Pin
George_George4-Jun-08 22:31
George_George4-Jun-08 22:31 
GeneralRe: float calculation issue Pin
The Nightcoder4-Jun-08 23:08
The Nightcoder4-Jun-08 23:08 
GeneralRe: float calculation issue Pin
George_George4-Jun-08 23:28
George_George4-Jun-08 23:28 
AnswerRe: float calculation issue (decimal is no solution) [modified] Pin
The Nightcoder28-May-08 23:12
The Nightcoder28-May-08 23:12 
GeneralRe: float calculation issue (decimal is no solution) Pin
George_George31-May-08 3:28
George_George31-May-08 3:28 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.