BIG BUG calculation in visual studio

Question

1.00/5 (2 votes)

See more:

, +

Hi guys.
I give 14 in all below codes. But it is wrong. It must be 15 !
Anybody help please?

What I have tried:

VB

MessageBox.Show("Result = " & Math.Truncate(500 / (1000 / 30)))
MessageBox.Show("Result = " & Math.Floor(500 / (1000 / 30)))
MessageBox.Show("Result = " & Int(500 / (1000 / 30)))
MessageBox.Show("Result = " & CInt(500 / (1000 / 30)))

Posted 11-Jun-24 23:47pm

Sh.H.

Updated 12-Jun-24 0:16am

v2

Add a Solution

Comments

Dave Kreskowiak 12-Jun-24 9:29am

This has nothing to do with Visual Studio. Visual Studio doesn't execute code.

Though, it has everything to do with your understanding of the code.

2 solutions

Solution 3

in C# with

MessageBox.Show("Result = " & Math.Truncate(500 / (1000 / 30)))

I get 15 using

Console.WriteLine("Result = " + Math.Truncate(500 / (1000 / 30)))

Posted 14-Jun-24 10:39am

charles henington

Comments

Richard Deeming 17-Jun-24 5:06am

Because C# is using integer division[^].

1000 / 30 == 33
500 / 33 == 15

You don't even need the Math.Truncate call.

The equivalent in VB.NET would use the integer division operator[^]:

500 \ (1000 \ 30)

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Richard Deeming · Accepted Answer · 2024-06-12T00:09:00

Solution 1

The result of your calculation[^] is 14.999999999999998.

(See What Every Computer Scientist Should Know About Floating-Point Arithmetic[^] if you don't understand why.)

If you truncate that number, you will get 14.

If you take the floor of that number, you will get 14.

The Int function is equivalent to the Math.Truncate function:

Int, Fix functions (Visual Basic for Applications) | Microsoft Learn[^]:
Both Int and Fix remove the fractional part of number and return the resulting integer value.

The CInt function[^] rounds the value to the nearest whole number, and is therefore the only version that will produce a result of 15.

In this specific case, your best option is to rearrange the calculation to (30 * 500 / 1000), or simply replace it with the pre-calculated result of 15.

Posted 12-Jun-24 0:09am

Richard Deeming

Comments

Sh.H. 12-Jun-24 6:13am

@Richard Deeming
Wrong!
The result is not 14.98 ! It is 15.0000015 !

Richard Deeming 12-Jun-24 6:19am

Did you even look at the link?
Division | C# Online Compiler | .NET Fiddle[^]

That clearly shows that the result of your calculation is 14.999999999999998.

The computer doesn't care what you think it is; that is the result of your calculation according to the rules of floating-point arithmetic for computers.

Sh.H. 12-Jun-24 6:26am

@Richard Deeming
Yes I see the link.
This is my exact question.
Why the real result is different to the calculated result?

Richard Deeming 12-Jun-24 6:31am

Because computers don't calculate in decimal; they calculate in binary. And even numbers which have a precise and finite representation in decimal don't always have a precise and finite representation in binary, hence the rounding errors.

All of which is explained in the second link in my answer.

Your "trick" works by using the System.Decimal type[^], which is slower and has a smaller range than regular floating-point representations, but results in fewer unexpected rounding issues like this.

CPallini 12-Jun-24 13:21pm

Hint: Have another look at the linked document.

CPallini 12-Jun-24 13:20pm

5.

Maciej Los 17-Jun-24 14:34pm

5ed!