Format longeng - rounding error

Hello All,

When using "format longeng", I am getting strange results to basic maths but when using "format short" the correct / expected answer is returned:

>> format short
>> 0.088 - 0.08
ans =
    0.0080

>> format longeng
>> 0.088 - 0.08
ans =
    7.99999999999999e-003

Is this a bug? Am I missing something? would this make a difference to my simulations?

For reference, I am running Microsoft Windows 7 Version 6.1 (Build 7601: Service Pack 1) on an Intel Xeon 64bit W3520 2.67GHz

MATLAB Version: 8.1.0.604 (R2013a)

Answer 1

[Roger Stafford]

Whatever your matlab version and whatever your operating system, you are attempting to do decimal arithmetic on a machine that uses binary numbers. For that reason it cannot represent 0.008 exactly. For that matter it can represent neither 0.088 nor 0.08 exactly. If you display the number the machine actually uses to sufficiently many digits, you will always be able to see the difference. However (for double precision numbers) the error tends to be out in the fifteenth or sixteenth decimal place. It is something all computer users must become accustomed to.

[Dave]

Roger,

I have also been looking at http://support.microsoft.com/kb/42980 - this helped me to understand.

Many thanks, Dave