Addition errors in IEEE754 floating point representation

Question

So in class, we were talking about the idea of floating point precision in IEEE754 format, and how, when some numbers are added, precision is lost. My professor then gave the following example of a large absolute error between what we think the answer might be, and what the answer actually is when our program (written in Java) actually compiles it:

$0.4 \times10^{30} + 1.1 \times10^{30}$

Compiling this in Java, i get $1.5000001 \times 10^{30}$

My question is, why does this happen? Printing out the IEEE754 format of the two numbers, I get:

$\space\space\space0[11100010]10111100010010010101011=+1.10111100010010010101011\times2^{99}$ $+0[11100001]01000011000111100001000=+1.01000011000111100001000\times2^{98}$

So, I suppose when you add these, you end up loosing precision, but exactly how? Moreover, how can I even tell, before adding, that the addition of two numbers will make them loose precision? Or is the only way to just plus into a program and see what happens? None of this was really covered in class unfortunately.

Answer 1

Printing out the IEEE754 format of the two numbers, I get:

$\space\space\space0[11100010]10111100010010010101011=+1.10111100010010010101011\times2^{99}$ $+0[11100001]01000011000111100001000=+1.01000011000111100001000\times2^{98}$

Compare the exact values:

$$ 1.1011110001001001010101011000111111010110011111001110000010110110100111 \times 2^{99} \\ 1.0100001100011110000011111010111001101101011100100001011111001010101 \times 2^{98} $$

Observe that both of the numbers are already rounded to represent them in IEEE 754, and it happens that both of them have rounded up.

Also note that since the values have different exponents, they've been rounded at different places.

Now add the rounded values:

  1.10111100010010010101011 ×299
+ 0.101000011000111100001000×299
 -------------------------------
 10.01011101110110000101111 x299
 11  1         1111

There's overflow, so the answer will have exponent $2^{100}$ and we have to round. Again, we're rounding up, to $1.00101110111011000011000\times2^{100}$.

So, I suppose when you add these, you end up loosing precision, but exactly how?

You will nearly always lose precision when adding two floating point numbers. If they're both positive and have the same exponent, the output will have a larger exponent and you lose some bits from the end. If they have different exponents, the smaller one will lose bits from the end. And if they have opposite signs and the same exponent, the output has a smaller exponent and you lose precision because you need bits beyond the end which were lost earlier. Essentially the only cases where you don't lose precision are when you add a number to itself, or when the input numbers and the result can all be represented exactly in the width of the type.

Answer 2

You were using single precision floating point numbers. Single precision floating point numbers look like this: You start with a mantissa, which is a number 1 ≤ m < 2 which is a multiple of $2^{-23}$, then you multiply by a power of 2, then you multiply by +1 or -1.

So when you write 0.4e30, that is not a possible floating-point number at all. Instead, you get the floating-point number closest to 0.4e30. So we have an error. And we have another error in 1.1e30. What you are adding is really the floating-point number closest to 0.4e30, plus the floating-point number closest to 1.1e30.

Now 0.4e30 has a smaller exponent than 1.1e30. If you add the two numbers, then the two last bits of 0.4e30 must be lost. So you add not quite the numbers you think you are adding, and you get an error during the rounding. If by coincidence the last two bits of 0.4e30 were both zeroes, then there's no rounding error.

There are some cases where we know there is no rounding error: If you add x+y, and x, y, and x+y are all integers up to 2^24, then you can prove that you will get the correct result with zero rounding error. If you subtract x-y where x and y are positive numbers and x/2 ≤ y ≤ 2x, then you can also prove that there is no rounding error. If you add a zero then there is no rounding error. But usually you just assume there will be a rounding error, and it will be up to one half of the value of the last mantissa bit of the result.