Confused by Floating Point Spacing

Question

I'm currently taking a numerical analysis class in college and we're covering floating point systems. For the most part, I have a good grasp on it. However, something I can't seem to visualize, and haven't seen any totally lucid explanations about after searching extensively, is spacing between floating point numbers. Also of note is that I'm talking about IEEE-754 here, but it applies to general systems too.

The Things I Do Understand:

• The area between $[-1,1]$ is a denormalized area.
• The areas after $1$ and less than $-1$ are where the normalized floating point numbers reside.
• The floating point numbers between perfect powers of the base are uniformly spaced, but the spacing varies from one perfect power of the base to another.
• The spacing between values between two perfect powers is proportional to the power on the left for positive numbers and the power on the right for negative numbers. (i.e. on a number line, the uniformly-spaced values between two low powers are closer together than between a higher power.)

What I'm Struggling to Understand

• From my understanding, the machine epsilon $\epsilon$ is a fundamental unit of spacing with respect to the floating point number line. That is, between $[1,B]$ where $B$ is the base, all values are $\epsilon$ apart. Then, you can scale any arbitrary floating point number by that fundamental machine epsilon and that product is the uniform spacing for that floating point number's associated power range. Is this even a correct interpretation?

  I also read that $\epsilon$ is an upper bound for relative error, so I'm not really sure how that fits into my explanation of it being an indivisible spacing unit.

• One of the questions I haven't been able to answer is what the minimum and maximum spacing between two positive floating point numbers is. I can trick myself into thinking I understand why multiplying the x's associated $B^e \cdot \epsilon$, where x is an arbitrary floating point number and $e$ is that number's corresponding exponent, yields the upper bound on error and therefore spacing, so $B^e \cdot \epsilon$ would be the maximum spacing.

  Minimum spacing truly boggles my mind right now, though. If the machine epsilon is the indivisible unit of spacing, then for example, how could we have more minimal spacing than between $1$ and $1 + \epsilon$? Wouldn't that just be left to the rounding rule used (if round-to-nearest, it would depend whether the number is closer to $1$ or $1 + \epsilon$, since it'll be rounded to one of those two).

Basically, if you could explain this in plain-english it would really help me get a solid understanding of what's going on at the number line level.

Answer 1

This is best explained in the base 10 equivalent: scientific notation

In scientific notation you have a mantissa and a exponent such that the value is $\mathrm{mantissa} \cdot 10^{\mathrm{exponent}}$.

In a computer floating point the mantissa is always the same size of significant digits (a double precision has around 16 digits) and the exponent is bounded.

Let's take a arbitrary format for our examples: a 4 digit mantissa and a 1 digit exponent

• 1 is then represented by $1.000 \cdot 10 ^ 0$

• the maximum value you can represent is $9.999 \cdot 10 ^ 9$

• the minimum absolute value you can represent in a normalized fashion (first digit is not 0) is $1.000 \cdot 10 ^ {-9}$

• $2\,147\,483\,648$ is most closely represented by $2.147 \cdot 10 ^ 9$ (note that this is actually $2\,147\,000\,000$)

  Here you'll see that to represent that large number there is an absolute error of $483\,648$ or a relative error of $0.0002$.

A denormalized value would be when the first digit is 0 like with $0.001 \cdot 10 ^ {-9}$, here it isn't that significant but in binary the first digit of the mantissa is assumed to be $1$. So that first bullet point is false, denormalised is all values in $(0,B^{\mathrm{minExp}}) $and $(-B^{\mathrm{minExp}},0)$.

Our machine epsilon here is $0.001$ (going from 1 to the next representable value 1.001).

So for numbers with exponent $0$ there is an accuracy of $0.001$, while for numbers with exponent 5 there is a accuracy of $100$.