I'm currently taking a numerical analysis class in college and we're covering floating point systems. For the most part, I have a good grasp on it. However, something I can't seem to visualize, and haven't seen any totally lucid explanations about after searching extensively, is spacing between floating point numbers. Also of note is that I'm talking about IEEE-754 here, but it applies to general systems too.
The Things I Do Understand:
- The area between $[-1,1]$ is a denormalized area.
- The areas after $1$ and less than $-1$ are where the normalized floating point numbers reside.
- The floating point numbers between perfect powers of the base are uniformly spaced, but the spacing varies from one perfect power of the base to another.
- The spacing between values between two perfect powers is proportional to the power on the left for positive numbers and the power on the right for negative numbers. (i.e. on a number line, the uniformly-spaced values between two low powers are closer together than between a higher power.)
What I'm Struggling to Understand
From my understanding, the machine epsilon $\epsilon$ is a fundamental unit of spacing with respect to the floating point number line. That is, between $[1,B]$ where $B$ is the base, all values are $\epsilon$ apart. Then, you can scale any arbitrary floating point number by that fundamental machine epsilon and that product is the uniform spacing for that floating point number's associated power range. Is this even a correct interpretation?
I also read that $\epsilon$ is an upper bound for relative error, so I'm not really sure how that fits into my explanation of it being an indivisible spacing unit.
One of the questions I haven't been able to answer is what the minimum and maximum spacing between two positive floating point numbers is. I can trick myself into thinking I understand why multiplying the x's associated $B^e \cdot \epsilon$, where x is an arbitrary floating point number and $e$ is that number's corresponding exponent, yields the upper bound on error and therefore spacing, so $B^e \cdot \epsilon$ would be the maximum spacing.
Minimum spacing truly boggles my mind right now, though. If the machine epsilon is the indivisible unit of spacing, then for example, how could we have more minimal spacing than between $1$ and $1 + \epsilon$? Wouldn't that just be left to the rounding rule used (if round-to-nearest, it would depend whether the number is closer to $1$ or $1 + \epsilon$, since it'll be rounded to one of those two).
Basically, if you could explain this in plain-english it would really help me get a solid understanding of what's going on at the number line level.