Is there such data type for low range float?

Question

For integers, we have unsigned integers to represent positive integers, including zero, and we have signed integers to represent negative and positive. There are always trade-offs between them.

For example, an 8-bit unsigned integer has a range value between [0, 255]; it can be cast into a signed integer version, hence the value has a range with an interval [-128, 127], making the maximum absolute value reduced by half.

This is the same for float32, for example, that is represented by IEEE754; the trade-off lies in the allocation between the mantissa bit and the exponent bit. Where the allocation for float32 is like the picture below:

If we allocate the mantissa with higher space, for example, 30- bits, leaving the exponent bit to the 1 bit, then we have a higher granularity of computation precision. But in trade-off, the lower maximum absolute range.

This is crucial for machine learning, data science, deep learning, or similar fields, where the min-max normalization with a range [0,1] or [-1,1] is commonly used. If we use a standard float and play with the normalization value, we are sure there is no out-of-bounds value (there is no data manipulation that leads to underflow by -1 and overflow by 1). We have unused information in the bit level. It's the most significant bit of the exponent bit!

So, let's say I define the ufloat32 type to represent an unsigned 32-bit float operation that has a range [0, 1]. We performed an illegal or out-of-bounds addition instruction like the example below:

ufloat32 a = 0.5312523213
ufloat32 b = 0.999999
a + b //returning overflow error

This is analogous to:

uint8 a = 250
uint8 b = 255
a + b // overflow

This makes me wonder if the vanishing/exploding gradient problem in deep learning actually comes from the practical rather than the theoretical due to loss of information at the bit level.

So, is there such a floating point standard that maximizes mantissa representation for precision granularity rather than scientific notation?

I imagine in the higher level of the numpy library in Python, there is such a dtype: np.ufloat64 and np.ifloat64.

Answer 1

You essentially are asking for a fixed point number data type, which should exist for most common programming languages (maybe not out of the box).

However, if your range is indeed just $[0, 1]$ or $[-1, 1]$, then a regular integer would suffice by simply treating the integer number as your mantissa.