I've been trying to wrap my head around machine numbers like the unit roundoff (u) and epsilon (e) in combination with the IEEE 754 standard. My textbook states some things that don't really make sense to me.

Unit roundoff according to my textbook is:

  • for single precision (mantissa is 23 bit): u = 6e-8
  • for double precision (mantissa is 52 bit): u = 2e-16

I've been trying to derive a formula for these results with two relations:

  • my textbook states: "In binary arithmetic with rounding we usually have e = 2*u"
  • e = 2^-n, n being the amount of mantissa bits

These combined results would then give: u = 2^-(n+1), again with n being the amount of mantissa bits. Checking this formule with the given results of u for different precisions:

for single: u = 2^-(23+1) = 5.96e-8, this result checks out. for double: u = 2^-(52+1) = 1.11e-16, this result doesn't check out.

Could someone please help me derive a correct formule for the unit roundoff, or point me to some mistakes I have been making? All help is appreciated.

  • $\begingroup$ Perhaps they are just rounding up? $\endgroup$ – Yuval Filmus Dec 9 '18 at 0:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.