1
$\begingroup$

I was reading that a quantity $x$ is $0$ upt to numerical precision. What does this statement formally mean -- especially in the context of numerical methods or real computers.

I looked up in google what that means but nothing useful came up. Then I read the precision page in wikipedia but it didn't really help:

In computer science, the precision of a numerical quantity is a measure of the detail in which the quantity is expressed. This is usually measured in bits, but sometimes in decimal digits. It is related to precision in mathematics, which describes the number of digits that are used to express a value.

If something is exactly zero instead of a number like 1e-7, my guess is that it's so close to zero that the machine couldn't measure it. Is that not right?

What does that a quantity is some value up to numerical precision unambiguously?

PS: I was thinking this could go on SO but given that it's not directly about programming but yes about concrete real values reported by computer, I went for CS. If it belongs somewhere else feel free to comment or help me move it.

cross posted: https://www.quora.com/unanswered/What-does-it-mean-unambiguously-that-a-number-is-value-0-up-to-numerical-precision

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ It probably means that $x$, as calculated on a computer, has the floating point value of $0$ (or $-0$, there is a difference). $\endgroup$
    – Yuval Filmus
    Apr 7, 2022 at 21:16
  • 1
    $\begingroup$ Where did you read it? Who said it? $\endgroup$
    – user253751
    Apr 7, 2022 at 21:21
  • 1
    $\begingroup$ Where did you read that statement? Please update the question. $\endgroup$
    – John L.
    Apr 7, 2022 at 21:25
  • $\begingroup$ It is common to say the results of two measurements are the same up to (or within) the precision of measurement. In other words, the difference as a number is 0 up to some precision. $\endgroup$
    – John L.
    Apr 7, 2022 at 21:41

1 Answer 1

Reset to default
2
$\begingroup$

I would imagine that it means that given your number format (IEEE-754 single/double/quadruple/etc precision) then you could have numbers which are smaller than machine precision (less than machine epsilon) which are represented by 0 but aren't 0 in reality. Numbers are logarithmically spaced by the machine epsilon value (for double precision it's like 1e-16 or $2^{-53}$) and you can't represent things with finer precision using that many bytes so it's just 0.

In most cases when you have some numerical method you have a tolerance for the method to converge and in some cases it's just machine epsilon. So people may say the error or whatever goes to 0 but they really mean it went to a number that we can no longer represent.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.