74
$\begingroup$

Normally in algorithms we do not care about comparison, addition, or subtraction of numbers -- we assume they run in time $O(1)$. For example, we assume this when we say that comparison-based sorting is $O(n\log n)$, but when numbers are too big to fit into registers, we normally represent them as arrays so basic operations require extra calculations per element.

Is there a proof showing that comparison of two numbers (or other primitive arithmetic functions) can be done in $O(1)$? If not why are we saying that comparison based sorting is $O(n\log n)$?


I encountered this problem when I answered a SO question and I realized that my algorithm is not $O(n)$ because sooner or later I should deal with big-int, also it wasn't pseudo polynomial time algorithm, it was $P$.

$\endgroup$
  • 3
    $\begingroup$ If you are going to count the complexity of comparing numbers, you should also write your complexity bounds in terms of the bit size of the input. So given $N$ $w$-bit numbers, the bit size of the input is $n=Nw$ and sorting can be done in $O(Nw \log N) = O(n \log n)$ time. $\endgroup$ – Sasho Nikolov Jun 9 '13 at 6:22
  • 2
    $\begingroup$ there are basically two "realms" or "regimes" of complexity study. generally $O(1)$ operations are assumed for "fixed-width" operations which is a reasonable approximation for most computer languages which have fixed-width number representations including for floating point eg 2-4 bytes (see eg IEEE standards). then there is "arbitrary precision arithmetic" where numbers have arbitrary size and there is more careful/precise study of the complexity of operations. the former context is more in applied analysis & the latter is more in theoretical/abstract analysis. $\endgroup$ – vzn Sep 14 '13 at 18:12
73
$\begingroup$

For people like me who study algorithms for a living, the 21st-century standard model of computation is the integer RAM. The model is intended to reflect the behavior of real computers more accurately than the Turing machine model. Real-world computers process multiple-bit integers in constant time using parallel hardware; not arbitrary integers, but (because word sizes grow steadily over time) not fixed size integers, either.

The model depends on a single parameter $w$, called the word size. Each memory address holds a single $w$-bit integer, or word. In this model, the input size $n$ is the number of words in the input, and the running time of an algorithm is the number of operations on words. Standard arithmetic operations (addition, subtraction, multiplication, integer division, remainder, comparison) and boolean operations (bitwise and, or, xor, shift, rotate) on words require $O(1)$ time by definition.

Formally, the word size $w$ is NOT a constant for purposes of analyzing algorithms in this model. To make the model consistent with intuition, we require $w \ge \log_2 n$, since otherwise we cannot even store the integer $n$ in a single word. Nevertheless, for most non-numerical algorithms, the running time is actually independent of $w$, because those algorithms don't care about the underlying binary representation of their input. Mergesort and heapsort both run in $O(n\log n)$ time; median-of-3-quicksort runs in $O(n^2)$ time in the worst case. One notable exception is binary radix sort, which runs in $O(nw)$ time.

Setting $w = \Theta(\log n)$ gives us the traditional logarithmic-cost RAM model. But some integer RAM algorithms are designed for larger word sizes, like the linear-time integer sorting algorithm of Andersson et al., which requires $w = \Omega(\log^{2+\varepsilon} n)$.

For many algorithms that arise in practice, the word size $w$ is simply not an issue, and we can (and do) fall back on the far simpler uniform-cost RAM model. The only serious difficulty comes from nested multiplication, which can be used to build very large integers very quickly. If we could perform arithmetic on arbitrary integers in constant time, we could solve any problem in PSPACE in polynomial time.

Update: I should also mention that there are exceptions to the "standard model", like Fürer's integer multiplication algorithm, which uses multitape Turing machines (or equivalently, the "bit RAM"), and most geometric algorithms, which are analyzed in a theoretically clean but idealized "real RAM" model.

Yes, this is a can of worms.

$\endgroup$
  • 3
    $\begingroup$ I know I'm just supposed to vote, but can't stop myself from saying it: This is the best answer. The trick is that (1) arithmetic operations are constant time by definition and that's OK because in theory you can choose any model, and (2) you should have some reasons for choosing a certain model, and this answer explains what they are. $\endgroup$ – rgrig May 4 '12 at 10:30
  • $\begingroup$ I agree with rgig, (Also I'm just supposed to vote), but a little problem is input size is not related to input numbers, e.g If I have an $n$ input my biggest number is $m$, and if I choose the computation model as the way I like, this causes to pseudo polynomial time algorithm became $P$, am I right? $\endgroup$ – user742 May 4 '12 at 11:03
  • 1
    $\begingroup$ If your input consists of numbers with more than $w$ bits, then to fit the model you have to split them into $w$-bit chunks, just like in real life. For example, if your input consists of $N$ integers between $0$ and $M$, then your true input size is $N\log_w M = (N\lg M)/(\lg w)$. Thus, pseudo-polynomial running times like $O(NM)$ time are still exponential in the input size when $M$ is big. $\endgroup$ – JeffE May 4 '12 at 11:48
  • $\begingroup$ Are there any algorithms analyzed in the Real RAM model that are not secretly "Order Type RAM" algorithms? I have never thought about it much, but can't quickly come up with an example that isn't. $\endgroup$ – Louis May 4 '12 at 19:45
  • 1
    $\begingroup$ @Louis: Yes, lots: Voronoi diagrams, Euclidean shortest paths, recursive cuttings, simplicial partition trees,.... But the best example is Gaussian elimination, which runs in $O(n^3)$ time on the real RAM model (or the unit-cost integer RAM, but needs $O(n^4)$ time on the integer RAM. $\endgroup$ – JeffE May 4 '12 at 21:32
23
$\begingroup$

It just depends on the context. When dealing with bit level complexity of algorithms, we do not say that the addition of two $n$ bits numbers is $O(1)$, we say it is $O(n)$. Similarly for multiplication etc.

$\endgroup$
  • $\begingroup$ From your referenced article: "can be measured in two different ways: one in terms of the integers being tested or multiplied, and one in terms of the number of binary digits (bits) in those integers", but this is not true, we should always measure by size of input. $\endgroup$ – user742 May 3 '12 at 9:04
  • 1
    $\begingroup$ @SaeedAmiri: it just depends on the encoding used. In the article, for instance, if the input is an integer $n$ specified using unary encoding, trial division will require only $\theta(n^{1/2})$. This is polynomial in the size of the input! Does this means that factoring by trial division is in $P$? No, the algorithm is pseudo-polynomial. Using the common binary encoding, you get an exponential algorithm, again in the size of the input. As stated, this happens because the number of bits in the input $n$ has become exponentially smaller changing its encoding. $\endgroup$ – Massimo Cafaro May 3 '12 at 11:45
  • $\begingroup$ By the way, pseudo-polynomial algorithms may actually be useful, if the order of magnitude of their parameters in actual instances is reasonably low. The most famous example is probably the pseudo-polynomial algorithm to solve the knapsack problem. $\endgroup$ – Massimo Cafaro May 3 '12 at 11:51
  • $\begingroup$ First I should mention that your referenced wiki page is not good because it doesn't have any good references, Also I don't know why you think I'm talking about pseudo-polynomial time algorithms, may be because input size normally is buttleneck in this cases? but I'm not talking about them, I'm talking mostly about problems which are in $P$ even by assumption of input size, like sorting, anyway because we can't cheating and say NPC problem is in $P$ I think we shouldn't say sorting is $O(n \log n)$ except we have a formal proof to ignore comparison. $\endgroup$ – user742 May 3 '12 at 12:01
  • $\begingroup$ I am discussing pseudo-polynomial algorithms to focus your attention on the size of the input, to show you that it can be misleading. Here is another example. You are given a natural number as input, say $n$, and the algorithm runs a loop in which it does $O(1)$ time operations for $n$ iterations. The complexity of this simple loop algorithm measured as a function of the input size, is $O(n) = O(2^{lg n})$. Since $lg n$ is the input size, the algorithm is exponential in the input size! Think about this. Now you may understand what I mean with "It just depends on the context". $\endgroup$ – Massimo Cafaro May 3 '12 at 12:41
16
$\begingroup$

To answer the question as stated: algorithmists just do it, fairly often, by using the RAM model. For sorting, in many cases, people even analyze the simpler comparison model, which I discuss a little more in the linked answer.

To answer the implicit question about why they do it: I would say that this model has pretty good predictive power for certain types of combinatorial algorithms, in which the numbers are all "small" and, on real machines, fit in registers.

To answer implicit follow-up about numerical algorithms: No, the plain old RAM model is not the standard here. Even Gaussian elimination can require some care. Typically, for rank computations, the Schwartz Lemma will enter (e.g., Section 5 here). Another canonical example is the analysis of the Ellipsoid Algorithm, which requires some care to analyze.

And finally: People have thought about string sorting before, even recently.

Update: The problem with this question is that "we" and "assume" are not so precisely specified. I would say that the people who work in the RAM model are not doing numerical algorithms or complexity theory (where determining the complexity of division was a celebrated result).

$\endgroup$
  • $\begingroup$ Hmmmm, seems it's an interesting answer .... $\endgroup$ – user742 May 3 '12 at 12:17
  • $\begingroup$ Is there a reason it doesn't completely answer the question? $\endgroup$ – Louis May 4 '12 at 9:14
7
$\begingroup$

I couldn't find any studies of this, but Kozen says in the introduction to "The Design and Analysis of Algorithms" that the $O(1)$ model "reflects experimental observation more accurately [than the log-cost model] for data of moderate size (since multiplication really does take one unit of time)." He also gives a reference to this paper as an example of how the $O(1)$ model can be abused.

This is absolutely not a legit evaluation (not least because it's Python), but here's some numbers from running python -mtimeit "$a * $b" for $a in $10^{\{1,2,...,66\}}$ and $b = 2*$a. (I stopped at 66 because that's when Python syntax stops accepting integer literals and I'd have to slightly switch my evaluation code, so I didn't. :p)

Each number is the mean of 10,000,000 loops, where it takes the best time of 3 runs in each loop. I'd do error bars or something but that'd be more effort. :p In any case, it looks pretty constant to me, even out to $10^{50}$ -- slightly surprising, since $\log_{10}(\tt{sys.maxint})$ is 43, which reinforces my suspicion that maybe this evaluation is particularly bogus and I should do it in C.

$\endgroup$
  • $\begingroup$ May be in work experience in this special case you are right (but I'm not sure:), but e.g take a look at this, seems it's $O(n)$ but it's not, it also is a practical problem. Also did you see any paper which saying sorting is $O(n \cdot \log n \cdot \log m)$? $\endgroup$ – user742 May 3 '12 at 9:15
7
$\begingroup$

You are right, in general we cannot assume they are $O(1)$.

Strictly speaking, if we want to sort an array with N numbers using comparisons, and the largest number is M, then on the worst case, each comparison may involve $O(\log M)$ comparisons on the bit level. And if our algorithm does $O (N \log N)$ comparisons, then its total complexity will be $O (N \log N \log M)$.

However, you will notice the difference only for very large values of $M$, that cannot be stored in a single register, as you can see from Dougal's experiment.

$\endgroup$
  • $\begingroup$ $O(\log m)$ not $O(\log n)$, I mean biggest number in the set is $m$. $\endgroup$ – user742 May 3 '12 at 9:01
  • $\begingroup$ Yes, but if there are N DIFFERENT numbers to sort, then the size of the largest is $O(log N)$ bits. $\endgroup$ – Erel Segal-Halevi May 3 '12 at 9:07
  • $\begingroup$ No it's not related to number of input, also if you want relate it to number of input I have $n$ input number and biggest one is $n^{n^n}$ $\endgroup$ – user742 May 3 '12 at 9:11
  • $\begingroup$ You are right, I corrected my answer. $\endgroup$ – Erel Segal-Halevi May 3 '12 at 13:01
4
$\begingroup$

I would say that we typically assume O(1) arithmetic operations because we're usually doing things in the context of 32-bit integers or 64-bit integers or IEEE 754 floating point numbers. O(1) is probably a pretty good approximation for that kind of arithmetic.

But in general, that's not true. In general you need an algorithm to perform addition, subtraction, multiplication and division. Boolos, Burgess and Jefferies' Computability and Logic springs to mind as a way to understand the proof(s) of that, in terms of a couple of different formal systems, Recursive Functions and Abacus Machines, at least, in my 4th Edition copy.

You can look at the lambda-calculus terms for subtraction and division with Church Numerals for an easy-to-see explanation of why those two operations aren't O(1). It's a bit harder to see for addition and multiplication and exponentiation, but it's there if you consider the form of Church Numerals themselves.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy