2
$\begingroup$

I notice occasionally in blogs or articles comparing different languages, algorithms, etc. that the author will divulge info about the processor used in the testing. Is this meaningful?

Shouldn't the relative performance differences be hardware-agnostic? That is, if Language A executes in twice the time as Language B, shouldn't that roughly persist no matter the hardware?

$\endgroup$
  • $\begingroup$ Yes, and no. See some related discussions here and here. $\endgroup$ – Raphael Apr 2 '15 at 7:17
  • $\begingroup$ @Raphael Very interesting material; thanks! $\endgroup$ – America Apr 2 '15 at 11:38
1
$\begingroup$

The answer to your question is much dependent on specifics.

I certainly agree with D.W. answer that for comparing performances of complex systems, you need at least that kind of information (hardware specs), if you are interested in software performance through blackbox measurements. If box A has CPU, memory and disks that are twice as fast as box B, while being otherwise identical, it is clear that the same program will run twice as fast on A than it does on B. So if you compare two programs, one running on A and the other on B, you have to be aware of that.

However, the information you get from blackbox testing is very limited. For one thing, you may have a faster CPU, but a slower disk. Then, it is much harder to compare two software variants,

Generally the observed performance depends on very many factors that are hard to identify, and hardware specs may be grossly insufficient. Among other factors you have the quality of the algorithms used, the quality of the programming, the quality of the compiler (optimizing or not), the bias of the benchmark used, and probably some more.

Then the techniques for comparing a system (say a word processor), for comparing language implementations and for comparing algorithms may be quite different, due to complexity and also to formal definibility.

My personal opinion is that comparison of algorithms should, as much as possible, not be hardware dependent. If possible, it should use formal cost analysis, but that is not always simple to achieve. Short of doing that, it should be based on benchmarking on a virtual machine with appropriate primitives, so that the virtual machine can count precisely the number of instructions of each type executed. Of course that leaves the implementation quality bias and the benchmark bias, but the process is formal enough to be disputable on objective grounds. And the results are sometimes surprising.

It is to be noted that I refer here to cost analysis, rather than asymptotic complexity analysis. Asymptotic complexity analysis is cost unit independent and can only measure computational cost scaling with input size, i.e. compare an algorithm with itself as input changes. But comparing two algorithms, or two algorithm implementations, requires defining common computational unit(s), which implies a precisely defined common computational framework (as discussed also in this answer). A common virtual machine seems the right way to do it.

Regarding more complex software, I know of cases where the benchmark bias plays an important role. I have a friend who participates in natural language parsing competitions, and his technology is clearly put at a disadvantage by the way the benchmarks are provided (though that is by no means intentional). Implicit assumption in benchmark creation can play an important role. And this has nothing to do with harware specs.

To conclude, hardware specs can be very important in some cases, and it is usually better to divulge it. But it is far from being the whole story, and should not be taken as such. And there are situations when it should be irrelevant and may just be an indication that no proper evaluation technique has been actually used for comparisons. Or at least, you should be wary of hasty conclusions regarding the technology used in the software and its quality.

$\endgroup$
2
$\begingroup$

Yes, hardware specs are relevant. No, relative performance differences are not hardware-agnostic.

Suppose I find that editing a certain movie on my laptop takes twice as long as printing a certain Word document on my laptop. Does this mean if I do those two tasks on my desktop computer, the movie editing task will also take twice as long as printing the Word document? No, not necessarily.

Maybe my desktop computer has a great GPU (graphics card) but my laptop doesn't have a GPU. The GPU can be expected to speed up the movie-editing task, but it won't speed up printing from Word. So, on my desktop it's even possible that the movie-editing task might be faster than the printing task.

This is one reason why hardware specs are relevant.

$\endgroup$
1
$\begingroup$

Performance comparisons are not hardware agnostic because performance does not scale proportionately for all applications across all computers.

Even if the relative performance of a particular program is believed to be independent of hardware reasonably likely to be used for the problem the program addresses, providing this information can help when attempting to reproduce results. Just as one would include compiler, libraries, runtime environment, data set, and other software aspects used in the test system even if one believed their impact was minor.

If one's belief about hardware independence is wrong or becomes wrong, not providing information about the hardware makes it more difficult to determine whether failure to reproduce results was a consequence of an improper run or of other factors.

Many hardware aspects can disproportionately impact performance.

If one algorithm uses a multi-target indirect branch where another uses conditional branches, a processor which uses a target predictor with one entry per jump instruction address will likely favor the latter program more than a processor with an indirect jump target predictor using some global history (allowing multiple targets per jump instruction).

The amount of memory available to the program can influence the frequency of garbage collection and this frequency can be linked to the algorithm. One algorithm might be significantly faster if memory capacity can be profligately wasted while another might take care (costing performance) to be thrifty with memory.

The size, indexing methods, and associaitivity of various levels of cache can influence performance differently for different algorithms: the active working set and conflict rate can differ. (Similarly, hardware prefetching and cache replacement policies can have different impacts on different algorithms.)

The execution width and pipeline depth of a processor can have dramatically different performance impacts on different algorithms. A scalar (one-wide) processor with a shallow pipeline favors reducing executed instruction count; a wide and deep processor decreases the cost of executing instructions but increases the cost of branch mispredictions and cache misses (more execution opportunities are lost).

The optimizing ability of compilers also varies among different hardware targets. An implementation that maps simply to a compiler's model of the hardware may perform artificially better (and a later compiler improvement may remove this artificial difference).

One factor that may not be well-recognized relates to the full use of system resources. Single-threaded programs run on an unloaded system may utilize cache capacity and memory bandwidth that would be shared if the system was running other tasks. Such a test would favor using these free resources for small performance benefits, clearly introducing a potential unintended bias in a comparison. (This is similar to the garbage collection/memory capacity consideration.)

Performance comparisons are also not merely about determining relative speed. Absolute runtimes are significant. A slower program may be preferred if the performance is good enough and other factors (e.g., ease of integrating with an existing system, generality of the algorithm, scaling) favor the slower program. In addition, the faster program may still be too slow for the considered use.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.