I'm doing research work for my last year in high school. My work is about processors and for the experimental part i've coded an app that can mesure how many Floating Point Operation can a processor do in one second. My problem is that i get really different results depending when I execute my benchmark, I can get 5,6 GFlops and then 5 seconds later I can get 1,6 GFlops. How is this possible? Isn't measuring a processor aproximate speed with how many floating point operations per second can do useful? If not, what ways are useful to get an aproximate result of how fast is a CPU.
$\begingroup$
$\endgroup$
3
-
1$\begingroup$ Many CPUs slow down when the processor gets hot. Yes, five seconds later the processor can be hot, and the processor slow. $\endgroup$– gnasher729Commented May 4, 2018 at 7:24
-
$\begingroup$ @Raphael As I cannot really vote for open, I'll just mention here that IMO bench-marking in general is important in e.g. practical evaluation of algorithms. Bench-marking a processor is more of a grey area, but on-topic, IMO. Oh and those three comments of yours look like an answer to me (for the 'why does this happen' question, at least), perhaps you should make it one. $\endgroup$– Discrete lizard ♦Commented May 4, 2018 at 9:54
-
$\begingroup$ Maybe reading a little on what goes into a CPU can help paint the picture you need. $\endgroup$– RaphaelCommented May 7, 2018 at 14:45
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
While benchmarking metal is mostly outside of the realms of computer science -- we usually don't want to worry about issues of materials and physics -- there are some general remarks here.
Tl;dr: benchmarks are inherently fickle.
Your benchmark process is not alone on the machine. Other processes and interrupts can cause your process to be stalled non-deterministically¹.
Therefore, if your benchmark is only a few seconds long (or less?), random events can dominate the measurement. You want to measure for long periods of time (so you get "average noise") on a machine where almost nothing else is running (so there's little noise) and compare with exactly the same system but the CPU (so the noise is the same).
Flops figures in the literature (or ad leaflets) are often theoretical peak figures. They just look at the CPU construction and say, okay, ideally (no pipeline stalls, no context switches, no memory delay, ...) at this frequency (which a store-bought, flawed chip may not be able to attain) the FP-ALU can compute this many operations per second.
This number is never attained. You can of course try to determine a "real" peak number by running different benchmarks and taking the maximum. Averages probably depend heavily on the benchmark.
Flops are only one type of operation. What about all the others?
Real computation performance depends on many factors that range from the architecture of the CPU to the design of the program being run on it. One notable example would be the length of the pipeline. All else being equal, a chip with a longer pipeline will outperform one with a shorter pipeline on long chains of operations without any conditional jumps -- and exactly the other way around if there are (many) such jumps!
-
If not, what ways are useful to get an aproximate result of how fast is a CPU.
I am not sure that there is a general measure for "CPU speed". Different CPUs excel at different things. Also, relative measures like "computation power" per Watt become increasingly relevant.
That said, I'm sure books have been written about all these issues. I can't recommend one, but doing some literature research may pay off -- especially at your level, since it'll point you towards many other great things to learn!
- It's not really non-deterministally or random, of course. But when looking at a CPU as a black box, it might as well be.
-
$\begingroup$ PS: All of those points, and more, apply to runtime measurements of algorithms. Which is why we usually don't do those in analysis of algorithms, if we can avoid it at all, but prefer combinatoric cost measures. Meaningful runtime experiments are possible, but tough; cf. McGeoch's work. $\endgroup$– RaphaelCommented May 7, 2018 at 20:48