Skip to main content
Commonmark migration
Source Link
  1. Your benchmark process is not alone on the machine. Other processes and interrupts can cause your process to be stalled non-deterministically¹.

    Your benchmark process is not alone on the machine. Other processes and interrupts can cause your process to be stalled non-deterministically¹.

    Therefore, if your benchmark is only a few seconds long (or less?), random events can dominate the measurement. You want to measure for long periods of time (so you get "average noise") on a machine where almost nothing else is running (so there's little noise) and compare with exactly the same system but the CPU (so the noise is the same).

  2. Flops figures in the literature (or ad leaflets) are often theoretical peak figures. They just look at the CPU construction and say, okay, ideally (no pipeline stalls, no context switches, no memory delay, ...) at this frequency (which a store-bought, flawed chip may not be able to attain) the FP-ALU can compute this many operations per second.

    This number is never attained. You can of course try to determine a "real" peak number by running different benchmarks and taking the maximum. Averages probably depend heavily on the benchmark.

  3. Flops are only one type of operation. What about all the others?

  4. Real computation performance depends on many factors that range from the architecture of the CPU to the design of the program being run on it. One notable example would be the length of the pipeline. All else being equal, a chip with a longer pipeline will outperform one with a shorter pipeline on long chains of operations without any conditional jumps -- and exactly the other way around if there are (many) such jumps!

  5. If not, what ways are useful to get an aproximate result of how fast is a CPU.

    I am not sure that there is a general measure for "CPU speed". Different CPUs excel at different things. Also, relative measures like "computation power" per Watt become increasingly relevant.

Therefore, if your benchmark is only a few seconds long (or less?), random events can dominate the measurement. You want to measure for long periods of time (so you get "average noise") on a machine where almost nothing else is running (so there's little noise) and compare with exactly the same system but the CPU (so the noise is the same).

  1. Flops figures in the literature (or ad leaflets) are often theoretical peak figures. They just look at the CPU construction and say, okay, ideally (no pipeline stalls, no context switches, no memory delay, ...) at this frequency (which a store-bought, flawed chip may not be able to attain) the FP-ALU can compute this many operations per second.

This number is never attained. You can of course try to determine a "real" peak number by running different benchmarks and taking the maximum. Averages probably depend heavily on the benchmark.

  1. Flops are only one type of operation. What about all the others?

  2. Real computation performance depends on many factors that range from the architecture of the CPU to the design of the program being run on it. One notable example would be the length of the pipeline. All else being equal, a chip with a longer pipeline will outperform one with a shorter pipeline on long chains of operations without any conditional jumps -- and exactly the other way around if there are (many) such jumps!

  3. If not, what ways are useful to get an aproximate result of how fast is a CPU.

I am not sure that there is a general measure for "CPU speed". Different CPUs excel at different things. Also, relative measures like "computation power" per Watt become increasingly relevant.

  1. Your benchmark process is not alone on the machine. Other processes and interrupts can cause your process to be stalled non-deterministically¹.

Therefore, if your benchmark is only a few seconds long (or less?), random events can dominate the measurement. You want to measure for long periods of time (so you get "average noise") on a machine where almost nothing else is running (so there's little noise) and compare with exactly the same system but the CPU (so the noise is the same).

  1. Flops figures in the literature (or ad leaflets) are often theoretical peak figures. They just look at the CPU construction and say, okay, ideally (no pipeline stalls, no context switches, no memory delay, ...) at this frequency (which a store-bought, flawed chip may not be able to attain) the FP-ALU can compute this many operations per second.

This number is never attained. You can of course try to determine a "real" peak number by running different benchmarks and taking the maximum. Averages probably depend heavily on the benchmark.

  1. Flops are only one type of operation. What about all the others?

  2. Real computation performance depends on many factors that range from the architecture of the CPU to the design of the program being run on it. One notable example would be the length of the pipeline. All else being equal, a chip with a longer pipeline will outperform one with a shorter pipeline on long chains of operations without any conditional jumps -- and exactly the other way around if there are (many) such jumps!

  3. If not, what ways are useful to get an aproximate result of how fast is a CPU.

I am not sure that there is a general measure for "CPU speed". Different CPUs excel at different things. Also, relative measures like "computation power" per Watt become increasingly relevant.

  1. Your benchmark process is not alone on the machine. Other processes and interrupts can cause your process to be stalled non-deterministically¹.

    Therefore, if your benchmark is only a few seconds long (or less?), random events can dominate the measurement. You want to measure for long periods of time (so you get "average noise") on a machine where almost nothing else is running (so there's little noise) and compare with exactly the same system but the CPU (so the noise is the same).

  2. Flops figures in the literature (or ad leaflets) are often theoretical peak figures. They just look at the CPU construction and say, okay, ideally (no pipeline stalls, no context switches, no memory delay, ...) at this frequency (which a store-bought, flawed chip may not be able to attain) the FP-ALU can compute this many operations per second.

    This number is never attained. You can of course try to determine a "real" peak number by running different benchmarks and taking the maximum. Averages probably depend heavily on the benchmark.

  3. Flops are only one type of operation. What about all the others?

  4. Real computation performance depends on many factors that range from the architecture of the CPU to the design of the program being run on it. One notable example would be the length of the pipeline. All else being equal, a chip with a longer pipeline will outperform one with a shorter pipeline on long chains of operations without any conditional jumps -- and exactly the other way around if there are (many) such jumps!

  5. If not, what ways are useful to get an aproximate result of how fast is a CPU.

    I am not sure that there is a general measure for "CPU speed". Different CPUs excel at different things. Also, relative measures like "computation power" per Watt become increasingly relevant.

Source Link
Raphael
  • 72.9k
  • 30
  • 181
  • 393

While benchmarking metal is mostly outside of the realms of computer science -- we usually don't want to worry about issues of materials and physics -- there are some general remarks here.

Tl;dr: benchmarks are inherently fickle.

  1. Your benchmark process is not alone on the machine. Other processes and interrupts can cause your process to be stalled non-deterministically¹.

Therefore, if your benchmark is only a few seconds long (or less?), random events can dominate the measurement. You want to measure for long periods of time (so you get "average noise") on a machine where almost nothing else is running (so there's little noise) and compare with exactly the same system but the CPU (so the noise is the same).

  1. Flops figures in the literature (or ad leaflets) are often theoretical peak figures. They just look at the CPU construction and say, okay, ideally (no pipeline stalls, no context switches, no memory delay, ...) at this frequency (which a store-bought, flawed chip may not be able to attain) the FP-ALU can compute this many operations per second.

This number is never attained. You can of course try to determine a "real" peak number by running different benchmarks and taking the maximum. Averages probably depend heavily on the benchmark.

  1. Flops are only one type of operation. What about all the others?

  2. Real computation performance depends on many factors that range from the architecture of the CPU to the design of the program being run on it. One notable example would be the length of the pipeline. All else being equal, a chip with a longer pipeline will outperform one with a shorter pipeline on long chains of operations without any conditional jumps -- and exactly the other way around if there are (many) such jumps!

  3. If not, what ways are useful to get an aproximate result of how fast is a CPU.

I am not sure that there is a general measure for "CPU speed". Different CPUs excel at different things. Also, relative measures like "computation power" per Watt become increasingly relevant.

That said, I'm sure books have been written about all these issues. I can't recommend one, but doing some literature research may pay off -- especially at your level, since it'll point you towards many other great things to learn!


  1. It's not really non-deterministally or random, of course. But when looking at a CPU as a black box, it might as well be.