This is a soft, and probably also an opinion based question.

Suppose I am writing a library in C++ for the purposes of working with financial data. My goals are:

  1. Since I work with big data, I want to make it as fast as possible by using parallel computing principle inside the functions.
  2. I want my colleagues to use this library without understanding/tweaking the parallelism underneath the functions.

The resulting questions are:

  1. Is this a good idea to make parallelism the default behaviour? For instance sum(vector) to parallelise the summation without asking the user.
  2. If yes, is there any rule of thumb that would cover the default behaviour of the task split between the processors? I.e. how many processors I should use?

Thanks in advance.


2 Answers 2


Parallelism helps with response time (the time it takes to complete one run of the algorithm), but it hurts throughput (the number of times you can run the algorithm per unit of time, if you want to run a large number of instances). It hurts throughput because synchronization between the threads itself costs time. The least expensive way to parallelize “run this algorithm once on each of these N inputs” is usually to run each input in full on a single processor. (There are exceptions, in particular if there are partial results that can be shared between runs.)

So if you expect that some users will want to do other things with their computer while they're running your software, there needs to be a way to limit how parallel it goes. “Other things” includes both running instances of your software, and running different software. In practice, this can usually be handled at the level of the operating system, but it isn't always convenient, especially if your software is chained with other software for which different parallelism settings are optimal.

Another reason why users might want to limit parallelism is if it causes a bottleneck in a factor other than running time, for example in memory usage. If each thread has a non-negligible memory cost, users may run out of memory before they run on processors. You can't predict the ratio of memory to processor count, so you can't predict what range will work out.

Furthermore, parallelism hurts for small instances: it takes time to instantiate threads and let them synchronize. Just because your software is intended for “big data” doesn't mean that every run will be on a large instance.

The main conclusion of what I've written here is that parallelism definitely needs to be tunable. More is not always better and there is no one-size-fits-all. This doesn't mean that maximum parallelism isn't a good default, just that it has to be configurable. But there is a reason not to make maximum parallelism the default: if it hits a threshold for another resource such as memory, it can make the program unusable. So in most cases, the default should be to use a single CPU, with an easy way to configure the number of CPUs or to use all available CPUs.

It's typically different with GPUs because they don't have much resource contention (GPUs don't have shared RAM) and if you're using them for computation, you're usually doing a single thing at a time. There, it makes sense for maximum parallelism to be the default. But it should still be tunable, in particular for users who want to run multiple parallel instances.


Have fun. If you use multiple threads you need to make sure that the overhead doesn’t outweigh the savings. The number of processors to use depends on the number of available processors, their properties (hyperthreading, performance vs. power saving), and the usage of the machine including the OS. And making one task fast may have negative side effects on the machine as a whole. Summary: It’s difficult.

Set your priorities: First make it work. Second make it maintainable. Third if required measure to find performance problems and fix them by using code that is not stupid (in the last ten years 99% of my performance problems were caused by stupid code and fixed by using code that isn’t stupid), then by using good code, then by using optimised vector code. Using multiple threads should be your very last resort.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.