2
$\begingroup$

Basically, I using an algorithm called 'miranda' to look at miRNA targets and it only runs on a single thread. It compares everything in one file against everything in another file, produces a file as an output and runs off of the command line in terminal. The process took roughly 20 hours to create the output file.

I was advised by my supervisor that if i split one of the files up into say 4 equally sized parts, and ran them in four separate terminal windows this would decrease the overall time it took for the process to be completed.

I found that when I was using a single terminal window, the process would take up about 100-120% of the CPU. However, when running four terminal windows, each individual process only takes between 30-40% of the CPU.

How much effect does splitting the file up like this have in the overall time it takes to run the process? Although I split it across four threads, will the effect only be an increase in speed of about 1.5 times?

$\endgroup$
2
$\begingroup$

What you describe (split job into parts, total CPU usage stays the same) indicates that your task is (or gets) I/O (or perhaps RAM) bound. Try splitting among several machines, not just several processes on the same machine. If the program you are using comes from somewhere else, see if there are newer versions, bug reports on performance, or perhaps tricks and suggestions, on its webpage or user sites.

Use the tools on your operating system to see what the bottleneck really is. As you don't say what operating system this is, I can't give concrete suggestions. Perhaps look into uxix.stackexchange.com, superuser.com, or serverfault.com (all part of the stackexhange network).

$\endgroup$
0
$\begingroup$

It sounds like your data set is very large. However, you might be able to significantly speed up the program if you read everything into memory before working on it. File I/O is very slow. Rather then running four terminal windows, still split the file up, but run multiple threads from the same program. You'll see decreased overhead.

If you're using C++, a good library to check out is OpenMP for threading. As vonbrand mentioned above, you could also split the program across multiple machines. If you have a cluster available, the best way to do this would still to run a single program but parallelize it. In C++, the library OpenMPI is useful for this.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.