From Wikipedia
SPMD (single program, multiple data) is a technique employed to achieve parallelism; it is a subcategory of MIMD. Tasks are split up and run simultaneously on multiple processors with different input in order to obtain results faster.
Does a program running on SPMD runs as either multiple processes or a process with multiple threads? If former, are the multiple processes runs the same program?
Wikipedia tries to compare SPMD with SIMD:
In SPMD, multiple autonomous processors simultaneously execute the same program at independent points, rather than in the lockstep that SIMD imposes on different data. With SPMD, tasks can be executed on general purpose CPUs; SIMD requires vector processors to manipulate data streams. Note that the two are not mutually exclusive.
What does "at independent points" mean? Does point means time? If yes, isn't "at independent points" contradict "simultaneously"?
Wikipedia also compares SPMD with SMP:
Unlike SPMD, shared memory multiprocessing, also called symmetric multiprocessing or SMP, presents the programmer with a common memory space and the possibility to parallelize execution by having the program take different paths on different processors. The program starts executing on one processor and the execution splits in a parallel region, which is started when parallel directives are encountered. In a parallel region, the processors execute a single program on different data. A typical example is the parallel DO loop, where different processors work on separate parts of the arrays involved in the loop. At the end of the loop, execution is synchronized, only one processor continues, and the others wait. The current standard interface for shared memory multiprocessing is OpenMP. It is usually implemented by lightweight processes, called threads.
From Section 6.3 of Computer Organization and Design, Fifth Edition: The Hardware/Software interface by David A. Patterson, John L. Hennessy:
programmers normally write a single program that runs on all processors of an MIMD computer, relying on conditional statements when different processors should execute different sections of code. This style is called Single Program Multiple Data (SPMD), but it is just the normal way to program a MIMD computer.
I don't quite understand how SPMD and SMP are different as stated in Wikipedia, and maybe some paraphrasing may help? In particular, Wikipedia says in SMP, a program can take different paths on different processors. In Patterson's book, in SPMD, a program can also run different sections of code on different processors.
Wikipedia says "The current standard interface for shared memory multiprocessing is OpenMP". Is multithreading by OpenMP or PThread on multiple processors SPMD or SMP?
Thanks.