Computer Science Stack Exchange is a question and answer site for students, researchers and practitioners of computer science. It only takes a minute to sign up.
Sign up to join this community
Parallel computing is not new but it is becoming common now a days. This is essentially driven by the need of the users (they need to process more data now) and also because of physical limitations on chip designing e.g power dissipation limits forcing to manufacturers to move to multicore CPUs.
My question is how is the parallel computing different now from what they used to do 30 years back?
30 years ago research into cache-coherent shared memory was just beginning. There were machines where you could send messages between processors. By 1983 there were a few (room sized) machines where processors could communicate through a shared memory interface (for example, the Cray X-MP), but they didn't have cache coherence. Most communication between processors in the early to mid '80s was done using message passing APIs. Today almost everyone uses the MPI standard for message passing APIs. The MPI standardization process started in 1992.
Cache-coherent shared memory multiprocessors and shared memory programming (for performance (as opposed to multi-threaded programming, which can be a convenient way to organize programs that need to interact asynchronously with the outside world)) started becoming somewhat common in the 1990s for systems with 10s of processors. The OpenMP standard for specifying parallel computation in Fortran or C was first published in 1997. Today (2013) you can buy a shared-memory machine with 8 hardware contexts for a few hundred dollars, and you can buy a shared-memory machine with 10s of hardware contexts for a few thousand dollars.
Hardware offload (to GPUs) started becoming popular when NVidia released the CUDA SDK for their GeForce GPUs in 2007. The computing model with hardware offload is heterogeneous and in flux (currently some computing is done on the main processor, which tends to be shared-memory, then data gets copied back and forth between the processor and the GPU, some computing is done in the many vector lanes on the GPU, which can share some memory, but also have some private memories, and then the results are copied back to the main processor.) It appears as if most vendors are trying to come up with approaches to make the movement between cpu and gpu more streamlined.
Meanwhile in the last 30 years the advent of techniques to dramatically improve single-thread performance have also internally leveraged a great deal of parallelism. Deep pipelining, superscalar and branch speculation techniques allow tens of instructions from a single thread to be in flight simultaneously. Each CPU in the Cray X-MP in 1983 could complete a theoretical peak of 200MFLOPS with a 105MHz clock, whereas a single core today might be able to do 12 GFLOPS with a 3GHz clock. So, many computations that would have needed to be hand-parallelized 30 years ago do not need to be explicitly parallelized today.
a main difference is that the hardware itself is highly parallelized whereas decades ago parallelism was more "simulated" at the OS level with multitasking on hardware that was not inherently parallel. so there were many processes for different users on a multiuser system, but the parallelism happened through context switching. for many years there was significant research into optimizing context switching techniques etc, optimizing single core CPUs to minimize context switching overhead, etc.
now a CPU generally inherently consists of multicore technology whereas pre mid 2000s most CPUs were single core. now there is a new Moores-law like scaling increase in CPU cores. a useful scientific ref that is helpful in conceptualizing and understanding the historical "paradigm shift" in parallel processing wrt CPUs & other hardware is The landscape of parallel computing, the view from Berkeley, 2006
