Computer Science Stack Exchange is a question and answer site for students, researchers and practitioners of computer science. It only takes a minute to sign up.
Sign up to join this community
When looking at concurrent programming, two terms are commonly used i.e. concurrent and parallel.
And some programming languages specifically claim support for parallel programming, such as Java.
Does this means parallel and concurrent programming are actually different?
Distinguish parallelism (using extra computational units to do more work per unit time) from concurrency (managing access to shared resources). Teach parallelism first because it is easier and helps establish a non- sequential mindset.
From " A Sophomoric∗ Introduction to Shared-Memory Parallelism and Concurrency" by Dan Grossman (version of November 16, 2013)
Conurrency and parallelism differ in the problems they solve and cause, but they are not independent.
Executing two tasks concurrently means that individual steps of both tasks are executed in an interleaved fashion. If you disregard parallelism, you can assume that only one statement is executed at any point in time, but you have (a priori) no guarantee which task gets to execute the next step.
This is useful in some regards:
Some of the main challenges are:
Executing two tasks in parallel means that statements are executed at the same time. This is mainly useful for:
Key challenges include:
See also this question for distinguishing parallel and distributed computing.
In addition to Nish's answer, let me recommend Simon Marlow's book on Parallel and Concurrent Programming in Haskell or his shorter tutorial. They answer your first question from Haskell's perspective, so they could be better suited for theoretically inclined readers (Haskell is a purely functional, lazy programming language that is much closer to Mathematics than other languages).
Quoting from there:
In many fields, the words parallel and concurrent are synonyms; not so in programming, where they are used to describe fundamentally different concepts.
A parallel program is one that uses a multiplicity of computational hardware (e.g. multiple processor cores) in order to perform computation more quickly. Different parts of the computation are delegated to different processors that execute at the same time (in parallel), so that results may be delivered earlier than if the computation had been performed sequentially.
In contrast, concurrency is a program-structuring technique in which there are multiple threads of control. Notionally the threads of control execute "at the same time"; that is, the user sees their effects interleaved. Whether they actually execute at the same time or not is an implementation detail; a concurrent program can execute on a single processor through interleaved execution, or on multiple physical processors.
I recommend reading the rest in the tutorial (p.4), but let me quote some of the remainder of this section, as it connects both programming paradigms with quantitative and qualitative characteristics of programs, such as efficiency, modularity, and determinism.
While parallel programming is concerned only with efficiency, concurrent programming is concerned with structuring a program that needs to interact with multiple independent external agents (for example the user, a database server, and some external clients). Concurrency allows such programs to be modular; the thread that interacts with the user is distinct from the thread that talks to the database. In the absence of concurrency, such programs have to be written with event loops and callbacks --- indeed, event loops and callbacks are often used even when concurrency is available, because in many languages concurrency is either too expensive, or too difficult, to use.
The notion of "threads of control" does not make sense in a purely functional program, because there are no effects to observe, and the evaluation order is irrelevant. So concurrency is a structuring technique for effectful code; in Haskell, that means code in the IO monad.
A related distinction is between deterministic and nondeterministic programming models. A deterministic programming model is one in which each program can give only one result, whereas a nondeterministic programming model admits programs that may have different results, depending on some aspect of the execution. Concurrent programming models are necessarily nondeterministic, because they must interact with external agents that cause events at unpredictable times. Nondeterminism has some notable drawbacks, however: programs become signifficantly harder to test and reason about.
For parallel programming we would like to use deterministic programming models if at all possible. Since the goal is just to arrive at the answer more quickly, we would rather not make our program harder to debug in the process. Deterministic parallel programming is the best of both worlds: testing, debugging and reasoning can be performed on the sequential program, but the program runs faster when processors are added. Indeed, most computer processors themselves implement deterministic parallelism in the form of pipelining and multiple execution units.
While it is possible to do parallel programming using concurrency, that is often a poor choice, because concurrency sacriffices determinism. In Haskell, the parallel programming models are deterministic. However, it is important to note that deterministic programming models are not sufficient to express all kinds of parallel algorithms; there are algorithms that depend on internal nondeterminism, particularly problems that involve searching a solution space. In Haskell, this class of algorithms is expressible only using concurrency.
A slightly idealised answer, perhaps...
Concurrency is a property of how a program is written. If a program is written using constructions like forks/joins, locks, transactions, atomic compare-and-swap operations, and so on, then it is concurrent.
Parallelism is a property of how a program executes. If a program executes on more than one computational unit simultaneously, then it is executing in parallel.
There are loads of answers on this, but it can be confusing. I like to think of it this way, and maybe it helps?:
Concurrent programming is code that does not care about the order of execution. Java is a poor language for concurrent programming, but there are libraries and frameworks to help. JavaScript is an excellent language for concurrent programming, and it's often difficult when you want to write something that isn't concurrent (e.g., if you want to force the order of execution). Concurrent programming is great for event-driven programming (where order of execution is determined by event listeners, like code running in your browser that acts when you click a button or type into a box).
An example would include creating a hundred HTTP requests. In NodeJS, the simplest solution is to open all 100 requests at once with a callback method, and when the responses come back, a method is executed each time. That's concurrent programming. In Ruby, the simplest (most common) solution is to open a request and handle the response, open the next request and handle the response, etc. For many requests, NodeJS is easier to do in a timely fashion, although you have to be careful to avoid hammering the server or maxing out your outbound connections (easy to do by mistake). You can write the Ruby in a concurrent way, but it's not how most Ruby code is written, and it hurts a little to do it.
Parallel programming is code that can be run simultaneously in multiple threads or processes. This allows you to optimise performance by running the code across multiple CPUs (often including multiple machines, as you might with something like Akka). Because NodeJS is not multi-threaded and there's no parallel execution, you don't have to worry about writing threadsafe code (and most JavaScript code I've seen is not threadsafe) . In Java, even though the language doesn't make concurrent programming the normal pattern, parallel programming is very much built in, and you do often have to worry about thread-safety. If you are writing a Website in Java, typically this will be run in a container that runs each request in a separate thread in the same memory, so anything shared across requests in memory (such as an in-memory cache or configuration) must be thread-safe.
Some of the above depends on the scope and boundaries you are talking about. I work on Websites. Most Java code I see is not concurrent programming. Sure, if you zoom out enough, the order that the customer requests come in is not important, but if you zoom in any further than that, the order that things are executed is dictated by the code. But the code is written so that the requests can execute in parallel with lots of shared objects that must be thread-safe.
Meanwhile, most JavaScript code I see is concurrent: it is written in way that the order of execution is unimportant at many levels. But it is not written to support parallel execution in shared memory. Sure, you can execute the same code in parallel across multiple processes, but the objects are not shared, so it is not parallel programming in any meaningful sense.
For additional reading, I do really like the illustrations in the top answer to this question here: https://www.quora.com/What-are-the-differences-between-parallel-concurrent-and-asynchronous-programming
