In addition to Nish's answer, let me recommend Simon Marlow's book on Parallel and Concurrent Programming in Haskell or his shorter tutorial. They answer your first question from Haskell's perspective, so they could be better suited for theoretically inclined readers (Haskell is a purely functional, lazy programming language that is much closer to Mathematics than other languages).
Quoting from there:
In many fields, the words parallel and concurrent are synonyms; not so
in programming, where they are used to describe fundamentally
A parallel program is one that uses a
multiplicity of computational hardware (e.g. multiple processor cores)
in order to perform computation more quickly. Different parts of the
computation are delegated to different processors that execute at the
same time (in parallel), so that results may be delivered earlier than
if the computation had been performed sequentially.
In contrast, concurrency is a program-structuring technique in which
there are multiple threads of control. Notionally the threads of
control execute "at the same time"; that is, the user sees their
effects interleaved. Whether they actually execute at the same time
or not is an implementation detail; a concurrent program can execute
on a single processor through interleaved execution, or on multiple
I recommend reading the rest in the tutorial (p.4), but let me quote some of the remainder of this section, as it connects both programming paradigms with quantitative and qualitative characteristics of programs, such as efficiency, modularity, and determinism.
While parallel programming is concerned only with efficiency,
concurrent programming is concerned with structuring a program that
needs to interact with multiple independent external agents (for
example the user, a database server, and some external clients).
Concurrency allows such programs to be modular; the thread that
interacts with the user is distinct from the thread that talks to the
database. In the absence of concurrency, such programs have to be
written with event loops and callbacks --- indeed, event loops and
callbacks are often used even when concurrency is available, because
in many languages concurrency is either too expensive, or too
difficult, to use.
The notion of "threads of control" does not make sense in a purely
functional program, because there are no effects to observe, and the
evaluation order is irrelevant. So concurrency is a structuring
technique for effectful code; in Haskell, that means code in the IO
A related distinction is between deterministic and nondeterministic
programming models. A deterministic programming model is one in which
each program can give only one result, whereas a nondeterministic
programming model admits programs that may have different results,
depending on some aspect of the execution. Concurrent programming
models are necessarily nondeterministic, because they must interact
with external agents that cause events at unpredictable times.
Nondeterminism has some notable drawbacks, however: programs become
signifficantly harder to test and reason about.
For parallel programming we would like to use deterministic
programming models if at all possible. Since the goal is just to
arrive at the answer more quickly, we would rather not make our
program harder to debug in the process. Deterministic parallel
programming is the best of both worlds: testing, debugging and
reasoning can be performed on the sequential program, but the program
runs faster when processors are added. Indeed, most computer
processors themselves implement deterministic parallelism in the form
of pipelining and multiple execution units.
While it is possible to do parallel programming using concurrency,
that is often a poor choice, because concurrency sacriffices
determinism. In Haskell, the parallel programming models are
deterministic. However, it is important to note that deterministic
programming models are not sufficient to express all kinds of parallel
algorithms; there are algorithms that depend on internal
nondeterminism, particularly problems that involve searching a
solution space. In Haskell, this class of algorithms is expressible
only using concurrency.