A Turing machine is named after British mathematician Alan Turing; he first described them in his paper "On computable numbers, with an application to the Entscheidungsproblem". It serves as the mathematical basis of computation and allowed various notions of computability to be mathematically proven even before physical computers existed.
Using Turing machines we can reason about what algorithms are fundamentally unsolvable, which ones are solvable in a reasonable amount of time, and which ones are solvable but take an unreasonable amount of time. Here, "reasonable" means the algorithm goes through a number of steps which is polynomial in the length of the input. For example, being able to prove mathematically that it is very hard to calculate the two prime factors of a large composite number is the underpinning of the security of the internet, the RSA algorithm.
The basic Turing machine consists of a 'tape' divided into 'cells', where each cell contains either a symbol or a blank; and a read/write head which can read symbols from the tape, write symbols onto the tape and move left or right along the tape. The action of the head is given by a transition function, which basically has a bunch of rules like "if you are in state q1 and you read an X, replace the X with a Y, move the head one cell right, and change to state q2".
The Church-Turing thesis is what makes Turing machines useful: simply put, states that a function is algorithmically computable if and only if there is a Turing machine which can compute it.
A language is Turing complete if and only if it can be used to simulate any single-taped Turing machine. Due to the Church-Turing thesis, this means it can be used to compute any algorithmically-solvable function.
As an interesting aside, a machine with only the instruction "subtract and branch if less than or equal" (see one instruction set computer) is actually Turing complete: it has conditional branching and the ability to change arbitrary memory locations, which is all that is required.
I recommend Sipser's "Introduction to the Theory of Computation" for a grounding in this stuff.