Many very different Turing complete computation model are physically realizable (up to considering infinity as standing for unboundedness). Thus that cannot the point for choosing a model.
The answer by @jkff is appropriate in remarking that the Turing
Machine is intended as a theoretical device for the mathematical purpose of studying computability and provability
(arising actually in the context of Hilbert's
Entscheidungsproblem). But it is not quite accurate in the reasons for
choosing a simple formalism.
Proving in principle the Halting problem is not that much harder with more advanced models. In fact, our "proofs" are often just construction of a
solution. We do not go much into the actual (very tedious) arguments that these
constructions are correct. But anyone who writes an interpreter for a
Turing complete language does as much as any construction a universal
machine. Well, C can be a bit intricate, and we might want to streamline it a
bit for such a purpose.
The importance of having a simple model resides much more in the use
that can be made of the model, than in establishing its properties
(such as the Halting Problem, to take the example given by @jkff).
Typically, great theorem are often theorems that can be expressed very
simply and are applicable to a wide range of problems. But they are
not necessarily theorems that are easy to prove.
In the case of TM, the importance of simplicity is because many
results are established by reducing the Halting Problem, or other TM
problems, to problems we are interested in (such as ambiguty of
Context-free languages), thus establishing inherent limitations to
solving these problems.
In fact, though very intuitive (which is probably the main reason for
its popularity), the TM model is often not simple enough for use in such
proofs. That is one reason for the importance of some other, even
simpler models, such as the Post Correspondence Problem, less
intuitive to analyse, but easier to use. But this is because these
computational models are often used to prove negative results
(which goes back to the original Entscheidungsproblem).
However, when we want to prove positive results, such as the existence
of an algorithm to solve some given problem, the TM is much too
simplistic a device. It is much easier to consider mode advanced
models such as the RAM computer, or an associative memory computer, or
one of many other models, or even simply one of the many programming
languages.
Then the TM model comes only as a reference point, in particular for
complexity analysis, given the complexity of reducing these models to
the TM model (usually polynomial).The simplicity of the TM model
lends then much credibility to complexity measures (as opposed, to take an extreme example, to the reductions of Lambda-calculus).
In other words, the TM model is often too simplistic for designing and
studying algorithms (positive results), and often too complex for
studying computability (negative results).
But it seems to be at about the right place to serve as a central link
to connect it all together, with the great advantage of being rather
intuitive.
Regarding physical analogies, there is no reason to choose one model
over another. Many Turing complete computation model are physically realizable (up to unboundedness for memory infinity), since
there is no reason to consider a computer together with its software as
less physical than a "naked" computer. After all, the software has a
physical representation, which is part of the programmed computer.
So, since all computation models are equivalent from that point of
view, we might as well chose one that is convenient for the
organization of knowledge.
