In computability theory, computable functions are also called recursive functions. At least at first sight, they do not have anything in common with what you call "recursive" in day-to-day programming (i.e., functions that call themselfes).
What is the actual meaning of recursive in the context of computability? Why are those functions called "recursive"?
To put it in other words: What is the connection between the two meanings of "recursiveness"?
Define some basic functions:
zero function
$$ zero: \mathbb{N} \rightarrow \mathbb{N} : x \mapsto 0 $$
successor function
$$ succ: \mathbb{N} \rightarrow \mathbb{N} : x \mapsto x + 1 $$
projection function
$$p_i^n: \mathbb{N}^n \rightarrow \mathbb{N} : (x_1, x_2, \dots, x_n) \mapsto x_i $$
From now on I will use $\bar{x_n}$ to denote $(x_1, x_2, \dots, x_n)$
Define a composition:
Given functions
Construct the following function:
\begin{align} h: \mathbb{N}^k \rightarrow \mathbb{N}: \bar{x_k} \mapsto h(\bar{x_k}) = f ( & g_1(\bar{x_k}), g_2(\bar{x_k}), \dots, g_m(\bar{x_k}) ) \end{align}
Define primitive recursion:
Given functions
Construct the following (piecewise) function:
$$ h : \mathbb{N}^{k+1} \rightarrow \mathbb{N} : \\ (\bar{x_k}, y + 1) \mapsto \begin{cases} f(\bar{x_k}), & y + 1 = 0 \\ g (\bar{x_k}, y, h(\bar{x_k}, y)), & y + 1 > 0 \end{cases} $$
All functions that can be made using compositions and primitive recursion on basic functions, are called primitive recursive. It is called that way by definition. While a link with functions that call themselves exists, there's no need to try and link them with each other. You might consider recursion a homonym.
This definition and construction above was constructed by Gödel (a few other people were involved too) in an attempt to capture all functions that are computable i.e. there exists a Turing Machine for that function. Note that the concept of a Turing Machine was not yet described, or it was at least very vague.
(Un)fortunately, someone called Ackermann came along and defined the following function:
This function is computable, but there's no way to construct it using only the constructions above! (i.e. $Ack$ is not primitive recursive) This means that Gödel and his posse failed to capture all computable functions in their construction!
Gödel had to expand his class of functions so $Ack$ could be constructed. He did this by defining the following:
Unbounded minimisation
This last one may be hard to grasp, but it basically means that $g((x_1, x_2, \dots, x_k))$ is the smallest root of $f$ (if a root exists).
All functions that can be constructed with all the constructions defined above are called recursive. Again, the name recursive is just by definition, and it doesn't necessarily have correlation with functions that call themselves. Truly, consider it a homonym.
Recursive functions can be either partial recursive functions or total recursive functions. All partial recursive functions are total recursive functions. All primitive recursive functions are total. As an example of a partial recursive function that is not total, consider the minimisation of the successor function. The successor function doesn't have roots, so its minimisation is not defined. An example of a total recursive function (which uses minimisation) is $Ack$.
Now Gödel was able to construct the $Ack$ function as well with his expanded class of functions. As a matter of fact, every function that can be computed by a Turing machine, can be represented by using the constructions above and vice versa, every construction can be represented by a Turing machine.
If you're intrigued, you could try to make Gödel's class bigger. You can try to define the 'opposite' of unbounded minimisation. That is, unbounded maximisation i.e. the function that finds the biggest root. However, you may find that computing that function is hard (impossible). You can read into the Busy Beaver Problem, which tries to apply unbounded maximisation.
The founders of computability theory were mathematicians. They founded what is now called computability theory before there was any computers. What was the way mathematicians defined functions that could be computed? By recursive definitions!
So there were recursive function before there were any other model of computation like Turing machines or lambda calculus or register machines. So people referred to these function as recursive functions. The fact that they turned out to be exactly what Turing machines and other models can compute is a later event (mostly proven by Kleene).
We have the simple definition of a recursive function which is now called primitive recursive function. There were not general enough (e.g. Ackermann's function) so people developed more general notions like $\mu$-recursive functions and Herbrand-Gödel general recursive functions that did capture all computable functions (assuming the Church's thesis). Church claimed that his model of lambda calculus captured all computable functions. Many people, and in particular Gödel, were not convinced that these capture all functions that can be computed. Until Turing's analysis of computation and introduction of his machine model.
The name of the field used to recursion theory. However there has been a successful push in recent decades to change the name to something more appealing from recursion theory to something more computer sciency (vs. mathy). As a result the field is now called computability theory. However if you look at books, papers, conferences, etc. in the early decades they are called recursion theory and not computability theory. Even the title of Soare's own 1987 book (who was the main person behind the push to change the name to computability theory) is "Recursively Enumerable Sets and Degrees".
If you want to know more about the history a fun and good place to read about it is the first chapter of Classical Recursion Theory by Odifreddi.
Robert Soare wrote an essay (Wayback Machine) in 1995 about this issue. According to him, the term (general) recursive functions was coined by Gödel, who defined them using some sort of mutual recursion. The name stuck, though later on other equivalent definitions were found.
For more information, I recommend Soare's essay.
instead of putting a long comment decided to add an answer:
Because they are defined recursively, i.e "more complex functions are defined in terms of previously defined, simpler functions"
This kind of iterative or incremental procedure creates well-defined functions (in the mathematical sense)
This is the meaning of recursiveness in mathematical parlance. See below how this relates to recursion in programming parlance.
Compare this procedure with techniques and methods like (mathematical) induction which is also an example of recursiveness in mathematics.
Programming has a mathematical vein as well as an engineering one.
This (usualy constructive) procedure is also refered as "bootstrapping" in Operating Systems parlance.
However a runtime recursion of the same function (i.e caling itself during its runtime), since it must (hmm, should) happen on already computed values (or arguments), or in other words, in the part of the result set already computed, is also recursive in the above sense, i.e "defined w.r.t previously defined functions (and their values)"
Else is not well-defined, and leads to such things like Stack Overflow :))))
To give a further example from Operating Systems, a runtime recursion (calling itself) can be taken as the analog of an operating system rebooting after a certain update (e.g core update). Many OSes do the following procedure:
Auberon's beautiful answer demonstrates a procedure of this kind in more detail.
