Reductions are useful in studying computability not so much to prove that problems are computable (although that is also done), but to prove that problems are not computable.
Used to prove incomputability, a reduction proof is a particular kind of proof by contradiction. You take your problem P and a known incomputable problem X, and you show that if you had an algorithm that computed solutions to P you could use it to compute solutions to X. Since we already know that there there does not exist a method for computing solutions to X this proves that there cannot exist a method for computing solutions to P.
This isn't really "different" from the kind of reduction you're talking about in algebra; if you have an algebraic problem P and you show that a solution to P could be used to find a solution to X, and you already know that X has no solutions, then this would show that P has no solutions either. But you usually think about the notion of "problem" slightly differently in algebra, as well as about the way you use "reductions".
My experience in mathematics was that you're usually aiming to turn an equation you don't know how to solve in to a form that you do know how to solve, in order to find a particular solution (possibly with unknown constants). Whereas a "problem" in computer science is a specification for producing an output (usually yes/no) on instances of an infinite family of inputs.1 And for computability purposes you're interested not in finding any particular solution, but in showing whether or not there exists an algorithm that can compute the solution for any input.
Basically it has turned out that a large number of computational problems have been difficult to directly prove incomputable, but are relatively easy to reduce to known incomputable problems. Variants on the Halting Problem are particularly rich sources of undergraduate-level incomputability proofs. I'm not aware that this is done nearly so much in mathematics, hence the difference in emphasis.
1 Any problem that only has one instance (e.g. one specific algebraic formula, like 2x^8 - 9 = 83) is completely uninteresting from the point of view of computability theory, because finite problem series are all trivially computable by a stupid algorithm that simply has all the solutions hard-coded and immediately prints the one corresponding to its input.
My Theory of Computation lecturer liked to joke that the problem "does God exist?" is computable; the program that computes the solution is either a single-state DFA that immediately accepts, or a single-state DFA that immediately rejects. We just don't know which one it is!