Absolutely not. In fact, these are most useful when $B$ is undecidable.
Diagonalization was used to show the Halting problem undecidable, but most other problems were shown to be undecidable through a reduction:
- Suppose we have a subroutine solving $B$
- Show that we can use that subroutine to solve $A$.
- If $A$ is undecidable, then we have a contradiction, so we know $B$ must also be undecidable.
That said, the a proper Turing reduction should always halt. It's cheating if you say "I can use $B$ to make a program that either solves $A$ or runs forever". That's not useful in general.
(Although if you used $B$ to find a recognizer for $A$ where $A$ is not recursively-enumerable, you could show that $B$ is not decidable/recursively-enumerable, depending on your initial assumption about $B$.)
If $B$ is decidable, and you reduce $A$ to $B$, all you've done is create an algorithm deciding $A$. Which is useful, but you don't need a formal notion of reduction to do that.