There are $2^{\aleph_0}$ possible oracles, but when fixing an oracle $A\subseteq \Sigma^*$, there are only countably many Turing machines $M^A$ with access to the oracle $A$.
Remember that a Turing machine with access to some oracle is defined in the same manner as a normal Turing machine, but with an additional operation (they can query the oracle). They have a special tape on which they can write a query, and say, three additional special states $q_{ask},q_{yes},q_{no}$. Upon entering $q_{ask}$, if $s\in\Sigma^*$ is the content of the query tape, the machine enters either $q_{yes}/q_{no}$, depending on whether or not $s\in A$.
Nothing stops us from encoding such machines as strings, in the same way we encode regular Turing machines, hence we only have countably many such machines.