The idea is to use an algorithm whose running time is linear in the height, which is $O(\log n)$ in a red-black tree. As JarkkoL mentions, you can maintain a count of the number of children in every subtree without effecting the asymptotic running time of the usual operations (insert, delete and search). You then go down the tree. Suppose you're at a vertex $v$ whose two children have subtrees of sizes $L,R$. If $k = L+1$ then $v$ is the $k$th smallest. If $k < L$ then you descend to the left child. If $k > L$ then you descend to the right child and replace $k$ by $k-L-1$. This algorithm runs in time $O(\log n)$.