Raphael's suggestion to use 2-shortest path is correct but more complex than necessary.
Your approach is unfeasible, the number of cycles is factorial in the size of input.
Here's a sketch of a solution in time $O(|V|+|E|)$.
Find a path $p_1, \dots p_n$ from $x$ to $y$ in $O(|V|+|E|)$. If no such path exists, answer $\textsf{NO}$. Otherwise, mark all the nodes in the path. That path is indeed unique if and only if no node in that path reaches a node that comes after it. Starting from the first node of the path $p_1$, pick a random edge going out from $p_1$ other than the one that connects it to $p_2$ and start a DFS from there. If you reach a marked node, answer $\textsf{NO}$, if you never reach a marked node, cut that branch from the graph and pick another random outgoing edge. Continue in a similar fashion until all outgoing edges are exhausted, then start again with the following node until you reach $p_n$.
If you finish the procedure without ever answering $\textsf{NO}$, answer $\textsf{YES}$. The entire procedure has the cost of a DFS, $O(|V|+|E|)$, therefore the overall time is $O(|V|+|E|)$.