The notion you are looking for is that of critical pair (see the wikipedia entry for instance). The idea is to find the "most simple" term which allows both rewrite rules to be applied. In this case the rewrite rules are:
$$ f(f(x))\rightarrow x$$
$$ f(a) \rightarrow b$$
To find a critical pair, you look for the most general (in a precise sense) instance of the variables on the left-hand side which allow one term to be the (non-trivial) subterm of another. In this case, we can instantiate $x$ to $a$ and obtain
$$ a\leftarrow f(f(a)) \rightarrow f(b)$$
This is a most general instantiation: we can't apply both rules if $x$ isn't sent to $a$. The equations you gave imply $a = f(b)$. To obtain a confluent rewrite system, we therefore need to add one of two rules:
$$ a \rightarrow f(b)$$
$$ f(b)\rightarrow a $$
Note that either rule will not make more terms equal, as the equality already holds! We can choose either rule to add to our system, but it is tempting to add the second one, as it seems more "computational". Now you have made your critical pair harmless: $f(f(a))$ rewrites to $a$ regardless of our choice of rule applications.
But we are not done! We have a new critical pair:
$$ b\leftarrow f(f(b))\rightarrow f(a)$$
However that critical pair is harmless, as both sides reduce to $b$.
Now every critical pair is harmless, and so we have verified the property of local confluence. This by itself is not enough to have confluence! However in your case, you have the additional property of termination: no term has an infinite rewrite sequence. This, in combination with local confluence, implies confluence, thanks to Newman's lemma.
Note that this whole approach (find critical pairs, add rewrite rules, check for termination, repeat) can be made systematic, to attempt turning any rewrite system into a terminating one; it is often called Knuth-Bendix completion.