That's a tough one. I don't know what could cause that.
It can be hard to know why we see the results we do, when working with neural nets. Often the best we can do is form several hypotheses, and then devise experiments to try to test those hypotheses.
Some possible explanations that you could try to test:
Perhaps it is noise / random chance
Perhaps the ...