As mentioned in this thread here, this is more due to the electronics of how computers work. It's a lot easier to create an on-off switch then it is an on-off-middle switch. If you're really curious about this, I'd recommend reading about how a computer works at the electronic level, as just doing this on-off switch took a lot of time for people to make properly.
That being said, there are computers coming that work with more than one state, mainly in the form of quantum computers. They can have an on and off state, as well as a "both" state if I'm not mistaken. There are of course advantages such as potential speed boosts from computers like this (though mostly from the way they work physics wise), but then there are also physical limitations imposed as well, such as the fact that they need to be kept incredibly cold to work properly from what has been experimented with so far. Quantum computers are the "main focus" of how to solve the issue of Moore's Law currently though.
But long story short, yes, computers that "use a base higher than 2" could be faster and more effecient, and there are computers like that, but they aren't ready for the mass populations of the world to use yet.