I was just reading this paper: Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks Rahul Dey and Fathi M. Salem
It seems to me that perhaps the architecture of LSTMs and GRUs are overly complicated. And that the same problems could probably be solved with a simpler architecture.
I get the theory behind LSTMs and GRUs as they are kind of trying to model the short term memory. But really all the need to do is get rid of the gradient exploding problem of RNNs.
What is the latest research. Is there something simpler than a GRU?
Edit: Actually I found something called a MGU (minimal gated unit) which claims to be simpler. What's the latest?