For a while, all classification tasks in natural language processing were based on simple RNN's, which operate in a very word-by-word order. Adding gating mechanisms increased ability to "look back", and the newer addition of context vectors which can train attention to different words during the task have made classification of text less about "left-to-right" reading and more about selective focusing.

However, I have never seen a seq2seq or any other natural language generation system (machine translation, image2seq, etc) which generates the desired sequential output not in sequential order. It seems this would be very powerful. Are there any examples of using attention not only in encoders, but also in decoders?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.