Recently I came across the concepts like Attention and Transformer architecture. After studying the papers and many blogs, I understood the concepts, but recently one doubt is coming in my mind and that is related to Multi-headed attention.
In Multi-headed attention, there are different query, key and value layers (dense layers). In some blogs I have seen some illustrations which show that different heads learn to apply attention on different parts of a sentence(I am mainly concerned about attention in text data).

My question is, initially the weights of all dense layers are initialized randomly, so how does this mechanism ensure that different heads learn to apply different attention?
Isn't it more obvious that during applying back propagation, all heads will learn same thing, failing to satisfy the main purpose of applying multiple heads?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.