Transformer Architecture Features Quiz

What you'll be asked

1. What is the primary role of the self-attention mechanism in a Transformer model?
2. In a Transformer encoder, what is the purpose of the feed-forward network applied after the multi-head attention layer?
3. Consider a Transformer model being used for machine translation. During the decoding phase, why is masked multi-head attention used in the decoder's self-attention layer?
4. What is the key advantage of using multi-head attention over single-head attention in a Transformer?
5. Imagine you are fine-tuning a pre-trained Transformer model (like BERT) for a specific downstream task, such as sentiment analysis. You have a relatively small labeled dataset. Which of the following strategies would be MOST effective for preventing catastrophic forgetting of the pre-trained knowledge while adapting to the new task?

Answers and explanations come after you submit.