Batching and Streaming Quiz

What you'll be asked

1. What is the primary benefit of using streaming when deploying a GenAI model for user-facing applications?
2. When implementing continuous batching (also known as iteration-level batching) in an LLM inference engine, what happens when a sequence in the batch finishes generation?
3. Which of the following is a primary trade-off when increasing the batch size in a GenAI inference system?
4. In the context of KV cache management, what is the main advantage of using PagedAttention?
5. You are optimizing an LLM serving pipeline where requests have highly variable input and output lengths. Which strategy would most effectively minimize 'bubbles' (idle GPU time) in your computation?

Answers and explanations come after you submit.