Distributed Training Quiz

What you'll be asked

1. In the context of distributed machine learning, what is the primary purpose of 'Data Parallelism'?
2. When using synchronous SGD (Stochastic Gradient Descent) in a distributed setup, what is the main bottleneck that can occur?
3. What is the primary advantage of using a Parameter Server architecture over an All-Reduce architecture in distributed training?
4. Which of the following scenarios is most appropriate for using Model Parallelism instead of Data Parallelism?
5. In a Ring All-Reduce implementation, why is the communication cost independent of the number of worker nodes (N)?

Answers and explanations come after you submit.