Parallelism and sharding

Compute interpretation

Model, data, pipeline, and optimizer-state partitioning methods that make large models fit distributed accelerator clusters.

Supporting reading cards

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, multi_gpu_dense_training)
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour (2017, multi_gpu_dense_training)
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour with Batch Normalization (2018, multi_gpu_dense_training)
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (2018, tpu_accelerator_transformer_era)
• Mesh-TensorFlow: Deep Learning for Supercomputers (2018, tpu_accelerator_transformer_era)
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, hyperscale_dense_llm_training)
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (2019, hyperscale_dense_llm_training)
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, sparse_memory_efficient_scaling)
• PaLM: Scaling Language Modeling with Pathways (2022, hyperscale_dense_llm_training)
• Gemini: A Family of Highly Capable Multimodal Models (2023, hyperscale_dense_llm_training)
• The Llama 3 Herd of Models (2024, hyperscale_dense_llm_training)
• Kimi k1.5: Scaling Reinforcement Learning with LLMs (2025, inference_time_compute_post_training)
• Kimi K2: Open Agentic Intelligence (2025, sparse_memory_efficient_scaling)
• Kimi K2.5: Visual Agentic Intelligence (2026, inference_time_compute_post_training)

Obsolete or less central under later compute

Track this only through linked reading cards; do not treat this method page as standalone evidence.