Scaling laws

Compute interpretation

Empirical compute/data/model allocation rules for deciding how to spend dense training budgets.

Supporting reading cards

• Language Models are Unsupervised Multitask Learners (2019, hyperscale_dense_llm_training)
• Language Models are Few-Shot Learners (2020, hyperscale_dense_llm_training)
• Scaling Laws for Neural Language Models (2020, hyperscale_dense_llm_training)
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, sparse_memory_efficient_scaling)
• Training Compute-Optimal Large Language Models (2022, hyperscale_dense_llm_training)
• PaLM: Scaling Language Modeling with Pathways (2022, hyperscale_dense_llm_training)
• Gemini: A Family of Highly Capable Multimodal Models (2023, hyperscale_dense_llm_training)
• Textbooks Are All You Need (2023, hyperscale_dense_llm_training)
• A Survey of Large Language Models (2023, hyperscale_dense_llm_training)
• The Llama 3 Herd of Models (2024, hyperscale_dense_llm_training)
• Qwen3 Technical Report (2025, hyperscale_dense_llm_training)

Obsolete or less central under later compute

Track this only through linked reading cards; do not treat this method page as standalone evidence.