LLM Anatomy Atlas The structural anatomy of open-weight large language models

Source papers

Techniques in this atlas are not folklore — each has a primary publication. The list below gathers the papers behind the attention variants, normalization schemes, position encodings and MoE designs referenced across the corpus.

ModelYearTopicIDSee models
Attention Is All You Need 2017 transformer arXiv:1706.03762 MHA
Language Models are Few-Shot Learners (GPT-3) 2020 scaling arXiv:2005.14165 GPT-3 175B (paper)
Fast Transformer Decoding: One Write-Head is All You Need (MQA) 2019 attention arXiv:1911.02150 MQA
GQA: Training Generalized Multi-Query Transformer Models 2023 attention arXiv:2305.13245 GQA
RoFormer: Rotary Position Embedding (RoPE) 2021 position arXiv:2104.09864 RoPE
YaRN: Efficient Context Window Extension 2023 position arXiv:2309.00071 RoPE+YaRN
Root Mean Square Layer Normalization (RMSNorm) 2019 normalization arXiv:1910.07467 RMSNorm
GLU Variants Improve Transformer (SwiGLU) 2020 ffn arXiv:2002.05202 SwiGLU
LLaMA: Open and Efficient Foundation Language Models 2023 model arXiv:2302.13971 LLaMA 1 7B
Llama 2: Open Foundation and Fine-Tuned Chat Models 2023 model arXiv:2307.09288 Llama 2 70B
The Llama 3 Herd of Models 2024 model arXiv:2407.21783 Llama 3.1 70B Llama 3.1 8B
Mistral 7B 2023 model arXiv:2310.06825 Mistral 7B
Mixtral of Experts 2024 moe arXiv:2401.04088 Mixtral 8x22B
Switch Transformers: Scaling to Trillion Parameter Models 2021 moe arXiv:2101.03961
DeepSeekMoE: Towards Ultimate Expert Specialization 2024 moe arXiv:2401.06066 DeepSeekMoE 16B
DeepSeek-V2: A Strong, Economical, and Efficient MoE Language Model (MLA) 2024 attention arXiv:2405.04434 MLA DeepSeek-V2
DeepSeek-V3 Technical Report 2024 model arXiv:2412.19437 DeepSeek-V3 DeepSeek-V3 Base
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL 2025 model arXiv:2501.12948 DeepSeek-R1
Qwen2.5 Technical Report 2024 model arXiv:2412.15115 Qwen2.5 7B Qwen2.5 72B
Qwen3 Technical Report 2025 model arXiv:2505.09388 Qwen3 8B Qwen3 32B
GLM: General Language Model Pretraining with Autoregressive Blank Infilling 2021 model arXiv:2106.16239 GLM-4 9B
MiniMax-01: Scaling Foundation Models with Lightning Attention 2025 model arXiv:2501.08313 MiniMax-Text-01
Native Sparse Attention: Hardware-Aligned and Natively Trainable 2025 attention arXiv:2502.11089
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness 2022 efficiency arXiv:2205.14135