LLM Anatomy Atlas 開源大語言模型的結構解剖

論文溯源

本圖譜中的技術並非口耳相傳——每項都有原始文獻。下表彙整了語料中引用的注意力變體、正規化方案、位置編碼與 MoE 設計的原始論文。

模型Year主題ID相關模型
Attention Is All You Need 2017 transformer arXiv:1706.03762 MHA
Language Models are Few-Shot Learners (GPT-3) 2020 scaling arXiv:2005.14165 GPT-3 175B (paper)
Fast Transformer Decoding: One Write-Head is All You Need (MQA) 2019 attention arXiv:1911.02150 MQA
GQA: Training Generalized Multi-Query Transformer Models 2023 attention arXiv:2305.13245 GQA
RoFormer: Rotary Position Embedding (RoPE) 2021 position arXiv:2104.09864 RoPE
YaRN: Efficient Context Window Extension 2023 position arXiv:2309.00071 RoPE+YaRN
Root Mean Square Layer Normalization (RMSNorm) 2019 normalization arXiv:1910.07467 RMSNorm
GLU Variants Improve Transformer (SwiGLU) 2020 ffn arXiv:2002.05202 SwiGLU
LLaMA: Open and Efficient Foundation Language Models 2023 model arXiv:2302.13971 LLaMA 1 7B
Llama 2: Open Foundation and Fine-Tuned Chat Models 2023 model arXiv:2307.09288 Llama 2 70B
The Llama 3 Herd of Models 2024 model arXiv:2407.21783 Llama 3.1 70B Llama 3.1 8B
Mistral 7B 2023 model arXiv:2310.06825 Mistral 7B
Mixtral of Experts 2024 moe arXiv:2401.04088 Mixtral 8x22B
Switch Transformers: Scaling to Trillion Parameter Models 2021 moe arXiv:2101.03961
DeepSeekMoE: Towards Ultimate Expert Specialization 2024 moe arXiv:2401.06066 DeepSeekMoE 16B
DeepSeek-V2: A Strong, Economical, and Efficient MoE Language Model (MLA) 2024 attention arXiv:2405.04434 MLA DeepSeek-V2
DeepSeek-V3 Technical Report 2024 model arXiv:2412.19437 DeepSeek-V3 DeepSeek-V3 Base
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL 2025 model arXiv:2501.12948 DeepSeek-R1
Qwen2.5 Technical Report 2024 model arXiv:2412.15115 Qwen2.5 7B Qwen2.5 72B
Qwen3 Technical Report 2025 model arXiv:2505.09388 Qwen3 8B Qwen3 32B
GLM: General Language Model Pretraining with Autoregressive Blank Infilling 2021 model arXiv:2106.16239 GLM-4 9B
MiniMax-01: Scaling Foundation Models with Lightning Attention 2025 model arXiv:2501.08313 MiniMax-Text-01
Native Sparse Attention: Hardware-Aligned and Natively Trainable 2025 attention arXiv:2502.11089
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness 2022 efficiency arXiv:2205.14135