LLM Anatomy Atlas L'anatomie structurelle des grands modèles de langage à poids ouverts

Index des modèles

L'index complet des architectures couvertes par cet atlas. Triez n'importe quelle colonne, filtrez par laboratoire ou famille d'attention, et ouvrez un modèle pour consulter son profil structurel complet. Le nombre de paramètres désigne le total ; « Actifs » correspond aux paramètres sollicités par token pour les modèles épars.

80
GPT-2 OpenAI 1.24 1.24 1K MHA12:12 Dense
GPT-3 175B (paper) OpenAI 175 175 2K MHA96:96 Dense
LLaMA 1 7B Meta 6.74 6.74 2K MHA32:32 Dense
Llama 2 70B Meta 68.98 68.98 4K GQA64:8 Dense
Mistral 7B Mistral 7.24 7.24 32K GQA32:8 Dense
ChatGLM3 6B GLM 6.2 6.2 8K GQA32:2 Dense
DeepSeek-Coder 6.7B DeepSeek 6.7 6.7 16K MHA32:32 Dense
DeepSeek LLM 7B DeepSeek 5.9 5.9 4K MHA32:32 Dense
DeepSeekMoE 16B DeepSeek 16.2 2.6 4K MHA16:16 64e top6 +2shared
Qwen1.5 7B Qwen 6.6 6.6 32K MHA32:32 Dense
Mixtral 8x22B Mistral 140.6 39.1 64K GQA48:8 8e top2
DeepSeek-V2 DeepSeek 236 21 160K MLA 160e top6 +2shared
DeepSeek-V2 Lite DeepSeek 15.6 2.5 160K MLA 64e top6 +2shared
GLM-4 9B GLM 9.4 9.4 8K GQA32:2 Dense
Qwen2 72B Qwen 71.4 71.4 32K GQA64:8 Dense
Llama 3.1 70B Meta 70.55 70.55 128K GQA64:8 Dense
Llama 3.1 8B Meta 8.03 8.03 128K GQA32:8 Dense
Qwen2.5 72B Qwen 71.4 71.4 32K GQA64:8 Dense
Qwen2.5 7B Qwen 7.6 7.6 32K GQA28:4 Dense
DeepSeek-V3 DeepSeek 671 37 160K MLA 256e top8 +1shared
DeepSeek-V3 Base DeepSeek 671 37 160K MLA 256e top8 +1shared
DeepSeek-R1 DeepSeek 671 37 160K MLA 256e top8 +1shared
MiniMax-Text-01 MiniMax 456 45.9 10000K GQA64:8 32e top2
Moonlight 16B Kimi 15.8 2.8 8K MLA 64e top6 +2shared
Qwen3 0.6B Qwen 0.6 0.6 40K GQA16:8 Dense
Qwen3 235B A22B Qwen 234.6 21.7 256K GQA64:4 128e top8
Qwen3 30B A3B Qwen 30.4 3.2 256K GQA32:4 128e top8
Qwen3 32B Qwen 32.1 32.1 40K GQA64:8 Dense
Qwen3 8B Qwen 7.9 7.9 40K GQA32:8 Dense
MiniMax-M1 MiniMax 456 45.9 10000K GQA64:8 32e top2
GLM-4.5 GLM 360.4 32.7 128K GQA96:8 160e top8 +1shared
GLM-4.5-Air GLM 107.7 13 128K GQA96:8 128e top8 +1shared
Kimi K2 Base Kimi 1,040.2 32.8 128K MLA 384e top8 +1shared
Step-3 StepFun 321 38 64K MHA 48e top3 +1shared
DeepSeek-V3.1 DeepSeek 671 37 160K MLA 256e top8 +1shared
Hunyuan 7B Hunyuan 7.8 7.8 32K GQA32:8 Dense
GPT-OSS 120B OpenAI 116.8 5.1 128K GQA64:8 128e top4
GPT-OSS 20B OpenAI 20.9 3.6 128K GQA64:8 32e top4
DeepSeek-V3.2 DeepSeek 671 37 160K MLA 256e top8 +1shared
Kimi K2 0905 Kimi 1,040.2 32.8 256K MLA 384e top8 +1shared
Qwen3-Next 80B A3B Qwen 78.7 2.9 256K GQA16:2 512e top10 +1shared
GLM-4.6 GLM 360.4 32.7 198K GQA96:8 160e top8 +1shared
Kimi Linear 48B A3B Kimi 50 2.8 256K MLA 256e top8 +1shared
MiniMax-M2 MiniMax 228.3 10.6 192K GQA48:8 256e top8
GLM-4.7-Flash GLM 29.4 3.2 198K MLA 64e top4 +1shared
Qwen3-Coder-Next Qwen 78.7 2.9 256K GQA16:2 512e top10 +1shared
GLM-5 GLM 744 40 198K MLA 256e top8 +1shared
Kimi K2.5 Kimi 1,040.2 32.8 256K MLA 384e top8 +1shared
MiniMax-M2.5 MiniMax 228.3 10.6 192K GQA48:8 256e top8
Qwen3.5 122B A10B Qwen 119.9 7.6 256K GQA32:2 256e top8 +1shared
Qwen3.5 27B Qwen 23.7 23.7 256K GQA24:4 Dense
Qwen3.5 35B A3B Qwen 33.9 2.7 256K GQA16:2 256e top8 +1shared
Qwen3.5 397B A17B Qwen 392.6 13.6 256K GQA32:2 512e top10 +1shared
Qwen3.5 9B Qwen 7.9 7.9 256K GQA16:4 Dense
Qwen3.5 9B Base Qwen 7.9 7.9 256K GQA16:4 Dense
Step-3.5-Flash StepFun 196 11 256K MHA64:64 288e top8 +1shared
DeepSeek-V4-Flash DeepSeek 284 13 1024K MQA64:1 256e top6 +1shared
DeepSeek-V4-Flash Base DeepSeek 284 13 1024K MQA64:1 256e top6 +1shared
DeepSeek-V4-Pro DeepSeek 1,606.6 49 1024K MQA128:1 384e top6 +1shared
GLM-5.1 GLM 744 40 198K MLA 256e top8 +1shared
Hunyuan 3 Preview Hunyuan 296.3 19.9 256K GQA64:8 192e top8 +1shared
Hunyuan 3 Preview Base Hunyuan 296.3 19.9 256K GQA64:8 192e top8 +1shared
Kimi K2.6 Kimi 1,040.2 32.8 256K MLA 384e top8 +1shared
MiniMax-M2.7 MiniMax 228.3 10.6 200K GQA48:8 256e top8
Qwen3.6 27B Qwen 23.7 23.7 256K GQA24:4 Dense
Qwen3.6 35B A3B Qwen 33.9 2.7 256K GQA16:2 256e top8 +1shared
Hunyuan-TurboS 30B A3B Hunyuan 30.3 3.3 256K GQA32:4 128e top8 +1shared
Hunyuan-TurboS 7B Hunyuan 7.8 7.8 256K GQA32:8 Dense
Step-3.7-Flash StepFun 198 11 256K MHA 288e top8 +1shared
GLM-5.2 GLM 767.4 40.2 1024K MLA 256e top8 +1shared
MiniMax-M3 MiniMax 443.4 25.5 1024K GQA64:4 128e top4 +1shared
MiniMax-M3 MXFP8 MiniMax 443.4 25.5 1024K GQA64:4 128e top4 +1shared
Hunyuan 3 Hunyuan 296.3 19.9 256K GQA64:8 192e top8 +1shared
Kimi K3 Kimi 2,780 103 1024K MLA 896e top16 +2shared
GLM-5.3 GLM 767.4 40.2 1024K MLA 256e top8 +1shared
GLM-5.3-Flash GLM 328.9 13 1024K MLA 288e top8 +1shared
Hunyuan 4 Preview Hunyuan 770 49 1024K MLA 256e top8 +1shared
Qwen3.8 2.4T A95B Qwen 2,399.6 95 256K GQA64:4 512e top10 +1shared
Qwen3.8 27B Qwen 23.7 23.7 256K GQA24:4 Dense
Qwen3.8-Flash-Next Qwen 123.6 5.1 256K GQA24:2 512e top10 +1shared