LLM Anatomy Atlas The structural anatomy of open-weight large language models

Lineage by laboratory

Each laboratory's releases in chronological order, with the architectural signature of each generation. Read down a column to see a design lineage: what stayed constant (norm, position encoding, attention family) and what changed (context, sparsity, dtype).

DeepSeek · 13

2023-11 DeepSeek-Coder 6.7B · 2023-11 · 6.7B DeepSeek-Coder 6.7B 6.7B DeepSeek LLM 7B · 2023-11 · 5.9B DeepSeek LLM 7B 5.9B DeepSeekMoE 16B · 2024-01 · 16.2B DeepSeekMoE 16B 16.2B · MoE DeepSeek-V2 · 2024-05 · 236B DeepSeek-V2 236B · MoE DeepSeek-V2 Lite · 2024-05 · 15.6B DeepSeek-V2 Lite 15.6B · MoE DeepSeek-V3 · 2024-12 · 671B DeepSeek-V3 671B · MoE DeepSeek-V3 Base · 2024-12 · 671B DeepSeek-V3 Base 671B · MoE DeepSeek-R1 · 2025-01 · 671B DeepSeek-R1 671B · MoE DeepSeek-V3.1 · 2025-08 · 671B DeepSeek-V3.1 671B · MoE DeepSeek-V3.2 · 2025-09 · 671B DeepSeek-V3.2 671B · MoE DeepSeek-V4-Flash · 2026-04 · 284B DeepSeek-V4-Flash 284B · MoE DeepSeek-V4-Flash Base · 2026-04 · 284B DeepSeek-V4-Flash Base 284B · MoE DeepSeek-V4-Pro · 2026-04 · 1606.6B DeepSeek-V4-Pro 1606.6B · MoE 2026-04
ModelReleasedParamsActiveAttentionMoE
DeepSeek-Coder 6.7B2023-116.76.7MHADense
DeepSeek LLM 7B2023-115.95.9MHADense
DeepSeekMoE 16B2024-0116.22.6MHA64e top6 +2shared
DeepSeek-V22024-0523621MLA160e top6 +2shared
DeepSeek-V2 Lite2024-0515.62.5MLA64e top6 +2shared
DeepSeek-V32024-1267137MLA256e top8 +1shared
DeepSeek-V3 Base2024-1267137MLA256e top8 +1shared
DeepSeek-R12025-0167137MLA256e top8 +1shared
DeepSeek-V3.12025-0867137MLA256e top8 +1shared
DeepSeek-V3.22025-0967137MLA256e top8 +1shared
DeepSeek-V4-Flash2026-0428413MQA256e top6 +1shared
DeepSeek-V4-Flash Base2026-0428413MQA256e top6 +1shared
DeepSeek-V4-Pro2026-041606.649MQA384e top6 +1shared

GLM · 11

2023-10 ChatGLM3 6B · 2023-10 · 6.2B ChatGLM3 6B 6.2B GLM-4 9B · 2024-06 · 9.4B GLM-4 9B 9.4B GLM-4.5 · 2025-07 · 360.4B GLM-4.5 360.4B · MoE GLM-4.5-Air · 2025-07 · 107.7B GLM-4.5-Air 107.7B · MoE GLM-4.6 · 2025-10 · 360.4B GLM-4.6 360.4B · MoE GLM-4.7-Flash · 2026-01 · 29.4B GLM-4.7-Flash 29.4B · MoE GLM-5 · 2026-02 · 744B GLM-5 744B · MoE GLM-5.1 · 2026-04 · 744B GLM-5.1 744B · MoE GLM-5.2 · 2026-06 · 767.4B GLM-5.2 767.4B · MoE GLM-5.3 · 2026-08 · 767.4B GLM-5.3 767.4B · MoE GLM-5.3-Flash · 2026-08 · 328.9B GLM-5.3-Flash 328.9B · MoE 2026-08
ModelReleasedParamsActiveAttentionMoE
ChatGLM3 6B2023-106.26.2GQADense
GLM-4 9B2024-069.49.4GQADense
GLM-4.52025-07360.432.7GQA160e top8 +1shared
GLM-4.5-Air2025-07107.713GQA128e top8 +1shared
GLM-4.62025-10360.432.7GQA160e top8 +1shared
GLM-4.7-Flash2026-0129.43.2MLA64e top4 +1shared
GLM-52026-0274440MLA256e top8 +1shared
GLM-5.12026-0474440MLA256e top8 +1shared
GLM-5.22026-06767.440.2MLA256e top8 +1shared
GLM-5.32026-08767.440.2MLA256e top8 +1shared
GLM-5.3-Flash2026-08328.913MLA288e top8 +1shared

Hunyuan · 7

2025-08 Hunyuan 7B · 2025-08 · 7.8B Hunyuan 7B 7.8B Hunyuan 3 Preview · 2026-04 · 296.3B Hunyuan 3 Preview 296.3B · MoE Hunyuan 3 Preview Base · 2026-04 · 296.3B Hunyuan 3 Preview Base 296.3B · MoE Hunyuan-TurboS 30B A3B · 2026-05 · 30.3B Hunyuan-TurboS 30B A3B 30.3B · MoE Hunyuan-TurboS 7B · 2026-05 · 7.8B Hunyuan-TurboS 7B 7.8B Hunyuan 3 · 2026-07 · 296.3B Hunyuan 3 296.3B · MoE Hunyuan 4 Preview · 2026-08 · 770B Hunyuan 4 Preview 770B · MoE 2026-08
ModelReleasedParamsActiveAttentionMoE
Hunyuan 7B2025-087.87.8GQADense
Hunyuan 3 Preview2026-04296.319.9GQA192e top8 +1shared
Hunyuan 3 Preview Base2026-04296.319.9GQA192e top8 +1shared
Hunyuan-TurboS 30B A3B2026-0530.33.3GQA128e top8 +1shared
Hunyuan-TurboS 7B2026-057.87.8GQADense
Hunyuan 32026-07296.319.9GQA192e top8 +1shared
Hunyuan 4 Preview2026-0877049MLA256e top8 +1shared

Kimi · 7

2025-02 Moonlight 16B · 2025-02 · 15.8B Moonlight 16B 15.8B · MoE Kimi K2 Base · 2025-07 · 1040.2B Kimi K2 Base 1040.2B · MoE Kimi K2 0905 · 2025-09 · 1040.2B Kimi K2 0905 1040.2B · MoE Kimi Linear 48B A3B · 2025-10 · 50B Kimi Linear 48B A3B 50B · MoE Kimi K2.5 · 2026-02 · 1040.2B Kimi K2.5 1040.2B · MoE Kimi K2.6 · 2026-04 · 1040.2B Kimi K2.6 1040.2B · MoE Kimi K3 · 2026-07 · 2780B Kimi K3 2780B · MoE 2026-07
ModelReleasedParamsActiveAttentionMoE
Moonlight 16B2025-0215.82.8MLA64e top6 +2shared
Kimi K2 Base2025-071040.232.8MLA384e top8 +1shared
Kimi K2 09052025-091040.232.8MLA384e top8 +1shared
Kimi Linear 48B A3B2025-10502.8MLA256e top8 +1shared
Kimi K2.52026-021040.232.8MLA384e top8 +1shared
Kimi K2.62026-041040.232.8MLA384e top8 +1shared
Kimi K32026-072780103MLA896e top16 +2shared

Meta · 4

2023-02 LLaMA 1 7B · 2023-02 · 6.74B LLaMA 1 7B 6.74B Llama 2 70B · 2023-07 · 68.98B Llama 2 70B 68.98B Llama 3.1 70B · 2024-07 · 70.55B Llama 3.1 70B 70.55B Llama 3.1 8B · 2024-07 · 8.03B Llama 3.1 8B 8.03B 2024-07
ModelReleasedParamsActiveAttentionMoE
LLaMA 1 7B2023-026.746.74MHADense
Llama 2 70B2023-0768.9868.98GQADense
Llama 3.1 70B2024-0770.5570.55GQADense
Llama 3.1 8B2024-078.038.03GQADense

MiniMax · 7

2025-01 MiniMax-Text-01 · 2025-01 · 456B MiniMax-Text-01 456B · MoE MiniMax-M1 · 2025-06 · 456B MiniMax-M1 456B · MoE MiniMax-M2 · 2025-10 · 228.3B MiniMax-M2 228.3B · MoE MiniMax-M2.5 · 2026-02 · 228.3B MiniMax-M2.5 228.3B · MoE MiniMax-M2.7 · 2026-04 · 228.3B MiniMax-M2.7 228.3B · MoE MiniMax-M3 · 2026-06 · 443.4B MiniMax-M3 443.4B · MoE MiniMax-M3 MXFP8 · 2026-06 · 443.4B MiniMax-M3 MXFP8 443.4B · MoE 2026-06
ModelReleasedParamsActiveAttentionMoE
MiniMax-Text-012025-0145645.9GQA32e top2
MiniMax-M12025-0645645.9GQA32e top2
MiniMax-M22025-10228.310.6GQA256e top8
MiniMax-M2.52026-02228.310.6GQA256e top8
MiniMax-M2.72026-04228.310.6GQA256e top8
MiniMax-M32026-06443.425.5GQA128e top4 +1shared
MiniMax-M3 MXFP82026-06443.425.5GQA128e top4 +1shared

Mistral · 2

2023-09 Mistral 7B · 2023-09 · 7.24B Mistral 7B 7.24B Mixtral 8x22B · 2024-04 · 140.6B Mixtral 8x22B 140.6B · MoE 2024-04
ModelReleasedParamsActiveAttentionMoE
Mistral 7B2023-097.247.24GQADense
Mixtral 8x22B2024-04140.639.1GQA8e top2

OpenAI · 4

2019-02 GPT-2 · 2019-02 · 1.24B GPT-2 1.24B GPT-3 175B (paper) · 2020-05 · 175B GPT-3 175B (paper) 175B GPT-OSS 120B · 2025-08 · 116.8B GPT-OSS 120B 116.8B · MoE GPT-OSS 20B · 2025-08 · 20.9B GPT-OSS 20B 20.9B · MoE 2025-08
ModelReleasedParamsActiveAttentionMoE
GPT-22019-021.241.24MHADense
GPT-3 175B (paper)2020-05175175MHADense
GPT-OSS 120B2025-08116.85.1GQA128e top4
GPT-OSS 20B2025-0820.93.6GQA32e top4

Qwen · 22

2024-02 Qwen1.5 7B · 2024-02 · 6.6B Qwen1.5 7B 6.6B Qwen2 72B · 2024-06 · 71.4B Qwen2 72B 71.4B Qwen2.5 72B · 2024-09 · 71.4B Qwen2.5 72B 71.4B Qwen2.5 7B · 2024-09 · 7.6B Qwen2.5 7B 7.6B Qwen3 0.6B · 2025-04 · 0.6B Qwen3 0.6B 0.6B Qwen3 235B A22B · 2025-04 · 234.6B Qwen3 235B A22B 234.6B · MoE Qwen3 30B A3B · 2025-04 · 30.4B Qwen3 30B A3B 30.4B · MoE Qwen3 32B · 2025-04 · 32.1B Qwen3 32B 32.1B Qwen3 8B · 2025-04 · 7.9B Qwen3 8B 7.9B Qwen3-Next 80B A3B · 2025-09 · 78.7B Qwen3-Next 80B A3B 78.7B · MoE Qwen3-Coder-Next · 2026-01 · 78.7B Qwen3-Coder-Next 78.7B · MoE Qwen3.5 122B A10B · 2026-02 · 119.9B Qwen3.5 122B A10B 119.9B · MoE Qwen3.5 27B · 2026-02 · 23.7B Qwen3.5 27B 23.7B Qwen3.5 35B A3B · 2026-02 · 33.9B Qwen3.5 35B A3B 33.9B · MoE Qwen3.5 397B A17B · 2026-02 · 392.6B Qwen3.5 397B A17B 392.6B · MoE Qwen3.5 9B · 2026-02 · 7.9B Qwen3.5 9B 7.9B Qwen3.5 9B Base · 2026-02 · 7.9B Qwen3.5 9B Base 7.9B Qwen3.6 27B · 2026-04 · 23.7B Qwen3.6 27B 23.7B Qwen3.6 35B A3B · 2026-04 · 33.9B Qwen3.6 35B A3B 33.9B · MoE Qwen3.8 2.4T A95B · 2026-08 · 2399.6B Qwen3.8 2.4T A95B 2399.6B · MoE Qwen3.8 27B · 2026-08 · 23.7B Qwen3.8 27B 23.7B Qwen3.8-Flash-Next · 2026-08 · 123.6B Qwen3.8-Flash-Next 123.6B · MoE 2026-08
ModelReleasedParamsActiveAttentionMoE
Qwen1.5 7B2024-026.66.6MHADense
Qwen2 72B2024-0671.471.4GQADense
Qwen2.5 72B2024-0971.471.4GQADense
Qwen2.5 7B2024-097.67.6GQADense
Qwen3 0.6B2025-040.60.6GQADense
Qwen3 235B A22B2025-04234.621.7GQA128e top8
Qwen3 30B A3B2025-0430.43.2GQA128e top8
Qwen3 32B2025-0432.132.1GQADense
Qwen3 8B2025-047.97.9GQADense
Qwen3-Next 80B A3B2025-0978.72.9GQA512e top10 +1shared
Qwen3-Coder-Next2026-0178.72.9GQA512e top10 +1shared
Qwen3.5 122B A10B2026-02119.97.6GQA256e top8 +1shared
Qwen3.5 27B2026-0223.723.7GQADense
Qwen3.5 35B A3B2026-0233.92.7GQA256e top8 +1shared
Qwen3.5 397B A17B2026-02392.613.6GQA512e top10 +1shared
Qwen3.5 9B2026-027.97.9GQADense
Qwen3.5 9B Base2026-027.97.9GQADense
Qwen3.6 27B2026-0423.723.7GQADense
Qwen3.6 35B A3B2026-0433.92.7GQA256e top8 +1shared
Qwen3.8 2.4T A95B2026-082399.695GQA512e top10 +1shared
Qwen3.8 27B2026-0823.723.7GQADense
Qwen3.8-Flash-Next2026-08123.65.1GQA512e top10 +1shared

StepFun · 3

2025-07 Step-3 · 2025-07 · 321B Step-3 321B · MoE Step-3.5-Flash · 2026-03 · 196B Step-3.5-Flash 196B · MoE Step-3.7-Flash · 2026-05 · 198B Step-3.7-Flash 198B · MoE 2026-05
ModelReleasedParamsActiveAttentionMoE
Step-32025-0732138MHA48e top3 +1shared
Step-3.5-Flash2026-0319611MHA288e top8 +1shared
Step-3.7-Flash2026-0519811MHA288e top8 +1shared