The structural anatomy of open-weight large language models · GPT-2 → 2026
The structural anatomy of open-weight LLMs
This atlas dissects the architectures of 80 open-weight large language models released between GPT-2 (2019) and 2026. Instead of benchmark scores, it records structure: how attention evolved from multi-head to grouped-query, multi-head latent and hybrid linear designs; how sparse mixtures-of-experts replaced dense feed-forward blocks; and which operators each laboratory adopted and when. Every figure is traceable to the public config.json shipped with the model or to the original publication.
80models in the corpus
10Family
55Sparse MoE
2019–2026Released
Nine ways into the same corpus — pick the view that answers your question.
Models
A sortable index of 80 open-weight LLM architectures: parameters, active parameters, context window, attention variant, MoE layout, layers and heads.
Evolution
How LLM attention evolved from MHA to MQA, GQA, MLA and hybrid linear designs — and which operator each model adopted, with first adopters.
Operators
Every architectural operator in the corpus — attention, MoE, normalization, position and inference-side — with its mechanism, adoption count, first open-weight adopter and evidence.
Timeline
A chronological map of 80 open-weight LLM releases from 2019 to 2026, plotted by total parameters and colored by laboratory.
Lineage
Model families grouped by laboratory: OpenAI, Meta, Mistral, DeepSeek, Qwen, GLM, Kimi, MiniMax, StepFun and Hunyuan.
Similarity
Cosine similarity between model architecture feature vectors: attention class, MoE layout, normalization, position encoding, scale and context.
Papers
The primary publications behind every technique in the atlas: Transformer, GQA, MLA, RoPE, YaRN, RMSNorm, SwiGLU, Mixtral, DeepSeek-V2/V3 and more.
Multimodal
The vision-capable models in the corpus, identified objectively by vision fields shipped in their public config.json.
Compare
Compare up to four LLM architectures side by side: attention, MoE layout, layers, context, dtype and more.
Map
All 80 models on one map: release date versus total parameters (log scale), colored by laboratory.
Wall
Every model in the atlas as a card: family, date, parameters, attention class and MoE layout at a glance.
Config Explorer
Browse the full public config.json of any of the 80 open-weight models: every field, searchable and copyable.
Quiz
Test your knowledge of LLM architectures: attention variants, MoE layouts, laboratories and scale — questions generated from the atlas corpus.