Operator breakdown
This page takes the corpus apart operator by operator. For each operator: what it does, how many models in the corpus use it, who shipped it first in open weights, and where the evidence lives — the exact config.json field, plus the original paper where one exists.
Attention operators
| Operator | Mechanism & meaning | Model | First open-weight adopter | Evidence |
|---|---|---|---|---|
| MHA | Every query head owns a key/value head: full expressivity, the largest KV cache. | 10 | GPT-2 (2019-02) | num_attention_heads = num_key_value_heads · arXiv:1706.03762 |
| GQA | Query heads share key/value heads in small groups — most of MHA quality at a fraction of the cache. | 46 | Llama 2 70B (2023-07) | num_key_value_heads < num_attention_heads · arXiv:2305.13245 |
| MQA | All query heads share a single KV head: minimal cache, some quality cost. | 3 | DeepSeek-V4-Flash (2026-04) | num_key_value_heads = 1 · arXiv:1911.02150 |
| MLA | Keys/values are projected into a low-rank latent vector plus a small decoupled RoPE key; the cache shrinks by an order of magnitude. | 21 | DeepSeek-V2 (2024-05) | kv_lora_rank · arXiv:2405.04434 |
| SWA | Each token attends only within a sliding window of recent tokens; cache and cost grow linearly with context. | 12 | Mistral 7B (2023-09) | sliding_window · arXiv:2310.06825 |
| Linear attention (hybrid) | Recurrent-style linear-attention layers with near-constant memory, interleaved with a minority of full-attention layers. | 0 | — | linear_attn_config / layer types |
Feed-forward & MoE
| Operator | Mechanism & meaning | Model | First open-weight adopter | Evidence |
|---|---|---|---|---|
| MoE | The FFN becomes many expert networks; a router activates the top-k per token, decoupling capacity from compute. | 55 | DeepSeekMoE 16B (2024-01) | n_routed_experts · num_experts_per_tok · arXiv:2401.06066 |
| SwiGLU / SiLU | Gated linear units with SiLU — better quality per parameter than plain GELU/ReLU feed-forward blocks. | 70 | LLaMA 1 7B (2023-02) | hidden_act = silu · arXiv:2002.05202 |
Normalization & position
| Operator | Mechanism & meaning | Model | First open-weight adopter | Evidence |
|---|---|---|---|---|
| RMSNorm | Normalizes by root-mean-square only, no mean centering: cheaper and stable at scale. | 73 | LLaMA 1 7B (2023-02) | rms_norm_eps · arXiv:1910.07467 |
| RoPE | Encodes relative position by rotating query/key pairs; extrapolates naturally with context. | 58 | Mistral 7B (2023-09) | rope_theta · arXiv:2104.09864 |
| YaRN long-context | Rescales the RoPE frequency spectrum to stretch the context window far beyond pretraining length. | 0 | — | rope_scaling.rope_type = yarn · arXiv:2309.00071 |
Inference & precision
| Operator | Mechanism & meaning | Model | First open-weight adopter | Evidence |
|---|---|---|---|---|
| Multi-Token Prediction | Extra lightweight heads predict several future tokens per forward pass and speed up decoding. | 39 | DeepSeek-V3 (2024-12) | num_nextn_predict_layers · arXiv:2412.19437 |
| FP8 training | Weights and activations in 8-bit float: roughly half the memory and faster matmuls at frontier scale. | 18 | DeepSeek-V3 (2024-12) | torch_dtype / quantization_config |
Method note
Adoption is decided mechanically from the shipped config.json (the evidence column names the exact fields). "First adopter" is the earliest open-weight release in this corpus that carries the operator — earlier non-open precedents are noted on the Papers page.