Datasets
Open LLM architecture parameters
Parameter counts, active parameters, layers, KV heads, head dimension, derived KV-cache bytes per token and context limits for Qwen, Llama, DeepSeek and MiniMax models.
- Version
- 1.0
- Updated
- Rows
- 9
- License
- CC BY 4.0
Model-card configuration values; KV bytes derived as 2 × layers × KV heads × head dim × 2.
Fields
| Field | Description |
|---|---|
model | Model id |
name | Model name |
family | Model family |
architecture | dense | moe |
attention | gqa | mla | hybrid-linear |
params_b | Total parameters, billions |
active_params_b | Active parameters per token, billions |
layers | Transformer layers |
kv_heads | KV heads |
head_dim | Head dimension |
kv_bytes_per_token_16bit | Derived KV-cache bytes per token at 16-bit (null if not modelled) |
max_context | Maximum context, tokens |
hf_id | Hugging Face repository |
Preview
First 9 of 9 rows.
| model | name | family | architecture | attention | params_b | active_params_b | layers | kv_heads | head_dim | kv_bytes_per_token_16bit | max_context | hf_id |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| llama-3-1-8b | Llama 3.1 8B Instruct | llama | dense | gqa | 8.03 | 8.03 | 32 | 8 | 128 | 131072 | 131072 | meta-llama/Llama-3.1-8B-Instruct |
| llama-3-3-70b | Llama 3.3 70B Instruct | llama | dense | gqa | 70.6 | 70.6 | 80 | 8 | 128 | 327680 | 131072 | meta-llama/Llama-3.3-70B-Instruct |
| qwen2-5-7b | Qwen2.5 7B Instruct | qwen | dense | gqa | 7.61 | 7.61 | 28 | 4 | 128 | 57344 | 131072 | Qwen/Qwen2.5-7B-Instruct |
| qwen3-32b | Qwen3 32B | qwen | dense | gqa | 32.8 | 32.8 | 64 | 8 | 128 | 262144 | 131072 | Qwen/Qwen3-32B |
| qwen3-30b-a3b | Qwen3 30B-A3B | qwen | moe | gqa | 30.5 | 3.3 | 48 | 4 | 128 | 98304 | 131072 | Qwen/Qwen3-30B-A3B |
| qwen2-5-72b | Qwen2.5 72B Instruct | qwen | dense | gqa | 72.7 | 72.7 | 80 | 8 | 128 | 327680 | 131072 | Qwen/Qwen2.5-72B-Instruct |
| deepseek-r1-distill-qwen-32b | DeepSeek-R1-Distill-Qwen-32B | deepseek | dense | gqa | 32.8 | 32.8 | 64 | 8 | 128 | 262144 | 131072 | deepseek-ai/DeepSeek-R1-Distill-Qwen-32B |
| deepseek-v3 | DeepSeek-V3 | deepseek | moe | mla | 671 | 37 | 61 | — | — | 70272 | 131072 | deepseek-ai/DeepSeek-V3 |
| minimax-m1-80k | MiniMax-M1-80k | minimax | moe | hybrid-linear | 456 | 45.9 | — | — | — | — | 1000000 | MiniMaxAI/MiniMax-M1-80k |
Changelog
- v1.0 · — Initial release.