Skip to content
Cloud Provider Benchmarks

Benchmarks · GPU & AI

LLM inference benchmarks: Qwen, DeepSeek, MiniMax and Llama on cloud GPUs

This page covers the open-weight model families most often deployed for self-hosted inference: Qwen, DeepSeek, MiniMax and Llama. For each model we publish weight size by precision, KV-cache bytes per token from the published architecture, the minimum number of GPUs needed, and a bandwidth-derived decode ceiling. Measured serving runs use vLLM through an adapter interface so other engines can be added.

Updated:

Direct answer

Model memory = parameters × bytes per parameter, plus KV cache that grows with context length and concurrency. A 70B model needs about 70 GB in FP8 before any KV cache, so it fits one H200 comfortably, one H100 tightly and needs at least two L40S. MoE models such as Qwen3-30B-A3B read only their active parameters per token, which raises decode speed but not memory requirements.

What is measured

  • Weight memory in BF16, FP8 and INT4
  • KV-cache bytes per token and at 32K context
  • Minimum GPU count per GPU model (FP8)
  • Single-stream decode ceiling per GPU model
  • Verified TTFT, TPOT and throughput runs by model

Model memory footprint

Computed from published model architectures.

Model memory requirements and minimum GPU count (FP8 weights)
Model Params (B) BF16 (GB) FP8 (GB) INT4 (GB) KV / token (KiB) KV @32K (GB) Min H100 SXM5Min H200 SXMMin L40S
Llama 3.1 8B Instruct Dense · GQA 8 16 8 4 128 4.3 1×1×1×
Llama 3.3 70B Instruct Dense · GQA 70.6 141 71 35 320 10.7 2×1×2×
Qwen2.5 7B Instruct Dense · GQA 7.6 15 8 4 56 1.9 1×1×1×
Qwen3 32B Dense · GQA 32.8 66 33 16 256 8.6 1×1×1×
Qwen3 30B-A3B MoE · 3.3B active · GQA 30.5 61 31 15 96 3.2 1×1×1×
Qwen2.5 72B Instruct Dense · GQA 72.7 145 73 36 320 10.7 2×1×2×
DeepSeek-R1-Distill-Qwen-32B Dense · GQA 32.8 66 33 16 256 8.6 1×1×1×
DeepSeek-V3 MoE · 37B active · MLA 671 1,342 671 336 69 2.3 >88×>8
MiniMax-M1-80k MoE · 45.9B active · HYBRID-LINEAR 456 912 456 228 — — 8×4×>8

Weight size = parameters × bytes per parameter. KV = 2 × layers × KV heads × head dim × 2 bytes (latent size for MLA). Minimum GPUs: 85% usable memory and at least 4 GB of KV headroom per GPU. A capacity model, not a measurement.

KV cache is not modelled for MiniMax-M1-80k: hybrid/linear attention layers are not captured by the standard formula. These models are assessed by measurement only.

Decode ceilings

Memory-bandwidth ceiling for single-stream decode (FP8, tok/s) Theoretical upper bound — not a measurement
Model H100 SXM5H200 SXML40S
Llama 3.1 8B Instruct 417 598 108
Llama 3.3 70B Instruct 95 2× TP 68 24 2× TP
Qwen2.5 7B Instruct 440 631 114
Qwen3 32B 102 146 26
Qwen3 30B-A3B 1,015 1,455 262
Qwen2.5 72B Instruct 92 2× TP 66 24 2× TP
DeepSeek-R1-Distill-Qwen-32B 102 146 26
DeepSeek-V3 does not fit 1,038 8× TP does not fit
MiniMax-M1-80k 584 8× TP 418 4× TP does not fit

Ceiling = aggregate memory bandwidth ÷ active weight bytes read per token (batch 1, excluding KV reads and communication, assuming ideal tensor parallelism). Real engines reach a fraction of this bound.

Memory-bandwidth ceiling for single-stream decode (BF16, tok/s) Theoretical upper bound — not a measurement
Model H100 SXM5H200 SXML40S
Llama 3.1 8B Instruct 209 299 54
Llama 3.3 70B Instruct 95 4× TP 68 2× TP 24 4× TP
Qwen2.5 7B Instruct 220 315 57
Qwen3 32B 102 2× TP 73 26 2× TP
Qwen3 30B-A3B 508 727 262 2× TP
Qwen2.5 72B Instruct 92 4× TP 66 2× TP 24 4× TP
DeepSeek-R1-Distill-Qwen-32B 102 2× TP 73 26 2× TP
DeepSeek-V3 does not fitdoes not fitdoes not fit
MiniMax-M1-80k does not fit 418 8× TP does not fit

Ceiling = aggregate memory bandwidth ÷ active weight bytes read per token (batch 1, excluding KV reads and communication, assuming ideal tensor parallelism). Real engines reach a fraction of this bound.

Verified results