Skip to content
Cloud Provider Benchmarks

GPU & AI infrastructure

Best GPU for LLM inference: matching H100, H200 and L40S to model size

Which GPU to use for LLM inference by model size, precision, context length and concurrency — a sizing guide for Qwen, Llama, DeepSeek and MiniMax.

Published: Updated: Türkçe oku

Direct answer

For 7B–14B models, the L40S is usually sufficient. For 30B–32B models, use an H100 at FP8 or an H200 for more concurrency. For 70B models, use one H200 at FP8 or two H100s. For DeepSeek-V3 (671B) at FP8, use an 8× H200 node; MiniMax-M1 (456B) fits on 4× H200 or 8× H100.

Executive summary

There is no single best GPU for LLM inference; the right choice follows from model size, precision, context length and concurrency. Small models (7B–14B) run well on an L40S. Mid-size models (30B–32B) fit on one H100 at FP8 with moderate headroom. 70B-class models fit on one H200 at FP8 or on two H100s. Very large mixture-of-experts models need a full multi-GPU node: DeepSeek-V3 at FP8 fits on eight H200s but not on eight H100s, while MiniMax-M1 at FP8 fits on four H200s or eight H100s.

Key findings

  1. 01 Weights at FP8 take about 1 GB per billion parameters; usable device memory under the standard 90% utilisation and 5% reserve is about 85% of the card.
  2. 02 KV cache, not weights, is often the binding constraint at long context: Llama 3.3 70B needs about 0.33 MB per token, or 43 GB for one 128K-token sequence.
  3. 03 Mixture-of-experts models decouple memory from speed: Qwen3 30B-A3B needs ~30 GB for weights but reads only ~3.3 GB per token at FP8.
  4. 04 Measured throughput depends on the serving engine; these sizing rules identify feasible configurations, not winners.

A sizing rule in three steps

  1. Weights. Parameters × bytes per parameter: about 2 bytes at BF16, 1 at FP8 and 0.5 at 4-bit.
  2. KV cache. 2 × layers × KV heads × head dimension × bytes, per token, multiplied by context length and concurrent sequences.
  3. Usable memory. About 85% of the card once the serving engine’s utilisation limit and runtime reserve are applied.

If weights plus the KV cache you need exceed usable memory, add GPUs (tensor parallelism) or reduce precision.

Model memory across GPUs

Model memory requirements and minimum GPU count (FP8 weights)
Model Params (B) BF16 (GB) FP8 (GB) INT4 (GB) KV / token (KiB) KV @32K (GB) Min H100 SXM5Min H200 SXMMin L40S
Llama 3.1 8B Instruct Dense · GQA 8 16 8 4 128 4.3 1×1×1×
Llama 3.3 70B Instruct Dense · GQA 70.6 141 71 35 320 10.7 2×1×2×
Qwen2.5 7B Instruct Dense · GQA 7.6 15 8 4 56 1.9 1×1×1×
Qwen3 32B Dense · GQA 32.8 66 33 16 256 8.6 1×1×1×
Qwen3 30B-A3B MoE · 3.3B active · GQA 30.5 61 31 15 96 3.2 1×1×1×
Qwen2.5 72B Instruct Dense · GQA 72.7 145 73 36 320 10.7 2×1×2×
DeepSeek-R1-Distill-Qwen-32B Dense · GQA 32.8 66 33 16 256 8.6 1×1×1×
DeepSeek-V3 MoE · 37B active · MLA 671 1,342 671 336 69 2.3 >88×>8
MiniMax-M1-80k MoE · 45.9B active · HYBRID-LINEAR 456 912 456 228 — — 8×4×>8

Weight size = parameters × bytes per parameter. KV = 2 × layers × KV heads × head dim × 2 bytes (latent size for MLA). Minimum GPUs: 85% usable memory and at least 4 GB of KV headroom per GPU. A capacity model, not a measurement.

Recommendations by model size

Model class Examples Suggested starting point
7B–8B Qwen2.5 7B, Llama 3.1 8B 1× L40S (FP8 or BF16)
30B–32B dense Qwen3 32B, DeepSeek-R1-Distill-Qwen-32B 1× H100 (FP8) or 1× H200
30B MoE Qwen3 30B-A3B 1× H100 or 1× L40S (FP8)
70B–72B Llama 3.3 70B, Qwen2.5 72B 1× H200 (FP8) or 2× H100
450B–700B MoE DeepSeek-V3 (671B), MiniMax-M1 (456B) 8× H200 (DeepSeek-V3 FP8); 4× H200 or 8× H100 (MiniMax-M1 FP8)

These are starting points for capacity, not performance claims.

Context length changes everything

Long contexts multiply the KV cache. For Llama 3.3 70B one 8K-token sequence needs about 2.7 GB of KV cache, while a single 128K-token sequence needs about 43 GB — more than half of an H100.

Qwen3 32B · FP8 · 1× H100 SXM5 Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 35.2 GB
Context 15102550100 Max conc.
8K ✓ ✓ ✓ — — — 16
16K ✓ ✓ — — — — 8
32K ✓ — — — — — 4
64K ✓ — — — — — 2
128K ✓ — — — — — 1

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Speed: bandwidth first, compute second

Memory-bandwidth ceiling for single-stream decode (FP8, tok/s) Theoretical upper bound — not a measurement
Model H100 SXM5H200 SXML40S
Llama 3.1 8B Instruct 417 598 108
Llama 3.3 70B Instruct 95 2× TP 68 24 2× TP
Qwen2.5 7B Instruct 440 631 114
Qwen3 32B 102 146 26
Qwen3 30B-A3B 1,015 1,455 262
Qwen2.5 72B Instruct 92 2× TP 66 24 2× TP
DeepSeek-R1-Distill-Qwen-32B 102 146 26
DeepSeek-V3 does not fit 1,038 8× TP does not fit

Ceiling = aggregate memory bandwidth ÷ active weight bytes read per token (batch 1, excluding KV reads and communication, assuming ideal tensor parallelism). Real engines reach a fraction of this bound.

Decode at low batch sizes is bounded by memory bandwidth; prefill of long prompts is bounded by compute. Production throughput sits between these bounds and depends on batching, the attention kernel and the scheduler.

Where these GPUs are available

GPU availability and published per-GPU-hour list prices by provider
Provider NVIDIA H100NVIDIA H200NVIDIA L40S Türkiye infrastructure
PortvMind Cloud Generally available gpu.h100s / gpu.h100m profiles TRY 195.835 / GPU-h 1, 2, 3 or 4 GPUs per server; 80 GB (81,920 MB) VRAM per GPU; 26 vCPU and 496 GB RAM per GPU; local NVMe disk included. Generally available GPU / AI Compute (H200 option) Listed as an available GPU option; configurations and prices are not included in the public calculator table and are provided via the console or sales. Generally available GPU / AI Compute (L40S option) Listed as an available GPU option; configurations and prices are not included in the public calculator table and are provided via the console or sales. Yes
Amazon Web Services Generally available EC2 P5 Single-GPU p5.4xlarge and 8-GPU p5.48xlarge sizes; availability varies by region. Generally available EC2 P5e / P5en 8-GPU instances; availability varies by region. Generally available EC2 G6e 1 to 8 L40S GPUs per instance. Yes
Microsoft Azure Generally available NC H100 v5 / ND H100 v5 NC H100 v5 with one or two H100 NVL GPUs; ND H100 v5 with eight H100 GPUs. Generally available ND H200 v5 Eight H200 GPUs per VM. Not offered L40S SKUs are documented for Azure Local / Azure Stack Hub (customer-site hardware). No
Google Cloud Generally available A3 High / A3 Mega Up to 8 H100 GPUs per VM; zone availability varies. Generally available A3 Ultra 8 H200 GPUs per VM; zone availability varies. Not offered Announced
DigitalOcean Generally available GPU Droplet — NVIDIA HGX H100 $4.410 / GPU-h 1× or 8× H100 80 GB; 1× plan with 20 vCPU and 240 GiB RAM. Generally available GPU Droplet — NVIDIA HGX H200 $4.470 / GPU-h 1× or 8× H200 141 GB; 1× plan with 24 vCPU and 240 GiB RAM. Generally available GPU Droplet — NVIDIA L40S $1.570 / GPU-h 1× L40S 48 GB with 8 vCPU and 64 GiB RAM. No
Vultr Generally available Cloud GPU — NVIDIA HGX H100 $2.300 / GPU-h 8× H100 80 GB SXM (640 GB GPU RAM, 216 vCPU, 1,914 GB RAM). Unverified Not listed on the public pricing page H200 was not listed on the Vultr pricing page at the last review date. Generally available Cloud GPU — NVIDIA L40S $1.671 / GPU-h 1, 2, 4 or 8× L40S 48 GB; 1× plan with 16 vCPU and 180 GB RAM. No
Hetzner Not offered Not offered Not offered No

Methodology

Derived using the GPU and LLM inference methodology. Model architecture parameters come from official model cards. Measured results are reported on the LLM inference benchmarks page.

Limitations

  • KV-cache figures assume a 16-bit cache without prefix sharing; vLLM’s prefix caching can reduce memory for shared prompts.
  • MiniMax-M1 uses a hybrid attention design whose KV footprint is not modelled here.
  • DeepSeek-V3 uses multi-head latent attention; its KV size follows the published compressed dimension.
  • Accuracy effects of quantisation are task-dependent and not assessed.

Conclusion

Start from model size and the context and concurrency you need, compute memory, and pick the smallest GPU configuration that fits with headroom. Then measure: the sizing rules here narrow the options, and benchmarks decide among them.

Frequently asked questions

What is the best GPU for running Llama 3.3 70B?

At FP8, one H200 (141 GB) holds the weights with about 49 GB of KV budget. Two H100 80 GB GPUs are the common alternative. At BF16 you need two H200s or four H100s.

What GPU do I need for an 8B model?

Any of the three. An L40S holds an 8B model at FP8 with about 33 GB left for KV cache, enough for roughly 30 concurrent 8K-token sequences under the modelling assumptions.

Does quantisation change the answer?

Yes. FP8 halves weight memory versus BF16, and 4-bit formats halve it again, usually with some accuracy cost that should be evaluated per task.

Sources

  1. [1] NVIDIA H100 Tensor Core GPU — NVIDIA Vendor datasheet accessed:
  2. [2] NVIDIA H200 Tensor Core GPU — NVIDIA Vendor datasheet accessed:
  3. [3] NVIDIA L40S GPU — NVIDIA Vendor datasheet accessed:
  4. [4] DeepSeek-V3 model card — DeepSeek Official documentation accessed:
  5. [5] Qwen3 30B-A3B model card — Qwen Official documentation accessed:
  6. [6] MiniMax-M1-80k model card — MiniMax Official documentation accessed:
  7. [7] vLLM documentation — vLLM project Official documentation accessed: