Skip to content
Cloud Provider Benchmarks

GPU & AI infrastructure

NVIDIA H200 vs L40S: capacity, bandwidth and inference cost trade-offs

H200 vs L40S compared for LLM inference: 141 GB HBM3e vs 48 GB GDDR6, decode ceilings, model fit from 7B to 72B and published cloud prices.

Published: Updated: Türkçe oku

Direct answer

Pick the H200 for 32B–72B models, long contexts or high concurrency on a single GPU. Pick the L40S for models up to about 14B parameters, image and video generation, or when the lowest hourly price is the priority and throughput needs are modest.

Executive summary

The H200 and L40S sit at opposite ends of the NVIDIA data-centre range. The H200 has 141 GB of HBM3e at 4.8 TB/s — about 2.9× the memory and 5.6× the bandwidth of the L40S's 48 GB of GDDR6 at 864 GB/s. A 70B model at FP8 fits on one H200 but needs at least two L40S cards. The L40S is the lower-cost option for small models and visual workloads; the H200 is built for large models and long contexts.

Key findings

  1. 01 Bandwidth differs by about 5.6× (4,800 vs 864 GB/s), which is the dominant factor for token-generation speed at low batch sizes.
  2. 02 Llama 3.3 70B at FP8 fits on one H200 with about 49 GB of KV budget; on L40S it requires two cards and leaves about 11 GB of KV budget in total.
  3. 03 For an 8B model at FP8, one H200 holds about 104 concurrent 8K-token sequences in KV cache versus about 30 on one L40S.
  4. 04 The L40S has no NVLink, so scaling past one card relies on PCIe bandwidth.

Specifications

GPU specifications (NVIDIA datasheets)
GPU Architecture Memory (GB) Memory type Bandwidth (GB/s) FP8 (TFLOPS) BF16 (TFLOPS) FP32 (TFLOPS) Interconnect TDP MIG
H200 SXM SXM5 Hopper 141 HBM3e 4,800 1,979 989 67 NVLink 4 900 GB/s Up to 700 W Up to 7 MIGs @ 18 GB
L40S PCIe dual-slot Ada Lovelace 48 GDDR6 (ECC) 864 733 362 91.6 PCIe only 350 W Not supported

Tensor values are dense (non-sparse) peaks; NVIDIA publishes sparse figures at twice these values.

Property H200 SXM L40S Ratio
Memory 141 GB HBM3e 48 GB GDDR6 2.94×
Bandwidth 4,800 GB/s 864 GB/s 5.56×
FP8 dense TFLOPS 1,979 733 2.70×
Multi-GPU link NVLink 900 GB/s PCIe Gen4 —

Model fit across sizes

Model memory requirements and minimum GPU count (FP8 weights)
Model Params (B) BF16 (GB) FP8 (GB) INT4 (GB) KV / token (KiB) KV @32K (GB) Min H200 SXMMin L40S
Llama 3.1 8B Instruct Dense · GQA 8 16 8 4 128 4.3 1×1×
Llama 3.3 70B Instruct Dense · GQA 70.6 141 71 35 320 10.7 1×2×
Qwen2.5 7B Instruct Dense · GQA 7.6 15 8 4 56 1.9 1×1×
Qwen3 32B Dense · GQA 32.8 66 33 16 256 8.6 1×1×
Qwen3 30B-A3B MoE · 3.3B active · GQA 30.5 61 31 15 96 3.2 1×1×
Qwen2.5 72B Instruct Dense · GQA 72.7 145 73 36 320 10.7 1×2×
DeepSeek-R1-Distill-Qwen-32B Dense · GQA 32.8 66 33 16 256 8.6 1×1×
DeepSeek-V3 MoE · 37B active · MLA 671 1,342 671 336 69 2.3 8×>8
MiniMax-M1-80k MoE · 45.9B active · HYBRID-LINEAR 456 912 456 228 — — 4×>8

Weight size = parameters × bytes per parameter. KV = 2 × layers × KV heads × head dim × 2 bytes (latent size for MLA). Minimum GPUs: 85% usable memory and at least 4 GB of KV headroom per GPU. A capacity model, not a measurement.

The table shows the gap clearly: models up to about 14B parameters run comfortably on either GPU, 30B-class models are tight on the L40S, and 70B-class models cross from one GPU to two or more.

One GPU vs several

Llama 3.3 70B Instruct · FP8 · 1× H200 SXM Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 49.3 GB
Context 15102550100 Max conc.
8K ✓ ✓ ✓ — — — 18
16K ✓ ✓ — — — — 9
32K ✓ — — — — — 4
64K ✓ — — — — — 2
128K ✓ — — — — — 1

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Llama 3.3 70B Instruct · FP8 · 2× L40S Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 11.0 GB
Context 15102550100 Max conc.
8K ✓ — — — — — 4
16K ✓ — — — — — 2
32K ✓ — — — — — 1
64K — — — — — — 0
128K — — — — — — 0

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Two L40S cards do hold the 70B FP8 weights, but with only about 11 GB of KV budget across both, long contexts are not feasible and concurrency is limited. Four cards change the picture but add PCIe communication on every layer.

Decode ceilings

Memory-bandwidth ceiling for single-stream decode (FP8, tok/s) Theoretical upper bound — not a measurement
Model H200 SXML40S
Llama 3.1 8B Instruct 598 108
Llama 3.3 70B Instruct 68 24 2× TP
Qwen2.5 7B Instruct 631 114
Qwen3 32B 146 26
Qwen3 30B-A3B 1,455 262
Qwen2.5 72B Instruct 66 24 2× TP
DeepSeek-R1-Distill-Qwen-32B 146 26
DeepSeek-V3 1,038 8× TP does not fit

Ceiling = aggregate memory bandwidth ÷ active weight bytes read per token (batch 1, excluding KV reads and communication, assuming ideal tensor parallelism). Real engines reach a fraction of this bound.

Prices

Published per-GPU-hour list prices
Provider GPU Per GPU-hour Per GB-hour Basis
DigitalOcean NVIDIA H200 $4.470 $0.0317 On-demand, 1× H200 GPU Droplet. · 6 November 2026
DigitalOcean NVIDIA L40S $1.570 $0.0327 On-demand, 1× L40S GPU Droplet. · 6 November 2026
Vultr NVIDIA L40S $1.671 $0.0348 On-demand per-GPU rate, 1× L40S. · 6 November 2026

Where a provider lists the GPU but does not publish an hourly price, it does not appear in this table. Availability is shown separately on the GPU availability page.

Methodology

See the GPU and LLM inference methodology. All figures are derived from specifications; none are measurements.

Limitations

  • The H200 NVL (PCIe) variant has lower power and compute limits than the SXM part.
  • An FP8 KV cache roughly doubles the concurrency figures for both GPUs.
  • Multi-card L40S efficiency over PCIe depends heavily on the serving engine and server topology.

Conclusion

These GPUs answer different questions. The H200 answers “how do I serve a large model with long contexts on as few GPUs as possible?” The L40S answers “what is the cheapest capable card for a small model or a visual workload?” Choosing between them starts with model size.

Frequently asked questions

Can an L40S run a 70B model?

Only across multiple cards or with 4-bit quantisation. At FP8 the weights are about 70.6 GB, which exceeds one L40S's 48 GB. Two L40S cards fit the weights but leave little KV-cache space.

Is an H200 worth it over several L40S cards?

It depends on the model. For 70B-class models one H200 avoids tensor parallelism over PCIe and offers far more KV capacity. For small models several L40S cards may deliver more aggregate throughput per unit cost — this needs measurement on your workload.

Sources

  1. [1] NVIDIA H200 Tensor Core GPU — NVIDIA Vendor datasheet accessed:
  2. [2] NVIDIA L40S GPU — NVIDIA Vendor datasheet accessed:
  3. [3] Llama 3.3 70B Instruct model card — Meta Official documentation accessed:
  4. [4] DigitalOcean GPU Droplet pricing — DigitalOcean Official documentation accessed: