NVIDIA H100 vs H200 for LLM inference: memory, bandwidth and when the upgrade matters

Specifications that matter for inference

Both GPUs are Hopper parts. The H200 SXM keeps the H100 SXM’s compute complex — the same fourth-generation Tensor Cores, Transformer Engine and NVLink 4 at 900 GB/s — and changes the memory subsystem. The table below is generated from the vendor datasheets referenced in the sources.

Three numbers decide most inference outcomes:

Property	H100 SXM	H200 SXM	Ratio
Memory	80 GB HBM3	141 GB HBM3e	1.76×
Memory bandwidth	3.35 TB/s	4.8 TB/s	1.43×
FP8 dense Tensor Core	1,979 TFLOPS	1,979 TFLOPS	1.00×

Why memory decides LLM serving

An inference server holds two things in GPU memory: the model weights and a key-value (KV) cache for every active sequence. Weights are fixed — about one byte per parameter at FP8 and two at BF16. The KV cache grows with context length and with the number of concurrent requests. For Llama 3.3 70B the KV cache costs about 0.33 MB per token at 16-bit precision, so a single 8K-token sequence needs roughly 2.7 GB.

When weights plus KV cache exceed one GPU, the model must be split across GPUs with tensor parallelism, which adds inter-GPU communication and doubles the hardware per replica. The H200’s extra 61 GB moves that threshold.

A 70B model on one GPU

The clearest difference is a 70B-class dense model at FP8. On an H100 the weights alone consume nearly all usable memory, so the smallest practical configuration is two GPUs. On an H200 the model fits on one GPU with about 49 GB of KV budget — room for about 18 concurrent 8K-token sequences under these assumptions.

The same model on two H100s has more total KV memory (about 65 GB), so it is not strictly worse — but it uses two GPUs per replica. Whether one H200 or two H100s is cheaper depends entirely on each provider’s per-GPU-hour price.

Decode speed: a bandwidth ceiling

Generating each token requires reading the active weights from memory. At batch size 1 that gives an upper bound of memory bandwidth divided by weight bytes. Real throughput is lower because of kernel efficiency, attention reads and scheduling, but the ratio between GPUs is informative.

For Llama 3.3 70B at FP8 the ceiling is about 47 tokens/s per H100 SXM and about 68 tokens/s per H200. With batching, total throughput rises far above the single-stream figure until compute or KV capacity becomes the limit — which is where the H200’s larger KV budget matters again.

Availability and price

Availability differs more between providers than between GPU generations. The table below reflects provider documentation as of the date shown. A price per GPU-hour is shown only where the provider publishes one.

Prices are list prices in each provider’s own currency and billing basis; they are not converted, because exchange rates and tax treatment change the comparison.

Methodology

All values are derived from public specifications using the formulas in the GPU and LLM inference methodology. This page uses no benchmark results. Measured TTFT, TPOT, throughput and latency percentiles are reported in the LLM inference benchmarks once a run has been reproduced.

Limitations

• Capacity figures assume a 16-bit KV cache; an FP8 KV cache roughly doubles the concurrency shown.
• PCIe and NVL variants of both GPUs have different bandwidth and power limits than the SXM parts modelled here.
• Bandwidth ceilings ignore attention-cache reads, which grow with context length and reduce decode speed at long contexts.
• Provider instance shapes (GPUs per server, vCPU, host memory) affect real deployments and are not modelled.

Conclusion

The H200 is an H100 with more and faster memory. It changes outcomes when memory is the constraint: 70B-class models at FP8, long contexts and high concurrency. For smaller models that already fit on an H100 with headroom, the H200 mainly adds concurrency, and the decision comes down to price per GPU-hour from the providers available to you.