Skip to content
Cloud Provider Benchmarks

Benchmarks · GPU & AI

NVIDIA H200 cloud benchmarks: 141 GB HBM3e for large-model and long-context inference

The H200 keeps the Hopper compute of the H100 but upgrades memory to 141 GB of HBM3e at 4.8 TB/s. Identical Tensor throughput with 76% more memory and 43% more bandwidth makes it the stronger choice where model size, context length or concurrency is limited by memory rather than compute.

Updated:

Direct answer

H200 fits a 70B model in FP8 on one GPU with a large KV budget, and the 4.8 TB/s bandwidth raises the single-stream decode ceiling by about 43% over H100 SXM. It is listed by PortvMind Cloud, AWS, Azure, Google Cloud and DigitalOcean among the providers tracked here.

What is measured

  • Executable model × context × concurrency scenarios on 1, 2, 4 and 8 GPUs
  • Single-stream decode ceiling from memory bandwidth
  • Provider availability and published per-GPU-hour list prices
  • Verified vLLM serving runs (when published)

Overview

The H200 keeps the Hopper compute complex of the H100 SXM but replaces its memory subsystem with 141 GB of HBM3e delivering 4.8 TB/s. For LLM inference this changes two things: larger models or longer contexts fit on a single GPU, and memory-bound decode steps run faster in proportion to bandwidth. Compute-bound phases such as prefill of long prompts see little change versus H100 SXM because peak Tensor Core throughput is identical.

Suitable workloads

  • Serving 70B-class models in FP8 on a single GPU with room for KV cache
  • Long-context (64K–128K) inference and high-concurrency serving
  • Memory-bound generative AI and HPC workloads
  • Training jobs that benefit from larger per-GPU batch or activation memory

Considerations

  • Peak FP8/BF16 Tensor Core throughput equals H100 SXM; gains come from memory capacity and bandwidth
  • H200 NVL (PCIe) has lower power and compute ceilings than H200 SXM

Specifications

GPU specifications (NVIDIA datasheets)
GPU Architecture Memory (GB) Memory type Bandwidth (GB/s) FP8 (TFLOPS) BF16 (TFLOPS) FP32 (TFLOPS) Interconnect TDP MIG
H200 SXM SXM5 Hopper 141 HBM3e 4,800 1,979 989 67 NVLink 4 900 GB/s Up to 700 W Up to 7 MIGs @ 18 GB
H200 NVL PCIe dual-slot Hopper 141 HBM3e 4,800 1,670 835 60 NVLink bridge (2- or 4-way) 900 GB/s Up to 600 W Up to 7 MIGs @ 16.5 GB

Tensor values are dense (non-sparse) peaks; NVIDIA publishes sparse figures at twice these values.

Providers offering NVIDIA H200

GPU availability and published per-GPU-hour list prices by provider
Provider NVIDIA H200 Türkiye infrastructure
PortvMind Cloud Generally available GPU / AI Compute (H200 option) Listed as an available GPU option; configurations and prices are not included in the public calculator table and are provided via the console or sales. Yes
Amazon Web Services Generally available EC2 P5e / P5en 8-GPU instances; availability varies by region. Yes
Microsoft Azure Generally available ND H200 v5 Eight H200 GPUs per VM. No
Google Cloud Generally available A3 Ultra 8 H200 GPUs per VM; zone availability varies. Announced
DigitalOcean Generally available GPU Droplet — NVIDIA HGX H200 $4.470 / GPU-h 1× or 8× H200 141 GB; 1× plan with 24 vCPU and 240 GiB RAM. No
Vultr Unverified Not listed on the public pricing page H200 was not listed on the Vultr pricing page at the last review date. No

Decode ceiling (theoretical)

Single-stream upper bound derived from memory bandwidth. Not a measurement.

Memory-bandwidth ceiling for single-stream decode (FP8, tok/s) Theoretical upper bound — not a measurement
Model H200 SXM
Llama 3.1 8B Instruct 598
Llama 3.3 70B Instruct 68
Qwen2.5 7B Instruct 631
Qwen3 32B 146
Qwen3 30B-A3B 1,455
Qwen2.5 72B Instruct 66
DeepSeek-R1-Distill-Qwen-32B 146
DeepSeek-V3 1,038 8× TP
MiniMax-M1-80k 418 4× TP

Ceiling = aggregate memory bandwidth ÷ active weight bytes read per token (batch 1, excluding KV reads and communication, assuming ideal tensor parallelism). Real engines reach a fraction of this bound.

Memory-bandwidth ceiling for single-stream decode (BF16, tok/s) Theoretical upper bound — not a measurement
Model H200 SXM
Llama 3.1 8B Instruct 299
Llama 3.3 70B Instruct 68 2× TP
Qwen2.5 7B Instruct 315
Qwen3 32B 73
Qwen3 30B-A3B 727
Qwen2.5 72B Instruct 66 2× TP
DeepSeek-R1-Distill-Qwen-32B 73
DeepSeek-V3 does not fit
MiniMax-M1-80k 418 8× TP

Ceiling = aggregate memory bandwidth ÷ active weight bytes read per token (batch 1, excluding KV reads and communication, assuming ideal tensor parallelism). Real engines reach a fraction of this bound.

Executable scenarios

Which context and concurrency combinations fit in memory on one GPU.

Llama 3.1 8B Instruct · BF16 · 1× H200 SXM Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 103.8 GB
Context 15102550100 Max conc.
8K ✓ ✓ ✓ ✓ ✓ — 96
16K ✓ ✓ ✓ ✓ — — 48
32K ✓ ✓ ✓ — — — 24
64K ✓ ✓ ✓ — — — 12
128K ✓ ✓ — — — — 6

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Qwen3 32B · FP8 · 1× H200 SXM Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 87.0 GB
Context 15102550100 Max conc.
8K ✓ ✓ ✓ ✓ — — 40
16K ✓ ✓ ✓ — — — 20
32K ✓ ✓ ✓ — — — 10
64K ✓ ✓ — — — — 5
128K ✓ — — — — — 2

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Llama 3.3 70B Instruct · FP8 · 1× H200 SXM Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 49.3 GB
Context 15102550100 Max conc.
8K ✓ ✓ ✓ — — — 18
16K ✓ ✓ — — — — 9
32K ✓ — — — — — 4
64K ✓ — — — — — 2
128K ✓ — — — — — 1

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Verified results