Skip to content
Cloud Provider Benchmarks

Benchmarks · GPU & AI

NVIDIA H100 cloud benchmarks: LLM inference capacity, availability and pricing

The H100 (Hopper) is the reference data-centre GPU for LLM serving. The SXM variant pairs 80 GB of HBM3 with 3.35 TB/s of bandwidth and NVLink; the PCIe and NVL variants trade bandwidth and power for standard servers. This page shows which model and context combinations are executable on one or more H100s, and which providers offer it.

Updated:

Direct answer

A single 80 GB H100 holds 8B–32B models in BF16 or FP8 with room for long contexts; 70B-class models need FP8 on one GPU with limited KV cache or two GPUs for high concurrency. Among the providers tracked here, PortvMind Cloud (Türkiye), AWS, Azure, Google Cloud, DigitalOcean and Vultr list H100 instances; Hetzner does not.

What is measured

  • Executable model × context × concurrency scenarios on 1, 2, 4 and 8 GPUs
  • Single-stream decode ceiling from memory bandwidth
  • Provider availability and published per-GPU-hour list prices
  • Verified vLLM serving runs (when published)

Overview

The H100 introduced fourth-generation Tensor Cores and the Transformer Engine with FP8 precision. The SXM5 variant pairs 80 GB of HBM3 at 3.35 TB/s with 900 GB/s NVLink, while the PCIe variant uses 80 GB of HBM2e at 2.0 TB/s. H100 NVL raises memory to 94 GB. Cloud providers frequently expose either SXM (in HGX 8-GPU baseboards) or PCIe/NVL cards, so the variant behind an instance determines memory bandwidth and multi-GPU scaling.

Suitable workloads

  • LLM pre-training and fine-tuning in BF16 and FP8
  • High-throughput LLM inference for models up to ~70B parameters per GPU at FP8
  • Multi-GPU tensor-parallel serving over NVLink (SXM)
  • HPC with FP64 Tensor Core support

Considerations

  • 80 GB limits single-GPU BF16 serving to roughly 30B-parameter dense models once KV cache is reserved
  • SXM vs PCIe variants differ by ~1.7× in memory bandwidth, which dominates decode speed
  • Supports MIG partitioning into up to seven instances

Specifications

GPU specifications (NVIDIA datasheets)
GPU Architecture Memory (GB) Memory type Bandwidth (GB/s) FP8 (TFLOPS) BF16 (TFLOPS) FP32 (TFLOPS) Interconnect TDP MIG
H100 SXM5 SXM5 Hopper 80 HBM3 3,350 1,979 989 67 NVLink 4 900 GB/s Up to 700 W Up to 7 MIGs @ 10 GB
H100 PCIe PCIe dual-slot Hopper 80 HBM2e 2,000 1,513 756 51 NVLink bridge (2-way) 600 GB/s 300–350 W Up to 7 MIGs @ 10 GB
H100 NVL PCIe dual-slot Hopper 94 HBM3 3,900 1,670 835 60 NVLink bridge (2-way) 600 GB/s 350–400 W Up to 7 MIGs @ 12 GB

Tensor values are dense (non-sparse) peaks; NVIDIA publishes sparse figures at twice these values.

Providers offering NVIDIA H100

GPU availability and published per-GPU-hour list prices by provider
Provider NVIDIA H100 Türkiye infrastructure
PortvMind Cloud Generally available gpu.h100s / gpu.h100m profiles TRY 195.835 / GPU-h 1, 2, 3 or 4 GPUs per server; 80 GB (81,920 MB) VRAM per GPU; 26 vCPU and 496 GB RAM per GPU; local NVMe disk included. Yes
Amazon Web Services Generally available EC2 P5 Single-GPU p5.4xlarge and 8-GPU p5.48xlarge sizes; availability varies by region. Yes
Microsoft Azure Generally available NC H100 v5 / ND H100 v5 NC H100 v5 with one or two H100 NVL GPUs; ND H100 v5 with eight H100 GPUs. No
Google Cloud Generally available A3 High / A3 Mega Up to 8 H100 GPUs per VM; zone availability varies. Announced
DigitalOcean Generally available GPU Droplet — NVIDIA HGX H100 $4.410 / GPU-h 1× or 8× H100 80 GB; 1× plan with 20 vCPU and 240 GiB RAM. No
Vultr Generally available Cloud GPU — NVIDIA HGX H100 $2.300 / GPU-h 8× H100 80 GB SXM (640 GB GPU RAM, 216 vCPU, 1,914 GB RAM). No

Decode ceiling (theoretical)

Single-stream upper bound derived from memory bandwidth. Not a measurement.

Memory-bandwidth ceiling for single-stream decode (FP8, tok/s) Theoretical upper bound — not a measurement
Model H100 SXM5
Llama 3.1 8B Instruct 417
Llama 3.3 70B Instruct 95 2× TP
Qwen2.5 7B Instruct 440
Qwen3 32B 102
Qwen3 30B-A3B 1,015
Qwen2.5 72B Instruct 92 2× TP
DeepSeek-R1-Distill-Qwen-32B 102
DeepSeek-V3 does not fit
MiniMax-M1-80k 584 8× TP

Ceiling = aggregate memory bandwidth ÷ active weight bytes read per token (batch 1, excluding KV reads and communication, assuming ideal tensor parallelism). Real engines reach a fraction of this bound.

Memory-bandwidth ceiling for single-stream decode (BF16, tok/s) Theoretical upper bound — not a measurement
Model H100 SXM5
Llama 3.1 8B Instruct 209
Llama 3.3 70B Instruct 95 4× TP
Qwen2.5 7B Instruct 220
Qwen3 32B 102 2× TP
Qwen3 30B-A3B 508
Qwen2.5 72B Instruct 92 4× TP
DeepSeek-R1-Distill-Qwen-32B 102 2× TP
DeepSeek-V3 does not fit
MiniMax-M1-80k does not fit

Ceiling = aggregate memory bandwidth ÷ active weight bytes read per token (batch 1, excluding KV reads and communication, assuming ideal tensor parallelism). Real engines reach a fraction of this bound.

Executable scenarios

Which context and concurrency combinations fit in memory on one GPU.

Llama 3.1 8B Instruct · BF16 · 1× H100 SXM5 Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 51.9 GB
Context 15102550100 Max conc.
8K ✓ ✓ ✓ ✓ — — 48
16K ✓ ✓ ✓ — — — 24
32K ✓ ✓ ✓ — — — 12
64K ✓ ✓ — — — — 6
128K ✓ — — — — — 3

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Qwen3 32B · FP8 · 1× H100 SXM5 Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 35.2 GB
Context 15102550100 Max conc.
8K ✓ ✓ ✓ — — — 16
16K ✓ ✓ — — — — 8
32K ✓ — — — — — 4
64K ✓ — — — — — 2
128K ✓ — — — — — 1

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Llama 3.3 70B Instruct · FP8 · 1× H100 SXM5 Executable scenario matrix (KV-cache capacity model) · KV budget ≈ -2.6 GB
Context 15102550100 Max conc.
8K ✕ ✕ ✕ ✕ ✕ ✕ 0
16K ✕ ✕ ✕ ✕ ✕ ✕ 0
32K ✕ ✕ ✕ ✕ ✕ ✕ 0
64K ✕ ✕ ✕ ✕ ✕ ✕ 0
128K ✕ ✕ ✕ ✕ ✕ ✕ 0

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Verified results