Skip to content
Cloud Provider Benchmarks

Benchmarks

GPU cloud benchmarks: NVIDIA H100, H200 and L40S for LLM inference

The GPU section combines three kinds of evidence, each labelled: datasheet specifications from NVIDIA, deterministic capacity models derived from those specifications and from model architectures, and measured inference runs that have passed verification. Provider availability and list prices are sourced from each provider’s own pages with an access date.

Updated:

Direct answer

For LLM inference the deciding GPU properties are memory capacity (which models and context lengths fit) and memory bandwidth (how fast tokens are generated). H200 has 141 GB at 4.8 TB/s, H100 SXM 80 GB at 3.35 TB/s and L40S 48 GB at 864 GB/s, so H200 serves the largest models and longest contexts per GPU, while L40S suits models up to roughly 30B parameters in FP8.

What is measured

  • Time to first token (TTFT) and time per output token (TPOT), p50/p95/p99
  • Aggregate and per-request output throughput (tokens/s)
  • GPU memory used and maximum feasible concurrency per context length
  • Cost per million output tokens from published hourly prices
  • Success and error rates under load

Tools

  • vLLM — Serving engine (OpenAI-compatible API)
  • vllm bench serve — Load generator producing TTFT, TPOT and throughput percentiles
  • nvidia-smi — Driver, CUDA and memory telemetry

GPU pages and analyses

Specifications

GPU specifications (NVIDIA datasheets)
GPU Architecture Memory (GB) Memory type Bandwidth (GB/s) FP8 (TFLOPS) BF16 (TFLOPS) FP32 (TFLOPS) Interconnect TDP MIG
H100 SXM5 SXM5 Hopper 80 HBM3 3,350 1,979 989 67 NVLink 4 900 GB/s Up to 700 W Up to 7 MIGs @ 10 GB
H100 PCIe PCIe dual-slot Hopper 80 HBM2e 2,000 1,513 756 51 NVLink bridge (2-way) 600 GB/s 300–350 W Up to 7 MIGs @ 10 GB
H100 NVL PCIe dual-slot Hopper 94 HBM3 3,900 1,670 835 60 NVLink bridge (2-way) 600 GB/s 350–400 W Up to 7 MIGs @ 12 GB
H200 SXM SXM5 Hopper 141 HBM3e 4,800 1,979 989 67 NVLink 4 900 GB/s Up to 700 W Up to 7 MIGs @ 18 GB
H200 NVL PCIe dual-slot Hopper 141 HBM3e 4,800 1,670 835 60 NVLink bridge (2- or 4-way) 900 GB/s Up to 600 W Up to 7 MIGs @ 16.5 GB
L40S PCIe dual-slot Ada Lovelace 48 GDDR6 (ECC) 864 733 362 91.6 PCIe only 350 W Not supported

Tensor values are dense (non-sparse) peaks; NVIDIA publishes sparse figures at twice these values.

GPU memory capacity

Source: NVIDIA datasheets. Dense (non-sparse) values.

H100 SXM5 80 GB H200 SXM 141 GB L40S 48 GB
Data table
GPU GPU memory capacity (GB)
H100 SXM5 80
H200 SXM 141
L40S 48
Memory bandwidth

Source: NVIDIA datasheets. Dense (non-sparse) values.

H100 SXM5 3,350 GB/s H200 SXM 4,800 GB/s L40S 864 GB/s
Data table
GPU Memory bandwidth (GB/s)
H100 SXM5 3,350
H200 SXM 4,800
L40S 864
Dense FP8 Tensor throughput

Source: NVIDIA datasheets. Dense (non-sparse) values.

H100 SXM5 1,979 TFLOPS H200 SXM 1,979 TFLOPS L40S 733 TFLOPS
Data table
GPU Dense FP8 Tensor throughput (TFLOPS)
H100 SXM5 1,979
H200 SXM 1,979
L40S 733
Dense BF16 Tensor throughput

Source: NVIDIA datasheets. Dense (non-sparse) values.

H100 SXM5 989 TFLOPS H200 SXM 989 TFLOPS L40S 362 TFLOPS
Data table
GPU Dense BF16 Tensor throughput (TFLOPS)
H100 SXM5 989
H200 SXM 989
L40S 362

Availability by provider

GPU cloud ranking →
GPU availability and published per-GPU-hour list prices by provider
Provider NVIDIA H100NVIDIA H200NVIDIA L40S Türkiye infrastructure
PortvMind Cloud Generally available gpu.h100s / gpu.h100m profiles TRY 195.835 / GPU-h 1, 2, 3 or 4 GPUs per server; 80 GB (81,920 MB) VRAM per GPU; 26 vCPU and 496 GB RAM per GPU; local NVMe disk included. Generally available GPU / AI Compute (H200 option) Listed as an available GPU option; configurations and prices are not included in the public calculator table and are provided via the console or sales. Generally available GPU / AI Compute (L40S option) Listed as an available GPU option; configurations and prices are not included in the public calculator table and are provided via the console or sales. Yes
Amazon Web Services Generally available EC2 P5 Single-GPU p5.4xlarge and 8-GPU p5.48xlarge sizes; availability varies by region. Generally available EC2 P5e / P5en 8-GPU instances; availability varies by region. Generally available EC2 G6e 1 to 8 L40S GPUs per instance. Yes
Microsoft Azure Generally available NC H100 v5 / ND H100 v5 NC H100 v5 with one or two H100 NVL GPUs; ND H100 v5 with eight H100 GPUs. Generally available ND H200 v5 Eight H200 GPUs per VM. Not offered L40S SKUs are documented for Azure Local / Azure Stack Hub (customer-site hardware). No
Google Cloud Generally available A3 High / A3 Mega Up to 8 H100 GPUs per VM; zone availability varies. Generally available A3 Ultra 8 H200 GPUs per VM; zone availability varies. Not offered Announced
DigitalOcean Generally available GPU Droplet — NVIDIA HGX H100 $4.410 / GPU-h 1× or 8× H100 80 GB; 1× plan with 20 vCPU and 240 GiB RAM. Generally available GPU Droplet — NVIDIA HGX H200 $4.470 / GPU-h 1× or 8× H200 141 GB; 1× plan with 24 vCPU and 240 GiB RAM. Generally available GPU Droplet — NVIDIA L40S $1.570 / GPU-h 1× L40S 48 GB with 8 vCPU and 64 GiB RAM. No
Vultr Generally available Cloud GPU — NVIDIA HGX H100 $2.300 / GPU-h 8× H100 80 GB SXM (640 GB GPU RAM, 216 vCPU, 1,914 GB RAM). Unverified Not listed on the public pricing page H200 was not listed on the Vultr pricing page at the last review date. Generally available Cloud GPU — NVIDIA L40S $1.671 / GPU-h 1, 2, 4 or 8× L40S 48 GB; 1× plan with 16 vCPU and 180 GB RAM. No
Hetzner Not offered Not offered Not offered No

Metric definitions

Metric definitions
Metric Unit Better Definition
Time to first token ms Lower Request start to first generated token, includes queueing and prefill.
Time per output token ms Lower Mean inter-token latency after the first token.
Aggregate output throughput tok/s Higher Generated tokens per second across all concurrent requests.
Cost per million output tokens currency/1M tok Lower Hourly price ÷ (aggregate output tok/s × 3600) × 10⁶.

Verified results

Frequently asked questions

Which matters more for LLM inference, FLOPS or memory bandwidth?

Token generation (decode) at low batch sizes is memory-bandwidth bound: every token reads the active weights. Prefill and large batches are more compute bound. Both matter; bandwidth sets the single-stream ceiling.

Are the capacity tables measurements?

No. They are deterministic models computed from datasheet memory sizes and model architectures, and are labelled as such. Measured runs are listed in the results tables with their source and run metadata.