Skip to content
Cloud Provider Benchmarks

Methodology

GPU and LLM inference benchmark methodology

How LLM inference is measured on H100, H200 and L40S with vLLM: models, precisions, context and concurrency scenarios, metrics, feasibility rules and verification.

Version 1.0 Updated:

Scope

  • NVIDIA H100, H200 and L40S GPUs, single-GPU and multi-GPU tensor-parallel configurations
  • Qwen, DeepSeek, MiniMax and Llama model families in BF16, FP8 and 4-bit precisions
  • Concurrency levels 1, 5, 10, 25, 50 and 100
  • Context lengths of 8K, 32K, 64K and 128K tokens
  • TTFT, TPOT, end-to-end latency (p50/p95/p99), output tokens/s, error rate, peak GPU memory and cost per 1M output tokens
Tools
Tool Version
vLLM (vllm bench serve) Recorded per run
NVIDIA driver and CUDA Recorded per run
nvidia-smi Recorded per run

Feasibility first

Before any run, each model × precision × GPU × context × concurrency scenario is checked against a memory model:

  • Weights = parameters × bytes per parameter (BF16 = 2, FP8 = 1, 4-bit = 0.5).
  • KV cache per token = 2 × layers × KV heads × head dimension × 2 bytes (16-bit cache), or the model’s published compressed size.
  • Usable memory = device memory × (0.90 − 0.05), matching vLLM’s default gpu_memory_utilization minus a runtime reserve.
  • A scenario is feasible if usable memory − weights ≥ KV per token × context × concurrency.

Infeasible scenarios are not run and are reported as infeasible, not as failures. The same model generates the theoretical tables on GPU pages, which are labelled as derived.

Llama 3.3 70B Instruct · FP8 · 2× H100 SXM5 Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 65.4 GB
Context 15102550100 Max conc.
8K ✓ ✓ ✓ — — — 24
16K ✓ ✓ ✓ — — — 12
32K ✓ ✓ — — — — 6
64K ✓ — — — — — 3
128K ✓ — — — — — 1

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Serving configuration

  • vLLM OpenAI-compatible server, with version, CUDA, driver and container image digest recorded.
  • --max-model-len set to the scenario context; --tensor-parallel-size equal to the GPU count.
  • Prefix caching disabled for comparability; speculative decoding off.
  • Default scheduler settings unless a run explicitly states otherwise.

Load generation

vllm bench serve with a fixed random dataset: input length = 50% of the context, output length 512 tokens, --max-concurrency set to the scenario level, --num-prompts = max(200, 10 × concurrency). The client runs on the same host or the same private network; its location is recorded.

Metrics

Metric Definition
TTFT Time from request sent to first output token, p50/p95/p99
TPOT Mean time between output tokens after the first, p50/p95/p99
E2E latency Request sent to last token, p50/p95/p99
Output throughput Output tokens per second across all requests
Error rate Failed or truncated requests ÷ total
Peak GPU memory Maximum used memory from nvidia-smi during the run
Cost per 1M tokens Instance price per hour ÷ (output tokens/s × 3,600) × 10⁶, using the published list price at run date

Import and verification

Results are imported with npm run import:benchmarks from vLLM’s JSON output and a metadata file (provider, region, instance, GPU, model, precision, versions, price source). The importer validates the schema, copies raw output, and always writes the record as pending verification. A record is published only after an independent re-run reproduces output throughput within ±5% and p50 TTFT within ±10%.

Limitations

  • Synthetic random prompts do not reflect cache-friendly production traffic.
  • Throughput depends on vLLM version; results are not comparable across major versions without re-running.
  • MiniMax-M1’s hybrid attention is not covered by the KV formula; its scenarios are evaluated by measured peak memory only.
  • Cost per token uses list prices; discounts change the result.

Benchmarks using this methodology

Other methodologies