Cloud Provider Benchmarks Cloud infrastructure research

Menu

Benchmarks · GPU & AI

Multi-user LLM inference: concurrency and context scenarios on H100, H200 and L40S

Serving many users at once is limited first by KV-cache memory: every active sequence reserves cache proportional to its context length. The scenario matrices below show, for each GPU and model, which combinations of 1, 5, 10, 25, 50 and 100 concurrent requests and 8K to 128K context fit in memory. Only executable scenarios are scheduled for measurement.

Updated: 6 November 2026

Direct answer

Maximum concurrency ≈ (usable GPU memory − weights) ÷ (KV bytes per token × context length). On one H200, Llama 3.3 70B in FP8 sustains far more concurrent 8K sessions than on one H100, while 128K contexts reduce concurrency on every GPU by sixteen times relative to 8K.

What is measured

Scenario feasibility for 6 concurrency levels × 5 context lengths
KV-cache budget per GPU configuration
Measured TTFT and TPOT degradation as concurrency rises (verified runs)
Error rate under load

Llama 3.1 8B Instruct · single GPU

Scenario matrix for each GPU model.

Llama 3.1 8B Instruct · BF16 · 1× H100 SXM5 Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 51.9 GB
Context	1	5	10	25	50	100	Max conc.
8K	✓	✓	✓	✓	—	—	48
16K	✓	✓	✓	—	—	—	24
32K	✓	✓	✓	—	—	—	12
64K	✓	✓	—	—	—	—	6
128K	✓	—	—	—	—	—	3

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Llama 3.1 8B Instruct · BF16 · 1× H200 SXM Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 103.8 GB
Context	1	5	10	25	50	100	Max conc.
8K	✓	✓	✓	✓	✓	—	96
16K	✓	✓	✓	✓	—	—	48
32K	✓	✓	✓	—	—	—	24
64K	✓	✓	✓	—	—	—	12
128K	✓	✓	—	—	—	—	6

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Llama 3.1 8B Instruct · BF16 · 1× L40S Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 24.7 GB
Context	1	5	10	25	50	100	Max conc.
8K	✓	✓	✓	—	—	—	23
16K	✓	✓	✓	—	—	—	11
32K	✓	✓	—	—	—	—	5
64K	✓	—	—	—	—	—	2
128K	✓	—	—	—	—	—	1

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Qwen3 30B-A3B · single GPU

Scenario matrix for each GPU model.

Qwen3 30B-A3B · FP8 · 1× H100 SXM5 Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 37.5 GB
Context	1	5	10	25	50	100	Max conc.
8K	✓	✓	✓	✓	—	—	46
16K	✓	✓	✓	—	—	—	23
32K	✓	✓	✓	—	—	—	11
64K	✓	✓	—	—	—	—	5
128K	✓	—	—	—	—	—	2

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Qwen3 30B-A3B · FP8 · 1× H200 SXM Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 89.3 GB
Context	1	5	10	25	50	100	Max conc.
8K	✓	✓	✓	✓	✓	✓	110
16K	✓	✓	✓	✓	✓	—	55
32K	✓	✓	✓	✓	—	—	27
64K	✓	✓	✓	—	—	—	13
128K	✓	✓	—	—	—	—	6

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Qwen3 30B-A3B · FP8 · 1× L40S Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 10.3 GB
Context	1	5	10	25	50	100	Max conc.
8K	✓	✓	✓	—	—	—	12
16K	✓	✓	—	—	—	—	6
32K	✓	—	—	—	—	—	3
64K	✓	—	—	—	—	—	1
128K	—	—	—	—	—	—	0

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Llama 3.3 70B Instruct · single GPU

Scenario matrix for each GPU model.

Llama 3.3 70B Instruct · FP8 · 1× H100 SXM5 Executable scenario matrix (KV-cache capacity model) · KV budget ≈ -2.6 GB
Context	1	5	10	25	50	100	Max conc.
8K	✕	✕	✕	✕	✕	✕	0
16K	✕	✕	✕	✕	✕	✕	0
32K	✕	✕	✕	✕	✕	✕	0
64K	✕	✕	✕	✕	✕	✕	0
128K	✕	✕	✕	✕	✕	✕	0

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Llama 3.3 70B Instruct · FP8 · 1× H200 SXM Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 49.3 GB
Context	1	5	10	25	50	100	Max conc.
8K	✓	✓	✓	—	—	—	18
16K	✓	✓	—	—	—	—	9
32K	✓	—	—	—	—	—	4
64K	✓	—	—	—	—	—	2
128K	✓	—	—	—	—	—	1

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Llama 3.3 70B Instruct · FP8 · 1× L40S Executable scenario matrix (KV-cache capacity model) · KV budget ≈ -29.8 GB
Context	1	5	10	25	50	100	Max conc.
8K	✕	✕	✕	✕	✕	✕	0
16K	✕	✕	✕	✕	✕	✕	0
32K	✕	✕	✕	✕	✕	✕	0
64K	✕	✕	✕	✕	✕	✕	0
128K	✕	✕	✕	✕	✕	✕	0

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Qwen2.5 72B Instruct · 2× GPU

Scenario matrix for each GPU model.

Qwen2.5 72B Instruct · FP8 · 2× H100 SXM5 Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 63.3 GB
Context	1	5	10	25	50	100	Max conc.
8K	✓	✓	✓	—	—	—	23
16K	✓	✓	✓	—	—	—	11
32K	✓	✓	—	—	—	—	5
64K	✓	—	—	—	—	—	2
128K	✓	—	—	—	—	—	1

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Qwen2.5 72B Instruct · FP8 · 2× H200 SXM Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 167.0 GB
Context	1	5	10	25	50	100	Max conc.
8K	✓	✓	✓	✓	✓	—	62
16K	✓	✓	✓	✓	—	—	31
32K	✓	✓	✓	—	—	—	15
64K	✓	✓	—	—	—	—	7
128K	✓	—	—	—	—	—	3

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Qwen2.5 72B Instruct · FP8 · 2× L40S Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 8.9 GB
Context	1	5	10	25	50	100	Max conc.
8K	✓	—	—	—	—	—	3
16K	✓	—	—	—	—	—	1
32K	—	—	—	—	—	—	0
64K	—	—	—	—	—	—	0
128K	—	—	—	—	—	—	0

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Verified results

GPU & AI infrastructure

Best GPU for LLM inference: matching H100, H200 and L40S to model size

Which GPU to use for LLM inference by model size, precision, context length and concurrency — a sizing guide for Qwen, Llama, DeepSeek and MiniMax.

6 November 2026

GPU & AI infrastructure

Multi-user LLM inference: how concurrency, context and KV cache limit a GPU

How many concurrent users a GPU can serve: KV-cache capacity by context length, why latency rises with load, and how to benchmark concurrency properly.

6 November 2026