Skip to content
Cloud Provider Benchmarks

GPU & AI infrastructure

Multi-user LLM inference: how concurrency, context and KV cache limit a GPU

How many concurrent users a GPU can serve: KV-cache capacity by context length, why latency rises with load, and how to benchmark concurrency properly.

Published: Updated: Türkçe oku

Direct answer

The maximum number of concurrent sequences a GPU can hold is the KV budget divided by the KV size of one sequence at the target context length. For Llama 3.1 8B at FP8 on one H100 that is about 55 sequences of 8K tokens, but only about 3 of 128K tokens. Throughput and latency at those levels must be measured.

Executive summary

Serving many users at once is limited by three things: KV-cache memory (how many sequences fit), compute (how fast batched prefill and decode run) and scheduling (how the engine trades latency for throughput). KV capacity can be calculated from model architecture and GPU memory; the latency effects of load must be measured. This article explains the calculation and the benchmark design used for concurrency levels 1, 5, 10, 25, 50 and 100.

Key findings

  1. 01 KV capacity scales inversely with context length: 16× longer context means 16× fewer concurrent sequences.
  2. 02 Grouped-query attention matters: Qwen2.5 7B uses 4 KV heads and needs about 57 KB per token, less than half of Llama 3.1 8B's 131 KB.
  3. 03 Beyond KV capacity, engines queue or pre-empt requests, which raises time-to-first-token sharply; p95 and p99 latency reveal this before averages do.
  4. 04 Benchmarks only run concurrency/context combinations that fit in memory; infeasible combinations are reported as such rather than as failures.

What limits concurrency

Continuous-batching engines such as vLLM add requests to the running batch as long as there is KV-cache space. Each active sequence holds keys and values for every token it has seen. Once the cache is full, new requests wait or running ones are pre-empted.

KV capacity by context length

The matrix below shows, for each context length and target concurrency, whether the configuration fits, using the standard assumptions.

Llama 3.1 8B Instruct · FP8 · 1× H100 SXM5 Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 60.0 GB
Context 15102550100 Max conc.
8K ✓ ✓ ✓ ✓ ✓ — 55
16K ✓ ✓ ✓ ✓ — — 27
32K ✓ ✓ ✓ — — — 13
64K ✓ ✓ — — — — 6
128K ✓ — — — — — 3

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Qwen3 32B · FP8 · 1× H200 SXM Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 87.0 GB
Context 15102550100 Max conc.
8K ✓ ✓ ✓ ✓ — — 40
16K ✓ ✓ ✓ — — — 20
32K ✓ ✓ ✓ — — — 10
64K ✓ ✓ — — — — 5
128K ✓ — — — — — 2

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Architecture differences

Grouped-query attention reduces KV heads and therefore KV size. Models with fewer KV heads support more concurrent users on the same hardware.

Model memory requirements and minimum GPU count (FP8 weights)
Model Params (B) BF16 (GB) FP8 (GB) INT4 (GB) KV / token (KiB) KV @32K (GB) Min H100 SXM5Min H200 SXMMin L40S
Llama 3.1 8B Instruct Dense · GQA 8 16 8 4 128 4.3 1×1×1×
Llama 3.3 70B Instruct Dense · GQA 70.6 141 71 35 320 10.7 2×1×2×
Qwen2.5 7B Instruct Dense · GQA 7.6 15 8 4 56 1.9 1×1×1×
Qwen3 32B Dense · GQA 32.8 66 33 16 256 8.6 1×1×1×
Qwen3 30B-A3B MoE · 3.3B active · GQA 30.5 61 31 15 96 3.2 1×1×1×

Weight size = parameters × bytes per parameter. KV = 2 × layers × KV heads × head dim × 2 bytes (latent size for MLA). Minimum GPUs: 85% usable memory and at least 4 GB of KV headroom per GPU. A capacity model, not a measurement.

How load affects latency

At low concurrency, TTFT is mostly prefill time and TPOT is close to the bandwidth-bound decode time. As concurrency rises:

  • Throughput (output tokens per second across all users) rises, then plateaus when compute saturates.
  • TPOT rises gradually because each decode step processes a larger batch.
  • TTFT stays flat until the KV cache or the scheduler budget fills, then rises steeply as requests queue.

The point where p99 TTFT begins to climb is the practical capacity of the deployment.

Benchmark design

The multi-user inference benchmark runs vllm bench serve at concurrency 1, 5, 10, 25, 50 and 100 and contexts of 8K, 32K, 64K and 128K, only where the scenario fits. It records TTFT, TPOT, end-to-end latency at p50/p95/p99, output throughput, error rate and peak GPU memory. Results are reported once a run has been reproduced.

Limitations

  • Capacity figures assume every sequence reaches the full context length; real traffic with mixed lengths fits more sequences.
  • Prefix caching, FP8 KV cache and speculative decoding change capacity and latency and are recorded as run parameters.
  • Network latency between client and server is excluded from server-side metrics.

Conclusion

Concurrency capacity is calculable; concurrency performance is not. Use KV arithmetic to choose feasible configurations, then measure tail latency at increasing load to find where the deployment stops meeting its target.

Frequently asked questions

How many users can one H100 serve?

It depends on model and context. With Llama 3.1 8B at FP8 and 8K-token sequences, about 55 sequences fit in KV cache. Whether latency at that concurrency meets your target must be measured.

What is TTFT and TPOT?

Time to first token measures the delay until the first output token, dominated by queueing and prefill. Time per output token measures the interval between subsequent tokens, dominated by decode.

Why report p95 and p99?

Under load, a minority of requests wait in the queue. Averages hide them; tail percentiles show the experience of the slowest users.

Sources

  1. [1] vLLM documentation — vLLM project Official documentation accessed:
  2. [2] Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., SOSP 2023 Secondary source accessed:
  3. [3] Llama 3.1 8B Instruct model card — Meta Official documentation accessed:
  4. [4] Qwen2.5 7B Instruct model card — Qwen Official documentation accessed: