Skip to content
Cloud Provider Benchmarks

GPU & AI infrastructure

NVIDIA H100 vs L40S: which GPU fits your inference workload?

H100 vs L40S compared on memory, bandwidth, FP8 compute and model fit, with derived decode ceilings and published cloud prices per GPU-hour.

Published: Updated: Türkçe oku

Direct answer

Use the L40S for small models (up to roughly 14B parameters at FP8), low-to-moderate concurrency and visual workloads where its lower hourly price matters. Use the H100 for 30B+ models, long contexts, latency-sensitive serving at high concurrency or any multi-GPU tensor-parallel deployment.

Executive summary

The L40S is an Ada Lovelace PCIe card with 48 GB of GDDR6 at 864 GB/s; the H100 SXM is a Hopper part with 80 GB of HBM3 at 3.35 TB/s and NVLink. The H100 has about 3.9× the memory bandwidth and 2.7× the dense FP8 Tensor Core throughput. The L40S is a cost-oriented option for 7B–14B models, image generation and mixed graphics/AI work; the H100 is the better fit for 30B+ models, high concurrency and multi-GPU serving.

Key findings

  1. 01 Memory bandwidth differs by about 3.9× (3,350 vs 864 GB/s), so the single-sequence decode ceiling for an 8B FP8 model is about 417 tokens/s on H100 SXM versus about 108 on L40S.
  2. 02 At FP8 a 32B dense model fits on one L40S with only about 8 GB of KV budget — about three concurrent 8K-token sequences under the assumptions used here.
  3. 03 The L40S has no NVLink and no MIG; multi-GPU serving uses PCIe, which limits tensor-parallel scaling.
  4. 04 Published L40S list prices are lower per GPU-hour than H100 prices at the same provider, but price per unit of bandwidth often favours the H100.

Two different design points

The H100 is a data-centre training and inference accelerator with HBM memory and NVLink. The L40S is a universal data-centre GPU built on Ada Lovelace: a PCIe card with GDDR6 memory, RT cores and video engines alongside FP8-capable Tensor Cores. They overlap for inference, but they are not substitutes at every model size.

GPU specifications (NVIDIA datasheets)
GPU Architecture Memory (GB) Memory type Bandwidth (GB/s) FP8 (TFLOPS) BF16 (TFLOPS) FP32 (TFLOPS) Interconnect TDP MIG
H100 SXM5 SXM5 Hopper 80 HBM3 3,350 1,979 989 67 NVLink 4 900 GB/s Up to 700 W Up to 7 MIGs @ 10 GB
L40S PCIe dual-slot Ada Lovelace 48 GDDR6 (ECC) 864 733 362 91.6 PCIe only 350 W Not supported

Tensor values are dense (non-sparse) peaks; NVIDIA publishes sparse figures at twice these values.

Property H100 SXM L40S H100 ÷ L40S
Memory 80 GB HBM3 48 GB GDDR6 1.67×
Bandwidth 3,350 GB/s 864 GB/s 3.88×
FP8 dense TFLOPS 1,979 733 2.70×
Interconnect NVLink 900 GB/s PCIe Gen4 only —

Model fit

Weights at FP8 take about one byte per parameter. With 48 GB of device memory, the L40S’s usable budget under the standard assumptions (90% utilisation, 5% reserve) is about 41 GB, so models up to about 14B parameters leave meaningful KV-cache space. A 32B model fits, but with little room left for concurrent sequences.

Model memory requirements and minimum GPU count (FP8 weights)
Model Params (B) BF16 (GB) FP8 (GB) INT4 (GB) KV / token (KiB) KV @32K (GB) Min H100 SXM5Min L40S
Llama 3.1 8B Instruct Dense · GQA 8 16 8 4 128 4.3 1×1×
Llama 3.3 70B Instruct Dense · GQA 70.6 141 71 35 320 10.7 2×2×
Qwen2.5 7B Instruct Dense · GQA 7.6 15 8 4 56 1.9 1×1×
Qwen3 32B Dense · GQA 32.8 66 33 16 256 8.6 1×1×
Qwen3 30B-A3B MoE · 3.3B active · GQA 30.5 61 31 15 96 3.2 1×1×

Weight size = parameters × bytes per parameter. KV = 2 × layers × KV heads × head dim × 2 bytes (latent size for MLA). Minimum GPUs: 85% usable memory and at least 4 GB of KV headroom per GPU. A capacity model, not a measurement.

Llama 3.1 8B Instruct · FP8 · 1× L40S Executable scenario matrix (KV-cache capacity model) · KV budget ≈ 32.8 GB
Context 15102550100 Max conc.
8K ✓ ✓ ✓ ✓ — — 30
16K ✓ ✓ ✓ — — — 15
32K ✓ ✓ — — — — 7
64K ✓ — — — — — 3
128K ✓ — — — — — 1

✓ scenario fits the memory model · — KV cache exceeded (not executed) · ✕ weights do not fit · n/a context above model maximum. Model: 90% memory utilisation, 5% runtime reserve, 16-bit KV cache.

Decode ceilings

Because decode streams the active weights for each token, bandwidth sets the upper bound at low batch sizes. The ratio follows bandwidth almost exactly.

Memory-bandwidth ceiling for single-stream decode (FP8, tok/s) Theoretical upper bound — not a measurement
Model H100 SXM5L40S
Llama 3.1 8B Instruct 417 108
Qwen2.5 7B Instruct 440 114
Qwen3 32B 102 26
Qwen3 30B-A3B 1,015 262

Ceiling = aggregate memory bandwidth ÷ active weight bytes read per token (batch 1, excluding KV reads and communication, assuming ideal tensor parallelism). Real engines reach a fraction of this bound.

Mixture-of-experts models such as Qwen3 30B-A3B are an interesting middle case: only about 3.3B parameters are active per token, so the decode ceiling is high even on the L40S — but all 30.5B parameters must still fit in memory.

Price per GPU-hour

Published per-GPU-hour list prices
Provider GPU Per GPU-hour Per GB-hour Basis
PortvMind NVIDIA H100 TRY 195.835 TRY 2.4479 gpu.h100s.xlarge (1× H100) hourly list price, tr-ist-01, excluding tax. Multi-GPU profiles price from ₺141.12 per GPU-hour (3× and 4× profiles). · 6 November 2026
DigitalOcean NVIDIA H100 $4.410 $0.0551 On-demand, 1× H100 GPU Droplet. · 6 November 2026
DigitalOcean NVIDIA L40S $1.570 $0.0327 On-demand, 1× L40S GPU Droplet. · 6 November 2026
Vultr NVIDIA H100 $2.300 $0.0287 On-demand "starting at" per-GPU rate, 8-GPU HGX configuration. · 6 November 2026
Vultr NVIDIA L40S $1.671 $0.0348 On-demand per-GPU rate, 1× L40S. · 6 November 2026

The L40S is cheaper per hour wherever both are listed. Dividing price by memory bandwidth gives a rough cost per unit of decode capacity; on that basis the gap narrows or reverses, because the H100 offers almost four times the bandwidth. The right answer depends on whether your workload saturates the GPU.

When each GPU fits

  • L40S: 7B–14B chat or embedding models, retrieval-augmented generation with modest concurrency, Stable Diffusion-class image generation, video transcoding with AI post-processing, and development environments.
  • H100: 30B–70B models, long-context workloads, high-concurrency APIs with strict latency targets, fine-tuning, and tensor-parallel serving across NVLink.

Methodology

Derived values use the formulas in the GPU and LLM inference methodology. Measured runs are reported on the L40S and H100 benchmark pages.

Limitations

  • The H100 PCIe variant (2.0 TB/s) narrows the bandwidth gap to about 2.3×; providers do not always state which variant they use.
  • Prefill-heavy workloads (long prompts, short answers) depend more on compute than bandwidth.
  • Sparse-TFLOPS figures are not used; all compute values are dense.
  • Prices are list prices at the access date and exclude committed-use discounts.

Conclusion

The L40S is a sensible, lower-cost choice when the model is small and the workload is not bandwidth-saturated. Once models reach 30B parameters, contexts grow long or concurrency rises, the H100’s memory capacity, bandwidth and NVLink make it the more predictable platform.

Frequently asked questions

Is the L40S good for LLM inference?

For small models, yes. An 8B model at FP8 leaves about 33 GB of KV budget on one L40S. For 30B+ models the 48 GB capacity and 864 GB/s bandwidth become limiting, and two or more cards connected over PCIe are needed.

Does the L40S support FP8?

Yes. The L40S has fourth-generation Tensor Cores with FP8 support through the Transformer Engine, at a lower peak throughput than the H100.

Why compare price per GB/s instead of price per hour?

Token generation is usually memory-bandwidth-bound, so the cost of moving bytes is a better first-order proxy for decode cost than the hourly price alone. It is still a proxy — measured cost per million tokens is the definitive metric.

Sources

  1. [1] NVIDIA H100 Tensor Core GPU — NVIDIA Vendor datasheet accessed:
  2. [2] NVIDIA L40S GPU — NVIDIA Vendor datasheet accessed:
  3. [3] DigitalOcean GPU Droplet pricing — DigitalOcean Official documentation accessed:
  4. [4] Vultr Cloud GPU — NVIDIA L40S — Vultr Official documentation accessed:
  5. [5] Amazon EC2 G6e instances — AWS Official documentation accessed: