Skip to content
Cloud Provider Benchmarks

Methodology

Compute benchmark methodology

How CPU single-thread and multi-thread throughput, memory bandwidth and provisioning time are measured on cloud virtual machines, and how results are verified.

Version 1.0 Updated:

Scope

  • General-purpose virtual machines with 2, 4 and 8 vCPU and 4 GB RAM per vCPU, or the closest published shape
  • CPU single-thread and multi-thread throughput (sysbench cpu)
  • Memory bandwidth (sysbench memory)
  • Provisioning time from API request to SSH availability
  • Sustained-load behaviour (stress-ng) to detect throttling or noisy neighbours
Tools
Tool Version
sysbench 1.0.20
stress-ng Recorded per run

Instance selection

For each provider the shape closest to the target vCPU and memory is selected from the general-purpose family. Shared-vCPU (burstable) and dedicated-vCPU offerings are tested separately and never mixed in one comparison. The instance type, region, CPU model reported by /proc/cpuinfo, kernel version and image are recorded with each run.

Image and preparation

  • Ubuntu 24.04 LTS from the provider’s official image catalogue.
  • Packages updated at boot; no tuning beyond defaults.
  • A 5-minute idle period after boot before measurements start.

Procedure

  1. Provisioning time. Timestamp at the create API call; timestamp when TCP port 22 accepts connections. Repeated for 5 instances.
  2. CPU single-thread. sysbench cpu --threads=1 --time=60 run; the events per second value is recorded.
  3. CPU multi-thread. Same, with --threads equal to the vCPU count.
  4. Memory bandwidth. sysbench memory --memory-block-size=1M --memory-total-size=100G --threads=<vCPU> run.
  5. Sustained load. stress-ng --cpu <vCPU> --timeout 15m followed immediately by a repeat of step 3, to compare throughput after sustained load.

Repetitions and statistics

Each measurement runs 5 times on each of 3 separately provisioned instances (15 samples). The median is published with the minimum, maximum and coefficient of variation. A coefficient of variation above 10% is flagged on the result.

Verification

A result moves from pending verification to verified only when a second operator re-runs the procedure on a fresh instance and the median falls within ±5% of the original. Raw tool output is stored with the result.

Limitations

  • sysbench CPU is a synthetic prime-number workload; it does not represent every application.
  • Results apply to the region and date of testing; providers change hardware without notice.
  • Shared-vCPU results depend on neighbouring load at the time of the run.

Benchmarks using this methodology

Other methodologies