Skip to content
Spike Documentation

Reproducible benchmarking

A useful benchmark answers a narrow question under recorded conditions. It is not a timeless claim about Spike, a GPU model, or a serving engine. Shared cluster load, software versions, input shape, precision, caching, and the selected resource all affect the result.

Define the question first

Section titled “Define the question first”

Examples of answerable questions:

  • Does this input pipeline keep one approved GPU busy?
  • Does four-GPU DDP reduce time per epoch for this fixed model and dataset?
  • Does a new image tag improve median latency without changing numerical output?
  • At what concurrency does an SCC-managed inference endpoint exceed its agreed latency target?

Choose one primary metric before collecting results. Common metrics include step time, samples per second, tokens per second, time to first token, peak GPU memory, and end-to-end job duration.

Freeze the test conditions

Section titled “Freeze the test conditions”

Record these fields with every result:

Area Required record
Workload Run:ai project, workload name, date, compute-resource preset, GPU and worker count
Software Immutable image tag, code revision, framework, CUDA, driver, and library versions
Model Model revision, precision, compilation settings, sequence or input shape
Data Dataset version, sample count, preprocessing, cache state, and data location
Method Warm-up, measured iterations, repeats, synchronization, concurrency, and summary statistic
Result Raw observations, median or percentile, variance, failures, and known limitations

Capture the software and visible GPU environment inside the workload:

Terminal window
python -m torch.utils.collect_env
nvidia-smi \
  --query-gpu=name,uuid,driver_version,memory.total \
  --format=csv

NVIDIA documents all supported nvidia-smi queries. Do not change GPU clocks, power limits, MIG layout, or other administrative settings.

Use a repeatable timing method

Section titled “Use a repeatable timing method”

For a small PyTorch operation, use torch.utils.benchmark rather than a single wall-clock measurement:

from torch.utils.benchmark import Timer


measurement = Timer(
    stmt="model(batch)",
    globals={"model": model, "batch": batch},
).blocked_autorange(min_run_time=2.0)


print(measurement)

For a full training loop:

  1. Warm up model loading, kernels, data workers, and caches.
  2. Synchronize CUDA before starting and stopping a manual timer.
  3. Measure enough steps to include normal input and checkpoint behavior.
  4. Repeat the complete run at least three times when cost permits.
  5. Preserve raw samples and report median plus spread, not only the best run.

Use NVIDIA Nsight Systems only when timeline-level CPU, CUDA, or I/O profiling is required. Profiling adds overhead, so keep profiled runs separate from performance runs.

Diagnose input bottlenecks and GPUDirect Storage

Section titled “Diagnose input bottlenecks and GPUDirect Storage”

Do not assume that storage is the bottleneck because GPU utilisation is low. First compare GPU utilisation, data-wait time, CPU usage, storage throughput, and step time with both cold and warm caches. CPU-side decoding, augmentation, small-file access, and an undersized shared-memory allocation can all leave a GPU waiting.

NVIDIA GPUDirect Storage can reduce copies through CPU memory on a supported storage, driver, container, and application path. Ordinary Python file reads do not start using GDS automatically. Tools such as the NVIDIA DALI NumPy reader can use it in a compatible pipeline, but CPU transformations may remove much of the benefit.

Ask your SCC AI Engineer to confirm the approved image, mount, and input pipeline before testing GDS. Researchers must not install host drivers or change the platform configuration. Benchmark the same dataset, batch, precision, and model with and without the approved path, and report cold-cache and warm-cache results separately. NVIDIA’s GDS getting-started guide describes the platform prerequisites.

Measure distributed scaling

Section titled “Measure distributed scaling”

Compare the same work at one GPU and at the approved larger topology:

speedup = single_gpu_time / distributed_time
efficiency = speedup / number_of_gpus

Keep the global batch size fixed when measuring strong scaling. State clearly when the batch size grows with GPU count, because that is weak scaling and answers a different question. Include checkpoint time and data loading; hiding them overstates application throughput.

Inference load tests

Section titled “Inference load tests”

Inference benchmarks must use an endpoint and concurrency limit agreed with the SCC AI Engineer who manages the deployment. Record:

  • Prompt and generated-token length distributions.
  • Concurrency and arrival pattern.
  • Warm versus cold cache state.
  • Time to first token, inter-token latency, end-to-end latency, and throughput.
  • Error, timeout, and rate-limit counts.

Do not upload sensitive prompts or outputs to a third-party benchmark service. Do not present a result from one model, image, or date as a general platform guarantee.

Reporting template

Section titled “Reporting template”

Publish the exact command, configuration, raw measurements, and limitations with the chart or table. A short report should let another project member rerun the same image and understand why a future result may differ.