Benchmarking

MLPerf is the industry-standard benchmark suite for measuring ML system performance. Developed by a consortium of industry and academic organizations, MLPerf provides standardized workloads, datasets, and measurement rules that enable fair comparison across different hardware platforms, frameworks, and system configurations.

Training and Inference Benchmark Suites

MLPerf includes separate benchmark suites for training and inference. The training benchmarks measure time-to-train for workloads spanning image classification (ResNet-50), object detection (SSD, RetinaNet), natural language processing (BERT), recommendation (DLRM), and reinforcement learning. The inference benchmarks measure throughput and latency across similar workloads under different scenarios.

Standardized benchmarks serve a critical role in the ML ecosystem by providing objective comparison points. However, they also have limitations. Benchmark workloads may not represent actual production workloads. Optimizations specific to benchmark rules may not generalize. And the competitive pressure to achieve top benchmark results can distort engineering priorities.

Benchmarks shape the industryDeeper InsightMLPerf results directly influence hardware purchase decisions worth billions of dollars annually. When a chip vendor posts a record-breaking MLPerf score, it can shift procurement decisions at hyperscalers and enterprises worldwide. The benchmarks don't just measure performance — they define what "good performance" means, steering R&D investment and architectural choices across the entire ML hardware ecosystem. Click to collapse

Domain-Specific Benchmarks

Beyond MLPerf, domain-specific benchmarks serve particular needs. DAWNBench focuses on training cost efficiency. DeepBench isolates individual operation performance. Application-specific benchmarks in areas like autonomous driving, medical imaging, and natural language understanding provide more relevant metrics for practitioners in those domains.

• MLPerf — Industry-standard suite for cross-platform comparison of training and inference.
• DAWNBench — Focuses on cost-to-accuracy, measuring dollar efficiency of training.
• DeepBench — Micro-benchmarks isolating individual DNN operations (GEMM, convolution).
• Application benchmarks — Domain-specific workloads for autonomous driving, NLP, medical imaging.

Profiling is the systematic measurement of where time and resources are spent during model training or inference. Without profiling, optimization is guesswork. Profiling tools reveal the actual bottlenecks, which are often surprising and not where engineers initially expect them.

Hardware-Level Profilers

GPU profilers like NVIDIA Nsight Systems and Nsight Compute provide detailed visibility into GPU utilization, memory bandwidth usage, kernel execution times, and communication overhead. Nsight Systems gives a timeline view of the entire application, showing how CPU, GPU, and network activities overlap. Nsight Compute provides deep analysis of individual kernel performance.

Framework-Level Profilers

Framework-level profilers like PyTorch Profiler and TensorBoard Profiler operate at a higher level of abstraction, showing time spent in specific operations, memory allocation patterns, and data loading bottlenecks. These tools are easier to use than hardware-level profilers and are sufficient for identifying most common performance issues.

import torch
from torch.profiler import profile, ProfilerActivity

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    schedule=torch.profiler.schedule(wait=class="tok-number">1, warmup=class="tok-number">1, active=class="tok-number">3),
    on_trace_ready=torch.profiler.tensorboard_trace_handler(&class="tok-comment">#class="tok-number">39;./log&#class="tok-number">39;),
    record_shapes=True,
    with_stack=True,
) as prof:
    for step, batch in enumerate(dataloader):
        train_step(model, batch)
        prof.step()

Systematic Profiling Methodology

Effective profiling requires a systematic methodology. Start with a high-level timeline to identify whether the bottleneck is in data loading, forward pass, backward pass, or communication. Then drill down into the bottleneck using more detailed tools. Measure under realistic conditions because bottlenecks shift with configuration.

1. Capture a high-level timeline (Nsight Systems or framework profiler) to see the big picture.
2. Identify the dominant bottleneck: data loading, forward pass, backward pass, or communication.
3. Drill down with a detailed profiler (Nsight Compute for GPU kernels, VTune for CPU).
4. Optimize the identified bottleneck and re-profile to confirm improvement.
5. Repeat — fixing one bottleneck often reveals the next one.

The roofline model is a visual performance analysis framework that determines whether a workload is compute-bound or memory-bound. It plots achievable performance (FLOP/s) as a function of operational intensity (FLOPs per byte of memory accessed), with the hardware's peak compute and peak memory bandwidth forming the "roofline."

Memory-Bound Workloads

A workload that falls on the sloped portion of the roofline is memory-bound: performance is limited by memory bandwidth rather than compute capability. Optimization should focus on reducing memory accesses through techniques like operator fusion, data layout optimization, or increased computation reuse.

Compute-Bound Workloads

A workload that falls on the flat portion of the roofline is compute-bound: performance is limited by the hardware's peak computation rate. For compute-bound workloads, optimization should focus on using more efficient operations, lower precision formats, or hardware with higher peak throughput.

In practice, most neural network operations fall into one of two categories. Large matrix multiplications (convolutions, attention, linear layers with large batch sizes) tend to be compute-bound. Element-wise operations, batch normalization, and small matrix operations tend to be memory-bound. Understanding this distinction guides optimization strategy for each part of the model.

Accurate performance measurement requires careful methodology to avoid common pitfalls. Warmup runs must precede measurement to ensure caches are populated, JIT compilation is complete, and GPU clock rates have stabilized. Without warmup, initial runs will be misleadingly slow.

Statistical Rigor

Statistical rigor is essential for meaningful performance comparisons. Report confidence intervals, not just averages. Use sufficient repetitions to establish statistical significance. Measure with and without competing system load to understand sensitivity. Small differences (under 5%) between configurations may not be meaningful given measurement noise.

Controlling the Environment

Benchmarking environments must be carefully controlled. GPU clock rates, power limits, driver versions, framework versions, and competing processes all affect measurements. Documenting and controlling these factors enables reproducible benchmarks.

• Lock GPU clock rates using nvidia-smi to prevent frequency scaling during measurement.
• Record driver version, CUDA version, cuDNN version, and framework version.
• Disable GPU boost or set a fixed power limit for consistent thermal behavior.
• Use process isolation (taskset, numactl) to pin CPU cores and avoid NUMA effects.
• Use containerized environments (Docker, Singularity) to standardize the software stack.
• Report hardware SKU, memory configuration, and interconnect topology.

Common Measurement Mistakes

Common measurement mistakes include cherry-picking the best run instead of reporting statistics, measuring artificial workloads that do not represent production use, ignoring data loading and preprocessing overhead, and comparing across different batch sizes or precision without normalization.

Effective benchmarking starts with clearly defining what you want to measure and why. Are you comparing hardware platforms? Evaluating optimization techniques? Selecting between model architectures? The measurement approach should be tailored to answer the specific question at hand.

End-to-End vs. Micro-Benchmarks

End-to-end benchmarking captures the true system performance by measuring the complete pipeline from data loading through inference and postprocessing. Micro-benchmarks isolate individual operations or components for detailed analysis. Both perspectives are valuable, but end-to-end measurements are what ultimately matter for production performance.

Reproducibility

Reproducibility is a key quality indicator for benchmarks. Others should be able to independently verify your results given the same hardware, software, and configuration. Publishing benchmark code, configurations, and raw measurement data (not just summary statistics) enables community verification and builds trust in the results.

Continuous Benchmarking in CI/CD

Benchmarking should be integrated into the CI/CD pipeline as a continuous practice, not a one-time event. Performance regression detection catches optimizations that inadvertently slow other parts of the system. Trend tracking over time reveals the compounding effect of many small improvements or regressions.

# Example: GitHub Actions benchmark workflow
name: Performance Regression Check
on: [pull_request]
jobs:
  benchmark:
    runs-on: [self-hosted, gpu]
    steps:
      - uses: actions/checkout@v4
      - name: Run benchmark suite
        run: python benchmarks/run_all.py --output results.json
      - name: Compare with baseline
        run: python benchmarks/compare.py --baseline main --current results.json --threshold 5