ML Wiki

Benchmark

4 min read

The Problem

You built a model. Is it any good? Is it better than the last model? Is it good enough to ship? Without a fixed reference test, every claim of progress is unverifiable. The general problem: how to construct a single, fixed measurement that tracks model quality over time and across systems.

The Key Insight

A benchmark is a fixed task + a fixed dataset + a fixed metric. By keeping all three constant, the benchmark turns model quality into a one-dimensional comparison. The artifact — the dataset, the test set, the evaluation harness, the leaderboard — is a public good that lets the field share a yardstick.

Mechanism in Plain English

A good benchmark consists of:

  1. A task definition — what is the model supposed to do? “Classify images into one of 1000 categories” (ImageNet); “answer multiple-choice questions across 57 subjects” (MMLU).
  2. A dataset split — train, validation, test. Test labels are held out (or only used for final evaluation).
  3. A metric — accuracy, F1, nDCG, BLEU, etc. Single-number summary that ranks models.
  4. A baseline — the simplest plausible model, to anchor what “better than nothing” means.
  5. A leaderboard — public record of all submissions, ideally with code/weights for reproducibility.

The benchmark is not the same as evaluation. Evaluation is the broader practice; a benchmark is one specific operationalization.

ASCII Diagram

BENCHMARK STRUCTURE:

  TASK: What does the model do?
       (e.g., "Retrieve the 10 most relevant passages for a query")

  DATASET: train + val + test
       (e.g., MS MARCO: 8.8M passages, 7K test queries with labels)

  METRIC: How is performance measured?
       (e.g., MRR@10 - mean reciprocal rank of the first relevant doc in top-10)

  HARNESS: How is the evaluation run?
       (e.g., the BEIR or MTEB toolkit)

  LEADERBOARD: Public ranking of submissions.
       (e.g., paperswithcode.com/sota/passage-retrieval-on-ms-marco)

  --> Submit a model, get a single number, compare to others.

Concrete Walkthrough

ImageNet, the canonical benchmark:

TASK:        Image classification, 1000 classes.
DATASET:     ~1.2M training images, 50K validation, 100K test.
METRIC:      Top-1 and Top-5 accuracy.
BASELINE:    Random guessing = 0.1% top-1.

PROGRESSION:
  2010 (pre-deep-learning):    72.8% top-5  (handcrafted features)
  2012 (AlexNet):              84.7% top-5  (first big CNN)
  2015 (ResNet-152):           96.4% top-5  (residual connections)
  2020 (NoisyStudent):         98.1% top-5  (semi-supervised)
  2024 (saturation):           99.0%+ top-5 (no further progress meaningful)

ImageNet was retired as a primary benchmark around 2020. Replaced by:
  - ImageNet-V2 (out-of-distribution test)
  - JFT-3B (Google's larger internal benchmark)
  - DataComp / LAION evaluation suites

What’s Clever

The first clever recognition: a benchmark must be hard enough that no current model solves it. ImageNet in 2009 was hard — best models hit 75% top-5. By 2017 it was solved at 95%+. Once solved, further “improvement” is overfitting, not progress.

The second clever recognition: benchmarks must be public and reproducible. Internal Google benchmarks (JFT-300M, JFT-3B) drove a lot of progress but couldn’t compare cross-organization. Public benchmarks (ImageNet, GLUE, MMLU) enable the field-wide leaderboard race that drives progress.

The third recognition: single-number rankings are both feature and bug. They make comparison easy, but they hide multidimensional model behavior. A model with 90% accuracy and high robustness is preferable to a model with 91% accuracy and brittle to distribution shift — but a single accuracy number doesn’t capture this. HELM and similar holistic evaluations push back against single-number rankings.

The fourth recognition: benchmarks decay. Three failure modes:

  • Saturation: top models reach human ceiling; further “progress” is overfitting.
  • Contamination: test data leaks into pretraining data, especially for web-scale LLMs.
  • Goodhart pressure: when a metric becomes a target, it ceases to measure what you wanted. Models are optimized for the benchmark, not the underlying capability.

Healthy benchmarks anticipate decay: refresh test sets, keep some tasks held-out, design for forward compatibility.

Key Sources

Open Questions

  • How to design contamination-resistant benchmarks? For LLMs trained on web-scale data, almost any public benchmark may have leaked. Solutions: dynamic benchmarks (changing periodically), held-out private tests (hard to reproduce), or contamination-aware metrics.
  • When to retire a benchmark? A clean rule would help, but this is currently driven by community consensus.
  • Can benchmarks measure “alignment” or “safety”? These are values-laden and contested. Hard to operationalize as a single metric.
  • Live benchmarks vs. fixed benchmarks: live (continuously refreshed) benchmarks resist contamination but lose comparability over time. Fixed benchmarks are comparable but contamination-prone. Hybrid designs are an active research direction.

Graph View

Backlinks