How to read an AI benchmark claim

Only as far as the test was run fairly and resembles your own work. A score is a measurement on a fixed set of tasks under one setup. Research finds three recurring problems. Test material leaks into training data, top models bunch near the ceiling and some tests do not measure what their names suggest. The seven questions below filter most weak claims. The strongest evidence is a small evaluation on your own tasks.

A benchmark is a fixed collection of questions or tasks with a scoring rule. It lets different models be compared on the same material. That is useful. It is also the source of each of the problems below.

Validity: does the test measure the thing?

A large multi-institution review, Measuring what Matters, was presented at NeurIPS 2025. Twenty-nine expert reviewers examined 445 benchmarks from leading conferences. They found recurring patterns in how phenomena are defined, how tasks are chosen and how results are scored. Those patterns undermine the claims built on them. The reviewers set out eight recommendations for benchmark designers. The practical lesson for readers is that a benchmark with reasoning or safety in its name does not thereby measure reasoning or safety.

Saturation: can the test still separate models?

When leading models all score close to the maximum, the benchmark stops telling them apart. A study presented at ICML 2026, When AI Benchmarks Plateau, examined 60 language model benchmarks. It found that nearly half showed saturation, with the rate rising as benchmarks age. One finding cuts against intuition. Keeping test data private did not protect a benchmark from saturating, while expert curation of the questions did.

Contamination: has the model seen the answers?

Models are trained on very large amounts of public text. Public benchmarks are public text. Two studies show how this can be checked. The authors of a 2024 paper on grade-school arithmetic wrote a fresh set of problems mirroring a popular benchmark. Some model families scored up to 8% lower on the new set, a sign of overfitting to the original. The frontier models of the time showed little such effect. In 2025, The SWE-Bench Illusion tested models on a widely cited coding benchmark. It found they could identify the file containing a bug from the issue description alone with up to 76% accuracy. On repositories outside the benchmark the figure was 53%. The authors read that gap as evidence that part of the headline performance was memorisation.

Vendor-run evaluations: who held the pen?

Most launch-day numbers are produced by the company selling the model. That is not dishonest in itself. It does leave room for choices that flatter. The Leaderboard Illusion (2025) studied one prominent public leaderboard and reported that some providers tested many private model variants before release and published only the best. The paper counted 27 private variants from one provider ahead of a single launch. It also estimated that two large providers each received about a fifth of the leaderboard's data, while 83 open-weight models shared under a third between them.

Even an honest setup involves decisions that move a score. That means the prompt, the number of attempts and whether the model had tools. The UK's AI Security Institute has noted that it is hard to know whether an evaluation has found the ceiling of what a system can do. Prompting, fine-tuning and tool access all change the result. The same post says the science is too young for independent evaluations to work as certification.

Noise: is the difference real?

A benchmark is a sample of questions, so every score carries sampling error. A 2024 paper, Adding Error Bars to Evals, argues that evaluations should be treated as experiments and reported with confidence intervals and proper statistical comparisons. Launch tables rarely include them. Without them, a gap of a point or two between two models may not mean anything.

Benchmark tables drive purchasing decisions, press coverage and internal business cases. If the numbers are inflated by memorisation, flattened by saturation or selected from many private attempts, decisions built on them inherit the error.

There is also a gap between test performance and usefulness. Stanford's 2026 AI Index describes a jagged frontier. Models that have won gold at the International Mathematical Olympiad read analogue clocks correctly only about half the time. Strong results in one area say little about the next. The same caution applies to agents, as covered in the note on AI agents and the Model Context Protocol.

Standards bodies have started to respond. In January 2026 the US National Institute of Standards and Technology published a draft of NIST AI 800-2. It sets out voluntary practices for automated benchmark evaluations. They cover how to choose benchmarks, run them and analyse and report the results. Its stated audience includes people who read evaluation reports, such as procurement specialists, as well as the engineers who produce them. The NIST page still described it as an initial public draft when checked on 2 October 2026.

When a benchmark figure appears in a pitch or a press release, these seven questions are the ones I would put to it:

1. What exactly was measured? Read the task description instead of the benchmark's name.
2. Who ran it? The vendor, the benchmark's authors or an independent third party.
3. What was the setup? Prompts, number of attempts, tools and any scaffolding around the model. If this is not disclosed, the number cannot be compared with anyone else's.
4. How old is the benchmark and is it saturated? If every leading model scores within a few points of the top, the test has stopped discriminating.
5. Could the model have seen the test? Ask whether there is a fresh or held-out version and how scores change on it.
6. How large is the gap relative to the uncertainty? Ask for the number of questions and a confidence interval.
7. Does the task resemble your work? If not, the result is background at best.

The last question points to the most reliable option, which is also the least glamorous. Collect a few dozen examples of your own work. Define what a good answer looks like. Then compare candidate systems on those. That is my opinion. It is not a research finding. Even so, it follows from everything above. A test nobody else has seen cannot be contaminated. It measures the thing you care about.

The same practice of asking for primary evidence applies to regulation. See the EU AI Act timeline. The evaluation research and tools cited here are collected in the reading list.