Start with the test behind the score.
One model ranks first, another improves by nine points, and a team wants to know which to use. Before answering, open the test description. The score depends on the questions, prompt, tools, time allowed, and scoring rules, as well as the model. Changing those conditions can change the result.
PaperBench makes this easier to see. It asks AI agents to reproduce the results of twenty research papers, with the work divided into 8,316 tasks that can be graded. It also tests the AI judge used to score the submissions.[1] A PaperBench result tells you about performance on that set of research and engineering tasks. It does not measure every activity a researcher might need to do.
When reporting a score, add the conditions that make it meaningful: which model version, doing which tasks, with which tools and limits, judged how? Someone considering the result can then see whether it answers their question and whether another score was produced under comparable conditions.
Read the test conditions before treating a score as a reason to choose a model.
Decide what you need the system to do.
Suppose you need an AI tool to draft customer support replies. A factual question-answering test may reveal whether it knows certain facts. It does not tell you whether it follows your refund policy, asks for missing information, or recognizes when a person should take over. Choose the test around the work you need done.
Evaluation researchers call the underlying ability or behavior a construct; the number used to assess it is a metric. You can make the distinction without using either term: “We need replies that customers understand” is a goal; “reviewers chose this reply” is one observation. Ask what else you would need to check before treating that observation as evidence of the goal.
HELM evaluates several aspects of model behavior together, including accuracy, robustness, fairness, and efficiency.[2] Its approach makes trade-offs visible. A system may answer more questions correctly while costing more or performing poorly for a particular group. Select the measures that matter for your use, and keep their meanings separate.
Check which tasks and users the test includes.
A test labelled “coding” might mostly ask for short functions. Your team might need changes across a large existing project. A test labelled “reasoning” might use short English questions, while your service handles long conversations in Japanese. Read the actual tasks before relying on the label.
Look for repeated templates, unclear questions, incorrect answer keys, and important cases represented by only a handful of examples. A small difference between models may be hard to distinguish from variation in the sampled tasks. Bowman and Dahl’s review of benchmarking argues for sound annotation, sufficient statistical power, and attention to difficulty and bias—not merely harder questions.[3]
Make a simple coverage list for your intended work. Which task types, languages, users, tools, and task lengths appear in the test? Which are scarce or absent? Record deliberate exclusions too. If an important part of your use is missing, plan a separate check before extending the result to it.
Check when the test was made and whether it is still informative.
Public test questions can end up in training data. A higher score may then partly reflect prior exposure rather than an ability to handle new tasks. This is often called benchmark contamination. Even without exact duplicates, repeated practice on similar questions can make a familiar test less informative about unfamiliar work.
Check the dates of the test, its public release, the model, and the evaluation. When possible, compare results on older questions with results on fresh or held-back ones. Evidence of copied wording can raise concerns, but failing to find it does not prove the test was unseen. Say when you cannot establish whether prior exposure affected the result.
LiveBench reduces this risk by regularly adding questions from recent sources and scoring them against objective answers.[4] Updating a test also changes the comparison: a score on new questions is not automatically comparable with last month’s score. Keep track of test versions and use shared items where useful. Decide what change—a new model, a changed workflow, or signs of exposure—will require another evaluation.
An old score may answer a question your current workflow no longer asks.
Check the scoring as carefully as the answers.
A scoring rule can miss something important. Exact text matching can reject a correct answer expressed differently. Code tests check only the behaviors they cover. Expert review can assess richer work, but reviewers need clear criteria. An AI judge can review more answers quickly, yet may respond to presentation as well as substance.
Write the criteria before comparing named models. Have people with relevant expertise review a sample of both passes and failures, and examine disagreements. In side-by-side comparisons, swap answer order. Try a small wording change in the judging instructions. If these changes reverse the ranking, show that uncertainty instead of presenting a stable winner.
Chatbot Arena gathers human preferences through paired comparisons and examines their reliability.[5] That is useful evidence about which answers people prefer in that setting. It does not, by itself, establish that the preferred system is safer, more factual, or more suitable for your specialist task. Check who judged the answers and what they were asked to judge.
Look at where the failures happen.
An overall average can hide failures concentrated in one task or group. The calculation also matters. Giving each category equal weight produces a different summary from giving each individual question equal weight. Counting a task as successful if any of several attempts passes answers a different question from measuring first-attempt success.
The illustration below shows two hypothetical sets of twenty cases. Both have an 80% success rate, but in one set most failures fall in a single group. That is why a useful report shows results for the groups and tasks that matter, along with uncertainty and variation between runs. When comparing models, examine their answers to the same tasks and the seriousness of their mistakes.
The same score can hide different failures
Hypothetical cases · 4 failures out of 20 in each panel · 80% overall success
Spread across groups
Concentrated in one group
Read failed runs to understand what needs fixing: missing knowledge, a lost instruction, a tool error, or unclear grading. Check passes too. In a METR study, some code submissions passed the automated tests but still needed changes before maintainers could use them.[8] A test can correctly check one part of the work while missing other requirements.
The same overall score can hide very different failures.
Try promising systems in the work you actually do.
A benchmark can help you make a shortlist. The next question is whether its successes carry over to your workflow. Real tasks may involve private tools, incomplete requests, interruptions, changing information, and quality standards that an automatic check does not cover.
PaperBench includes understanding, implementation, and experiment execution in its replication tasks.[1] METR’s time-horizon work takes another approach: it relates an agent’s chance of success to how long a task takes a human expert.[6] The time describes human task length, not how long the AI runs. Both approaches reveal more than a short answer test, but still concern the tasks and conditions studied.
Try representative tasks with your actual tools and domain reviewers before relying on a system. Start where outputs can be reviewed without affecting users. Measure completion time, correction work, important errors, and whether the result is usable. In METR’s early-2025 study of experienced open-source developers, participants believed AI helped them work faster even though measured completion times increased.[7] The study concerns that setting; its broader lesson is to measure the outcome you care about rather than infer it from a score or an impression.
Use these questions to turn a score into a decision with clear limits. An unanswered question identifies evidence still needed.
Report what the result supports and when to check again.
End with a decision someone can assess. For example: “These results support a supervised pilot for routine English replies; we have not tested refund exceptions or other languages.” This hypothetical conclusion is more useful than calling a model “best.” It names the use, the limit, and the next step.
Save enough detail to understand and repeat the evaluation: model version, prompts, tools and permissions, task set, time or cost limits, scoring rules, reviewer or judge version, and how repeated attempts were counted. Keep the calculation that produced the summary. If a material setting changes, label the new result accordingly.
Assign someone to review new evidence and specify when to retest. A model update, changed user group, revised tool, or failure outside the test’s coverage may be enough. The result may justify a pilot, a change to one workflow, further testing, or keeping the current system. State which of those decisions the evidence supports.