Keep the result and the question behind it.
Suppose an assistant scores 72% on your test. A few months later, a new version produces different answers. The old score still describes the original test, but may no longer guide today’s choice. Keep the model, test, settings, and date with the result so the reader knows what it refers to.
This problem predates today’s language models. HAPI collected more than 1.7 million predictions from commercial machine-learning services between 2020 and 2022. Its authors found substantial changes over time, including falling accuracy on some tests and changing error patterns even when the overall score stayed steady.[1] The same product name did not guarantee the same behavior.
Alongside the score, record what the study helped you understand. Which task became easier? Who benefited? Which errors still needed review? What could change that explanation? These questions give a later study something useful to test even when the original model is no longer available.
Keep the question and comparison that make a changed result understandable.
Say what the evidence actually shows.
One successful answer shows what happened on that attempt. A set of tested cases can say more about a model under those conditions. To know whether it improves a job, you need to study people completing the work, including review and correction. To know whether an organization makes better decisions, you need evidence about those decisions.
Keep these claims separate. A model that extracts information accurately may save search time without improving the final decision. A faster first draft may create extra checking later. Measure the outcome you are claiming, and do not use a narrow test as evidence for a broader benefit.
A useful finding might be that first-pass extraction reduced manual searching in a defined review process, while unusual cases still required specialist judgment. That is a relationship a successor model can be tested against. Describe why you think the improvement happened and what evidence supports that explanation, rather than making a claim so broad that any result would fit it.
Compare the same kind of completed work.
Use a measure that still makes sense when the model changes: completed cases, resolved requests, accepted decisions, or time spent on a defined task. Token counts and message counts can help explain system behavior, but fewer messages do not necessarily mean the job was done better.
METR’s work on task duration offers one example. It measures the length of tasks a system can complete at a stated success probability, expressing length as the time a skilled person would take.[4] This is not how long the AI runs. The shared task measure and reliability level let researchers compare systems using a consistent question.
Define completion, required quality, cost, and acceptable failure before testing the replacement. A system that succeeds half the time may be useful with close supervision and unsuitable for unattended work. If the task definition changes, keep some cases that can be scored both ways. That lets you distinguish a changed definition from improved performance.
Use the same task and required reliability when comparing models.
Keep familiar tests and add current ones.
An unchanged test gives you a basis for comparison, but can become outdated or too familiar to models. Replacing the whole test creates a different problem: you can no longer tell whether a changed score reflects the model or the new questions. Keep a fixed set for comparison alongside fresh cases from current work.
Dynabench lets people create valid examples that a current model struggles with. LiveBench draws new questions from recent sources and uses answers that can be checked objectively.[2][3] These approaches show two ways to keep tests informative as models change. Neither removes the need to understand what a particular test measures.
Give each group a clear purpose: unchanged cases for comparison, fresh cases for current conditions, and difficult cases for known failure types. Keep a record of questions removed because their answers or tasks no longer apply. Report the groups separately so an easier set of new cases cannot quietly inflate the overall result.
Check the effect on people doing the work.
A preregistered experiment on professional writing found that access to ChatGPT reduced average task time by 40% and increased assessed quality by 18%.[5] These are results for the people, tasks, and tool tested. A later study can ask whether the same kind of help remains useful, without assuming that every new model will produce the same percentages.
A workplace study covering more than 5,000 customer-support agents found an average increase in issues resolved per hour, with larger benefits for less experienced workers. The most experienced workers had small speed gains and small declines in quality.[6] For a replacement system, check both who benefits and what changes in their work. Faster handling alone may not mean better support.
An experiment with consultants found that AI assistance helped on some tasks and hurt on a task beyond the system’s capabilities.[7] Keep the description of those tasks and how people decided to rely on the tool. A new model may move the boundary, but you still need to find where its help becomes unreliable.
Check who benefits, which work improves, and where help becomes unreliable.
Compare what the systems accomplish under the required conditions, while allowing their wording and methods to differ.
Plan how you will compare a replacement.
While the current system is still available, choose unchanged cases, fresh cases, and important past failures. Test the current and candidate systems with comparable tools, access, time, and human help. Compare completed work, reliability, cost, waiting, correction effort, and serious errors. Repeat enough cases to understand ordinary output variation, and hide the version labels from reviewers where feasible.
Research on model selection over time examines how to combine recent and historical data when conditions change. Adaptive rolling-window methods adjust how much past data is used.[9] For a practical review, the useful principle is to consider both: historical cases reveal lost capabilities, while current cases show what the service needs to handle now. The appropriate balance depends on the task.
Decide in advance what would justify replacing the system. If the candidate improves on fresh cases but fails old ones, check whether those old cases still matter. If both systems make the same serious error, investigate shared data and tools as well as the models. Matching errors do not establish a shared cause. If the new system solves the problem differently, revise the explanation of how it helps.
Find out what changed besides the model.
A result can change because the model changed, the application changed, different people used it, or the task’s correct answer changed. A new policy may make an old answer wrong even if the model behaves exactly as before. Record changes in source material, interface, procedures, and human practice alongside the model version.
The WILDS benchmark documents performance gaps when machine-learning systems encounter changed settings, such as different hospitals, regions, or time periods.[8] For your comparison, keep some inputs unchanged and collect cases from the present setting too. You need to know what stayed constant before explaining why performance moved.
Monitor the case types, outcomes, corrections, requests for help, costs, and complaints that matter to users. Look at important groups separately even when the overall score holds steady: HAPI found that errors could move between types of data beneath a stable average.[1] Those checks can reveal when an earlier conclusion stops applying.
Show readers when a finding needs another check.
At the end of the report, state the task, people, period, system version, comparison, and result. Explain where the finding applies and what remains uncertain. Name changes that should trigger review, such as a new model, policy, interface, price, user group, or serious failure.
Give the finding a clear status. It may still be supported after a new test, need revision, have been replaced by a better explanation, or no longer concern a current task. If a review is due but has not happened, say that its current status is unknown. An old report’s continued availability is not evidence that its claims still hold.
Keep earlier versions and explain what changed in the evidence, interpretation, and recommended action. Readers should be able to see which parts remain useful and why others were revised. Research can stay useful through model updates when it gives the next reviewer a clear way to check the answer again.