Decide what should stay the same.
A colleague reruns your AI-assisted analysis and gets a differently worded summary. If the same evidence supports the same findings, the wording may not matter. If an important exception disappears, a familiar-looking summary may hide a changed conclusion. Decide what you need to reproduce before comparing two outputs.
The National Academies distinguishes computational reproducibility—consistent results from the same data and computational steps—from replication, which uses newly collected data to investigate the same question. Generalization asks whether findings apply to other groups or settings.[1] There is also a more basic practical task: recovering what happened in the original run. A complete record helps with that task, but does not by itself pass the other tests.
State what must match and what variation is acceptable. You might require the same extracted values, a calculation within a stated tolerance, or the same central findings with supporting passages. Choose criteria that fit the method. Set them before the rerun so you do not accept a changed result simply because you prefer it.
Decide which result must stay the same before you compare two runs.
Save the work as you do it.
A report shows the final answer. It may not show which sources were excluded, which translation was used, or why a researcher corrected an AI suggestion. Those steps can change the result. Preserve the materials and decisions behind them while they are still available.
Give each run an identifier and a record linking its inputs, procedure, important intermediate outputs, and final result. Keep the originals separate from cleaned or translated files. A file hash is a digital fingerprint that helps check whether a file’s contents have changed. Record it when the file is used; calculating one months later cannot prove which version was used then.
The NeurIPS reproducibility program combined checklists, code submission, and independent attempts to reproduce results.[2] These practices work best when recording starts with the research. A checklist completed at publication cannot recover a temporary search result or an unrecorded correction. Use the run record as you work, rather than trying to reconstruct everything at the end.
Record the model version and what it returned.
A model family name is not enough to identify what ran. OpenAI’s API documentation notes that behavior can change between model snapshots and recommends pinned versions and task evaluations for consistency.[4] When available, save the exact version identifier, provider, API address, request time, and generation settings. Keep request or response identifiers too. If the provider returns an identifier for the serving configuration, often called a system fingerprint, save that with the run.
Using a fixed snapshot limits one source of change, but does not guarantee identical answers or permanent access. A hosted version may eventually be retired. Decide how you would assess a replacement and record which comparisons it must pass. If no available replacement meets those criteria, explain which part of the original result you can no longer reproduce.
For a model run locally, keep the model files and their hashes, software versions, and relevant hardware settings. Include the tokenizer, which converts text into model inputs, and any compression or generation settings that could affect results. For either a hosted or local model, save the raw response before parsing or editing it. This lets a reviewer distinguish what the model produced from what happened afterward.
Save the model version, the full input, and the response it actually returned.
Keep the full instructions, not just the last question.
Two people can type the same question and send different inputs to a model. One system may add earlier messages, examples, retrieved passages, tool instructions, or a required output format. Save the complete message sequence and the template or code that assembled it, including the order of the material.
Sclar and colleagues found that small formatting changes could substantially affect performance in tasks where prompts supplied a few examples.[6] This does not mean every prompt is equally fragile. It does mean that formatting and prompt selection can be part of the method. Keep the versions you tried, how you chose between them, and whether the reported result was typical or selected from many attempts.
When you change a prompt, give it a new version and check it on a small, fixed set of examples. Save both the reusable instructions and the material added for that run. If you revise instructions after seeing an unwanted result, record the revision and start a new run. A reviewer should be able to see how the method developed.
Preserve the sources and the steps that changed them.
Repeating a search query later may return different pages or a different order. Save the queries, result identifiers, ranks, access times, and inclusion decisions. Keep copies or extracts of the sources you actually used where permission allows. A URL alone may no longer show the evidence you saw.
Follow the material from source to model input. Was a scan converted to text, a passage translated, a document split into sections, or part of it cut to fit? For changes that affect the result, record the tool, settings, input, output, and any failure. If the data are restricted, explain the access procedure and provide a safe example with the same structure. That example can test whether the process runs; it cannot reproduce the protected data’s findings.
ReproEvalCard reviewed fifty-five papers about systems that use language models across multiple steps. It found that 75% of the papers did not report controls for random variation, and 61% lacked records of intermediate steps.[5] Such gaps make it difficult to tell where a rerun changed. Save the records needed to compare the important steps, not just the final answer.
The run record explains what happened. The pass criteria explain what should remain consistent when someone tries again.
Check whether variation changes the finding.
AI outputs can vary even when the request stays the same. Repeat a representative set of cases and keep all the outputs. Compare the result you said mattered: extracted values, included claims, rankings, or recommendations. Report how often and how much it changes, including the groups or task types where errors appear.
Record random seeds and settings intended to reduce variation when the system supports them. A seed helps control random sampling, but is not a general guarantee of identical results. Also test changes your method is supposed to tolerate, such as a different ordering of passages or an equivalent prompt. If those changes alter the conclusion, report that dependency.
Decide how you will judge differences before running the check. A calculated table may need exact agreement. A qualitative account may need review of whether the same findings remain supported and whether exceptions changed. If results differ, compare intermediate outputs to locate the first important change. A score for how similar the final sentences look cannot explain the cause.
Check whether the finding changes, not just whether the sentences match.
Record the researcher’s decisions too.
People correct transcripts, choose sources, reject generated code, combine categories, and decide when to stop searching. Those choices belong in the method. Record what the person saw, what they changed, and why. Keep the original suggestion alongside the accepted version when it affected the result.
The W3C provenance model describes how records, activities, and responsible people or systems relate to one another.[7] A simple way to apply that idea is to link each important output to its inputs, the operation performed, and who performed it. A short change log may be enough; the reviewer needs to follow the transformation, not learn a new documentation system.
A well-recorded analysis can still use a biased sample or make a mistaken interpretation. Reproducibility makes those errors easier to inspect; it does not certify that the conclusion is correct. Review the method as well as the execution, and retain unsuccessful runs when omitting them would misrepresent how the final result was obtained.
Ask someone else to follow the method.
Start the handover guide with the result to check and how to recognize success. Then point to the materials, access requirements, instructions, software versions, and important intermediate outputs. Include a small example for checking setup, the known limits, and permitted uses. A structured file list can support automatic version and hash checks alongside the written guide.
Heil and colleagues describe progressively stronger standards, from making data, models, and code available to documenting the environment and automating execution.[3] Make setup explicit and automate repeatable steps where useful. Let a failed run identify the step that stopped, so another person can investigate without starting the entire process again.
Give the package to someone who did not build it. ACM’s artifact-review guidance separates available materials from reviewed, usable materials and independently reproduced results.[8] Note where that person has to guess, lacks access, or cannot tell whether an output is acceptable. Repair those gaps and report what was actually reproduced, including remaining differences. That attempt is much better evidence than a folder labeled “reproducible.”