Reproducibility in AI search research: fixed questions, versions and archives

AI answers vary, so you cannot reproduce a study by asking again and expecting the same words. Reproducibility here means something more precise: the questions are fixed and public, everything that could change an answer is recorded, every output is archived, and anyone can recompute the published numbers from the archive. A rerun should land within the expected variation.

What does reproducible mean for AI answers?

There are two levels, and it helps to keep them apart.

• Computational reproducibility. Given the stored answers and the published method, another person gets the same numbers. This is achievable and should be expected.
• Replication. Given the same questions asked fresh, another person gets results consistent with the original, within the stated uncertainty. This is realistic, but not identical. See margins of error in AI measurement, explained simply.

A claim such as "the assistant recommended brand X" cannot be reproduced exactly. A claim such as "brand X was recommended in about this share of answers to these questions, with this interval" can be checked.

Why fix the questions?

A fixed question set protects against two quiet problems.

1. Cherry-picking. If you can choose questions after seeing answers, you will drift towards the ones that tell a story. Fixing the set in advance removes the temptation.
2. Moving targets. If the questions change, change over time cannot be told apart from change in the questions.

Practical rules help: write questions the way people type them, keep brand names out so that no brand is favoured by the wording, and publish the list. Our guide on building a benchmark set of questions covers this in detail.

For long-running work, keep a stable subset that never changes, so the same question can be compared across years. See anchor questions.

What has to be recorded?

Anything that could change an answer. The list is longer than most people expect.

• the question and its version;
• the assistant, the model and its version;
• the exact request, including any instruction added to it;
• whether search was on, and any limits on it;
• location and language settings, or the fact that none were sent;
• the date and time;
• the full answer, citations and any consulted pages the provider reports;
• how the answer was read: who or what extracted brands, with which rulebook and version.

The last item is often forgotten. If the reader of the answers changes, the numbers can change even when the answers did not. See one reader for four AIs.

Why archive every answer?

Because the live web and the assistants both move. A page cited in an answer may be edited or deleted. A model may be replaced. If you stored only the final numbers, nobody could check them later.

An archive makes three things possible.

• Recomputing a number years later, with the original data.
• Re-reading old answers under a better method, without asking again.
• Auditing a result by looking at the words the assistant actually said.

Archiving also means keeping snapshots of the cited pages, not only their addresses, since the page you cite today may read differently next month. The reasoning is in why every AI answer should be kept.

How should versions be handled?

Treat anything that could change results as a version, and keep old versions readable.

• Model changes. When a provider replaces a model, mark the date on every timeline and label comparisons that cross it.
• Method changes. Changing the instruction sent, the settings or the reading rules makes a new method version. Do not quietly mix versions.
• Question changes. Edited questions become a new benchmark version. Anchors stay unchanged.

Citewake follows these rules, and its methodology states them. Each edition on the editions page says which method version produced it.

What are the honest limits?

State them. They are part of reproducibility.

• Answers collected through official interfaces describe those interfaces, not every consumer experience.
• A location-free request is not a worldwide request. See does location change AI answers?
• A study of a fixed set of questions says nothing about questions outside it.
• Providers report different evidence, so some measures are unavailable for some assistants. Unavailable is not the same as zero.

What does a reviewer look for?

A sceptical reviewer will usually ask the same handful of questions. Were the questions chosen before the results? Is everything that could change an answer written down? Can the numbers be recomputed from the stored data? Are the uncertainty ranges based on the right unit, which is the question and not each separate run? Are limits stated, and do the conclusions stay within them? If you can answer yes to each and point to where, you are in good shape. If you cannot, the gap is worth closing before publication, because these are the same questions your readers will ask, and an answer you only have in your head is no answer at all.

A short checklist for your own study

1. Write and publish the questions before collecting.
2. Fix the method and give it a version number.
3. Ask each assistant the same way, in one pass.
4. Store everything, unedited.
5. Report rates with intervals and the number of questions.
6. Make the data available so others can recompute.

If you want a quick, free first look at how assistants describe a site, the Citewake checker is there, though a real study needs the steps above.