Margins of error in AI measurement, explained simply

A margin of error is a range around a number that shows how far it might move by chance. An AI visibility figure with no range looks more certain than it is. With a range, you can tell a real difference from a wobble.

Why do AI numbers wobble?

There are two sources of chance.

1. The AI varies. Ask the same question twice and the answer may differ. Why AI answers change every time you ask covers why.
2. The questions are a sample. No benchmark can ask every question a buyer might. A brand that does well on one set of questions might do less well on a slightly different set.

Both mean that if you ran the exact measurement again, you would not get the exact same figure. The margin of error says how different you might expect it to be.

What does a range look like?

Suppose, hypothetically, a brand is recommended in 12 of 60 questions, a rate of 20%. A report with no range says "20%". A report with a range might say "20%, with a range of 11% to 31%". That second version tells you the true figure could plausibly be a good deal lower or higher.

The numbers here are invented to illustrate the idea and are not results from any measurement. The point is the shape: a single figure hides a spread, and the spread decides what you can conclude.

What is the unit of measurement?

This is the point most people miss. Citewake's published method treats the question as the independent unit, not the answer.

Why? Answers to the same question are related. If a question strongly favours a certain brand, every answer to it will tend to name that brand, and getting eight answers to it does not give you eight independent pieces of evidence. Treating each answer as separate would make the result look far more certain than it is.

So the method resamples questions, keeping each question's answers together. It calls this a cluster bootstrap. In plain terms: imagine redrawing the benchmark many times by picking questions at random with replacement, recomputing the figure each time, and noting how much it moves. The spread of those figures gives the range. The method shows a 90% interval around each rate.

How should you read a range?

• Narrow range: the data pins the figure down. You can trust it more.
• Wide range: the figure is loose. Treat it as a rough indication.
• Ranges that overlap: the data can't separate the two figures with confidence.
• Ranges that don't overlap: the difference is likely real, within this benchmark.

Do not read the middle of a range as the truth and the edges as errors. Every value inside is plausible. The middle is only the best single guess.

How do ranges apply to change over time?

The same logic applies to change. If a figure moves from one month to the next, the question is whether the move is bigger than chance would produce.

The method compares each question's answer this edition with the same question's answer in the previous edition, from the same AI. It estimates the change and its range together. A change counts as a move only when its own range excludes zero. Otherwise the report says there is no clear change.

That is a restrained standard, and a deliberate one. It will sometimes say "no clear change" when something did change, but it will rarely announce a change that was only noise. Why AI visibility should be measured monthly, not daily explains the rhythm that supports it.

Why not just ask more often?

Asking the same question many more times narrows one source of chance, the AI's own variation, but not the other. If the questions themselves are few or alike, asking them ten times more leaves you with the same limited view of the category. Both matter: enough different questions, and enough care in how they are grouped. That is why the method puts its effort into a well-built set of questions and an honest way of counting them, rather than into volume alone.

What does a margin of error not tell you?

Be clear about the limits.

• It covers chance within the designed benchmark. It says nothing about how well the benchmark represents all buyers. A benchmark can be fair and still not be everyone.
• It does not cover personalised answers. A signed-in person with a history may see different things.
• It does not predict the future. It describes the edition measured. AIs change.

The method states these limits on its pages, and any report you read should too.

What should you ask of any AI visibility report?

1. Is there a range, or only a single figure?
2. What is it based on: how many questions, and how many answers per question?
3. Does it treat repeated answers to one question as independent?
4. When it reports a change, does it say whether the change is larger than chance?
5. What does it say it does not cover?

A report that answers these questions plainly can be trusted more than one with a bigger number and no range.

How can you apply this yourself?

If you track your own questions, ask each one several times, in each assistant, and count how often you appear. Do it across enough questions to see the spread. Resist concluding anything from one answer. How many times to ask AI before trusting an answer goes through the practicalities.

The full description is on the methodology page, written as a method ahead of the first edition in early 2027. To see what assistants say about your site now, try the Citewake checker.