Measuring recommendation rate with confidence intervals, in plain words

A recommendation rate is the share of valid AI answers, in a fixed set of questions, that recommend a brand. A confidence interval is the range that shows how far to trust that share. If a brand is recommended in 40% of answers with an interval of 35% to 45%, the real figure for that set of questions is probably in that range, not exactly 40%.

What is a recommendation rate, exactly?

It is a simple fraction.

recommendation rate = answers that recommend the brand ÷ all valid answers

Two details matter.

• The denominator is fixed. It is all valid answers in the benchmark, not just the ones where the brand appeared. A brand named in 5 answers out of 5 that mentioned it is not at 100%.
• "Recommend" has a definition. The answer presents the brand as a choice for the question. Naming it as history, as an example, or as one to avoid does not count. The distinction is covered in mentioned, recommended or cited, and in brand mentions vs. recommendations.

Rates can be shown overall and per assistant. The per-assistant view usually tells you more, because assistants differ. See why AI assistants disagree.

Why is a rate alone not enough?

Because it comes from a sample. You asked a certain set of questions a certain number of times. A different but equally reasonable set, or another run, would give a slightly different rate. A single number hides that.

Compare two claims: "Brand A is recommended in 42% of answers" and "Brand A is recommended in 42%, and we're fairly sure the true figure is between 36% and 47%." The second tells you whether a gap to Brand B at 38% means anything. The first invites you to read a lead that may not exist.

What is a confidence interval, in plain words?

Imagine you could repeat the whole study many times, each with a fresh draw of similar questions and answers. Each repeat would give a slightly different rate. The interval is the range that covers most of those repeats.

The word "confidence" is a statement about the method, not about any one result. A 90% interval comes from a procedure that, over many studies, captures the true value about nine times in ten. It does not mean there is a 90% chance the truth sits in this particular range, though for everyday reading, treating it as "a sensible range for the true figure" is a fair summary.

What makes an interval wide or narrow?

Three things.

1. How many questions. More distinct questions give a narrower interval. This is the biggest lever.
2. How much answers vary. If a brand is named in nearly all answers or almost none, the interval is tight. In the middle, it is wider.
3. How many repeats. More repeats of each question help, but with diminishing returns. See how many times should you ask an AI.

Why is the question the unit?

Because answers to one question are not independent. If you ask the same question ten times, you do not have ten independent pieces of evidence about the category. You have ten looks at one question. Counting them as ten would make the interval look far narrower than it should.

So Citewake's method resamples questions, keeping each question's answers together, and builds the interval from that. This approach is called a cluster bootstrap. In plain words: it re-draws the questions many times with replacement, recomputes the rate each time, and takes the range. The full description is on the methodology page, and a gentler introduction is in margins of error in AI measurement, explained simply.

How should you read two brands next to each other?

Use the intervals, not just the rates.

• Clearly apart: the intervals do not overlap, or barely touch. A difference is likely real for this question set.
• Overlapping a lot: do not claim a lead. Say there is no clear difference.
• In between: report the rates, note the overlap, and keep measuring.

Edition 0 reports every rate with its interval; the editions pages show them, and the findings guide uses a few.

A worked example in words

Suppose a benchmark has 60 questions, each asked of four assistants. A brand is recommended in a share of the valid answers, say a little over one in three. Resample the 60 questions with replacement many times, recompute the share each time, and note that most results fall within a band a few points either side. That band is the interval. If a competitor sits well inside it, you cannot say you are ahead. If the competitor is far outside it, you probably are. The same idea applies to each assistant separately, though with fewer answers behind each one, so the bands are wider. Wide bands are not a failure. They are the study telling you honestly how little it knows.

What does the interval not tell you?

It is easy to over-read. The interval covers uncertainty within the designed benchmark. It says nothing about:

• whether the questions represent all buyers;
• what signed-in, personalised chats would show;
• what the assistants will say next month;
• whether the reading of answers was accurate. That has its own checks.

Each of these is a limit to state beside the number. Two habits keep you honest: write down how many questions and answers sit behind every rate, and when a gap is small, say "no clear difference" rather than picking a winner. The Citewake checker can give you a first rough picture, but a rate worth quoting needs the steps above.