What a fair AI visibility benchmark looks like

A fair AI visibility benchmark uses a fixed, published set of realistic questions with no brand names in them, asks the same way of every assistant, keeps every answer, defines each measure precisely, and states its uncertainty. It lets anyone trace a published number to the words an assistant said and recompute it. It also never sells inclusion. If a benchmark fails these tests, its numbers may still be interesting, but they cannot be fully trusted.

Why does fairness matter?

Because the numbers influence decisions: which agency to hire, which tool to buy, where to put budget. If a benchmark favours some brands through its question choice or its method, those decisions are skewed.

AI answers add their own difficulties. They vary run to run, differ by assistant and are easy to cherry-pick. A fair benchmark is built to resist those problems.

What are the marks of a fair benchmark?

1. A fixed, versioned question set. The questions are written in advance and frozen. If any question changes, that is a new version, and results are compared only within a version.

2. Questions without brand names. A benchmark of discovery asks what a buyer asks before knowing any names. Putting a brand in the question tests recognition, not discovery.

3. Published questions. Anyone can see what was asked. This allows recomputation and exposes any bias in the set. The trade-off is that publishing invites gaming, so a fair benchmark guards against it, for example with a private validation set that checks for drift. How to build a benchmark set of questions covers construction.

4. The same method for every brand. Same assistants, same instructions, same search setting, same scoring. No special handling for anyone, including the benchmark's owner.

5. Every answer kept. Full text, sources, date and settings stored, so any number can be traced and, if needed, corrected openly.

6. Precise definitions. Recommendation, neutral mention, citation: each defined, with a stated denominator.

7. Stated uncertainty. Intervals or margins that reflect the fact that answers vary and the question set is a sample.

8. No paid inclusion. Payment cannot buy a place, a score or a correction. Inclusion follows evidence.

What should the questions look like?

They should mirror real buyer language and cover a spread of intents: best, compare, how to, price and specific use. They should range from broad questions, where leaders tend to win, to narrow ones, where specialists can. They should avoid leading words.

A mix of decision questions, where a good answer names products, and informational questions, where it may not, should be reported separately. Mixing them blurs what a recommendation rate means. See decision questions and informational questions.

How should uncertainty be handled?

Honestly. Because answers vary, any rate is an estimate. And because the questions are a sample of all possible buyer questions, the estimate carries uncertainty beyond run-to-run noise.

A good benchmark treats the question as the unit of independence, keeps a question's repeated answers together, and shows intervals. It also states what the interval does not cover, such as how well the benchmark represents all buyers. Margins of error in AI measurement, explained simply gives the plain version.

What are the warning signs of an unfair one?

• A single score with no definition behind it.
• Hidden questions that cannot be checked.
• Questions that name the brand being scored.
• Results from a single run, presented as fact.
• No saved answers to trace.
• Inclusion or rank for sale.
• No mention of how answers vary.
• Comparisons across periods when the questions or method changed.

Any one of these should make you ask for more detail before you rely on the figures.

How can you judge a benchmark you did not build?

Ask for the methodology page and read it as you would a financial prospectus. Is the question set available? Can you tell what was asked of which assistant, and how many times? Are measures defined with denominators? Is uncertainty stated? Are corrections handled openly, with the original kept visible?

Ask also about conflicts. If the benchmark's owner sells a product in the category it measures, the document should say so and explain how the owner is treated by the same code as everyone else.

Then test one claim yourself. Pick a published number and try to trace it back to the answers behind it. If you can, the benchmark takes its own rules seriously. If you cannot, treat its numbers as opinion, however polished the presentation. A fair benchmark invites this kind of scrutiny. An unfair one tends to resist it.

What does a fair benchmark leave out on purpose?

It does not pretend to measure everything. It describes answers to a designed set of questions, asked through defined means, at a stated time. It does not capture every conversation, personalised answers in signed-in apps or real-world demand. A fair benchmark says so.

Citewake's methodology is written to this standard: published questions, kept answers, separate measures and stated limits. You can read it and judge for yourself. To see where your own brand appears, a quick first look is available through the Citewake checker.

What is the short version?

A fair benchmark publishes its questions, keeps brand names out of them, treats every brand the same, saves every answer, defines each measure, states its uncertainty and never sells inclusion. If a benchmark cannot show you the words behind its numbers, treat the numbers with caution.