What is a benchmark set of questions?

A fixed list of the questions buyers ask about a category, asked of AI assistants in the same way each time, so results can be compared across brands and across months.

How many questions do you need?

Enough to cover the category's range of needs. Citewake's published method uses 60 scored questions per category, in three tiers of 20.

Should a benchmark question name a brand?

No. A question that names a brand measures that brand's name, not whether an AI would choose it. Branded questions belong in a separate diagnostic.

Changing a question changes what is measured. A version number makes clear which results can be compared.

How to build a benchmark set of questions: Citewake

A good benchmark is a fixed set of buyer questions, written the way people actually ask, with no brand names in them, spread across easy and hard cases and across different intents, then frozen and versioned so every result can be compared with the last.

Why do you need a fixed set at all?

Because a measurement needs a ruler that doesn't change. If you ask whatever comes to mind each time, you can't tell whether a brand improved or you asked an easier question. A fixed set removes that doubt. Every brand faces the same questions, and every month is compared with the last.

Where do the questions come from?

From real behaviour, not from what a marketing team wishes buyers asked. Citewake's published method draws on:

real search demand, such as suggestions, related questions and keyword data;
public forums, where people describe their needs in their own words;
the anonymised, aggregated shape of customers' questions, never a customer's question word for word.

The method records the source of each question: observed demand, the aggregated shape of customers' questions, a customer request, research, or an editorial hypothesis. A hypothesis is checked against observed demand before the set is frozen.

You can do the same on a smaller scale. Collect questions from sales calls, support tickets, search suggestions and community threads, and keep the wording buyers use.

What makes a question good?

First person and problem first. "I run a small agency and need to show clients how they appear in AI answers" is closer to real use than "best AI visibility tool".
No brand name. A brand in the question measures recognition of the name. Branded checks, such as asking whether a named product is any good, are useful but they belong in a separate diagnostic.
No leading words. "The best", "the cheapest" and similar words can push an answer. Use them only where people really do.
One need at a time. A question with three needs gets a muddled answer.

How should the set be structured?

By difficulty and by intent.

Three tiers

Head: broad buying questions, where large, established brands tend to win.
Mid: a use, a budget or a type of buyer.
Tail: a specific need, where specialists and newer products can win.

Mixed intents inside each tier

best and recommend;
compare and alternatives;
how to;
price;
specific use.

The published method uses 60 scored questions per category: 20 in each tier. Reporting results per tier, as well as overall, shows where a brand wins. A brand that only appears in the tail has a different story from one that only appears in the head.

What about tagging?

Tag each question before it is ever run. The method uses three tags:

Intent: what the person wants to do.
Context: the situation, such as an agency, a free plan or no mobile signal, or none.
Type: decision questions, where a good answer names products, or informational ones, where it may name none. Decision questions and informational questions: why the difference matters explains why this matters.

Setting tags in advance prevents sorting questions to suit the results afterwards.

What should a benchmark leave out?

Anything that would let the result be steered. That means no questions written around a particular brand's strengths, no questions chosen because a certain brand does well on them, and no questions that depend on a feature only one product has unless buyers genuinely ask for it. It also means resisting the urge to add a question whenever a new topic is trending. A benchmark is a standing instrument. If you want to explore a new topic, build a second set for it.

How do you keep the set honest?

Freeze it. Once the set is frozen as a version, don't edit it. Changing any question makes a new version, and results are compared within a version only.
Publish it. If the scored questions are public, anyone can rerun them and check a number.
Keep a private check set. The method keeps a separate private set, written to the same rules, which is never used for published numbers. If a brand does well on the public set and badly on the private one, that triggers an investigation. It isn't treated as proof of gaming, since the cause may be the benchmark itself.
Log every change. At a version change, record what changed and why.

Anchor questions: comparing AI answers across years covers how to keep some questions fixed across versions.

How do you build a smaller version?

You can do this on a small scale for your own business.

List twenty or thirty questions buyers ask, from real sources.
Remove brand names and leading words.
Spread them across broad, mid-level and specific needs.
Tag each one.
Write the questions in a file, give it a version and a date.
Run them on a regular schedule, in the same way each time.

How to find the questions your customers ask AI helps with step 1, and what a fair AI visibility benchmark looks like shows the standard to aim for.

Where can you read the full method?

On the methodology page. It describes the approach as a method. The first published edition is due in early 2027. To see how assistants treat your own site today, you can try the Citewake checker.