One reader for four AIs: why consistent reading matters

Comparing ChatGPT, Claude, Gemini and Perplexity is only fair if one reader applies one standard to all four answers. If different systems read different assistants, the differences between readers get mistaken for differences between the assistants.

What does reading an answer involve?

An AI answer is free text. To turn it into something you can count, a reader has to decide:

• which brands appear;
• in what role, whether recommended, mentioned neutrally or presented as one to avoid;
• in what order, and whether the order expresses a preference;
• which sources were cited.

None of this is obvious. "A strong option for agencies" is a recommendation. "Another well-known tool is X" probably is not. "X was popular years ago but has been overtaken" is a negative. Reasonable readers will sometimes disagree.

What goes wrong with separate readers?

Suppose one system reads ChatGPT's answers and another reads Perplexity's. If the first is stricter about what counts as a recommendation, ChatGPT will look less generous even if both assistants write the same thing. Every comparison between them is now contaminated, and nobody can say by how much.

The same applies to naming. If one reader merges two names into one brand and the other keeps them apart, shares will differ for reasons unrelated to the assistants. When is a brand the same brand? Entity resolution in AI answers covers why that matters.

How does Citewake's published method do it?

Under the method, the four assistants only answer. One reader reads everything. For each question, it reads all four answers side by side, with each answer's citations. That means:

• a brand is resolved to the same entity across all four;
• "recommended" is judged by a single standard.

The reader works from a fixed schema and a versioned rulebook that holds definitions, role rules and the category's brand and alias list. Its model, rulebook and brand list are recorded on every reading. Changing any of them makes a new method version, and the labelled test set is scored again before it ships.

Does the reader judge which brand is better?

No, and that is deliberate. The method says the reader classifies and does not judge. It records what each answer says. It never adds an opinion about which brand is better, never adds a brand that isn't in the text, and works only from the answer and the fetched page.

A further check: a brand is kept only if its name, alias or domain appears in the answer and the quoted words are found word for word. That proves the words exist, though not that the role is right, which is why accuracy is measured separately.

What about cited pages?

Pages that an answer cited are read under the same rulebook, for the brands they list, in what role and in what order. The method uses a smaller, faster reader for these long, list-like pages, and sends every page it marks uncertain, plus a random sample of the rest, to the main reader for a ruling. How often the two disagree is published. Pages that can't be fetched, are blocked or are paywalled are recorded as unusable, left out of the base and counted openly.

What if the reader is one of the assistants it reads?

That is a fair worry, and the method names it. The reader is itself an AI model, and it may come from the same provider as one of the four assistants. So the method reports its accuracy separately for each assistant's answers. If it reads one assistant's answers measurably differently from the others, the reader is changed, or a second reader from another provider reads those answers. Either way it becomes a method version.

The lesson is general. No reader is neutral by assertion. Neutrality has to be shown, with numbers, and shown by group.

How is the reader checked?

Against a hand-labelled test set of complete answers and complete cited pages, labelled by two reviewers who resolve and log their disagreements. The set deliberately includes hard cases: brands that are easy to miss, incorrect additions, recommendations against neutral and negative mentions, ambiguous names, parent and product cases, and citations with no brand named in the prose.

For each category the method reports precision, recall, role accuracy and entity accuracy with confidence intervals, and sets bars a category must meet before it is published. A random sample of each edition's readings is also checked by hand, as a check for drift, not a certificate of accuracy.

Why not let each assistant describe itself?

Because an assistant is not a neutral reporter of its own answers. Asking an assistant whether it recommended a brand invites a fresh answer, not a record of the old one. The reading has to be done on the saved text, by a reader working to written rules, so that the same words always receive the same label. Written rules also let anyone challenge a label by pointing to the rule and the words.

What should you ask of any cross-assistant comparison?

1. Who or what reads the answers, and is it the same for every assistant?
2. How is "recommended" defined?
3. How accurate is the reading, and is that shown for each assistant?
4. Can you see the answer behind the label?

If a report compares assistants and cannot answer these, treat the comparison as a rough impression. How to compare answers across ChatGPT, Claude, Gemini and Perplexity covers doing it by hand.

Where is the full method?

On the methodology page, described as a method ahead of the first published edition in early 2027. To see how the assistants describe your own site, try the Citewake checker.