Why AI assistants disagree with each other about the best product

AI assistants disagree because they are not four copies of one thing. Each runs its own searches, finds and trusts different pages, draws on different training knowledge, and writes in a different style. Change any of those and the brands named change. For a buying question with no single right answer, disagreement is the normal result, not a malfunction.

Where does the disagreement come from?

Follow a question through the process described in how AI assistants search the web before they answer. There are at least five points where two assistants can part ways.

1. Whether they search. One may search and another may answer from memory. See when an AI answers without searching.
2. What they search for. Each rewrites the question into its own queries. See search queries vs. prompts.
3. What they find. Different search services return different results.
4. What they trust. Each weighs sources differently.
5. How they write. One assistant may name two brands and stop. Another may name seven and add caveats.

A difference at any step carries forward. By the time of the final answer, small differences have compounded.

Does the style of the answer change who is named?

Yes, and it is easy to overlook. An assistant that writes short answers has room for few names, so only the strongest candidates appear. An assistant that writes long, balanced answers names more, so smaller brands get a mention.

That means a brand can look weak in one assistant partly because of its format, not its judgement of the brand. When comparing assistants, count how many brands each typically names before reading too much into who is missing. The Edition 0 report measures this; see the editions and the findings guide.

Is it just random variation?

Some of it is. The same assistant, asked the same question twice, can name different brands. That is covered in why AI answers change every time you ask. So when two assistants disagree, there are two possible explanations: a real, consistent difference between them, or ordinary variation between askings.

You separate the two by asking many questions, several times each. A consistent difference appears again and again. A random one washes out. This is the same logic behind asking more than once before trusting an answer.

What does disagreement tell you?

Quite a lot, if you read it as data instead of treating it as a problem.

• Strong agreement across assistants suggests a brand has support in many kinds of sources. It is the more robust position.
• Strong support from one assistant only suggests the brand is well covered by the sources that assistant favours, and weak elsewhere.
• Wide disagreement on the first pick suggests the category has no settled winner.

The last point is why there is no real rank #1 in AI answers. If four assistants rarely agree on a first pick, a single "number one" is not a stable fact about the category.

How do you compare assistants fairly?

A fair comparison needs the same footing for each.

• The same questions, word for word.
• The same moment, so none has had longer to change.
• One reader. If a different person or tool judges each assistant's answer, differences in the judging can look like differences in the assistants. See one reader for four AIs.
• Per-assistant results, not just a blend. A blended number hides exactly the disagreement you want to see.

Citewake reports each rate per assistant as well as overall for this reason. The methodology explains how.

Does disagreement mean the AI is unreliable?

Not necessarily. Disagreement means the question has no single settled answer, and the assistants are showing you that. Think of asking four well-informed friends which laptop to buy. They would reasonably name different models, depending on what they had read and how they weigh price against weight. What does matter is whether an assistant is wrong about facts, such as inventing a feature or naming a product that does not exist. That is a different problem from preferring a different product. Keep the two apart: check facts against primary sources, and treat preferences as a range of informed opinions rather than a verdict to be corrected.

Can the same assistant disagree with itself?

Yes, and it is a useful check. If an assistant gives a different first pick each time you ask, its disagreement with another assistant tells you little. If it gives the same pick nearly every time, and another assistant gives a different one nearly every time, the difference is real and stable. The rule of thumb is to compare differences between assistants against the differences within each one. Only when the first is clearly larger than the second should you describe the assistants as disagreeing, instead of simply being noisy.

What should a brand do with this?

Do not chase an average. Look at where you stand with each assistant, and ask why the weakest one differs.

• If one assistant leaves you out, look at which sources it cites in your category, and whether you appear there.
• If one cites a competitor's pages heavily, read them. What do they say that yours do not?
• If your result swings between askings, treat it as marginal and keep measuring.

The wider lesson is that "what does AI say about us?" is the wrong question. The useful question is "what does each assistant say, how often, and why?" The Citewake checker offers a first look at how assistants describe a site.