How AI recommendations change over time, and how to track it

AI recommendations change for four reasons: the assistants are updated, the web they search changes, brands launch and fade, and answers vary naturally between askings. To track change properly, ask the same questions of the same assistants on a regular schedule, compare each question with its own earlier answer, and call something a move only when it is larger than the normal variation.

Why do recommendations change?

Separate the causes, because they call for different responses.

1. Model updates. A provider replaces or retunes a model. Behaviour can shift in one step: longer answers, a different habit of searching, a new set of favourites.
2. Changes in the web. New articles, reviews and threads appear. Old ones disappear or are edited. Search results move, and with them what an assistant finds.
3. Changes in the market. Brands launch, rename, merge or close. Features change.
4. Noise. Even with nothing else changing, an assistant can answer differently on a different day. See why AI answers change every time you ask.

The hard part is that these look the same from outside: a brand is gone from an answer. Only a careful method can tell you which it was.

How do you track change without fooling yourself?

Use the discipline of a repeated measure.

• Fix the questions. Same wording each time. For long-running work, keep a set of anchor questions that never change. See anchor questions.
• Keep the setup constant. Same assistants, same request, same settings, same time of the month if you can.
• Collect in one pass. All assistants within a short window, so a period reflects one moment.
• Compare question by question. Match each question's answer to its own answer last time, from the same assistant.
• Record the context. Note model updates and method changes on the timeline.

Matching question by question is the key step. It removes the question set as a source of change, leaving the assistants and the world.

How often should you check?

Less often than people expect. Checking daily creates a jittery chart that mostly shows noise, and it invites reactions to differences that mean nothing. The real drivers, models, sources and markets, change over weeks. For most purposes a monthly rhythm is enough. We make the case in why AI visibility should be measured monthly, not daily, and the practical setup is in how to track AI visibility over time.

What should you watch for?

A short list of signals that carry information.

• Brands arriving or leaving a category's answers.
• Rises and falls in how often a brand is recommended, per assistant.
• New sources appearing, or familiar ones disappearing.
• Changes in style: longer answers, more caution, more brands named.
• Shifts around a model update, which may affect every brand at once.
• First appearances: the first time an assistant names or recommends a brand. These are worth keeping as a record.

More on reading these is in how AI recommendations evolve, and what to watch for.

When is a change real?

When it is larger than the uncertainty around it. Each rate has an interval, and so does each change. If the interval for the change includes zero, the honest label is "no clear change". If it excludes zero, you have a move.

Two cautions. First, a model update can move many brands at once, so a drop that follows one may say little about you. Second, a single edition cannot show a trend. Look for a change that persists across more than one. See measuring recommendation rate with confidence intervals.

What does a tracking log look like?

Keep it simple. For each edition, record the date, the assistants and their versions, the method version, and any known events such as a model release or a major change on a cited site. Beside it, keep the rate for each brand you follow, per assistant, with its interval. Add a column for notes where you write what you think happened and why, and mark which of those notes you later confirmed. After a few months the log shows patterns that no single result could: which changes persisted, which faded, and which lined up with outside events. The notes column matters most, because it records your reasoning when it was fresh, not rebuilt after the fact.

What if a brand disappears from the answers?

Check the plain explanations first. Did the question set change? Did the assistant's model change? Did the assistant simply decide not to search on that run? Then read the new answers and the sources they cite, and compare them with the earlier ones. Often the cause is visible: a roundup that listed you was updated, or a competitor's page took its place. If you cannot find a cause and the drop is within the normal range of variation, wait for the next edition before acting. Patience here is cheaper than a rewrite built on noise.

What do you do with a real change?

Work out which cause it was before you react.

• If it follows a model update, check whether competitors moved too.
• If it follows a new source, read that source.
• If a competitor rose, read what changed on the pages the assistants cite for them.
• If nothing identifiable changed, keep watching.

Do not rewrite your whole site on one month's data. Citewake's method compares each edition with the previous one, as described in the methodology. The first published figures are in the editions, and the Edition 0 numbers are a baseline, not yet a trend. For a free first look at a site today, the Citewake checker is available.