How can I test whether our content work changed what AI assistants recommend?

Treat it as an experiment, not a before-and-after screenshot. AI answers vary across runs and over time, so repeated, consistent measurements matter. (arxiv.org)

A practical test

1. Define what “recommend” means.

   Don’t count every brand mention as a recommendation. For each answer, record whether you were:

   • recommended as an option;
   • named but not recommended;
   • cited or linked as a source;
   • absent—or described inaccurately.

1. Build a fixed set of real buyer questions.

   Include non-branded prompts like “What’s a good…?”, “Compare…”, and “Alternatives to…”. Keep the wording, language, location, and intent consistent; don’t test only prompts that already name your company.

1. Create a comparison group.

   Select similar topics or pages that won’t receive the new content work yet. Measure both the pages you update and the untreated group before and after. If the treated group improves more than the control group, that’s stronger evidence of an effect than a simple before/after increase. If all your content is changed at once, you can still track outcomes—but it’s harder to attribute the change specifically to that work.

1. Run the same tests repeatedly.

   Ask the prompts across the assistants that matter to your audience, several times at baseline and after the content is live and has had time to be discovered. Record the date, assistant, model if visible, prompt, full answer, and cited URLs. Compare rates across runs, not one-off answers.

1. Track a small set of outcome metrics.

   For example:

   • Recommendation rate: answers that recommend you ÷ answers that returned a relevant recommendation.
   • Citation rate: answers citing your site ÷ relevant answers.
   • Position and competitor share: whether you’re a leading option and which competitors appear.
   • Qualified visits and conversions: where referral data is available.

1. Interpret the result cautiously.

   Look at the change for treated pages versus controls, and check whether it holds across prompts and assistants. Note other changes during the test—PR, product launches, site changes, or model updates—that could also explain a shift.

For Google Search’s AI features, use Search Console alongside your prompt tests. Google now documents a generative-AI performance report for AI Overviews and AI Mode, with impressions by page and date; that measures visibility in Google Search, not whether assistants generally recommend you. (support.google.com)

Minimum viable setup: a spreadsheet, a fixed prompt set, treated and control topics, repeated runs, and a rubric that separates “mentioned,” “recommended,” and “cited.”

You can answer this with a before-and-after test: run a fixed set of prompts through the assistants before and after your changes, then compare the results, accounting for the fact that AI answers vary from run to run.

1. Set a baseline before you change anything

• Pick 20 to 50 high-value commercial prompts. One source suggests starting manually with that many, then moving to a dedicated tool once prompt volume becomes unmanageable. Pick 20 to 50 high-value commercial prompts, run them across LLMs of choice, and calculate your visibility score as brand mentions divided by total answers tested, multiplied by 100. Move to a dedicated tool once the prompt volume becomes unmanageable.
• Include a mix of informational, comparative, and transactional prompts, since they influence perception and choice differently. What types of queries (informational, comparative, transactional) influence perception and choice?
• Record the date, the assistant, the model version, and whether you used a logged-in or logged-out session for every run.

2. Sample repeatedly, not once

• Answers are not deterministic. One tracker notes that tools differ on whether they scrape the chat interface or the API, models change constantly, and personalization means the same prompt can return different answers. Run each prompt several times per assistant, using fresh sessions, and use the same method at both baseline and follow-up.
• Metrics are estimates from repeated prompt sampling. Track them over time and per model, never from a single answer.

3. Score the answers, not just whether your name appears

• Mention rate: how often your brand appears at all.
• Recommendation rate: how often the answer actively suggests you. A neutral list entry differs from a recommendation. Calculate it as the number of answers that contain an explicit recommendation for your brand by the total prompts tested.
• Share of voice: how often you appear compared with named competitors in the same answers.
• Sources cited: which pages the assistant draws on. This matters most for your test, because it shows whether your new content is being used at all. Trace the data citations behind responses.
• Accuracy and sentiment: whether the description of your brand is correct and favorable.

4. Separate your effect from background noise

• Keep a control group of relevant prompts that your content work did not target. If the targeted prompts move and the control group does not, the change is more likely your work. If both move, the cause may be model updates or market shifts.
• Compare the change against your run-to-run variation. A 5-point move may be noise if repeated runs routinely differ by that much.
• Allow time for effects. Content may take weeks to be crawled, indexed, and picked up, so check at several intervals, not just once.

5. Connect it to business outcomes (optional)

• Track AI-referred traffic, branded search, and sales inquiries alongside visibility scores. Treat these as supporting evidence, since the causal link is weaker than in your prompt tests.

Tools: You can start with a spreadsheet. Dedicated trackers such as Ahrefs' Brand Radar, Amplitude's AI visibility product, or Meltwater's GenAI Lens automate the sampling. Check how each one collects answers (interface scraping versus API), because that affects comparability with your manual results. Free audits also exist, but they are one-off snapshots rather than longitudinal tests.

Caveat: A single-session check or one tool's score can mislead. The most convincing evidence is a consistent, repeated, controlled comparison across several assistants over time.

Testing whether your content updates have changed AI assistant recommendations requires shifting from traditional "rank tracking" to Answer Engine Optimization (AEO). Because AI answers are generated dynamically and vary by prompt and model, you cannot rely on a single, static position.

Instead, use the following framework to measure your impact over time.

1. Build a Fixed "Prompt Set"

To see if your content changes are actually working, you must test them consistently against a "ground truth" set of questions.

• Define Your Prompts: Create 20–40 prompts that your ideal customers actually ask (e.g., "What are the best tools for [Your Category]?", "How to solve [Specific Problem]?", or "Compare [Your Brand] vs. [Competitor]").
• Establish a Schedule: Run this same set of prompts across major platforms (ChatGPT, Perplexity, Gemini, Google AI Overviews, and Claude) on a fixed cadence—weekly or bi-weekly.
• Track Trends, Not Anecdotes: A single check is meaningless because AI responses are inconsistent. Record whether your brand is:
  • Mentioned: Simply named in the response.
  • Recommended: Positively framed or suggested as a solution.
  • Cited: Linked as a source (the gold standard for traffic and authority).

2. Use "Directional" Monitoring

Since there is no "AI tab" in Google Search Console, you must assemble a view from multiple sources to track the impact of your work.

• Manual Tracking (Free): Use a spreadsheet to log your prompt results over time. This is the most defensible method because it forces you to look at the qualitative shift in how the AI "talks" about your brand.
• Server Log Analysis: Check your server logs for "live retrieval fetches." Unlike bulk training crawls, these occur when an AI system is actively reading your page to answer a user’s live question. A spike in these specific fetches is a high-confidence signal that your content is being chosen as relevant.
• AEO Monitoring Tools: If you have the budget, tools like Geoptie, Profound, Ahrefs Brand Radar, or Conductor can automate the tracking of brand mentions, citation rates, and competitive share-of-voice across multiple AI engines.

3. Measure Business Outcomes

Visibility is a vanity metric if it doesn't lead to action. Connect your AI visibility efforts to business KPIs:

• AI Referral Traffic: Use Google Analytics 4 (or your preferred analytics platform) to isolate traffic from AI-driven domains (e.g., chatgpt.com, perplexity.ai).
• Branded Search Spikes: Monitor if your direct traffic or branded keyword volume increases when your content begins to earn more citations in AI summaries.
• Conversion Rates: Track if visitors arriving from these AI-engine referrers are higher quality (e.g., longer time on site, higher conversion rates) compared to traditional search traffic.

Summary Checklist for Your Team

Important Note: Avoid looking for a "ranking" number. Instead, look for consistency. If your brand was absent from your "category" prompt but starts appearing in 3 out of 10 runs after your content update, you are moving the needle.

Treat it like a before-and-after experiment, not a one-off “ask an AI and see” check.

1. Define what “changed the recommendation” means. Track separately whether an assistant:
   • mentions your brand,
   • recommends it (not just names it),
   • links to or cites your specific content,
   • recommends competitors instead,
   • describes your offering accurately.

1. Choose a fixed prompt set. Use real questions from customers, sales calls, support, and search queries. Include discovery, comparison, and purchase-intent prompts. Keep the wording and market context consistent.

1. Record a baseline before changing content. Run each prompt on the assistants and platforms that matter to your audience. Save the date, prompt, full answer, brands named, recommendation, and cited URLs. Use fresh chats and repeat each prompt several times—answers can vary between runs. A practical starting point is three to five runs per prompt per assistant. ⁹¹⁵

1. Make the content change and keep a change log. Note the page, what changed, publication date, and the prompt groups you expect it to affect.

1. Rerun the same test. After the content has had time to be found and reflected in results, use the same prompts, assistants, settings, and number of repetitions. Compare rates, not isolated answers:
   • Recommendation rate = runs that recommend you ÷ total runs
   • Citation rate = runs citing your page ÷ total runs
   • Mention rate = runs mentioning you ÷ total runs

   Break results down by assistant and prompt category, and track competitor mentions or citations too. ⁸¹⁰

1. Check business outcomes separately. Monitor visits and conversions from AI-referred traffic, alongside ordinary organic clicks and conversions. For Google’s AI search features, Search Console documentation currently describes reporting on impressions and pages shown; use analytics for visits and conversions. ²³

Be cautious about causality. AI answers vary, and platforms, retrieval systems, and competitors change over time. A better test is to update some comparable pages or topic groups while leaving others unchanged, then compare how both groups shift. Keep the observation window and measurement method the same, and report the sample size and dates. A rise in recommendations after an edit is evidence of a possible effect—not proof by itself. ⁸¹⁰