How can I test whether our content work changed what AI assistants recommend?

The cleanest test is a controlled before/after study, not a handful of prompts. AI answers vary between runs, and a mention isn’t the same as a recommendation—so track both consistently.

A practical test

1. Define “recommend.” For example: your brand is named as a suitable option, is included in a shortlist, or is preferred over competitors. Track simple mentions separately.
2. Build a fixed prompt set. Use real questions your audience asks, including comparison and “best option for…” prompts. Keep the exact wording and split prompts into topic groups.
3. Record a baseline before publishing. Run the prompts across the assistants that matter to you, repeating each prompt several times on different days. Save the full answers, citations, and URLs—not just a yes/no score.
4. Use a comparison group. Update some comparable pages or topics and leave others unchanged for a while. Compare the change in outcomes for updated topics against the change for untouched topics. This helps distinguish the content’s effect from general shifts in models, competitors, or the web.
5. Repeat the same test after publication. Keep the assistant, settings, search/browsing mode, prompt wording, and sampling schedule as consistent as possible. Record model/version and date when available.
6. Score the answers. Useful measures include:
   • recommendation rate: share of runs where your brand is recommended
   • shortlist share: share of runs where you appear among the options
   • citation rate: share citing your page, and which page
   • recommendation position and accuracy of the claims
   • competitor share, as a comparison—not a substitute for your own results

Because results fluctuate, compare averages across repeated runs and report the sample size or uncertainty. A single answer—or a single “AI visibility” score—can be misleading. (arxiv.org)

Check whether visibility led to business impact

Separately monitor AI referrals, conversions, and branded search. These are useful downstream signals, but they don’t prove the content caused a change in recommendations: many people may see an answer without clicking. OpenAI says ChatGPT search referral URLs can include utm_source=chatgpt.com, which may help identify some visits in analytics. (help.openai.com)

For Google specifically, Search Console’s generative AI performance report provides impressions for links shown in AI Overviews and AI Mode, grouped by dimensions such as page and date. That measures visibility in those features, not recommendations across all assistants. (support.google.com)

A compact experiment: choose 20–50 representative prompts, collect repeated baseline answers, update a subset of relevant pages, keep matched pages unchanged, then repeat the prompt set on a regular schedule. Report the treatment-versus-control change in recommendation and citation rates. That gives you a much more credible answer to “did our content work change what assistants recommend?” than a one-off spot check.

Treat this as a before-and-after experiment. A single AI answer tells you very little, because the same prompt can produce different answers on different runs.

1. Lock down the prompt set before you change anything

• Use a fixed set of prompts, roughly 20 to 50 commercial prompts. One guide suggests starting there and moving to a dedicated tool once volume gets unmanageable. Pick 20 to 50 high-value commercial prompts, run them across LLMs of choice, and calculate your visibility score as brand mentions divided by total answers tested, multiplied by 100.
• Ground the prompts in real buyer language. Pull from Search Console queries, site-search logs, sales and support questions, and "People also ask" items.
• Split prompts into two groups: unbranded category prompts (e.g., "best CRM for startups") that test whether you get discovered, and branded prompts that test how you're described. Brand and nonbrand prompts should not share one unqualified total. Within the unbranded group, separate comparison and selection prompts from educational ones, since content often moves one more than the other.

2. Record a baseline, and run each prompt several times

• Run every prompt multiple times on each engine you care about (ChatGPT, Gemini, Perplexity, Google AI Mode, Claude, Copilot, etc.). Run-to-run variation is large. One study found identical brand recommendations in under 1 in 100 repeated runs, so treat single-run rankings as noise. SparkToro research running nearly 3,000 prompts across multiple AI platforms found that these produce identical brand recommendations less than 1 in 100 times.
• Log the date, engine, model version if visible, location, and whether the interface was web-enabled. Save the full answer text and the cited sources.

3. Score a small set of metrics

• Mention rate: share of prompts where your brand appears in the answer text.
• Citation rate: share of prompts where your domain is linked as a source. Keep this separate from mentions, because a model can know your brand without citing your pages.
• Share of voice: your mentions as a proportion of all brands named in those answers.
• Position and sentiment: whether you're the first recommendation or a footnote, and whether the description is accurate and favorable.

4. Make the change and re-run on a schedule

• Re-run the identical prompts, on the same engines, with the same scoring, at fixed intervals after publishing. Repeating panels at fixed intervals lets you see drift as well as the effect of your changes.
• Keep a control. Leave some prompt families untouched, or change content only for a subset of topics. If visibility rises across the board, including in the untouched group, the cause may be model updates or market noise rather than your work.

5. Decide whether the change is real

• Compare the before and after rates against the run-to-run noise. With enough repeats, a two-proportion test or a confidence interval on each rate will tell you whether a movement of a few points is meaningful. Most small prompt sets won't clear that bar for modest changes, so expect to need more runs or a larger set.
• Compare the treated prompts against the control prompts, not just against your own past numbers.

6. Check downstream signals too

• Track AI referrals in your analytics by creating a channel group for known AI referrers. This undercounts, because referrers are often stripped, so treat it as a floor.
• Check whether the pages that gained citations are the ones you changed.

Tools vs. DIY: A spreadsheet and manual runs are enough to start. Dedicated trackers (Semrush, Profound, Scrunch, Meltwater, Muck Rack's Generative Pulse, and others) automate repeated sampling and scoring, which matters once you have hundreds of prompts. Whichever you use, make sure you can export the raw answers, since you'll want to audit them.

Testing whether your content work influences AI assistant recommendations requires a shift from traditional "ranking" metrics to "visibility" and "citation" tracking. Because AI models generate answers dynamically rather than serving a fixed list of links, you must measure presence, accuracy, and sentiment across a variety of simulated user journeys.

To effectively test and track your impact, use the following framework:

1. Build a "Prompt Library" (The Baseline)

You cannot measure improvement without a consistent baseline. Create a spreadsheet of 20–40 high-intent prompts that your ideal customers actually use. Categorize these to track performance trends:

• Category Questions: "What is the best [product category]?"
• Use-Case Questions: "How do I solve [specific problem] using [tool type]?"
• Comparison Questions: "[Your Brand] vs. [Competitor]?"
• Problem Questions: "Why is [pain point] happening?"

Testing Method: Run these exact prompts periodically across major AI assistants (ChatGPT, Gemini, Perplexity, Copilot, etc.). Record whether your brand is mentioned, if it is recommended, and if you are cited as a source. A single test is an anecdote; a trend across weeks of testing is a measurement.

2. Monitor Key Visibility Metrics

Move beyond "rank" and focus on these four indicators of AI-driven visibility:

• Mention Rate: How often the AI names your brand in its response.
• Citation Frequency: How often the AI links to your website as a source for its claims.
• Share of Voice (SOV): How often you appear relative to your competitors for the same prompts.
• Sentiment/Accuracy: Is the AI describing your value proposition correctly, or is it hallucinating or favoring a competitor?

3. Use Purpose-Built AI Visibility Tools

Manual testing is a great start, but enterprise or dedicated AI tracking tools can automate the process and provide deeper insights into "why" a model is—or isn't—citing you.

• Specialized Platforms: Tools like ZipTie, Lumar, Otterly.AI, Profound, and Ahrefs Brand Radar are built to monitor citations and brand sentiment across LLMs.
• Integrated Solutions: Many SEO platforms (e.g., Semrush, Similarweb) have expanded to include AI visibility features that bridge the gap between traditional search and generative engine optimization (GEO).

4. Track "Hidden" AI Referral Traffic

AI-driven traffic is often misclassified in analytics tools like Google Analytics 4 (GA4).

• Check Referrals: Look for traffic from sources like chatgpt.com, perplexity.ai, or gemini.google.com in your referral reports.
• Account for "Direct" Spikes: A significant portion of AI-driven traffic is often bucketed under "Direct" or "Organic Search" because the user may see a brand mention in an AI answer and then search for the brand by name later. Cross-reference traffic spikes with the dates you published new, highly targeted content.

5. Optimize for "Answer Engine" Mechanics

If your testing shows low visibility, your content may need to be restructured for how AI models "read" the web (often called Generative Engine Optimization or GEO):

• Structure for Clarity: AI favors self-contained, concise answers. Use clear H2/H3 headers phrased as questions, followed by a direct, evidence-based answer in the first two sentences.
• Use Schema Markup: Implementing Article, FAQ, and Product schema helps AI crawlers parse your site’s data more accurately.
• Focus on "E-E-A-T": AI models prioritize content that is perceived as authoritative. Ensure your content includes expert insights, real-world case studies, and citations from other high-authority domains.

Summary Checklist for Your Next Review

Treat it as an experiment, not a one-off “ask an AI” check. AI answers vary, and a simple before-and-after comparison can’t show that your content caused a change. ³¹⁰

1. State a specific hypothesis. For example: “Adding a clear pricing comparison will increase our inclusion in answers to pricing-comparison prompts.”
2. Choose a fixed prompt set that reflects real customer questions. Use the exact same prompts, AI products or modes, locale, and account context each time; record the date and any model/version details shown. ³⁷
3. Record a baseline. Run the prompts repeatedly before publishing, saving the full answers and citations—not just a visibility score. A practical starting point is a few weeks of baseline observations, though the right amount depends on how variable the answers are. ⁴⁸
4. Change one thing at a time. Log what you changed, when it went live, and which pages and prompts it was meant to affect. If possible, leave comparable pages unchanged as controls. ⁴⁸
5. Check that the change can be found. Confirm the updated pages are accessible and eligible for indexing, then allow time for the relevant search systems to pick them up before measuring. ⁸
6. Repeat the same measurements under the same conditions. Compare brand inclusion, recommendation position or grouping, cited URLs, and whether the answer accurately reflects the content. Keep “mentioned,” “cited,” and “recommended” as separate outcomes. ⁷¹⁰
7. Compare against controls. Look at how the changed pages or prompt group moved relative to the unchanged group. If both move similarly, an outside change may explain the shift. Repeat the pattern before treating it as a reliable finding. ⁸¹⁰

A simple primary metric could be the percentage of eligible runs in which your brand is recommended for a defined prompt group. Track citations and accuracy alongside it. Report results as evidence consistent with an effect, not proof, unless the comparison design supports a causal conclusion. ⁷¹⁰