How Citewake works: the index, the engine and the measures

0First principle: the index is the product

Citewake is only as good as its index and the way the engine compiles it. Every published number summarises the index and has no value beyond it. So the order of priorities is fixed:

1. Collect the right questions, the same way, on schedule.
2. Read every answer correctly: which brands, in what role, with which sources.
3. Measure each thing for what it is, with fixed denominators and stated uncertainty.
4. Show it.

Nothing in step 4 may hide a weakness in steps 1 to 3. We'd rather publish modest, trustworthy numbers than an impressive score whose meaning shifts between brands and AIs.

1What we measure, and what we don't yet

AI assistants usually name three to five brands per answer, mostly the biggest. A single "were you named" score would put nearly everyone at 0 and the few named near the top. Rather than force a spread with a formula, Citewake measures three distinct things, each with its own plain meaning, so that a young brand has real numbers that move before it is ever recommended.

Alongside them, as supporting metrics, clearly labelled:

• Neutral Mention Rate and Negative Mention Rate, kept separate from recommendations.
• Competitive Share: the brand's share of all positive recommendations in the benchmark. It's relative, so it can fall when rivals gain even if the brand's own Recommendation Rate holds, and we say so wherever it's shown.
• Reported Retrieval Presence: shown per AI, only where that AI's API reports the URLs it surfaced or consulted (section 6.3).

How the measures read for a young brand. Recommended for 2 of 120 questions by one AI, cited on three roundups and named neutrally twice, it shows exactly that, and each number moves as it grows. A brand with no trace shows "no evidence in this benchmark", which is distinct from "not indexed" (section 9).

2Definitions

Category

A market defined by the job a buyer wants done (for example, "AI search visibility tools"). Curated by us.

Market

Where the question is asked from. "Location unspecified" (no location sent) is the default. A named country or city is a separate market. It isn't a sample of the world.

Benchmark

The fixed, versioned set of scored questions for a category. Public.

Validation set

A separate private question set used only to check the benchmark, never to produce published numbers.

Observation

One question, one AI, one collection: the full response and everything it returned.

Valid answer

An observation that completed and returned an answer (section 9).

Cycle

4 weeks. Every scored question is asked of every AI once per cycle.

Brand

A canonical entity: name, aliases, products, parent company and domains.

3The index: categories and questions

3.1 Categories

• Curation: each category is defined by a buyer's job, in one sentence, with stated boundaries.
• How brands join: only through evidence. A brand enters a category's index when an AI names it in an answer, or a cited page lists it. Nobody can submit, pay for, or request inclusion. A brand can be in several categories.
• Indexed on demand: a category is indexed only when there is demand: a paying customer working in it, or repeated lookups of brands in it. Each one is approved by us. The index grows with revenue, not ahead of it.

3.2 Scored questions

120 per category, in three tiers of 40:

• Head: broad buying questions, where leaders win.
• Mid: a use, a budget or a buyer type.
• Tail: a specific need, where specialists and new products can win.

Each tier mixes intents: best and recommend; compare and alternatives; how to; price; specific use.

Rules:

• First person, problem first, the way people type.
• No brand name in any scored question. Branded diagnostics ("is X any good?") are a separate, labelled tool, outside the discovery benchmark.
• No leading words.

Question sources: real search demand (suggestions, related questions, keyword data); public forums; and the anonymised, aggregated shape of aipozi customers' questions, never a customer's question verbatim.

Tier weighting is reported, not baked in. Each measure is shown overall and per tier, so a reader sees where a brand wins. The overall figure weights the tiers equally. Any other weighting needs evidence and a version change.

3.3 Versioning and the validation set

• Versions: a benchmark is frozen as v1, v2 and so on. Changing any question makes a new version. Numbers are compared within a version only.
• Public scored set: all scored questions are published, so anyone can recompute a published number.
• Private validation set: a separate private set of 30 questions per category, written to the same rules, asked on the same schedule, and never used for published numbers. A persistent gap between a brand's results on the two sets triggers an investigation. It isn't treated as proof of gaming; the cause may be the benchmark itself.
• Retiring the validation set: it's retired and published at each version change, so its past checks can be verified after the fact.

4Collection

• The AIs: ChatGPT, Claude, Gemini and Perplexity, through their official APIs, with live web search on. Each observation records the provider, model, model version and every setting.
• Clean requests: no memory, no custom instructions, no steering system prompt. The question goes in exactly as written.
• The schedule: a 4-week rotating cycle. The 120 scored questions are split into four fixed groups of 30, balanced by tier and intent, and collected in turn across the cycle. Every question is asked once per AI per cycle. The validation set rotates the same way.
• When numbers are published: after a cycle completes, as a monthly edition. Changes are compared question by question against the same question's previous answer (section 8).
• Idempotent and resumable: a run is keyed by category, version, market, week and group. A retry never doubles an observation, and a crash resumes where it stopped.
• The cost cap: each run stops at twice its expected cost. A run that doesn't complete its group isn't published as comparable. It finishes on retry, or that period is marked incomplete and the figures say so.

4.1 What we keep, forever

A published number can be recomputed from these, even after the live web pages change.

4.2 The three compartments

Every observation passes through three separate compartments. Each has different economics and different rules.

Compartment 1: the prompt. There is no prompt caching: a scored question is about 30 tokens, too short to cache. An answer is never cached or reused. Every observation is a fresh request, because a reused answer is invented data.

Compartment 2: the search. Most of the cost of collection is here, and it can't be shared.

• The temptation: run one search ourselves and give all four AIs the same results.
• Why we don't: that would measure our search, not theirs. Which pages each AI finds and trusts is a large part of why the AIs recommend different brands, and that difference is what Citewake measures.
• What we do instead: each AI's search settings (how many searches it may run, how much context it reads) are fixed, recorded on every observation, and changed only by a method version.
• Batch processing: it's used for this compartment only where a provider is confirmed to run live web search identically in batch mode. Each provider is checked in the pilot.

Compartment 3: the analysis. The grand master's long, fixed rulebook (definitions, role rules, the category's brand and alias list) is cached. Analysis isn't time-sensitive and uses no web search, so it runs through batch processing.

5Extraction: reading the answers

The strictest rules in this document apply here.

5.0 The grand master

The four AIs only answer. One judge reads everything.

• The model: Claude Opus 5.5, our strongest. The index is only as good as this reading, so it gets the best reader.
• What it reads, together: for each question, all four AIs' answers side by side, with each one's citations. Reading them together means a brand is resolved to the same entity in all four, and "recommended" is judged by one standard across ChatGPT, Claude, Gemini and Perplexity. That's what makes the measures comparable across AIs.
• Cited pages: most third-party pages are long, plain lists. A small, fast model (Claude Haiku 5.5) reads them under the same rulebook. Every page Haiku marks as uncertain, plus a random 10% of the rest, goes to the grand master, which rules on it. How often the two disagree is published.
• It classifies; it doesn't judge. The grand master records what each answer says. It never adds an opinion about which brand is better, never adds a brand that isn't in the text, and works only from the answer and the fetched page.
• Versioned: the grand master's model, rulebook and brand list are versioned and recorded on every extraction (section 4.1). Changing any of them makes a method version, and the labelled test set is re-scored before it ships.

5.1 Answers

1. Find every brand. For each AI's answer, in a fixed schema: every brand in the answer, in order; the exact words it was named with; its role, being recommended (presented as a choice to make for the question), neutral mention (named without being recommended, as history, comparison context or an example) or negative (presented as one to avoid, inferior or unsuitable); whether its order expresses preference (section 7.2); and any of its domains cited.
2. Check that the words exist. A brand is kept only if its name, alias or domain appears in the answer text and its quoted words are found verbatim. This proves the words exist. It doesn't prove the role or the entity is right, which is why section 5.3 exists.
3. Resolve the entity. Brands are matched by domain first, then by alias, with parent and product relationships recorded. A product named in an answer counts for that product, and rolls up to its parent only where we show the parent. Ambiguous names are held, never guessed.

5.2 Cited pages

Each fetched third-party page is read for the brands it lists, in what role (recommended, neutral or negative) and in what order, using the same schema and checks. Pages that can't be fetched, are blocked, are paywalled or are unparseable are recorded as unusable and excluded from the denominator, and the unusable rate is published.

A brand found on a cited page tells us one thing only: Citewake found it in its fetched copy of a page the answer cited. It doesn't establish that the AI saw that passage or considered the brand.

5.3 Measuring accuracy

The labelled test set is a hand-labelled set of complete answers and complete cited pages, labelled by two reviewers, with disagreements resolved and logged. It deliberately includes brands the extractor might miss; incorrect additions; recommendations against neutral and negative mentions; ambiguous names; parent and product cases; citations with no brand named in the prose; and brands named negatively on third-party pages.

What it reports, per category, with 95% confidence intervals: precision (extracted brands that are right), recall (real brands that were found), role accuracy, entity accuracy, and source-page extraction accuracy, reported on its own.

The spot check. A random sample of new extractions is checked by hand on a regular schedule. It's an operational check for drift. It doesn't certify accuracy, and it's described that way.

Bars before anything is public: recall and precision each with a lower confidence bound of 90% or more; role accuracy with a lower bound of 85% or more. The pilot sets the final bars from the data. A category below its bars isn't published.

6The evidence, described exactly

We use precise names for what each signal is and isn't.

6.1 Recommendation evidence

Comes only from the answer text, by role (section 5.1).

6.2 Citation evidence

• What counts: a URL the answer attributes something to, as the API structures its citations.
• Inline against listed: inline citations and a wider list of sources are different things where the API distinguishes them, and they're kept apart.

6.3 Reported retrieval presence

• What it is: a URL that the provider's API reports as surfaced or consulted while answering. It means the retrieval system surfaced it, nothing more.
• Where the API reports nothing: the measure is unavailable for that AI. It's never filled in from citations or anything else.
• Per provider: it's shown separately for each AI, because providers report different things, and it isn't combined across AIs until they are shown to be comparable.

6.4 Cited-page presence

A brand listed in Citewake's fetched copy of a third-party page that an answer cited (section 5.2).

What is a hypothesis. That presence on cited pages tends to come before being recommended is something we expect but haven't shown. It's tested prospectively in the pilot (section 11), not claimed.

7How each measure is computed

7.1 Fixed denominators

Cited-page Presence is also shown weighted by how many answers cited each page, labelled as weighted. Every rate is shown per AI, per tier and overall. The overall figure weights each AI and each tier equally.

7.2 Position

Being listed first isn't evidence of preference by itself; lists can be alphabetical, chronological or descriptive.

• What counts as preference: order counts only where the answer expresses it ("the best is…", "my top pick", numbered rankings described as rankings, "if you only choose one").
• What we report: Top-pick Rate, the valid answers where the brand is the expressed first choice ÷ valid answers.
• What happens to order otherwise: it's recorded but not used.

7.3 Uncertainty

• What's independent: the unit is the question. Repeated answers to the same question are dependent and aren't treated as independent.
• Intervals: every rate is shown with a 90% interval from a cluster bootstrap that resamples questions, keeping each question's answers together.
• What the interval covers: uncertainty within this designed benchmark. It says nothing about how well the benchmark represents all buyers, and each page states that.

8Change over time

• A matched comparison. A change is measured question by question: each question's answer this cycle against the same question's answer last cycle, from the same AI. The change and its interval are estimated together, by a cluster bootstrap over questions.
• When a change counts: a change is called a move only when its own interval excludes zero. Otherwise the page says "no clear change".
• Calibration runs: during the pilot we ask the same group several times within a few days. That separates how much answers vary on their own from real change over time, and sets how much change is detectable.
• Model changes: when a provider ships a new model, the date is marked on every chart, and comparisons across it are labelled.

9Missing data and zero results

Each observation has exactly one status, and each status is counted and published.

• No quiet redistribution: missing observations are never filled in from the ones that remain.
• Completion rate: each cycle publishes it per AI. A cycle below 95% complete for any AI is marked incomplete for that AI.
• Unavailable telemetry is never a zero.

What a brand lookup can return:

Measured

The brand appears in the evidence for an indexed category; its measures are shown.

No evidence in this benchmark

The category is indexed and complete, and the brand wasn't found in any answer or usable cited page. Shown with the counts ("0 of 480 valid answers").

Insufficient coverage

The category is indexed, but the current cycle is incomplete or provisional.

Not indexed

We don't index this category yet. We say which category the site seems to belong to, and log the request as demand.

10Integrity

• No paid inclusion and no overrides. Nobody can buy a place, a number or a correction.
• Our own brands get no special treatment. Products made by our team are in the categories indexed first: aipozi and Citewake itself in AI search visibility tools, Biotree in link-in-bio tools, and livedot (and Windline and RunFast, where they offer live following) in live race tracking. Each is measured by the same code, has no manual adjustment and is always included in the spot check. We state the conflict here.
• Sources built to game the measures: a page only counts if an answer cited it. Weighted presence depends on how many answers cited the page. Sudden changes driven only by newly cited, rarely cited pages are reviewed before publishing.
• Corrections are added, not written over. The original publication is kept and the corrected figures are published beside it, labelled with the date, the cause and the evidence. The current result reflects the correction. History stays visible.
• Disputes: a brand can show us an extraction error. If the answer or the page snapshot proves it, we correct it as above. A brand can't dispute an AI's opinion.

11Validation

In this order:

1. Each measure counts its stated event. Shown by extraction accuracy against the labelled test set (section 5.3) and by completion rates.
2. Each measure is repeatable. The calibration runs (section 8) show how much answers vary on their own, and how large a change the cycle-on-cycle comparison can detect.
3. Relationships with outcomes. Our own sites (aipozi, Biotree, livedot) test the plumbing: their measured AI referral visits against their measures. Three sites can't validate anything across categories. Wider validation needs a larger sample of consenting sites across categories: tune on one set of sites and evaluate on a separate one, or the result only fits the examples. Referrals and recommendations are different outcomes, because traffic also depends on demand, links and clicks. A weak relationship is a finding, not a reason to bend the measures.
4. Prospective hypotheses are tested forward in time, not asserted. For example, that cited-page presence tends to come before recommendations.
5. The composite score is considered only after 1 to 4. Its weights are fitted on one dataset and tested on another, and it ships only if it adds information beyond the separate measures.

12Limits we state plainly

• The measures describe AI answers through the APIs to a designed benchmark. They don't measure all conversations, real demand, or personalised answers in signed-in consumer apps. Each month, we check a sample of questions by hand in the logged-out consumer apps and publish how closely they agree.
• "Location unspecified" isn't worldwide.
• Competitive Share is relative to rivals. The other rates aren't, but they still depend on the benchmark's questions.
• Providers report different evidence. Where an AI doesn't report something, that measure is unavailable for it.
• AIs change. The measures report what they said in the cycle, not what they'll say next.