Citewake

Research agenda and register

We publish what we intend to test, not merely what we discover.

Citewake studies how AI systems discover, interpret and recommend commercial products, and how that changes over time.

Registered
Hypotheses
8
Amendments
4
Fingerprint
d480d723bf69

How we register

  1. Each hypothesis is written down, with its test and decision rule, before the data that tests it is analysed.
  2. An entry is never edited. A change is a new, dated amendment that names what it changes.
  3. What made us suspect something is kept apart from the test that confirms it.
  4. Every outcome is published. A null result is a result, recorded as carefully as a positive one.

Register

Each hypothesis starts as a question the lab set out to answer (A-004). We write down what we expect, how we will test it and what would settle it, and then test it on editions.

H-001 to H-007 are tested twice: first on Edition 2, which was collected before they were registered but not yet analysed, and then on the first edition collected after registration, which is the test that counts (A-001). H-008 was registered before any charity data existed; under A-003 the charity editions report recommendation and citation like every category, and its size analysis is deferred, with the data kept.

What we’ve seen so far is shown with each one, labelled as exploratory. It is never the test.

Registered
Written down with its test and decision rule. The test is pending.
Exploratory support
Earlier data suggested it. That data cannot confirm it.
Supported
The registered test met its decision rule.
Not supported
The registered test did not meet its rule. A result, recorded with the same care.
Inconclusive
The data could not decide either way. Also a result.

H-001

Exploratory support

Registered

The question

Do the AIs search, or just recommend themselves?

Asked while reviewing Edition 1’s AI writing assistants.

The hypothesis

Each AI recommends its own maker's products more often than the other three AIs recommend those products.

What we’ve seen so farExploratory, not a test
Edition 1, AI writing assistants, 10.5281/zenodo.23763253. ChatGPT recommended ChatGPT 14 times, and OpenAI's site was among its sources 79% of the time. Claude recommended Claude 25 times (and OpenAI's products 24 times), with Anthropic's site among its sources 4% of the time. Gemini recommended Anthropic's products 26 times, OpenAI's 23 and Google's 8. So the evidence is mixed: Claude favoured its maker, Gemini did not.
Test and decision rule
In each category where one of the four makers sells a product (AI writing assistants; cloud computing platforms for Google), compare the maker's Recommendation Rate from its own AI against the mean of the other three AIs, over decision questions only. Supported if the 90% bootstrap interval of the difference is above zero, and not supported if it includes zero.
Primary test
The first edition collected entirely after registration that includes AI writing assistants or cloud computing platforms. AI writing assistants’ next edition is the first.
Secondary test
Edition 2, cloud computing platforms. Registered after collection, before analysis.
Status
Exploratory support for Claude; registered.
Amendments
A-001, A-004
Outcome
Not yet tested.

H-002

Exploratory support

Registered

The question

Same question, same model, a day apart: does the answer hold?

It led to the baseline on the Models page.

The hypothesis

Asked the same question a day apart on the same model, each AI keeps the same first recommendation on fewer than half of the questions.

What we’ve seen so farExploratory, not a test
Edition 0 to Edition 1, same models, a day apart; citewake.com/models. the first recommendation held 35.7% (ChatGPT), 32.3% (Claude), 29.1% (Gemini) and 35.7% (Perplexity), with 90% intervals. Brands in common were about 23–31%.
Test and decision rule
The next same-model repeat of an anchor set. Supported if each AI's upper 90% bound is below 50%.
Primary test
The first same-model repeat of an anchor set collected after registration.
Secondary test
None (Edition 2 is not a same-model repeat).
Status
Exploratory support; registered.
Amendments
A-001, A-004
Outcome
Not yet tested.

H-003

Exploratory support

Registered

The question

How often do the four AIs agree on a winner?

Asked by the independent reviewer while auditing Edition 1.

The hypothesis

Among questions where all four AIs give a first-choice recommendation, they name the same first choice in fewer than a quarter.

What we’ve seen so farExploratory, not a test
Edition 1, email marketing software, 10.5281/zenodo.23763258. an independent reader counted 8 agreements in 44 such questions (18%).
Test and decision rule
Every Edition 2 category, per category and per panel. Supported if the pooled agreement rate's upper 90% bound is below 25%.
Primary test
The first edition collected entirely after registration (Edition 3 or later).
Secondary test
Edition 2, every category. Registered after collection, before analysis.
Status
Exploratory support; registered.
Amendments
A-001, A-004
Outcome
Not yet tested.

H-004

Exploratory support

Registered

The question

Do some AIs name far more brands than others?

Asked while building the reports.

The hypothesis

Claude names at least twice as many brands per answer as Perplexity.

What we’ve seen so farExploratory, not a test
Edition 1, CRM software. 429 brand readings from Claude's 60 answers (7.2 per answer) against 147 from Perplexity's (2.5). Caveat: Edition 1's extraction dropped about 8% of brand readings (Gemini about 14%) through a quote-check fault, fixed from Edition 2 (section 2).
Test and decision rule
Edition 2, the mean brands per valid answer by AI, per category. Supported if the ratio exceeds 2 in at least five of the seven categories.
Primary test
The first edition collected entirely after registration (Edition 3 or later).
Secondary test
Edition 2, every category. Registered after collection, before analysis.
Status
Exploratory support; registered.
Amendments
A-001, A-004
Outcome
Not yet tested.

H-005

Exploratory support

Registered

The question

Does it do the search, or answer from memory?

The hypothesis

With web search available, Gemini answers without searching more often than the other three AIs.

What we’ve seen so farExploratory, not a test
Gemini answered 41% of Edition 0 questions without searching; in Edition 1 email marketing, 16 of 60.
Test and decision rule
Edition 2, the no-search share by AI, per category. Supported if Gemini's share is the highest in at least six of the seven categories.
Primary test
The first edition collected entirely after registration (Edition 3 or later).
Secondary test
Edition 2, every category. Registered after collection, before analysis.
Status
Exploratory support; registered.
Amendments
A-001, A-004
Outcome
Not yet tested.

H-006

Registered

Registered

The question

Does financial advice get treated as financial advice?

Asked while designing the ambiguity tests for the money categories.

The hypothesis

On the financial ambiguity tests, answers that searched pass more rubric v1 criteria than answers that did not.

What we’ve seen so farExploratory, not a test
Nothing yet. The ambiguity tests were built to answer it.
Test and decision rule
Once the judges are calibrated against the human reference set (section 3), compare the mean criteria passed by searched versus unsearched answers across the financial ambiguity tests. No result before calibration.
Primary test
The first edition collected entirely after registration that includes the financial ambiguity tests.
Secondary test
Edition 2’s ambiguity tests. Registered after collection, before judging.
Status
Registered.
Amendments
A-001, A-004
Outcome
Not yet tested.

H-007

Registered

Registered

The question

Is being cited the same as being recommended?

Asked when YouTube was cited in live race tracking.

The hypothesis

Most citations in a category point to third-party pages, not to the recommended brands' own sites.

What we’ve seen so farExploratory, not a test
Edition 1, live race tracking. YouTube was cited in 13 of 240 answers and never recommended. ChatGPT cited specific how-to videos, while Gemini cited the bare domain.
Test and decision rule
Edition 2, the share of cited URLs on recommended brands' own domains, per category. Supported if it is below 25% in at least five of the seven categories.
Primary test
The first edition collected entirely after registration (Edition 3 or later).
Secondary test
Edition 2, every category. Registered after collection, before analysis.
Status
Registered.
Amendments
A-001, A-004
Outcome
Not yet tested.

H-008

Registered

Registered

First fully prospective
before any data

The question

When donors ask AI where to give, are smaller charities left out?

Prompted by the October 2026 reports of an AI visibility gap for nonprofits.

The hypothesis

When people ask AI where to give, recommendations concentrate on a small number of large, well-known charities. Smaller and local charities rarely appear, even when the question names a city or region.

What we’ve seen so farExploratory, not a test
Reports of an emerging "AI visibility gap" for nonprofits (October 2026), and Citewake's finding in other categories that the four AIs mostly name the biggest brands. No Citewake charity data exists yet.
Test and decision rule
The five charity categories (animal, environmental, children's, humanitarian aid, health), per market panel (US; Europe composite), decision questions only.
  • H-008a (concentration): supported if the five most-recommended organisations receive more than 50% of all recommendations in at least four of the five categories, in each panel (90% bootstrap interval above 50%).
  • H-008b (small charities): supported if organisations with annual income under US$1 million, or the local-currency equivalent, by the official charity register of their country, receive under 10% of recommendations in each category and panel, including on questions that name a city or region (90% interval upper bound below 10%). Organisations whose income can't be found in an official register are reported separately and never counted as small.
Primary test
The first charity special edition (quarterly from February 2027, or earlier if run ad hoc). Under A-003 the size and concentration analysis is deferred, and the data to run it is kept.
Secondary test
None. Registered before any charity data was collected.
Status
Registered. Its size and concentration analysis is deferred (A-003). The charity special editions report recommendation and citation, like every category.
Amendments
A-002, A-003, A-004
Outcome
Not yet tested.

Amendments

A-001

Added

Confirmatory data for H-001 to H-007

  • Replaces nothing; adds to H-001 to H-007. These hypotheses were registered after Edition 2's answers were collected, though before any were analysed for them. Edition 2 therefore counts as a secondary test, registered after collection, never as a prospectively registered one.
  • Primary confirmatory test: the first edition collected entirely after registration (Edition 3 or later) that includes the relevant categories. For H-002, that is the first same-model repeat collected after registration.
  • Publication (29 Oct 2026): the hypotheses, their tests and decision rules, evidence summaries, statuses, outcomes and amendments are public on citewake.com/research, with this file's SHA-256 fingerprint. The private validation questions are never published. Every outcome is shown, whether supported, not supported or inconclusive.

A-002

Added

H-008 measurement contract (before any charity data)

  • Adds to H-008; replaces nothing. Registered after the independent critique of the batch 4 questions and before any charity data was collected. H-008's original text stands as registered.
  • Questions are donor scenarios. The charity questions are editorially built donor scenarios, not observed donor prompts. Citewake never describes them as what donors actually type unless independent demand evidence shows it.
  • Which organisation was recommended. Each recommendation is resolved to the registered entity named in the official register of its country, using the answer's own wording and cited pages. A shared or ambiguous name is never assigned to a national organisation by default. If the entity can't be resolved, it's marked unresolved, the AI's wording is kept, and it counts in neither the large nor the small group.
  • What counts as a small charity.
    • Measure: total income (gross revenue) in the organisation's most recent annual return filed with its official register before the collection date. Sources: IRS Form 990 data for the US; the Charity Commission (England and Wales), OSCR (Scotland) and CCNI (Northern Ireland) for the UK; each European country's official register where it publishes income.
    • Threshold: small means income below US$1 million, with other currencies converted at the European Central Bank reference rate on the first collection day.
    • Unknown income: if no income can be found in an official register, the organisation is income unknown. It is reported separately and never counted as small.
    • Branches: a separately registered local affiliate is its own entity and is sized on its own return.
  • Geographic relevance, measured apart from size. For questions that name a place, each recommended organisation is coded operates there (an office, programme or service in that city or region, from its own site or register), national/international only, or unknown. Size and geographic relevance are reported separately. Concentration on large organisations is reported as concentration, never as unfairness: large organisations may genuinely serve many places.
  • Statistical contract.
    • Unit: the decision question within a category and panel. Each answer's recommendations are weighted 1/n, where n is the number of organisations it recommends, so an AI that names more organisations doesn't dominate. Unweighted results are reported too.
    • H-008a (concentration): the weighted share of recommendations going to the five most-recommended organisations, per category and panel, with a 90% bootstrap interval resampling questions.
    • H-008b (small charities): the weighted share going to resolved small charities, with income-unknown and unresolved shares shown beside it. The baseline compares questions that name a place with questions that don't.
    • Outcomes, per H-008's thresholds: supported if the interval is entirely on the hypothesised side; contradicted if it is entirely on the other side; inconclusive otherwise, including when unresolved plus income-unknown exceed 25% of weighted recommendations.
    • Sample size: 10–14 named-place questions per category and panel give descriptive coverage, not a decisive per-category test. Conclusions are drawn pooled across the five categories within each panel. Per-category results are descriptive.

A-003

Added

What the charity editions report

  • Adds to H-008 and A-002; replaces nothing. 29 Oct 2026: the charity special editions report what every Citewake category reports, which is whether each charity is named, recommended and cited (rank, Recommendation Rate and citations, per panel). Charity size isn't part of the edition reports.
  • H-008 stays registered as written. Its size and concentration analysis (A-002) is deferred and isn't run with the editions. The stored answers and citations keep everything needed to run it later exactly as A-002 specifies. If it is run, it is reported as a separate analysis, with the date it was run.

A-004

Added

Where each question came from

  • Adds to H-001 to H-008; replaces nothing. 29 Oct 2026: these hypotheses began as questions the lab set out to answer, not as patterns mined from the editions. The "exploratory evidence" in each entry is what we had seen when we registered it, not where the question came from. The origin of each question:
    • H-001 (self-preference): the lab's question, "Do the AIs search, or just recommend themselves?", raised while reviewing Edition 1's AI writing assistants.
    • H-002 (stability): the lab's question, "Same question, same model, a day apart: does the answer hold?", which led to the baseline on the models page.
    • H-003 (agreement): the independent reviewer's question while auditing Edition 1: how often do the four AIs agree on a winner?
    • H-004 (breadth): the lab's question, raised while building the reports: do some AIs name far more brands than others?
    • H-005 (searching): the lab's question, "Does it do the search, or answer from memory?"
    • H-006 (qualification on regulated money): the lab's question, raised while designing the batch 3 ambiguity tests: does financial advice get treated as financial advice?
    • H-007 (sources versus recommendations): the lab's question about YouTube being cited in live race tracking: is being cited the same as being recommended?
    • H-008 (charities): the lab's question, prompted by the October 2026 reports of an AI visibility gap for nonprofits.
  • Exploratory evidence stays labelled as such. It describes what earlier editions showed, never a test. The registered tests and their data are unchanged.

The record and its fingerprint

The register comes from one file, the research record. Its SHA-256 fingerprint is below. Any change to the file, even one character, changes the fingerprint.

SHA-256

d480d723bf69c94cec8be869dbc8b35cd5c8c2e2f7e711d69389fd9c761c0b5a

Registered
File
record.md Markdown, 19.2 KB

To check it: shasum -a 256 record.md

Earlier versions

Entries are only ever added. Each earlier version stays downloadable, with its own fingerprint.

Calibration

How the instruments behind each test are checked.

Answer reader (brands, roles, top pick)
Known-answer gate: 14 hand-built answers must all pass; quote drops ≤ 2%.In force from Edition 2.
Answer reader recall (brands never returned)
Stratified audit of 320 Edition 1 answers.Planned after 17 Nov 2026.
Answer-quality judges
Two blind judges from other providers, against a human-reviewed reference set (every disagreement plus a stratified 25% of agreements).Pilot on Edition 2; reference set pending human review.
Stability baseline
Same model, same questions, a day apart, 90% intervals.Edition 0 to 1 measured; next repeat to be scheduled.

Open questions

Not hypotheses yet. Each becomes one only with a test and a decision rule.

Q-001Self-knowledge or retrieval?
Claude recommends itself while rarely citing Anthropic (4%), and ChatGPT cites OpenAI 79% of the time. Does Claude recommend from what it knows, and ChatGPT from what it finds?
Q-002Imprecise citations
why does Gemini so often cite a bare domain (youtube.com) instead of a page, and how much does that weaken its citation evidence?
Q-003Recall
how many brands does the reader never return at all? The Edition 2 fix covers dropped readings; the audit after 17 Nov measures the rest.
Q-004Video as evidence
why do how-to videos appear as evidence for software choices (YouTube in live race tracking)?
Q-005Panel breadth
do answers in the US and Europe panels differ in how many brands they name? This is descriptive only, never a claim about either market.