AI answers as a dataset: what researchers can study with them

Stored AI answers make a dataset you can study the way you would study any text corpus. With a fixed set of questions, asked the same way across assistants and over time, you can ask which brands are named, which sources are cited, how the assistants differ, how answer style varies, and how all of it changes. The data cannot show why an assistant chose what it did, or what private chats look like.

What makes a set of answers a dataset?

Collecting answers is not enough. A dataset is answers plus the structure that lets you compare them.

• Fixed questions with identifiers, so the same question can be followed across assistants and time.
• Metadata on every answer: assistant, model, version, date, settings, whether search ran.
• Evidence kept apart: the answer text, the citations, the pages the provider reported.
• A stated reading method: how brands were identified and what counts as a recommendation.

Without these, you have a folder of chats. With them, you have something others can check and extend. See reproducibility in AI search research.

What questions can researchers ask of it?

Here are some that this kind of data can answer well.

About brands. Which brands appear for a category? How concentrated is the field: do two or three names take most recommendations, or is it spread? Are there brands that one assistant knows and another never names?

About sources. Which domains are cited, and how are they distributed across source types? Do assistants share sources or diverge? What is the relationship between pages found and pages cited? See pages found vs. pages cited.

About behaviour. How often does each assistant search? How many searches does it run? How long are its answers, and how many brands does it name? Does it hedge?

About agreement. How often do assistants name the same first brand? Where do they agree most?

About change. What moves between editions: brands entering or leaving, new sources, longer answers, shifts after a model update? See how AI recommendations change over time.

About method itself. How much do repeated askings vary? How sensitive are results to wording or location? See does location change AI answers?.

What kinds of researchers find it useful?

• Marketing and communications scholars studying how brand information flows through AI.
• Information-science researchers interested in source selection, citation behaviour and search.
• Economists and strategists looking at concentration and how recommendations shape markets.
• Journalists who need citable, checkable evidence of what assistants say.
• Students learning measurement, with a real, current, openly documented example.

None needs special access. They need the questions, the answers, the method and a licence to use them.

What are the limits of this data?

Be clear about these in any write-up.

• It is a designed sample. The answers describe the questions asked, not everything people ask.
• It is API-level. Answers collected through official interfaces differ from signed-in consumer chats with memory and personalisation.
• Location is a variable. A request sent with no location is not a worldwide view.
• Providers report different evidence. Some measures are unavailable for some assistants.
• Reading adds error. Identifying brands and roles in free text is done by a reader, human or model, with an error rate. Good datasets report accuracy against hand-labelled examples.
• Time matters. Assistants change, so a snapshot dates quickly.

None of this weakens the data if it is stated. It tells readers what a finding means.

How should you handle the reading step?

The step from raw answer to structured data is where much of the quality is decided. Brand names have spellings, products roll up to companies, and an answer may name a brand as a warning. So it is worth using one consistent reader and one rulebook for all assistants, and checking it against a labelled set. See when is a brand the same brand? and one reader for four AIs.

What would a first project look like?

A modest first project is a comparison across assistants within one category. Take the published question set and answers, count how many brands each assistant names per answer, and compare the distributions. Then pick a second variable, such as whether search ran, and see how the distribution changes. Report the sample size, the intervals, and the limits. A project like this needs no new data collection, only careful reading of the data that exist, and it teaches the central lesson of the field: the same question can produce quite different behaviour from different assistants, and the differences themselves are the finding.

What should you write down before you start?

Write your hypothesis and your plan for analysis before opening the data, even if the data are already public. Say which comparisons you will make, which measures you will use and how you will treat missing values. Then keep a record of anything you add later, labelled as exploratory. This protects you from finding patterns that are really the product of looking in many places. It also makes your work easier for others to build on, since they can see which results were planned and which were discovered along the way.

Where can you find a dataset like this?

Citewake publishes the answers behind each edition, with the questions, the model and the method, so that others can recompute its figures or ask new questions of them. The first release is Edition 0, and the editions pages say what is available, the licence and how to cite it. The methodology explains how the data were collected and read. Before using any such data, read the limits note on the page and cite the exact edition you used. For a rough first look at a single site, there is also the free Citewake checker.