How to test AI answers properly: signed out, private, repeated

To test AI answers properly, remove everything that could tilt them, then ask the same fixed question several times and save every answer. That means a signed-out or private session with memory and custom instructions off, a new chat for each run, identical wording, one consistent search setting, and a written record of the full text and sources. Without these controls you are measuring your own history, not what a buyer would see.

Why does a clean session matter?

Assistants can adapt to you. A signed-in account may hold memory of past conversations, saved preferences and custom instructions. Those can change which brands an assistant names, how it phrases advice and what it assumes about you.

You are not the buyer. If you have spent months asking an assistant about your own company, it may be primed to mention it. A test run in that account would overstate your visibility, and you would not know by how much.

So the first rule is to make the session as neutral as the product allows.

What does "clean" mean in practice?

• Signed out, where the product allows it. If you must sign in, use a dedicated test account with no history.
• Memory off. Switch off any feature that remembers earlier chats.
• Custom instructions empty. Remove any standing instructions.
• A private or fresh browser window. This removes cookies and local history, which are minor influences but easy to control.
• No leading context. Do not tell the assistant who you are or what you sell before asking.

The point is not perfection. It is to avoid the large, avoidable distortions.

How do you keep the question fixed?

Write the question once and paste it every time. Small changes in wording change the answer, so "best project tool for agencies" and "what project tool should an agency use" are different tests.

Keep a fixed list, and give each question a number. When you re-test next month, paste the same text. A change in results then means something, instead of reflecting a slight rewording.

Never put your own brand name in a discovery question. How to check if ChatGPT recommends your business explains why.

Why start a new chat each time?

Within a conversation, each answer is shaped by what came before. If you ask a follow-up, the assistant is also responding to its earlier reply. For a clean sample, open a new chat for every run, and ask the one question.

The same applies when you move between assistants: treat each as its own fresh start. Do not paste in a previous answer or a hint.

Why repeat?

Because answers vary. The causes are covered in why AI answers change every time you ask, but the consequence is simple: one run is a sample of one. Repeat each question several times, in separate chats, and count how often you appear. How many times should you ask an AI goes into how many.

Should search be on?

It depends on what you want to learn. Many buyers use assistants with web search active, and recommendation questions often trigger it. If your aim is to see what customers are likely to see, test with search on. Whatever you choose, keep it constant across runs and note it. Changing it midway makes results incomparable.

What should you save?

Everything that produced the result:

1. the question text and its number;
2. the assistant and product version, as shown to you;
3. the date and the search setting;
4. the full answer, copied as text, not summarised;
5. the sources and links shown;
6. your reading: which brands, in what role.

Save the answer before you interpret it. Interpretation can be redone; a lost answer cannot be recovered, because the assistant will say something different next time. That principle runs through Citewake's methodology, which keeps every answer so any figure can be traced back to the words behind it.

Do you need to do all of this every time?

Not every time at the same depth. The controls matter most when you are producing numbers you plan to compare or share. For a quick curiosity check, a signed-out session and a couple of repeats are fine, as long as you do not treat the result as proof.

For anything you will report to a client, a boss or a board, do it properly: written question list, clean sessions, repeats, saved answers and a recorded setup. The extra time repays itself the first time someone asks how you got the number.

It also helps to write the procedure down once. A short checklist listing the session type, search setting, wording rule and what to save means that anyone on your team can repeat the test and produce comparable results. Consistency across people matters as much as consistency across time.

What are the limits of this kind of testing?

Be honest about them. A clean test shows what a neutral first-time user might see. It does not show what a particular signed-in person with their own history will see, and it cannot capture every location or device. Treat results as a fair, repeatable sample, not a complete picture.

To get started quickly, the Citewake checker offers a first look, and find the questions your customers ask AI helps you build the question list.

What is the short version?

Control what you can: signed out or a clean account, memory off, a new chat per run, fixed wording and one search setting. Then repeat, save everything and report rates. A careful test takes only slightly longer than a careless one, and it produces numbers you can defend when someone challenges them.