AI crawlers and robots.txt: who can read your site

Your robots.txt file tells crawlers which parts of your site they may fetch, and AI companies publish named crawlers you can allow or block individually. If you want to appear in AI answers that use live web search, make sure you are not blocking the crawlers that fetch pages for search and for user requests. Training crawlers are a separate choice.

This is one of the few technical levers that is fully in your hands, and one of the easiest to get wrong without noticing.

What does robots.txt actually do?

It is a plain text file at the root of your domain (yourdomain.com/robots.txt). It lists rules by user agent, the name a crawler announces, and says which paths that crawler may or may not fetch.

Two limits matter:

• It is a request, not a lock. Responsible crawlers honour it. It does not stop anyone determined, and it is not a security measure.
• It controls fetching, not mentions. Blocking a crawler stops it reading your pages. It does not stop anyone else from writing about you, and an assistant can still describe you from other sources.

Which AI crawlers exist?

The big AI companies document their crawlers publicly. The names and purposes below come from the publishers' own documentation. They can change, so always check the current pages.

Google-Extended works as a control token and is separate from Googlebot. Google states that it does not affect inclusion in Google Search. Treat the details as something to verify against Google's documentation.

The useful pattern is that most providers separate three jobs: gathering data to train models, building a search index, and fetching a page because a user asked about it. Each can be set separately.

Why does the distinction between those jobs matter?

Because your goals differ for each.

• Search and user-triggered crawlers are how an assistant reads your page to answer a question now. Blocking them means that assistant cannot read your site when it searches, so it relies on what others say about you.
• Training crawlers collect content that may be used to build future models. Some businesses are content to allow this; some, for example publishers selling content, are not.

If your aim is to be recommended in live-search answers, the search and user-triggered crawlers are the ones to leave open. The split between memory and live search is described in training data or live search.

How do you check your own file?

It takes five minutes.

1. Open yourdomain.com/robots.txt in a browser.
2. Look for User-agent: lines naming the crawlers above, and for User-agent: * (all crawlers).
3. Read the Disallow: rules under each group. Disallow: / blocks the whole site for that group.
4. Check whether a blanket rule is catching more than you meant. A broad Disallow under User-agent: * applies to every crawler without its own group.
5. Compare against each company's current documentation, since names and behaviour can change.

An example that allows search and user fetches while blocking the training crawler for OpenAI looks like this:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

That is an illustration of the syntax. Whether it suits you is your decision.

What else can block AI crawlers?

robots.txt is not the only gate. Check these too.

• Security and bot-protection layers. A firewall, CDN or bot manager may challenge or block automated visitors even when robots.txt allows them. Your server logs or your provider's dashboard will show what is being blocked.
• Login walls and paywalls. A crawler cannot read what it cannot reach. Key pages that answer buyer questions, such as pricing and product pages, should be public.
• Content that needs scripts to appear. A page whose text only appears after heavy client-side rendering may be harder for some fetchers to read. Make sure the core text is in the HTML.
• Meta tags and headers. Directives such as noindex can keep pages out of indexes.

If robots.txt looks fine but an assistant still never seems to read your site, these are the next places to look.

What should most businesses do?

For a business that wants to be found and recommended, a sensible default is:

1. Allow the search and user-triggered crawlers of the assistants your buyers use.
2. Make a deliberate choice about training crawlers, and write down the reason.
3. Keep pricing, product and answer pages public and quick to load.
4. Re-check after any change to your CDN, firewall or CMS, because those are common causes of accidental blocks.

Then test the result. Ask your buyers' questions and see whether the assistants cite your own pages for factual queries. If they never do, return to the checklist. The routine for checking answers is in how to check if ChatGPT recommends your business.

A related question is whether a file such as llms.txt helps. See llms.txt explained. To see how a site appears to assistants today, use the Citewake checker. On how Citewake itself handles pages it cannot fetch, including blocked pages being recorded as unusable, see the methodology.