Robots.txt for AI Crawlers: Let ChatGPT, Claude, and Perplexity In

Every major vendor now runs three separate crawlers

The single most expensive robots.txt mistake in 2026 is treating each AI company as one bot. OpenAI and Anthropic both run a three-way split, and the three do genuinely different jobs with genuinely different consequences when blocked.

*Perplexity does not publish a clean three-way split; PerplexityBot covers indexing and Perplexity-User covers user-initiated fetches.

The default failure mode is a blanket User-agent: GPTBot / Disallow: / copied from a 2023 opt-out template. That blocks training — which produces no citations anyway — while leaving OAI-SearchBot completely untouched. The reverse configuration is worse: blocking OAI-SearchBot while allowing GPTBot hands over your content for training and removes you from ChatGPT Search, which is the exact opposite of what almost every business wants.

The other tokens worth a deliberate decision

Two deprecations to clear out of old files: Claude-Web and anthropic-ai are legacy tokens no longer in use, and Google renamed Google-NotebookLM to Gemini Notebook, with the old user-agent deprecated and supported only through August 2026. Rules referencing the retired names do nothing.

Training, search, or user-fetch: which to allow

Decide these three independently. They are independent controls, and collapsing them into one policy is where most sites lose visibility they wanted to keep.

Search crawlers: allow, in almost every case

If you want to be cited when someone asks an assistant a question in your domain, the search crawler must be able to fetch the page. There is no alternative mechanism. Blocking OAI-SearchBot or Claude-SearchBot is functionally identical to blocking Googlebot in 2010.

User-fetch agents: allow, essentially always

These fire when a specific person pastes your URL and asks the assistant to read it. Blocking them breaks a request the user explicitly made about your page. Worth knowing: OpenAI's documentation states that because ChatGPT-User fetches are user-initiated, robots.txt may not apply to them — which puts that agent on the same footing as Google's user-triggered fetchers rather than being independently controllable. Anthropic, by contrast, states that all three of its bots honour robots.txt, including the non-standard Crawl-delay directive.

Training crawlers: a real business decision

This is the only one where blocking is defensible. Allowing training gets your brand, terminology, and positioning into the model's parametric knowledge — which is what makes an assistant mention you unprompted, without a live search. Blocking it protects content you consider a licensable asset. Publishers with archives worth licensing block. Businesses whose content is marketing rather than product almost always benefit more from being learned than from being withheld.

Note the asymmetry that makes the decision less dramatic than it feels: blocking training only affects future crawls. Anything already ingested stays in models already trained.

A robots.txt configuration you can copy

This is the allow-search, allow-fetch, allow-training posture that suits most commercial sites. Read the next section before deploying it, because the grouping behaviour is not what most people assume.

# --- AI search and retrieval: allowed ---
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search
Disallow: /*?*sort=

# --- AI training: allowed, adjust to your licensing stance ---
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Allow: /
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/

# --- Everything else ---
User-agent: *
Allow: /
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search
Disallow: /*?*sort=

Sitemap: https://example.com/sitemap.xml

To block training instead while keeping AI search visibility, change the second group's Allow: / to Disallow: / and leave the first group alone. That is the whole edit — and it is the configuration most publishers actually intend when they say they want to opt out of AI.

Why User-agent: * does not do what you think

This is the gotcha that turns a careful robots.txt into an accidental data leak, and it is barely mentioned in most AI crawler guides.

Robots.txt groups are not inherited. Under the Robots Exclusion Protocol standard, a crawler selects the single most specific matching user-agent group and obeys only that group. It does not merge the group with the * group. So this configuration:

User-agent: *
Disallow: /admin/
Disallow: /internal/
Disallow: /customer-data/

User-agent: GPTBot
Allow: /

...does not mean "GPTBot may crawl everything except the three protected paths." It means GPTBot may crawl everything, including /admin/, /internal/, and /customer-data/, because those Disallow lines live in a group GPTBot is no longer reading.

Every path you protect in the * group must be repeated in every named group. That's the reason the template above looks redundant — the redundancy is load-bearing. Any time you add a new named AI crawler group, re-copy the protected paths into it as part of the same edit, and diff the groups afterwards.

A related detail: within a group, the most specific rule wins by path length, not by order. Allow: /blog/ beats Disallow: / for URLs under /blog/ regardless of which line comes first. Ordering is a readability choice, not a precedence mechanism.

Verifying a crawler is real, not a spoofer

User-agent strings are a claim, not an identity. Anyone can send GPTBot in a header, and scrapers do exactly that to inherit whatever allowances you granted the real thing. If your allow rules are permissive, verification is not optional.

Each vendor publishes an IP list to check against:

• Anthropic publishes a combined list at claude.com/crawling/bots.json — one set of IPv4 prefixes with no per-bot breakdown, so you cannot distinguish ClaudeBot from Claude-User by IP alone.
• OpenAI publishes separate JSON ranges per agent for GPTBot, OAI-SearchBot, and ChatGPT-User in its developer documentation.
• Google publishes Googlebot ranges and supports reverse-DNS verification against googlebot.com and google.com.

The practical rule: enforce verification at the WAF, express preferences in robots.txt. Robots.txt is a request that well-behaved crawlers honour; it has never been an access control. If a path genuinely must not be read, it needs authentication, not a Disallow line — which, being publicly readable, actually advertises the path's existence.

One version detail that breaks log filters: OAI-SearchBot's user-agent now leads with a full Chrome desktop string rather than the older short form, and GPTBot is at version 1.4. Any log analysis matching on an exact legacy string will silently report zero hits from a crawler that is visiting normally.

Testing and monitoring the configuration

A robots.txt change is worthless if something upstream never lets the crawler read the file. Verify in this order:

1. Confirm the file is reachable by a non-browser client. curl -sS https://example.com/robots.txt should return plain text with a 200. A 202, 403, or an HTML challenge page means a WAF or hosting anti-bot layer is intercepting — that has to be fixed first, and it is the most common reason a correct robots.txt changes nothing. The diagnostic path is covered in the companion piece on why AI doesn't cite your website.
2. Fetch a real page with each crawler user-agent and confirm a 200 with full HTML, not a shell or a challenge.
3. Grep your access logs by user-agent weekly. Absence of a crawler that should be allowed is the earliest signal that something upstream regressed.
4. Re-audit quarterly. The token landscape moved three times in the last two years: OAI-SearchBot arrived in 2024, Claude-SearchBot split off in 2025, and vendor documentation on user-initiated fetches changed again in 2026. A file written once and forgotten is a file that is now wrong.
5. Re-check after every CMS or plugin upgrade. Security plugins and managed platforms routinely reset robots.txt to a default that reinstates the old block-everything posture.

Once access is settled, the next layer is stating what crawlers may do with content they are allowed to fetch — a separate mechanism from Allow and Disallow, covered in the piece on the Content-Signal directive.

Managing that cycle is part of what our SEO, AEO and GEO service handles — AI crawler rules, llms.txt, and structured data implemented and kept current on your platform, rather than delivered as a recommendations document. If you just want to know whether your current file is helping or hurting, a free AI-visibility consultation will tell you.