robots.txt for AI Crawlers: What to Allow and Block in 2026

Updated October 18, 2026: We rewrote this guide against each vendor's crawler documentation: training, search and user-triggered agents are now separate choices (GPTBot is not required for ChatGPT search), the robots.txt examples are corrected (anthropic-ai removed, Claude-User added), and the Cloudflare, Apple and agent-browser facts now link their sources. The September 25 update had added Applebot, Cloudflare's new AI-bot defaults and the end of OpenAI's Atlas browser; see why the GEO evidence changed.

If AI search crawlers cannot read your site, AI engines are far less likely to show or cite it — robots.txt is the gate, so check yours against the current list of AI user agents. A single security plugin update or CDN setting can silently block the crawlers ChatGPT, Perplexity and Claude use for search, which can remove or reduce your pages in their answers.

This guide sorts the 2026 AI user agents by what they do, gives you two robots.txt files to copy, and shows the two places where AI bots get blocked even when robots.txt says allow.

What are the four categories of AI user agents in 2026?

Training crawlers, search crawlers, user-triggered fetchers and control tokens — each does a different job, so blocking each one has a different effect.

The table follows each vendor's own documentation, checked in September 2026: OpenAI, Anthropic, Perplexity, Google's common crawlers and user-triggered fetchers, and Apple.

CCBot does not fit neatly in one row. It builds Common Crawl's open dataset, which others use both to train models and to index search, so Common Crawl says blocking it also blocks the search-indexing uses of that data (Common Crawl, 2024).

Whether your pages appear in Google's AI Overviews and AI Mode is not controlled by Google-Extended, although Search Console Help says Google-Extended is how you limit training of the models behind those features. Google says robots.txt rules for Googlebot are the control for Search, including its AI features, and Search Console has a separate "Search generative AI" control; the nosnippet, data-nosnippet, max-snippet and noindex controls limit what is shown from your pages (Google; Search Console Help). Microsoft lists no Copilot user agent (Bing's crawler list); its Copilot controls are the noarchive and nocache meta tags, not a separate robots.txt token (Microsoft).

Verdict: allow the search crawlers if you want to appear in AI answers, and decide on training separately. OpenAI says GPTBot and OAI-SearchBot are independent: a site "can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot" (OpenAI). Allowing training crawlers may put your pages into future models; blocking them signals that your content should not be used for training, but it does not stop models from learning about you from other sites or licensed data. OpenAI says robots.txt changes take about a day to affect ChatGPT search.

What should my robots.txt actually say?

Pick one of two files. If appearing in AI answers is your priority and you have no licensing or other reason to keep your content out of model training, Example B is the simpler choice; Example A adds that opt-out but also opts you out of grounding in Gemini Apps and Vertex AI and of the search-indexing uses of Common Crawl's data. Both are valid robots.txt following Google's robots.txt rules; replace the sitemap URL with yours.

Example A: allow search engines, AI search and user-triggered agents, and opt out of AI training (and Gemini grounding).

# Example A: allow search engines, AI search and user-triggered agents;
# opt out of AI model training (and Gemini grounding).

# Everyone not named below (Googlebot, Bingbot, Applebot, etc.)
User-agent: *
Allow: /

# AI search / indexing crawlers (allowed)
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# User-triggered agents (allowed)
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

# AI training crawlers and control tokens (blocked)
# Google-Extended also covers grounding in Gemini Apps and Vertex AI
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

What Example A does, according to the vendors:

• Googlebot, Bingbot and Applebot follow the * group, so Google Search (including AI Overviews and AI Mode eligibility), Bing and Apple search are unaffected.
• ChatGPT search stays open through OAI-SearchBot while GPTBot is blocked.
• Blocking Google-Extended also opts your pages out of grounding in Gemini Apps and Vertex AI, not only Gemini training. Delete that group if Gemini app answers matter to you.
• Blocking CCBot is a trade-off, because it also blocks search-indexing uses of Common Crawl's data. Delete the CCBot group if that matters more to you than the training opt-out.
• The Allow lines for ChatGPT-User and Perplexity-User state intent but control little, since those agents may not follow robots.txt. Claude-User does honor it.

Example B: allow everything, including AI training.

# Example B: allow all crawlers and agents, including AI training.
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

One warning about groups: a crawler obeys only the group that names it and falls back to * only when no group matches (RFC 9309). Google says user-agent-specific groups and the * group "are not combined" (Google), and Bing likewise ignores the generic section once it finds its own. If your * group has Disallow lines (for example /wp-admin/), repeat them in every named group. And a stray User-agent: * group with Disallow: / does not apply to the bots you named, but it still blocks every crawler not named in the file, including Googlebot and Bingbot, so remove it unless you mean it.

Why Applebot matters: Apple began rolling out Siri AI in beta, in English, on September 14, 2026, said it will "leverage information from across the web," and said Spanish is coming "next month," that is, October 2026 (Apple Newsroom, September 14, 2026). Apple's Applebot documentation (published September 4, 2026) says Applebot's data helps answer "broad world knowledge questions in Siri and Search," so Apple names Applebot, not Googlebot, as the crawler behind those answers. If your robots.txt mentions Googlebot but not Applebot, Applebot follows your Googlebot rules. To opt out of Apple model training only, disallow Applebot-Extended; Apple points to the nosnippet tag to keep content out of those Siri answers.

Why do AI bots get blocked even when robots.txt allows them?

Two silent blockers: the CDN layer and JavaScript-only rendering.

The CDN layer. robots.txt only states a preference, so your CDN or firewall can block AI crawlers at the network level whatever your robots.txt says (Cloudflare's robots.txt documentation). Cloudflare's free Bot Fight Mode challenges traffic that matches known bot patterns and can't be skipped with WAF custom rules or Page Rules, so for exceptions Cloudflare points to Super Bot Fight Mode (Pro plan and up), which supports Skip rules and a Verified bots setting; AI crawlers are handled by the separate Search, Agent and Training settings under Security Settings (Bot Fight Mode; Super Bot Fight Mode; Block AI bots). Sucuri documents no AI-crawler rule; its aggressive bot filter setting blocks "invalid user agents that do not match real browsers," such as empty ones, so check that setting if an allowed bot never shows up in your logs (Sucuri).

Since September 15, 2026, Cloudflare offers each new domain a preset based on whether the site earns money from ads. For ad-supported sites it keeps Search allowed, sets Training to "Disallow AI Training" (training-only crawlers such as OpenAI's GPTBot and Anthropic's ClaudeBot are blocked site-wide while Googlebot, Bingbot and Applebot keep crawling for search) and blocks Agent traffic, such as chat fetch bots and browser-use agents, on pages with ads; sites without ads get Allow for all three. Existing domains keep their settings, mapped onto the new controls (Cloudflare, September 15, 2026; this updates Cloudflare's July 1, 2026 changelog, which described Training as blocked only on pages with ads). So a new ad-supported site on Cloudflare blocks GPTBot and ClaudeBot by default even if robots.txt allows them: check your AI traffic settings either way.

Cloudflare's Bot Preference Sync, announced August 21, 2026 for every plan from Free to Enterprise and on by default for new customers, writes your Search, Agent and Training choices into the robots.txt your site serves, in a managed block, between comment lines that start with "# BEGIN Cloudflare" and "# END Cloudflare", placed above your existing rules, so check the live file, not just your own copy (Cloudflare, August 21, 2026).

If you only want to opt out of AI training, use Cloudflare's "Disallow AI Training" setting, never "Block" or "Block on pages with ads." Since September 15, 2026, those two settings also stop mixed-use crawlers, including Googlebot, Bingbot and Applebot, so they cut off search too; "Disallow AI Training" blocks training-only crawlers such as those of OpenAI and Anthropic while Googlebot, Bingbot and Applebot keep crawling (Cloudflare, September 15, 2026). Cloudflare adds that, until Microsoft's support launches (Cloudflare says Microsoft is targeting early 2027), this setting does not pass a no-training preference to Bing through robots.txt.

JavaScript-only content. Some AI crawlers do not execute JavaScript: in a Vercel/MERJ study (December 17, 2024) of crawler traffic to nextjs.org and Vercel's network, OpenAI's crawlers (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic's ClaudeBot and PerplexityBot did not render JavaScript (ChatGPT's and Claude's crawlers downloaded JS files but never ran them), while Googlebot, whose infrastructure Gemini uses, and Applebot did. Anthropic's developer docs also say Claude's API web fetch tool "currently does not support websites dynamically rendered with JavaScript." If your page is an empty div until JS runs, crawlers that don't run JavaScript see nothing — allowed or not. Test with a curl request: if your main content is not in the raw HTML response, you have a rendering problem, not a permissions problem.

How do I verify AI bots are actually visiting?

Check your server logs monthly for each agent name — allowing bots in robots.txt does not prove they are visiting. In cPanel raw access logs or your hosting analytics, search for GPTBot, ClaudeBot, OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot and Applebot. Do not look for Google-Extended or Applebot-Extended: they are control tokens and never appear in logs.

User-agent names can be faked, so confirm real visits with each vendor's published IP list (linked from the OpenAI, Anthropic, Perplexity, Google and Apple crawler pages) or your CDN's verified-bot label. Behind a CDN, your origin logs may show the CDN's addresses instead of the bot's.

The crawler pages of OpenAI, Anthropic and Perplexity don't say how often their bots visit, so use your own logs as the baseline and look into any sudden drop. If a bot that used to appear regularly disappears from your logs, look first at recent changes: a CMS security plugin update, a CDN setting change or a robots.txt edit.

What about agent-mode browsers?

robots.txt does not reliably control them — and you should not try to block them by user agent. Agent-mode browsers such as Perplexity Comet and Claude in Chrome act in real browser sessions on behalf of users; Claude in Chrome works inside the user's own Chrome "using your existing logins" (Anthropic, August 26, 2026). OpenAI announced on July 9, 2026 that it was deprecating its Atlas browser, which was scheduled to stop working on August 9, 2026, and moving browser-based agent work into ChatGPT and Codex, where the ChatGPT desktop app handles deeper agentic work and the ChatGPT Chrome extension or sidebar helps while browsing in Chrome (OpenAI Help Center, as of September 2026), plus a cloud browser that runs on OpenAI's servers (TechCrunch, July 9, 2026), now documented as ChatGPT Work's Cloud browser.

None of these agents has its own robots.txt token in the vendors' crawler documentation (checked September 2026). Perplexity's crawler page lists only PerplexityBot and Perplexity-User, and no vendor documents a token or user agent for Comet or Claude in Chrome. ChatGPT's Cloud browser is the one you can identify: it signs its requests with Web Bot Auth (HTTP Message Signatures, RFC 9421), so CDNs such as Cloudflare can verify it and let you allow or block it (OpenAI Help Center). Cloudflare's preset for new ad-supported domains blocks this Agent traffic on pages with ads.

Blocking by user-agent pattern risks blocking real human visitors. These are potential customers delegating a task to an AI — your job is to make the site work for them, not to keep them out.

Allow the search crawlers, decide on training, verify at the CDN, confirm in the logs. Fifteen minutes of robots.txt work plus a monthly log check is cheap insurance in GEO: every other technique depends on the engines being able to read your site at all.

If you want a second look at what AI engines can and cannot read on your site, request a visibility assessment. Before any work begins, we reply with a fit recommendation, the scope and the fee. The form sends a request; it does not run an automated audit.