Robots.txt and AI crawlers. Out of training, in the answers: the 25 names to write and what happens to each one.
Block GPTBot to protect your text and you may think you have left ChatGPT; block everything and you leave the answers too. Here is a commented example robots.txt and the token sheet, checked with each owner.
Every AI company uses different names for different jobs: one to train models, one for search, one to open a page when a user asks. In robots.txt you choose name by name, and blocking training does not take your site out of the answers.
The names and rules come from their owners’ own pages: OpenAI, Anthropic, Google, Perplexity, Apple, Amazon, Common Crawl, DuckDuckGo, Mistral and Microsoft, read on 10 November 2026. The standard is IETF RFC 9309.
The file asks; it cannot lock the door. Fetchers working on a user’s request say they may ignore it, and in the European Union the same line also counts as a reservation of rights.
The example robots.txt and the Excel sheet can be downloaded below, with no email required.
Written by: Mattia Esposito, AI and marketing consultant, founder of Itria AI. From Puglia, based in Turin since 2023, he works remotely in Italy and abroad. LinkedIn profile.
The example robots.txt and the AI crawler token sheet, to download
A ready-made robots.txt with a comment above every name, and an Excel sheet of 27 tokens where you pick Allow or Block and the line to copy writes itself. The example keeps the site out of model training and in search and AI answers. The second sheet has ten checks to run once the file is live.

| File | Contents | Link |
|---|---|---|
| Textto copy | The example robots.txt, commented line by line. Only the sitemap address needs changing. | |
| Excelto fill in, in Italian | My choice (27 tokens with purpose, stated rule, effect of blocking, official page and IP file), Checks, Example. | |
| PDFto read, in Italian | The example and the token table, to keep beside you while you write the file. |
User-agent: *
Allow: /
# Model training: out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: MistralAI-Training
User-agent: Amazonbot
Disallow: /
# Search and AI answers: in
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Several names above a single Disallow line form one group: RFC 9309 allows it, and each crawler follows the group that names it. The Allow lines for search matter when a blanket block sits higher up; otherwise they make the choice readable to the next person who opens the file.
On 7 August a friend sent me what ChatGPT said about me: “I can’t find a Mattia Esposito linked to itria.io as founder”. My site’s robots.txt said only User-agent: * and Allow: /, plus the sitemap address. No door was shut. I put it like this on 21 September: “I had written the llms.txt, the schema, all the nice stuff. As far as AI was concerned, I didn’t exist.” The full post.
Then I rewrote the file and named the crawlers one by one, in three groups: search engines, assistants that read when someone asks, training. I let them all in, training included, and today the file says so in its first line: “Itria’s content is public”. For a site that lives on being cited, it’s the right call. For a company with catalogues, price lists or copy to protect it may not be, and the example above does the opposite.
An open file did not make me appear on its own. The pages did: in 28 days the site’s impressions went from 24 to 1,222, up 4,992%, and between 11 October and 7 November 2026 came 268 citations in Copilot and 3,550 impressions in AI Overviews and AI Mode. Robots.txt decides who may come in. What they find inside decides who gets cited, and the guide on how to show up in ChatGPT covers it.
List of AI crawler user agents in 2026
There are 25 names to write after “User-agent:”, from ten companies, in three families: training, search and user requests. The tables gather them from the official pages, read on 10 November 2026. For a short definition of a crawler, see the glossary entry on AI crawlers.
| Training | Whose it is and what it does | Respects robots.txt |
|---|---|---|
| GPTBot | OpenAI. Collects pages that may be used to train its models. | Yes |
| ClaudeBot | Anthropic. Collects pages for training. | Yes |
| Google-Extended | Google. A control token with no crawler of its own: it decides whether your text is used for Gemini and its grounding. | Yes |
| Applebot-Extended | Apple. An opt-out from training only: it does not crawl by itself. | Yes |
| CCBot | Common Crawl, a non-profit. An open archive of the web that many use to train models. | Yes |
| MistralAI-Training | Mistral. Model training. | Yes |
| Amazonbot | Amazon. Improves products and services and may be used to train Amazon’s models. | Yes, file cached for up to 30 days |
| Search | Whose it is and what it does | Respects robots.txt |
|---|---|---|
| OAI-SearchBot | OpenAI. Finds pages for ChatGPT’s search answers. | Yes |
| Claude-SearchBot | Anthropic. Indexes pages for Claude’s search. | Yes |
| PerplexityBot | Perplexity. Builds the index behind its answers; stated as not used for foundation models. | Yes |
| Googlebot | Google. Search, including AI Overviews and AI Mode. | Yes |
| bingbot | Microsoft. Bing search, which Copilot draws on for the web. | Yes |
| Applebot | Apple. Spotlight, Siri and Safari; it may also feed AI answers. | Yes |
| Amzn-SearchBot | Amazon. Answers in Alexa and Amazon search, not training. | Yes |
| DuckAssistBot | DuckDuckGo. Reads in real time for AI answers; stated as not used to train models. | Yes, after 72 hours |
| MistralAI-Index | Mistral. Its search index; stated as not used for training. | Yes |
| User request | Whose it is and what it does | Respects robots.txt |
|---|---|---|
| ChatGPT-User | OpenAI. Opens a page when a ChatGPT user asks. | Not always |
| Claude-User | Anthropic. Opens pages for Claude users’ questions. | Yes |
| Perplexity-User | Perplexity. Opens a page at a user’s request. | Generally no |
| Google-Agent | Google. Agents hosted by Google that browse and act for the user. | Generally no |
| Google-GeminiNotebook | Google. Reads sources given to Gemini Notebook; until August 2026 it was called Google-NotebookLM. | Generally no |
| Amzn-User | Amazon. Opens pages for users’ requests. | Not always |
| MistralAI-User | Mistral. Governs which sites users’ requests can reach. | Yes |
Two names with a narrow job are left out: OAI-AdsBot, which OpenAI uses to check pages submitted as ads in ChatGPT, and Google-CloudVertexBot, which crawls only when the site owner asks for it to build a Vertex AI agent.
Brave has no name of its own: it does not crawl what Googlebot is not allowed to crawl. Meta publishes a page on its crawlers and ByteDance none for Bytespider: I could not check them word for word, so they are missing from the tables. Meta is in the sheet, marked to be rechecked.
Which AI bots should I block in robots.txt, and which not?
If you want to stay in assistants’ answers but keep your text out of training data, block the first table and let in the second. The example makes this choice. If you want the opposite, to be read and used as widely as possible, write no block at all: a robots.txt with no rule for a name already lets it in.
Two names are never touched for AI reasons: Googlebot and bingbot. Blocking them takes the site out of Google and Bing, and with them out of AI Overviews, AI Mode and Copilot. The effect goes further. Brave follows the rules given to Googlebot; Apple does the same when Applebot is not named. One wrong line on Googlebot empties three search engines.
You have little say over the third table. Those fetchers open a page because a user asked, and nearly all of them warn that they may not respect the file. Blocking them remains a signal. To leave a single assistant’s answers, its search crawler is what counts: the guides on Perplexity, Claude and Copilot go into each one.
Does blocking GPTBot remove me from ChatGPT?
No: GPTBot is for training, and ChatGPT search uses OAI-SearchBot. OpenAI keeps the settings separate, so a site can say no to the first and yes to the second. The reverse is the case to avoid: a site that blocks OAI-SearchBot will not appear in ChatGPT’s search answers, except as a navigational link.
Then there is ChatGPT-User, which opens a page when a user asks. OpenAI puts it plainly: “Because these actions are initiated by a user, robots.txt rules may not apply.” OpenAI says a change to the file takes about 24 hours to reach its systems, and publishes the IP addresses of each crawler in separate files.
Anthropic works the same way: blocking ClaudeBot takes your text out of training, not your site out of Claude, whose search uses Claude-SearchBot.
What does Google-Extended block, and what not?
Google-Extended decides whether your site’s text is used to train Gemini and to ground its answers in the Gemini apps and Vertex AI; it does not touch Google Search. Google says so on its crawlers page: it has no effect on inclusion in Search and is not a ranking signal. Google-Extended is a control token with no crawler of its own: Googlebot does the crawling.
So Google-Extended does not take your site out of AI Overviews or AI Mode, which are part of Search. Those follow the Search meta rules, covered in the guide on how to opt your site out of AI Overviews. On the same page Google announces that from early 2027 it will also support a shared training-preference signal from the IETF’s AIPREF working group.
Do AI crawlers respect robots.txt?
The training and search crawlers of the companies in the tables say they respect it; many user-request fetchers say they do not, or not always. The standard itself promises no more. RFC 9309, published by the IETF in September 2022 to formalise the method dating from 1994, says it in one line: “These rules are not a form of access authorization.”
The RFC also sets rules worth knowing when you write the file. Names are matched without regard to case. Between two rules the more specific wins, and on a tie Allow wins. If the file returns a 4xx error, the crawler may go anywhere; with a 5xx it must treat everything as disallowed. Google caches the file for up to 24 hours, ignores anything beyond 500 KiB and does not read crawl-delay.
Anyone can put GPTBot in a request header. To check a bot is who it says, compare its address with the published ones: OpenAI, Anthropic, Perplexity, Amazon, Common Crawl and DuckDuckGo publish IP files, and Common Crawl warns explicitly about fake CCBots.
A copy-and-paste trap: on Perplexity’s page, in the table heading, Perplexity-User appears with an unusual hyphen, and pasted like that into the file it does not match.
How to opt out of AI answers without leaving Google
With page meta rules, not robots.txt: nosnippet for Google and Apple, noarchive or nocache for Bing, noindex for Anthropic’s search partners. Robots.txt decides whether a crawler reads. The AI answers of an engine that has already read the page for search are governed page by page.
Google applies nosnippet, data-nosnippet, max-snippet and noindex to the AI features in Search. Apple, on the page updated on 4 September 2026, writes that nosnippet keeps a page out of generated answers in Siri and search, while Applebot-Extended only opts out of training.
Bing, in an announcement of September 2023, when the assistant was still called Bing Chat, explained that with noarchive content stays in the results but leaves answers and training, and with nocache only the URL, title and snippet appear.
Lastly, Anthropic writes that noindex reaches its search partners: a page with noindex does not enter Claude’s answers when it searches. For Copilot the route goes through the Bing index, covered in the guide to Bing Webmaster Tools and IndexNow.
Cloudflare’s managed robots.txt for AI bots: the file you didn’t write
If the site sits behind Cloudflare, the robots.txt it serves may not be the one in your CMS: Cloudflare’s managed robots.txt puts a block of its own in front. That block holds the Content Signals (search, ai-input, ai-train), a reservation of rights citing the EU copyright directive and Disallow lines for eight training crawlers.
Cloudflare sells CDN and security services, so it is a source with an interest. It splits AI traffic into three behaviours: Search, Agent and Training. In its documentation updated on 2 November 2026, all three start on Allow for new domains; for domains set up earlier, check the dashboard. CDN rules act before the file.
On my own site a different Cloudflare setting was removing the email address from the HTML: the case is in the invisible email lab. The real check is a single one: open yoursite.com/robots.txt in a browser and read what is there.
Is robots.txt legally binding for AI training? What EU law says
In the European Union, a robots.txt that excludes training crawlers is also a reservation of rights with legal weight. Article 4(3) of Directive (EU) 2019/790 allows text and data mining unless the rightholder has reserved it, by machine-readable means for content made public online.
The AI Act, Regulation (EU) 2024/1689, completes the picture: Article 53(1)(c) requires providers of general-purpose AI models to identify and respect that reservation. Elsewhere the law differs, while the tokens stay the same: the names and the owners’ pages are identical in Italy and in the English-speaking world.
| Where | What applies to text and data mining | Source |
|---|---|---|
| European Union | The exception does not apply if the rightholder has reserved it by machine-readable means; providers of general-purpose models must respect the reservation. | Dir. 2019/790, art. 4(3); Reg. 2024/1689, art. 53(1)(c) |
| Italy | The EU rule. The national text sits in the Italian copyright law, to be read on Normattiva. | Dir. 2019/790 |
| United Kingdom | The exception covers only research for non-commercial purposes. | CDPA 1988, s. 29A |
| United States | No statute read for this page; the tokens and the owners’ pages are the same. | Owners’ pages |
On the technical side, the IETF is drafting a shared vocabulary for AI usage preferences, with labels such as train-ai and search, plus a way to write them in robots.txt and in HTTP headers. They are still two working drafts. Anthropic has been applying them gradually since 30 October 2026, with rules that name ClaudeBot taking precedence; Google has announced early 2027.
From one page to five. On the morning of 25 September 2026, Claude’s web search had a single page of itria.io; by the evening, after I asked Brave to recrawl seven, it had five. Robots.txt was already open: the index was what was missing. Before and after in Brave and Claude.
The missing email. Cloudflare’s obfuscation removed the address from the HTML and put it back with a script: to an assistant the contact page had no contact details. The same flaw turned up on 14 of 121 producers’ sites, in the invisible email lab.
The result. Between 11 October and 7 November 2026, Bing’s AI Performance report counted 268 citations of itria.io pages in Copilot, and Search Console 3,550 impressions in AI Overviews and AI Mode. How the work is done on a real company is on the page about presence in search engines and AI assistants.
Questions and answers
Which AI bots should I block in robots.txt?
It depends on what you want to keep out. If you want to stay in assistants’ answers but out of training data, block the training tokens: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, MistralAI-Training and, if you wish, Amazonbot.
Let in the search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, and never touch Googlebot or bingbot, which put your site on Google and Bing.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot collects pages for training OpenAI’s models. ChatGPT search uses OAI-SearchBot, which has a separate setting: a site can block the first and let in the second. OpenAI says a change takes about 24 hours to reach its systems.
Does Google-Extended block AI Overviews?
No. Google writes that Google-Extended has no effect on inclusion in Google Search and is not a ranking signal: it governs whether your text is used for Gemini. AI Overviews and AI Mode are part of Search and follow Googlebot and the nosnippet, max-snippet and noindex rules.
Do AI crawlers respect robots.txt?
The training and search crawlers of the large companies say they do. Fetchers that open a page because a user asked do not, or not always: OpenAI says so for ChatGPT-User, Perplexity for Perplexity-User, Google for its user-triggered fetchers and Amazon for Amzn-User.
The standard, RFC 9309, points out that the file is a request and not access authorisation.
Is robots.txt legally binding against AI training?
In the European Union it counts as a reservation of rights. Article 4 of Directive (EU) 2019/790 removes the text and data mining exception when the rightholder has reserved use by machine-readable means, and the AI Act requires providers of general-purpose AI models to respect that reservation. This is general information, not legal advice.
Notes on sources
- IETF, RFC 9309, Robots Exclusion Protocol, September 2022: source of the quotation, case-insensitive matching, the most specific rule, 4xx and 5xx. Google, how it interprets the specification, updated 23 October 2026, CC BY 4.0: caching, 500 KiB, crawl-delay.
- OpenAI, Overview of OpenAI Crawlers: GPTBot, OAI-SearchBot, ChatGPT-User (source of the quotation), OAI-AdsBot, the 24 hours and the IP files.
- Anthropic, crawler page and noindex page, 3 September 2026.
- Google, common crawlers, updated 30 October 2026, and user-triggered fetchers, updated 19 August 2026, CC BY 4.0.
- Perplexity, Perplexity Crawlers; Apple, About Applebot, 4 September 2026; Amazon, Amazonbot; Common Crawl, CCBot; DuckDuckGo, DuckAssistBot; Mistral, robots; Brave, Brave Search crawler.
- Microsoft Bing, announcement on noarchive and nocache, September 2023: it refers to Bing Chat, Copilot’s name at the time.
- Cloudflare, AI crawlers and managed robots.txt, both updated 2 November 2026: documentation from a CDN and security vendor.
- IETF, drafts draft-ietf-aipref-vocab-08, 14 September 2026, and draft-ietf-aipref-attach-05, 19 August 2026: working documents, not standards.
- Directive (EU) 2019/790, art. 4(3); Regulation (EU) 2024/1689, art. 53(1)(c); Copyright, Designs and Patents Act 1988, s. 29A.
- The itria.io robots.txt can be read at itria.io/robots.txt. The figures come from Search Console (generative AI report) and Bing Webmaster Tools, window 11 October to 7 November 2026. All pages were read on 10 November 2026.
Robots.txt decides who comes in. What they find inside decides who gets cited. Positioning is built starting from the market.
Mattia Esposito, who wrote this guide, also works with Itria on positioning: in-depth market analysis to find where competitors are exposed and take that ground. He did it on his own site first. In August, asked who had founded itria.io, ChatGPT couldn’t find him. From 11 October to 7 November 2026 itria.io pages appeared 13,803 times in Google and Bing results and 3,818 times in AI answers: Google’s AI Overviews and AI Mode, and Microsoft Copilot.