Call · 30 min
MarketingMattia Esposito11 November 202614 min read

AI training and robots.txt. Out of the models, still in Google and ChatGPT: the shared signal of 30 October 2026 and the lines that already work today.

On 30 October Google, OpenAI and Microsoft announced a shared line against AI training, but none of the three applies it yet. Here is a ready-made robots.txt, with today’s lines and the one for 2027.

Updated 11 November 2026

In brief

To keep your content out of training without vanishing from AI answers, you block the training crawlers, one per company, and let the search crawlers in. This works today. The shared signal announced on 30 October 2026 sits alongside these lines and does not replace them.

The signal is written as a robots.txt group, User-Agent: __usage_ai-training, and comes from an IETF draft. Anthropic began recognising it on 30 October; Microsoft will from 1 February 2027, Google in 2027, OpenAI in the coming months.

With Google there is a price: refusing Gemini training also removes linked citations in the Gemini apps. AI Overviews and AI Mode do not change.

The ready-made robots.txt is below, as text and as PDF, with no email required.

Written by: Mattia Esposito, AI and marketing consultant, founder of Itria AI. From Puglia, based in Turin since 2023, he works remotely in Italy and abroad. LinkedIn profile.

A robots.txt to block AI training bots, ready to download

A robots.txt for a small business that wants to stay in Google, in Bing and in ChatGPT and Claude answers, while opting out of model training. It has the lines that already work and the group for the shared signal, ready for when Google, OpenAI and Microsoft start reading it. Every line has a comment above it.

The ready-made robots.txt for a small business: out of AI model training, still in search, with the group for the shared __usage_ai-training signal and the timeline of who reads it
FileContentsLink
Textto copy

The robots.txt with a comment on every line, in Italian. Change only the sitemap address.

addestramento-ai-robots-txt-pmi.txt

PDFto read, in Italian

The file, the table of names with what you lose by blocking each, and the timeline of the shared signal. In Italian.

addestramento-ai-robots-txt-pmi.pdf

The lines that set it apart from an ordinary robots.txt

# Training: out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

# Shared signal of 30 October 2026
User-Agent: __usage_ai-training
Disallow: /

Name in the fileIf you block itIn force from
GPTBot

OpenAI does not use your content for training; the site stays in ChatGPT search.

Already today

OAI-SearchBot

The site drops out of ChatGPT search answers. Let it in.

About 24 hours after the change

ClaudeBot

Future content is excluded from Anthropic’s training.

Already today

Claude-SearchBot

Less visibility in Claude’s answers. Let it in.

Already today

Google-Extended

Out of Gemini training and out of citations in the Gemini apps; Search stays.

Already today

Applebot-Extended

Out of training for Apple’s models; Applebot keeps reading for Siri and Spotlight.

Already today

CCBot

The site leaves Common Crawl’s open archive.

Already today

__usage_ai-training

No to training for anyone who recognises the signal; nothing lost in search.

Anthropic from 30 October 2026; Microsoft from 1 February 2027; Google in 2027; OpenAI with no date

The full list, with Amazon, Mistral, Perplexity and DuckDuckGo, is in the guide to robots.txt and AI crawlers. This page goes deep on the new signal: who reads it, from when, and what saying no to each company costs. What a crawler is, in two lines, is in the glossary.

On 7 September I published an analysis of the websites of 142 Italian companies that also sell abroad. On the same sites I counted the file that introduces a company to AI assistants: 28 out of 143 had one, and in 24 a plugin had written it. In the post I put it like this: “An assistant doesn’t browse a site. It reads it, and reports back what it finds written there.” The full post.

Robots.txt runs the same risk. The CMS, the SEO plugin or the CDN writes it, and the owner may never have opened it. The 30 October signal is one more line in that file: if nobody writes it, when Google and Microsoft start reading it they will find nothing to respect.

I built this guide only on pages published by the companies that read the file, plus the IETF: Google, OpenAI, Microsoft, Anthropic, Apple. On 21 September I put it this way: “AI cites the body that issues the rule or the firm doing the work. Never the person explaining it.” The post.

The same post held the reason for a downloadable file: a Word template had brought in 12 of the site’s 34 clicks, and an assistant cannot summarise a file. My own robots.txt, incidentally, makes the opposite choice and lets everyone in; why is explained in the guide to AI crawlers.

What is the shared AI training signal of 30 October?

The signal is a robots.txt group that carries a label instead of a crawler’s name, __usage_ai-training, and tells anyone who recognises it not to use the content to train generative models. Google, OpenAI and Microsoft announced it on the same day, 30 October 2026, each on its own blog, and describe it as a voluntary signal.

The syntax comes from an IETF draft, written by an author at Anthropic; the IETF is the body that writes the internet’s technical standards. It adds no new directives: a program that does not know the label ignores it as it would an unknown bot. On the Datatracker the draft is listed as individual, not yet adopted by the AIPREF working group, and it expires on 3 May 2027 unless renewed.

Four rules in the draft change how you write the file. A Disallow under User-agent: * does not count as a no to training. If the label is missing, no preference is expressed: silence is not a refusal. The label is not an access rule, so keeping a crawler out still needs its own group. Finally, it works by path as well, so you can refuse training on a single folder.

The draft leaves the names of individual crawlers valid, and they may be read as more specific rules. It covers only generative models: training a model that just classifies or ranks falls outside it. A licence agreement signed with a provider overrides what the file says.

Does the signal work yet? Who respects it, and from when

Not yet, except for Anthropic: Google, OpenAI and Microsoft have announced it but do not apply it yet. Each has given a different date, and until then opting out of training still runs through the groups for individual crawlers. Google, in its post: “We aim to support this new control along with other AI providers in 2027”. Its crawler documentation, updated the same day, says early 2027.

WhoWhat it has said about the shared signalWhat to write today
Anthropic

Recognises it from 30 October 2026, extending it gradually to all services. Rules naming ClaudeBot take precedence.

ClaudeBot

Microsoft

Will honour it from 1 February 2027, as Bingbot recrawls pages.

No name for training: today NoArchive or NoCache

Google

Will support it in 2027; with no group for Google-Extended it will read it as a block on Google-Extended.

Google-Extended

OpenAI

Will implement it in the coming months, with details in a later update; GPTBot and OAI-SearchBot remain.

GPTBot

Apple

No statement found on the shared signal.

Applebot-Extended

Common Crawl

No statement found on the shared signal.

CCBot

The timeline of the shared AI training signal: announced on 30 October 2026, Bing from 1 February 2027, Google in 2027, OpenAI in the coming months 30 Oct 2026announcement; Anthropicstarts reading it 15 Dec 2026Bing: temporarypreference 1 Feb 2027Microsofthonours it 1 Mar 2027Bing: NoArchive andNoCache no longer enough 3 May 2027IETF draft -00expires 30 Sep 2027Bing: temporarypreferences end OpenAI: “in the coming months”, no date Google: in 2027, “early 2027” in its documentation, no day
The timeline of the shared signal according to the companies that announced it. The dates in the dashed boxes have not been set: OpenAI and Google have not given a day.

Meta and Perplexity are not among the companies that announced the signal, and no source read says they will follow it. Their names, with what they declare today, are in the guide to AI crawlers. The llms.txt file has nothing to do with it either: it introduces the site to assistants and expresses no restrictions, as the guide to llms.txt explains.

How to stop AI training without disappearing from Google

On Google, Gemini training is controlled with Google-Extended, and blocking it does not remove the site from results, AI Overviews or AI Mode. Google states that Google-Extended does not affect inclusion in Search and is not a ranking signal. The AI features in Search follow Googlebot and the nosnippet, max-snippet and noindex rules.

The point missing from the guides on page one is a different one. Google-Extended bundles two things: training for Gemini models and grounding, meaning the answers in the Gemini apps that cite the site with a link. Block it and you leave both. The 30 October post says so plainly: sites that specifically allow Google-Extended are eligible to appear with links in Gemini app answers.

The shared signal does not untie the knot. Google has said a site with __usage_ai-training set to Disallow, and no group for Google-Extended, will be treated as if it had blocked Google-Extended. To stay cited in the Gemini apps you must write User-agent: Google-Extended with Allow: /, and accept training. Leaving AI Overviews as well takes a different route, covered in the guide to turning off AI Overviews for your site.

How to block AI training bots without disappearing from ChatGPT and Claude

Block GPTBot and let OAI-SearchBot in: for OpenAI they are two independent settings. GPTBot collects pages for training; OAI-SearchBot puts the site in ChatGPT search answers, and a site that blocks it appears only as a navigational link. A change to the file reaches search in about 24 hours.

Anthropic works the same way with ClaudeBot and Claude-SearchBot. OpenAI has said the shared signal will not replace the controls it already offers in robots.txt; Anthropic, that rules naming ClaudeBot take precedence over the general signal. So the file keeps both layers. How to get cited once you are in is covered in the guide to showing up in ChatGPT, and for Claude in showing up in Claude.

Some lists online include ChatGPT-User among the bots to block, the program that opens a page when a user asks for it. It is not the training crawler, and OpenAI warns that robots.txt rules may not apply to those requests. Blocking it does not protect your content from training.

Bing and Copilot: training and answers can be separated

With Microsoft the two choices are independent: from 1 February 2027 you will be able to refuse training and stay in Copilot answers. Bing’s post of 30 October defines two preferences, AI Training and AI Grounding, both allowed by default. Neither changes whether you appear in Bing’s classic results.

Training is refused with the shared signal in robots.txt, for the whole site or by path, or page by page with the meta name="Bingbot" content="ai-training:disallow" or the X-Robots-Tag header. Grounding is set only page by page: ai-grounding:disallow, or disallowextended, which excludes Microsoft products such as Copilot and leaves third-party services. There is no site-wide robots.txt setting for grounding.

Microsoft warns that refusing grounding means less brand visibility in AI experiences, and adds: “Neither choice guarantees inclusion, traffic, or payment.” Anyone using NoArchive or NoCache today has a deadline: from 1 March 2027 they no longer apply to training and grounding. A preference takes effect when Bingbot revisits the page; IndexNow helps without guaranteeing it, as the guide to Bing Webmaster Tools and IndexNow explains.

The mistakes that take a site out of AI answers

The worst is the blanket block, User-agent: * with Disallow: /, sometimes left over from a site under construction. It removes the site from Google, Bing and every answer that depends on them, and according to the IETF draft it does not even express a no to training. The second is blocking the search crawlers, OAI-SearchBot and Claude-SearchBot, along with the training ones.

Blocking by IP address at the firewall can stop crawlers reading the robots.txt itself: Anthropic says so on its page. The file also applies only to the domain or subdomain it is served from, so each one needs a copy. And it remains a preference: Cloudflare itself writes that signals in robots.txt are not technical countermeasures against scraping.

If you are on Cloudflare, the file may contain lines you never wrote: the managed robots.txt adds Content Signals, such as Content-Signal: search=yes, ai-train=no. They are a different syntax, launched by Cloudflare in September 2025, and no Cloudflare page read on 11 November says whether it will also add __usage_ai-training. To see what you are serving, open yoursite.com/robots.txt in a browser.

Does robots.txt remove content already used for AI training?

No: blocks and the signal apply to what crawlers collect from the moment you write them. Anthropic talks about future materials, Microsoft about future training, and the IETF draft ties the preference to the robots.txt in force when the content was collected. What a model has already learnt stays put. So write the signal group now.

Robots.txt and AI training: what the law says in Italy and beyond

In Italy and the European Union, a no to training written in a machine-readable form is a reservation of rights, with legal weight. Article 4 of Directive (EU) 2019/790 allows text and data mining of lawfully accessible works unless the rightsholder has reserved it, which for online content means by machine-readable means.

Italian copyright law repeats this in Article 70-quater, and Article 70-septies, added in 2025, extends the rule to mining carried out with artificial intelligence, including generative AI. The AI Act, in Article 53, requires providers of general-purpose models to respect the reservation. The EU Code of Practice of 10 July 2025, signed by Google, Microsoft, OpenAI and Anthropic among others, calls for crawlers that follow robots.txt.

WhereWhat applies to training on the site’s contentSource
Italy

Mining allowed unless the rightsholder has expressly reserved it, including when a generative AI system does it.

Law 633/1941, Articles 70-quater and 70-septies

European Union

The reservation is expressed by machine-readable means; providers of general-purpose models must respect it.

Dir. 2019/790, art. 4(3); Reg. 2024/1689, art. 53(1)(c)

EU Code of Practice

Signatories use crawlers that follow robots.txt and machine-readable protocols that are widely used or adopted by standards bodies.

GPAI Code, Copyright chapter, Measure 1.3

United Kingdom

The text and data mining exception covers only non-commercial research; robots.txt works as a preference.

CDPA 1988, s. 29A

United States

No law read for this page: the signal works as a voluntary commitment by providers.

Posts by Google, OpenAI, Microsoft

Whether __usage_ai-training is enough on its own as a reservation, no court or the Commission has said yet: Google presents it as consistent with the Code. On 13 July 2026 the Commission published a feasibility study on an EU registry of opt-outs, to sit alongside existing solutions, without deciding anything. For related topics there are guides on AI images and copyright and on the AI Act for generated content.

What is confirmed and what is not

Confirmed: the announcement, the syntax and Microsoft’s dates. No day yet from Google or OpenAI, no statement from Apple, Meta, Perplexity or Common Crawl. The table lists each fact with its date and source, read on 11 November 2026. This page will be reviewed by 1 February 2027 and after any announcement from Google or OpenAI.

FactDate and sourceStatus
Announcement of the shared signal

30 October 2026; blogs of Google, OpenAI and Bing

Confirmed

Syntax User-Agent: __usage_ai-training

30 October 2026; IETF draft -00 and Bing’s post

Confirmed, draft not yet a standard

Anthropic recognises it

From 30 October 2026; Claude Help Center

Confirmed, being rolled out gradually

Bing: temporary grounding preference

15 December 2026; Bing blog

Announced, not live yet

Microsoft honours it

1 February 2027; Bing blog

Announced, not live yet

NoArchive and NoCache no longer enough on Bing

1 March 2027; Bing blog

Announced

Google supports it

In 2027, early 2027 in the documentation; Google Search Central

Announced, no day given

OpenAI implements it

In the coming months; OpenAI blog

Announced, no date

Apple, Meta, Perplexity, Common Crawl take part

No source found

Not confirmed

The signal counts as a reservation of rights in the EU

No decision; Google calls it consistent with the Code

Not decided

EU registry of opt-outs

Study of 13 July 2026; European Commission

Study only

Two cases measured on itria.io

An open door is not enough. On the morning of 25 September 2026, Claude’s web search had a single page of itria.io; by the evening, after I asked Brave to recrawl seven, it had five. The robots.txt already let everyone in: what was missing was the index. The before and after is in the lab piece on Brave and Claude.

The CDN that rewrites the page. A Cloudflare setting stripped the email address from the HTML and put it back with a script: to an assistant, the contact page had no contact details. The same flaw turned up on 14 of 121 producers’ sites, in the lab piece on the invisible email.

The result. Between 11 October and 7 November 2026, Bing’s AI Performance report counted 268 citations of itria.io pages in Copilot, and Search Console 3,550 impressions in AI Overviews and AI Mode. How the work is done for a real company is on the page about presence in search engines and AI assistants; the other guides are in the marketing for small businesses index.

Questions and answers

Does the __usage_ai-training signal work yet?

Only in part. Anthropic began recognising it on 30 October 2026, rolling it out gradually. Microsoft will honour it from 1 February 2027, Google has announced 2027 and OpenAI “the coming months”.

Until then, opting out of training still runs through the groups for individual crawlers: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot.

If I block Google-Extended, do I drop out of AI Overviews?

No. Google-Extended governs training for Gemini and citations in the Gemini apps; Google states that it does not affect inclusion in Search. AI Overviews and AI Mode follow Googlebot and the nosnippet, max-snippet and noindex rules.

Can I refuse Gemini training and still be cited in the Gemini apps?

Not today. Google-Extended covers training and grounding in the Gemini apps together, and Google has said the shared signal, with no group for Google-Extended, will count as a block on Google-Extended. To be cited in Gemini you have to allow Google-Extended, and training with it.

Does robots.txt remove my content from models already trained?

No. A block applies to what crawlers collect from the moment it is written. Anthropic talks about future materials, and the IETF draft ties the preference to the robots.txt in force when the content was collected.

Does robots.txt carry legal weight against AI training in Italy?

Yes, as a reservation of rights: Italian copyright law, in Articles 70-quater and 70-septies, and Directive (EU) 2019/790 allow text and data mining unless the rightsholder has reserved it, and the AI Act requires providers of general-purpose models to respect that reservation. Whether the new signal is enough on its own as a reservation, no authority has said yet. This is information, not legal advice.

Notes on sources

  1. Google Search Central, Supporting a shared AI training control for websites, 30 October 2026, CC BY 4.0: the quotation, how Google-Extended is treated and grounding in the Gemini apps. Google, common crawlers, updated 30 October 2026 (early 2027), and AI features and your website.
  2. OpenAI, Making it easier for websites to express AI training preferences, 30 October 2026, and Overview of OpenAI Crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, the 24 hours.
  3. Microsoft Bing, Your content, your choice, Krishna Madhavan, 30 October 2026: the quotation, the two preferences, the meta syntax and the dates of 15 December 2026, 1 February, 1 March and 30 September 2027.
  4. IETF, draft-kelley-aipref-training-preference-signal-00, K. Kelley (Anthropic), 30 October 2026, an individual draft expiring on 3 May 2027: a working document, not a standard.
  5. Anthropic, crawler page, updated in the week of 11 November 2026: ClaudeBot, Claude-SearchBot, Claude-User, recognition from 30 October, blocking by IP. The page does not give the label’s name, which comes from the IETF draft.
  6. Apple, About Applebot, 4 September 2026; Common Crawl, CCBot; Cloudflare, Content Signals Policy, 24 September 2025, and managed robots.txt: a provider’s documentation of its own product.
  7. Directive (EU) 2019/790, Article 4; Law 633/1941, Article 70-quater and Article 70-septies; Regulation (EU) 2024/1689, Article 53; CDPA 1988, s. 29A.
  8. European Commission, General-Purpose AI Code of Practice, published on 10 July 2025, with the list of signatories; study on the opt-out registry, 13 July 2026 (presentation page read).
  9. The posts by Mattia Esposito cited here are on LinkedIn, dated 7 and 21 September 2026. The itria.io figures come from Search Console (generative AI report) and Bing Webmaster Tools, window 11 October to 7 November 2026. All pages were read on 11 November 2026.
·Who does it for you

Refusing AI training is one line. Getting cited is the rest of the site. Positioning is built starting from the market.

Mattia Esposito, who wrote this guide, also works with Itria on positioning: in-depth market analysis to find where competitors are exposed and take that ground. He did it on his own site first. In August, asked who had founded itria.io, ChatGPT couldn’t find him. From 11 October to 7 November 2026 itria.io pages appeared 13,803 times in Google and Bing results and 3,818 times in AI answers: Google’s AI Overviews and AI Mode, and Microsoft Copilot.