Call · 30 min
GlossaryMattia Esposito10 November 20264 min read

AI crawlers. The programs that read your site for assistants, and what they are called.

An AI crawler downloads a site’s pages for an artificial intelligence company: for search, for training or at a user’s request. Here are the names, and what blocking them changes.

In brief

An AI crawler is the program that reads a site’s pages for an artificial intelligence company.

OpenAI, Anthropic and Perplexity each use one for search and one for user requests, and OpenAI and Anthropic a separate one for training. Blocking the training crawler doesn’t take the site out of the answers.

They read the HTML and don’t run JavaScript, according to a Vercel measurement of 1.3 billion requests.

Written by: Mattia Esposito, AI and marketing consultant, founder of Itria AI. From Puglia, based in Turin since 2023, he works remotely in Italy and abroad. LinkedIn profile.

The three jobs of an AI crawler

Artificial intelligence companies use different crawlers for different jobs, and each one has its own name. One serves the assistant’s search, one model training, one opens a page when a user asks. The table brings together the names declared by OpenAI, Anthropic, Perplexity and Google, read on 10 November 2026.

CompanySearchTrainingOn request
OpenAI

OAI-SearchBot

GPTBot

ChatGPT-User

Anthropic

Claude-SearchBot

ClaudeBot

Claude-User

Perplexity

PerplexityBot

not declared

Perplexity-User

Google

Googlebot

Google-Extended (a control token, not a separate crawler)

not applicable

Perplexity writes that PerplexityBot is not used to train models. Google-Extended, according to Google, governs the use of text for Gemini and doesn’t affect presence in Search or in AI Overviews. The lines to write for each are in the guide on robots.txt and AI crawlers.

robots.txt, the crawlers’ front door

A crawler stops or comes in according to robots.txt, the file at the root of the site standardised by the IETF in RFC 9309 in September 2022. OpenAI says so for ChatGPT search: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers”. In front of robots.txt there is often a CDN such as Cloudflare, with its own rules on AI crawlers that apply first.

In Itria’s study of 121 food producers’ websites, every one let in the search crawlers of ChatGPT, Claude and Perplexity. The problem was inside the page: 11.6% had an email address invisible to assistants, hidden by obfuscation or missing from the code. The itria.io case is in the piece on the email address AI assistants can’t see.

What they can’t read

AI crawlers read the HTML as it arrives and don’t run JavaScript. Vercel measured this across about 1.3 billion requests in a month. Text, prices and contacts that only appear after a script don’t exist for an assistant. How the pages they read make it into answers is explained in the guide on how to show up in ChatGPT; the file proposed to guide them is llms.txt, and the work as a whole is called GEO.

Questions and answers

What is an AI crawler?

A program that visits websites and downloads their pages on behalf of an artificial intelligence company. Every crawler has a name, its user agent, which you use in robots.txt to let it in or keep it out.

What is the difference between GPTBot and OAI-SearchBot?

They are two OpenAI crawlers with different jobs. GPTBot collects pages that may be used to train models; OAI-SearchBot finds pages for ChatGPT search. OpenAI writes that a site blocking OAI-SearchBot is not shown in search answers.

Will blocking AI crawlers take me out of ChatGPT?

It depends which ones you block. Blocking GPTBot, ClaudeBot or Google-Extended keeps your text out of training and leaves the site in the answers; blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes it from that assistant’s search.

Do AI crawlers respect robots.txt?

The search and training crawlers of OpenAI, Anthropic and Perplexity say they do. For crawlers that open a page at a user's request the rules change: Perplexity writes that Perplexity-User generally ignores it.

Do AI crawlers read JavaScript?

Generally not. Vercel measured about 1.3 billion requests in a month: the main AI crawlers, including those of OpenAI, Anthropic and Perplexity, read the HTML as it arrives, without running scripts.

Notes on sources

  1. OpenAI, Overview of OpenAI Crawlers: the quotation. Anthropic, crawler page, updated 30 October 2026. Read on 10 November 2026.
  2. Perplexity, Perplexity Crawlers; Google, common crawlers (Google-Extended), CC BY 4.0.
  3. IETF, RFC 9309, Robots Exclusion Protocol, September 2022. Vercel, The rise of the AI crawler.
  4. Itria’s study of producers’ websites dates from 26 September 2026.
·Who does it for you

Opening the right doors takes an afternoon. Getting cited instead of your competitors is the work that follows. Positioning is built starting from the market.

Mattia Esposito, who wrote this guide, also works with Itria on positioning: in-depth market analysis to find where competitors are exposed and take that ground. He did it on his own site first. In August, asked who had founded itria.io, ChatGPT couldn’t find him. From 11 October to 7 November 2026 itria.io pages appeared 13,803 times in Google and Bing results and 3,818 times in AI answers: Google’s AI Overviews and AI Mode, and Microsoft Copilot.