Call · 30 min
MarketingMattia Esposito10 November 202614 min read

How to show up in Claude. Which Anthropic bots to let in, where Claude gets its results, and nine checks to run on your site.

ClaudeBot, the Anthropic bot many sites block, has little to do with Claude’s search: two other names matter. Here is which ones, where Claude gets its results, and a nine-point checklist in Word and PDF.

In brief

To show up in Claude, your site has to let in Claude-SearchBot, which indexes pages for search, and Claude-User, which opens them when a user asks a question. ClaudeBot is for training: blocking it does not take you out of search.

Claude does not search with its own bots alone. Anthropic writes that noindex reaches its search partners, without naming them, and that images come from Bing. In Anthropic’s list of providers, read on 10 November 2026, web search runs through three providers, Brave Search among them, and Brave does not crawl what Googlebot cannot crawl.

A page that can only be read with JavaScript stays out of the tool Claude uses to open websites, according to the API documentation.

The nine-point checklist can be downloaded below, in Word and PDF, with no email required.

Written by: Mattia Esposito, AI and marketing consultant, founder of Itria AI. From Puglia, based in Turin since 2023, he works remotely in Italy and abroad. LinkedIn profile.

The nine-point checklist, to download

One sheet with the four names Anthropic uses to visit websites, the nine checks with where to look, and the robots.txt lines, in Word to fill in and PDF to print. You need access to your site’s control panel and to your CDN’s, if you have one. It takes half an hour; point 8 is repeated every two weeks.

The nine-point checklist for showing up in Claude, with the table of Anthropic’s bots and what happens if you block them
FileContentsLink
Wordto fill in, in Italian

The table of Anthropic’s bots, the nine checks with where to look, the robots.txt lines.

come-comparire-su-claude-checklist.docx

PDFto print, in Italian

The same checklist, to tick off with a pen.

come-comparire-su-claude-checklist.pdf

N.What to checkWhere to look
1

robots.txt lets in Claude-SearchBot and Claude-User.

yoursite.com/robots.txt

2

A decision has been made on ClaudeBot: blocking it takes you out of training, not out of search.

yoursite.com/robots.txt

3

Googlebot is not blocked, because Brave follows its rules.

yoursite.com/robots.txt

4

No noindex on the pages you want found.

Page source, meta robots

5

The text of key pages sits in the initial HTML, readable without JavaScript.

Page source (Ctrl+U)

6

Firewall and CDN do not block Anthropic’s IPs; a decision has been made on Claude-Agent.

CDN or hosting dashboard

7

Brave knows the site: search site:yoursite.com on Brave and count the pages.

search.brave.com

8

Test on Claude with search on: three to five fixed questions, site among the sources yes or no, cited yes or no.

claude.ai, every two weeks

9

If a page has to come out of the answers, the request to Anthropic with proof that the domain is yours.

Email from the domain

On 25 September 2026 I asked Claude, with web search on, the same questions twice, in the morning and in the evening. In the morning, among the results Claude was reading, itria.io had a single page. Between the two rounds I asked Brave to recrawl seven addresses on the site. By the evening there were five different pages.

On the eight questions common to both rounds, the site appeared among the results in two answers in the morning and in four in the evening. Cited in the answer: zero times, both rounds. One pair of measurements proves no cause, and I wrote it up that way in the Brave and Claude lab note. I wrote the conclusion there too: being found and being cited are two different steps.

Claude draws on several sources, and each has its own doors. This page lines up the ones Anthropic and Brave document, as read on 10 November 2026. For ChatGPT, Copilot and Google the doors are different, and they are in the guide on how to show up in ChatGPT.

How do you show up in Claude’s answers?

Claude cites its sources when it uses web search: it searches, reads the results and puts links to the pages it drew on in every answer. Anthropic launched it on 20 March 2025 for paid plans in the United States and opened it to all plans worldwide from 27 May 2025. In the new Claude experience there is no switch: it searches when it needs to.

How a page reaches a Claude answer: the site’s doors, the indexes of Anthropic and its partners, Claude’s search, the citation The pagetext in the HTML The doorsrobots.txt and firewall The indexesAnthropic and partners The searchClaude searches and opens The citationthe link to the source points 4-5points 1-3, 6point 7 point 8checks 8-9
The path of a page to a Claude answer with a citation, and which checklist points act on each step.

On Team and Enterprise plans, the organisation’s owner has to switch search on first, in the settings. With search on, Claude can also open in full a page whose address the user types in. It can use your approximate location from your IP too, so the same question asked from Bari and from London can come back with different results.

Developers building products on Claude pay 10 dollars per 1,000 searches, plus tokens. The documentation estimates one to three searches for a factual question and ten or more for a comparison. Every result carries its address, title and page_age, the date the page was last updated: a written, current date travels with the link.

What search engine does Claude use?

Anthropic does not name a single engine: it writes that Claude’s search relies on partners with their own index, and names Bing for images only. The line about partners is on the page about removing content from Claude, where noindex “tells our partners not to index your content so that they don’t send it to us in response to your web search query”.

For images, the help page on search is explicit: “Image search is powered by Bing”. For text, no help page read for this guide names an engine, and the API documentation, which describes the search tool line by line, names no index provider. The providers’ names turn up in another document.

In Anthropic’s list of providers, read on 10 November 2026, web search runs through three providers: Brave Search, TurboPuffer and Parallel Web Systems. Parallel was not on the list I read on 25 September, when I measured the case described below. Alongside them there is Claude-SearchBot, the bot Anthropic uses to index pages for search. Providers and bot coexist, and no source says in what proportion, so the checklist opens every door rather than picking one.

What Anthropic saysWhere it says itDate
Claude-SearchBot indexes pages for search

Help page on its bots

updated 4 November 2026

noindex reaches the search partners

Page on removing content

3 September 2026

Images in the results come from Bing

Help page on web search

updated 9 November 2026

Brave Search, TurboPuffer and Parallel Web Systems for web search

List of providers

read on 10 November 2026

No index provider named

API search tool documentation

read on 10 November 2026

Every answer that uses search carries citations and links

Announcement and help page on web search

20 March 2025

Results can take your location from your IP into account

Help page on web search

updated 9 November 2026

ClaudeBot vs Claude-SearchBot vs Claude-User: what’s the difference?

Anthropic has three bots with three jobs, and for showing up, Claude-SearchBot and Claude-User are the two that count. The first indexes pages for search; the second opens a page when a user asks a question; ClaudeBot collects pages for training. Of Claude-User, Anthropic writes that turning it off “may reduce your site’s visibility for user-directed web search”.

NameWhat it does, according to AnthropicIf you block it
Claude-SearchBot

Indexes pages for Claude’s search.

Pages are not indexed for search

Claude-User

Fetches a page in response to a user’s question.

Less visibility in user-directed search

ClaudeBot

Collects pages to train the models.

Out of training, still in search

Claude-Agent

The Anthropic-hosted browser that does tasks on the web for the user.

Decided on the server or firewall

Anthropic says it respects robots.txt and does not try to get past CAPTCHAs. It publishes its bots’ addresses in a file, bots.json: read on 10 November 2026, it held 14 IPv4 address blocks, created on 20 October 2026. The file does not say which bot each block belongs to, so it helps you spot a fake Claude, not tell the three apart.

For a bot behaving differently from its description, Anthropic gives an address, [email protected], to write to from an email on the domain concerned. The help page also exists in Italian, with the same rules. The list of bots for every assistant is in the glossary entry on AI crawlers.

Does blocking ClaudeBot remove my site from Claude search?

No: ClaudeBot is for training, and the settings for the three bots are separate. A site can say no to training and stay in search with a few lines in robots.txt, like these. Order does not matter: each bot reads the block with its own name.

In robots.txt: in Claude’s search, out of training

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

A robots.txt that does not name these bots already lets them in. The lines matter when there is a blanket block higher up, such as “User-agent: *” with “Disallow: /”, which can come from a plugin or from a configuration years old. The checklist looks for it at point 1; all the lines for the other assistants are in the guide to robots.txt and AI crawlers.

In the European Union, saying no to training also carries legal weight. Article 4(3) of the 2019 copyright directive lets rightholders reserve text and data mining through machine-readable means, and Article 53 of the AI Act requires model providers to respect that reservation. For search, on the other hand, the rules are the same in Italy and in the English-speaking world.

What is the Claude-Agent user agent, and how do you recognise it?

Claude-Agent is the name of the real browser Anthropic hosts so that Claude can do tasks on the web for a user, for example in Claude Duet. Its requests carry the user agent of an ordinary Chrome with Claude-Agent/1.0 added. Anthropic advises recognising it by the name: “match on the Claude-Agent token rather than the full string”.

The IP addresses for this traffic are published in a separate file, claude.com/.well-known/sail, in a format based on a technical draft that Anthropic itself describes as not yet a standard. A user agent can be faked, so to be sure you check the IP. Preferences for Claude-Agent are independent of those for the three bots.

Anthropic’s page talks about allowing or denying this traffic and does not say it reads robots.txt: the decision is made on the server or in the firewall. For a company that sells, a browser filling in a form or comparing a price on a buyer’s behalf is traffic to let through. For a customer-only area, the choice may differ.

Does Claude use Brave Search? Why blocking Googlebot also shuts out Brave

Brave’s bot has no name of its own and follows the rules written for Googlebot: a page closed to Googlebot stays closed to Brave. Brave puts it like this: “if a domain or page is not crawlable by Googlebot, then Brave Search’s bot will not crawl it either”. It does so to avoid being shut out by sites that only let Google in.

For anyone who wants to show up in Claude, the consequence is practical. A Disallow meant for Googlebot, even on a single section of the site, applies to Brave, and so to the part of Claude’s search that runs through its index. To take a page out of Brave’s index robots.txt is not enough: you need noindex, and Brave has to revisit the page to read it.

Brave has a public page for asking it to recrawl an address, one at a time. The quickest check is on its own search: site: followed by your domain, then count the pages that come up. If one or two come up on a site with dozens, Brave barely knows the site. Its crawler, Brave writes, is also fed by the Web Discovery Project, a browser option switched off by default.

Two cases measured on itria.io

From one page to five. On the morning of 25 September 2026, for Claude’s web search, itria.io had a single page among the results; in the evening, after asking Brave to recrawl seven, it had five. Citations in the answer: zero and zero. The before and after, with the limits of the data, are in Brave and Claude.

The email address that wasn’t there. Cloudflare’s obfuscation took the address out of the HTML and put it back with a script: for an assistant, the contact page had no contact details. The case, and the same flaw on 14 of 121 producers’ sites, is in the piece on the email address AI assistants can’t see.

The result. Between 11 October and 7 November 2026 itria.io pages appeared 3,818 times in Google and Microsoft AI answers: 3,550 impressions in AI Overviews and AI Mode according to Search Console, 268 citations in Copilot according to Bing Webmaster Tools.

Can Claude read JavaScript websites?

The tool Claude uses to open pages cannot read them, according to the API documentation. Anthropic says so in one line: “The web fetch tool currently does not support websites dynamically rendered with JavaScript.” Text, prices and contact details that appear only after a script runs are left out of what Claude reports.

The page describes the API, meaning Claude inside other companies’ products; that the claude.ai app behaves the same way is plausible, but not written down. A real browser such as Claude-Agent’s, on the other hand, runs the page. The tool reads text, HTML and PDF, and opens only addresses that have already appeared in the conversation, in the user’s message or in search results. An address barred by robots.txt comes back with an error.

I wrote it on 7 September, after looking at the websites of 142 Italian companies that also sell abroad: “An assistant doesn’t browse a site. It reads it, and reports what it finds written there.” The same day I counted how many had the file that explains the company to assistants: 28 out of 143, and in 24 a plugin had written it. The full post (in Italian).

How to remove your site or a page from Claude’s answers

With noindex for pages, with robots.txt for images and video and, for an address that has already appeared, with a written request to Anthropic. The help page of 3 September 2026 gives [email protected] and asks for proof that the domain is yours: the ICANN or WHOIS registration, the domain receipt, the SSL certificate or an email from the domain itself.

For images and video, Anthropic writes that its search partners only index files their bots can crawl, and that you exclude them with robots.txt. The safest way not to appear, it adds, is still to remove the content or put it behind a password. Read in reverse, the same levers are the ones to check when a page does not show up.

Then there is an exit the site does not control. Anyone building a product on Claude can restrict search to a list of allowed domains or exclude others, with the allowed_domains and blocked_domains settings. A site can therefore stay out of that product by the builder’s choice, whatever its own robots.txt says.

How do I know if Claude cites my website?

By asking Claude, with search on, the same three to five questions a few weeks apart, and noting whether the site shows up among the sources and whether it is cited. The Anthropic documents read for this guide offer no dashboard for site owners like Search Console or Bing’s AI Performance report. Answers change from one round to the next, so the series is what counts.

Server logs show visits from Claude-SearchBot and Claude-User, and how often they come back to the same pages. Between appearing among the sources and being cited there is the content: pages that answer a precise question, with figures, dates and sources. How to work on that step is in the guide on how to get cited by AI assistants.

For the other assistants the doors are different: Perplexity has its own index, Copilot goes through Bing. How to build all this for a real company is on the page about presence in search engines and AI assistants; the questions to ask anyone selling it are on the page about GEO consultancy.

Questions and answers

How do I show up in Claude?

Let in Claude-SearchBot, the bot Anthropic uses to index pages for search, and Claude-User, which opens a page when a user asks a question. Keep Googlebot unblocked, because Brave follows its rules, and put your text in the HTML so it can be read without JavaScript.

To be cited as well, the page has to answer a precise question with facts that can be checked.

What search engine does Claude use?

Anthropic writes that Claude’s search relies on partners with their own index, without naming them, and that images come from Bing. In Anthropic’s list of providers, read on 10 November 2026, web search runs through three providers: Brave Search, TurboPuffer and Parallel Web Systems. Anthropic also runs a bot of its own, Claude-SearchBot, which indexes pages for search.

If I block ClaudeBot, do I disappear from Claude?

No. ClaudeBot collects pages to train the models; for search, Claude-SearchBot and Claude-User are what count. The settings are separate: you can say no to training and stay in the answers.

What is Claude-SearchBot?

Claude-SearchBot is Anthropic’s bot for indexing pages for Claude’s search. If you block it in robots.txt, Anthropic writes, your pages are not indexed for search. The IP addresses of Anthropic’s bots are published at claude.com/crawling/bots.json.

Can Claude read JavaScript websites?

The tool Claude uses to open pages cannot, according to Anthropic’s API documentation: text and contact details that only appear after a script runs are left out. The documentation says nothing either way about the claude.ai app, so it still makes sense to put your text in the HTML.

Notes on sources

  1. Anthropic, Claude Help Center, page on Anthropic’s bots, updated 4 November 2026: ClaudeBot, Claude-User, Claude-SearchBot, robots.txt, CAPTCHAs, the reporting address; source of the line on Claude-User. IP addresses in bots.json. Read on 10 November 2026.
  2. Anthropic, Anthropic-hosted browsers, updated 4 November 2026: Claude-Agent, the line on the token and the IP file.
  3. Anthropic, Enable and use web search, updated 9 November 2026: plans, location, images from Bing; web search announcement, 20 March 2025, updated to 27 May 2025.
  4. Anthropic, Report, block, and remove content from Claude, 3 September 2026: search partners, noindex, the removal request; source of the line on partners.
  5. Anthropic, API documentation: web search tool (price, page_age, allowed and blocked domains) and web fetch tool (JavaScript, formats, robots.txt).
  6. Anthropic, list of providers, read on 10 November 2026: Brave Search, TurboPuffer and Parallel Web Systems with the web search function.
  7. Brave, Brave Search Crawler: Googlebot’s rules, noindex, the Web Discovery Project; source of the quoted line. Read on 10 November 2026.
  8. Directive (EU) 2019/790, Article 4(3), and Regulation (EU) 2024/1689, Article 53(1)(c), on EUR-Lex.
  9. The Brave and Claude case is a single pair of measurements, on 25 September 2026; the itria.io figures come from Search Console (generative AI report) and Bing Webmaster Tools, window 11 October to 7 November 2026.
·Who does it for you

Opening the doors to Claude is the first step. Getting cited instead of your competitors is the work that follows. Positioning is built starting from the market.

Mattia Esposito, who wrote this guide, also works with Itria on positioning: in-depth market analysis to find where competitors are exposed and take that ground. He did it on his own site first. In August, asked who had founded itria.io, ChatGPT couldn’t find him. From 11 October to 7 November 2026 itria.io pages appeared 13,803 times in Google and Bing results and 3,818 times in AI answers: Google’s AI Overviews and AI Mode, and Microsoft Copilot.