Article
Claude reaches your website in two ways. The first is the data Anthropic trains its models on. The second is live search: a user asks a question, Claude searches the web, reads a few pages and answers with citations. The second way brings traffic and recommendations today, so that is where I start.
Everything I write about the bots here I checked in Anthropic’s documentation. The links are in the text and in the sources below the article.
How Claude searches the web
When search is on, Claude decides for itself when to go to the web. According to Anthropic’s help pages, every such answer comes with citations, that is links to the pages Claude took the information from. That is where you want to be.
Where does Claude get its list of pages to read? In March 2025 Anthropic added Brave Search to its list of subprocessors, and Simon Willison checked that Claude’s citations matched Brave’s results for the same query. TechCrunch reported on it too. Anthropic does not publish exactly how it picks sources, so treat this as a hint, not a recipe. The practical takeaway is simple: check how your site does in Brave Search for the queries that matter to you.
Anthropic’s three bots
Anthropic describes three bots in its article on crawling. Each does something different, so each needs its own decision.
| Bot | What it does | What happens if you block it |
|---|---|---|
ClaudeBot |
Collects public content that may be used to train models | New content from your site stays out of training data |
Claude-SearchBot |
Indexes pages to improve search results in Claude | Your site may appear less often and less accurately in search answers |
Claude-User |
Fetches a page when a user asks a question that needs it | Claude will not read the page, even when a user asks it to |
A few more things from the same documentation:
- Anthropic’s bots respect robots.txt.
- Anthropic supports the non-standard
Crawl-delaydirective if you want to slow the bots down. - robots.txt works separately for every subdomain, so
shop.example.comneeds its own. - The bots’ IP addresses are published in claude.com/crawling/bots.json. Anthropic advises against blocking by IP, because then the bots may not be able to read your robots.txt.
robots.txt: three decisions, not one
The most common mistake I see is one rule for all AI bots. Blocking training is fine if that is what you want. Blocking search cuts you out of the answers, and that is a decision to give up traffic.
If you want to be in Claude’s answers but do not want your content used for training, robots.txt can look like this:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
If you do not mind training, simply do not block any of them. On this site I let all three in, because I sell visibility in AI and have to be visible myself. You can see my file at deadnet.pl/robots.txt.
Two things quietly break this:
- Firewall and CDN. If your site sits behind Cloudflare or another CDN, check the AI bot blocking settings. They can block bots regardless of what your robots.txt says. When I set up this site, I made sure Cloudflare’s managed robots.txt did not overwrite my file.
- SEO plugins. Some CMS plugins add rules for AI bots on their own. Look at the final file at
/robots.txt, because the plugin settings do not show everything.
llms.txt: a cheap extra, not a requirement
The llms.txt file is a proposal by Jeremy Howard from September 2024. It is a Markdown file in the site root: the site name as a heading, a short description in a quote and lists of links to the most important pages.
Anthropic’s crawler documentation does not say Claude uses it. Google says outright that you do not need any AI text files to appear in AI Overviews. I have an llms.txt on this site, because it generates itself from the content and costs nothing. Do not expect the file alone to change anything, though. What counts is what is on the pages it links to.
Content that can be quoted
Claude reads several pages and builds an answer from them. It quotes the passages that can be lifted out of context and still make sense. A few rules I use with clients:
- Definition at the top. The first sentence of a service or category page says what it is. “An SEO audit is a review of a site’s technical setup, content and links” can be quoted. “Welcome to our store” cannot.
- A paragraph that stands on its own. One paragraph, one answer, no references like “as mentioned above”.
- Numbers with units. Compare two sentences. “Our backpack is the perfect choice for every trip.” “The backpack holds 30 liters, weighs 1.1 kg and fits a 15.6 inch laptop.” The second answers a specific question, so it has a chance of ending up in an answer.
- Headings phrased like customer questions. “How much does a drone roof inspection cost” instead of “Pricing”.
- Author and dates. Who wrote it, when it was published and when it was last updated. This article has both dates under the title.
One technical point that matters for PWA and headless stores: I found nothing in Anthropic’s documentation about whether the bots run JavaScript. It is safer to assume they do not. If a product description only appears after a script runs, the bot may see an empty page. You can check this by turning JavaScript off in your browser or fetching the page with curl.
Structured data
Anthropic does not say Claude reads JSON-LD. Structured data is still worth having: it organizes facts about your company, products and authors for search engines, and Claude uses search engines. Just make sure the JSON-LD matches what is visible on the page. A mismatch between the two signals that the page cannot be trusted.
How to check whether Claude quotes your site
-
Ask Claude. Write down 10 to 20 questions your site should answer and ask Claude with search turned on. Note who gets quoted. Repeat every month with the same list.
-
Go through your server logs.
Claude-SearchBotvisits show which pages Claude indexes.Claude-Uservisits are the moments when someone asked a question and Claude came to your page for the answer. On a server with a typical access log, this is enough:grep -E "ClaudeBot|Claude-SearchBot|Claude-User" access.log \ | awk '{print $7}' | sort | uniq -c | sort -rn | head -20A bot’s name is easy to fake, so for important decisions check the IP against
bots.json. -
Check your analytics. Visits from
claude.aiare people who clicked a citation. Some browsers do not pass the referrer, so this number will be on the low side. -
Check Brave. Type the same questions into Brave Search and see whether your site shows up.
Where to start
- Open
/robots.txton your site and check the rules forClaudeBot,Claude-SearchBotandClaude-User. - Check the AI bot settings in your CDN and SEO plugins.
- Fetch a few important pages with
curland see whether the content is in the HTML without JavaScript. - Rewrite the first paragraphs of your service and category pages so they can be quoted.
- Write down a list of questions and check who Claude quotes instead of you today.
This is the first article in a series on visibility in AI assistants. The next ones cover Grok, ChatGPT, Perplexity and Gemini with AI Overviews.