Article

robots.txt blocks GPTBot and allows OAI-SearchBot, a firewall lets in addresses from the OpenAI list, and a ChatGPT answer cites the store as a source.

ChatGPT answers questions in two ways: from what the model learned earlier and from live search. In search, ChatGPT gives sources with links, so that is where you can win traffic. OpenAI describes its bots in its crawler documentation, and this article is based on it.

OpenAI’s four bots

Bot What it does Does it follow robots.txt
OAI-SearchBot Shows sites in ChatGPT search results Yes. A site blocked for it will not appear in search answers
GPTBot Collects content that may be used to train models Yes. Blocking it means the content should not be used for training
ChatGPT-User Visits a page when a user asks a question or uses a GPT action Possibly not, because a person starts the visit
OAI-AdsBot Checks landing pages of ads in ChatGPT The documentation does not say. It only visits pages submitted as ads

The most important line in the documentation: whether you appear in ChatGPT search is decided by OAI-SearchBot, not by ChatGPT-User. OpenAI also writes that search changes to robots.txt take effect after about 24 hours.

If you want to appear in ChatGPT answers but do not want to hand over your content for training:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

I leave ChatGPT-User without rules, because it is a visit at a person’s request. If you block it, someone who pasted a link to your site into the chat gets an answer without its content.

OpenAI also recommends not blocking the IP addresses in openai.com/searchbot.json. If you run a firewall with an allow list or aggressive bot protection, add those ranges. A bot’s name is easy to fake in the logs, so when blocking or unblocking, check the IP, not just the name.

Where ChatGPT gets its results

Apart from its own crawler, OpenAI’s documentation does not say whose indexes ChatGPT search uses. Tests from 2024, described among others by Windows Latest, showed results very similar to Bing’s. That is not an official confirmation. That is why I treat Bing Webmaster Tools as a cheap step worth taking: add the site, submit a sitemap and check that Bing indexes your important pages. It takes a quarter of an hour and helps beyond ChatGPT too.

Content ChatGPT picks as a source

In search mode, ChatGPT reads several pages and cites the ones that best answer the question. The rules are the same as with other assistants:

  • The answer in the first paragraph, the details below.
  • Concrete data: prices, dimensions, deadlines, model names, dates.
  • One topic per page. A page that answers everything at once loses to one that answers exactly one question.
  • Content in the HTML. If a product description only loads through JavaScript, check whether it is in the page source. The easiest way is curl or your browser’s view source.

In stores I most often see two problems. The first is category descriptions that do not exist or are copied from the manufacturer. The second is comparisons and guides hidden in PDFs or in tabs that do not have their own URL.

How to check the results

  1. A list of questions. Write down the questions customers ask before buying and check every month who ChatGPT cites. Keep the results in a spreadsheet, because the answers change.
  2. Logs. OAI-SearchBot visits show which pages go into the index. ChatGPT-User visits are the moments when someone asks about your topic and ChatGPT comes to your site for the answer.
  3. Analytics. Visits from chatgpt.com are people who clicked a source. Some browsers do not pass the referrer, so this number will be on the low side.

Where to start

  1. Check robots.txt for OAI-SearchBot and GPTBot.
  2. Check that your firewall does not block the addresses in searchbot.json.
  3. Add your site to Bing Webmaster Tools and submit a sitemap.
  4. Rewrite the first paragraphs of your most important pages so they answer specific questions.

This article is part of a series on visibility in AI assistants. Start with the article on Claude, where I write more about the content itself and how to check it.

Sources

I checked the documentation on 3 November 2026. Vendors change it without notice, so it is worth checking again before you act on it.

  1. OpenAI: Overview of OpenAI Crawlers
  2. Windows Latest: ChatGPT search engine uses Bing (1 November 2024)
  3. Bing Webmaster Tools

Want me to look at your site?

Tell me which site it is and what matters to you. I will reply with what I see at first glance and what to fix first.

hello@deadnet.ovh

30 minutes online, pick a free slot in my calendar.