Article

Posts on X and a Grok answer that quotes the brand's account and website. Next to it, robots.txt with a question mark, because xAI does not name its crawler.

With Grok you have to start with what is unknown. On 3 November 2026 I found no official xAI documentation giving the crawler’s name, its user agent, its IP addresses or whether it respects robots.txt. Bot directories online list various names, but they do not agree with each other and none of them cites xAI. So in this article I rely only on what xAI publishes itself.

How Grok looks for information

xAI describes search in its API documentation. Grok has two built-in tools that it runs on its own while answering:

  • Web Search searches the web in real time and reads pages to find information.
  • X Search searches posts on X: by keywords, meaning, users and whole threads.

The tools overview shows that both run on xAI’s side and Grok reaches for them automatically. It is documentation for developers, but it shows well where Grok gets its knowledge from: the web and X. The second part is what sets it apart from other assistants.

What this means for robots.txt

With no official bot name, there is no reliable rule for robots.txt. You can add a group for a name from one of the directories, but you do not know whether xAI uses it or obeys it. For a site that wants to be visible, that is not a problem anyway. Just do not block all bots with a User-agent: * rule, and do not run aggressive bot protection that stops unknown clients.

If you want to block Grok, robots.txt will not guarantee it. That leaves server logs and firewall rules, but without official IP ranges it is hard to tell an xAI bot from someone pretending to be one.

What you can do

Since Grok reads the web and X, two things work:

  • A site that is easy to quote. The same rules as with other assistants: the answer in the first paragraph, concrete numbers, a separate page for each important question, content in the HTML without waiting for JavaScript. I write more about this in the article on Claude.
  • Presence on X. A brand account that answers questions from your industry, posts specifics and links to pages with the full answer. X Search goes through posts and threads, so what the brand writes on X can end up in Grok’s answers.

How to check the results

  1. Ask Grok. Use the same list of customer questions you use with other assistants. Note whether your site or your X account shows up.
  2. Go through your logs. Look for unknown user agents containing xAI or Grok. Write down their IP addresses, because they will help if xAI ever publishes official ranges.
  3. Check your analytics. Visits from grok.com and x.com show traffic from Grok and from X itself.

Where to start

  1. Make sure robots.txt does not block all bots with a single rule.
  2. Rework the first paragraphs of your most important pages so they answer specific questions.
  3. If the brand has an X account, pick a few customer questions and answer them in posts with a link to your site.

I will update this article when xAI publishes documentation for its crawler. The next part of the series covers ChatGPT.

Sources

I checked the documentation on 3 November 2026. Vendors change it without notice, so it is worth checking again before you act on it.

  1. xAI: Web Search
  2. xAI: X Search
  3. xAI: Tools Overview

Want me to look at your site?

Tell me which site it is and what matters to you. I will reply with what I see at first glance and what to fix first.

hello@deadnet.ovh

30 minutes online, pick a free slot in my calendar.