Article
With Grok you have to start with what is unknown. On 3 November 2026 I found no official xAI documentation giving the crawler’s name, its user agent, its IP addresses or whether it respects robots.txt. Bot directories online list various names, but they do not agree with each other and none of them cites xAI. So in this article I rely only on what xAI publishes itself.
How Grok looks for information
xAI describes search in its API documentation. Grok has two built-in tools that it runs on its own while answering:
- Web Search searches the web in real time and reads pages to find information.
- X Search searches posts on X: by keywords, meaning, users and whole threads.
The tools overview shows that both run on xAI’s side and Grok reaches for them automatically. It is documentation for developers, but it shows well where Grok gets its knowledge from: the web and X. The second part is what sets it apart from other assistants.
What this means for robots.txt
With no official bot name, there is no reliable rule for robots.txt. You can add a group for a name from one of the directories, but you do not know whether xAI uses it or obeys it. For a site that wants to be visible, that is not a problem anyway. Just do not block all bots with a User-agent: * rule, and do not run aggressive bot protection that stops unknown clients.
If you want to block Grok, robots.txt will not guarantee it. That leaves server logs and firewall rules, but without official IP ranges it is hard to tell an xAI bot from someone pretending to be one.
What you can do
Since Grok reads the web and X, two things work:
- A site that is easy to quote. The same rules as with other assistants: the answer in the first paragraph, concrete numbers, a separate page for each important question, content in the HTML without waiting for JavaScript. I write more about this in the article on Claude.
- Presence on X. A brand account that answers questions from your industry, posts specifics and links to pages with the full answer. X Search goes through posts and threads, so what the brand writes on X can end up in Grok’s answers.
How to check the results
- Ask Grok. Use the same list of customer questions you use with other assistants. Note whether your site or your X account shows up.
- Go through your logs. Look for unknown user agents containing xAI or Grok. Write down their IP addresses, because they will help if xAI ever publishes official ranges.
- Check your analytics. Visits from
grok.comandx.comshow traffic from Grok and from X itself.
Where to start
- Make sure robots.txt does not block all bots with a single rule.
- Rework the first paragraphs of your most important pages so they answer specific questions.
- If the brand has an X account, pick a few customer questions and answer them in posts with a link to your site.
I will update this article when xAI publishes documentation for its crawler. The next part of the series covers ChatGPT.