How to audit your robots.txt for AI crawlers

Find the file

It is at yourdomain.com/robots.txt. If you get a 404, everything is allowed by default and you have made no decision at all. If it returns a server error (5xx), the robots.txt standard, RFC 9309, tells crawlers to assume the whole site is disallowed.

The crawlers that matter

Each AI company runs separate agents for separate jobs, and each can be allowed or blocked on its own.

Search crawlers govern whether you appear in AI search answers. OAI-SearchBot does this for ChatGPT, Claude-SearchBot for Claude and PerplexityBot for Perplexity. OpenAI states that sites opted out of OAI-SearchBot are not shown in ChatGPT search answers.

Training crawlers collect content for future models. GPTBot feeds OpenAI's models and ClaudeBot feeds Anthropic's. Blocking them keeps your content out of training. It does not remove you from ChatGPT or Claude search answers, which the search crawlers govern.

Google-Extended is the one people miss. It is a robots.txt token with no crawler of its own, so it never appears in your logs. It governs whether content Google crawls is used to train Gemini and to ground answers in the Gemini app. Blocking it does not affect your normal Google rankings, and allowing it does not help them either. AI Overviews and AI Mode follow your Googlebot rules and snippet controls instead.

User-triggered agents fetch a page when someone asks an assistant to read it: ChatGPT-User, Claude-User and Perplexity-User. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Anthropic says Claude-User follows them.

Reading what you have

A user-agent block naming a crawler followed by Disallow: / blocks it entirely. A crawler you do not name follows your User-agent: * group, so it is allowed unless that group disallows it. A crawler you do name ignores the * group completely, so repeat any rules you still want it to follow. A wildcard User-agent: * with Disallow: / blocks every crawler without a group of its own, which occasionally survives from a staging configuration and is worth checking for on its own merits.

Making the decision

Publishers whose product is the writing have a commercial argument for blocking, because an answer built on their work substitutes for the visit. Brands that sell something further down the funnel rarely gain from blocking the search crawlers, because absence from the answer means absence from the shortlist. The training crawlers are a separate call.

Write down which crawlers you allow and why, and set a date to review the list. The case for each side is argued in Is your robots.txt blocking the AI crawlers you want?

After you change it

Changes are read within about a day. OpenAI says its search results take around 24 hours to reflect a robots.txt update, Perplexity says up to 24 hours, and Google generally caches the file for up to 24 hours. Pages still have to be crawled after that, so give it a few weeks before judging the effect, and track whether your brand starts appearing in AI answers for your priority questions.