OpenAI introduced GPTBot in August 2023. If nobody has revisited your robots.txt since, it is making decisions about your AI visibility that nobody took.
The file is older than the problem
Ask when your robots.txt was last edited. If the answer is a replatform from before GPTBot existed, the file was written to manage crawl budget for Googlebot and Bingbot. It takes no view on whether a language model should read your product pages, because nobody asked it to.
The crawlers worth a decision
The distinction that matters is between crawlers that train models and crawlers that fetch pages for answers. OpenAI, Anthropic and Perplexity document them separately, and each can be allowed or blocked on its own. Blocking GPTBot or ClaudeBot keeps your content out of future training. It does not take you out of ChatGPT or Claude search answers, which OAI-SearchBot and Claude-SearchBot govern. PerplexityBot does the search job for Perplexity and, by Perplexity's account, is not used for training.
Google's training control is Google-Extended. It controls whether your content trains and grounds Gemini, and it is separate from Googlebot, so blocking it does not affect your normal search rankings. Nor does it govern AI Overviews, which follow your Googlebot rules. Our guide How to audit your robots.txt for AI crawlers lists each agent and what it controls.
There is no universally right answer
A publisher whose product is the words has a real argument for blocking. A brand whose product is sold further down the funnel rarely has one for blocking the search crawlers, because being absent from the answer means being absent from the consideration set entirely. Training is a separate question, closer to licensing, and reasonable brands answer it both ways.
What is never right is a default nobody chose. If you block Google-Extended without knowing it, you are opting out of Gemini's grounding for no reason. If you allow everything without knowing it, you have made a licensing decision by accident.
The check takes ten minutes
Open your robots.txt and look for user-agent lines naming these crawlers. If none appear, they follow your User-agent: * rules, so they are allowed unless that group blocks them. Decide whether that is what you want, write it down with the reasoning, and revisit it when the commercial picture changes.
Either answer can be right, provided somebody made the call deliberately.
