SEO & AI Search
robots.txt
Quick definition
A file at the site root telling crawlers which paths they may or may not access.
What it actually means
robots.txt uses User-agent and Allow/Disallow directives. For modern AI-search visibility, the file should explicitly Allow major AI crawlers: GPTBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot, OAI-SearchBot, Google-Extended for Google AI training, and Applebot-Extended. Blocking these crawlers, intentionally or by accident, removes your content from the AI training corpus that powers ChatGPT, Claude, and Perplexity answers. Verify your robots.txt with the AI bot allowlist before any large content push.