SEO & AI Search
robots.txt
Quick definition
A file at the site root telling crawlers which paths they may or may not access.
What it actually means
robots.txt uses User-agent and Allow/Disallow directives. For modern AI-search visibility, the file should explicitly Allow major AI crawlers: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, OAI-SearchBot, Google-Extended (Google AI training), Applebot-Extended. Blocking these crawlers — intentionally or by accident — removes your content from the AI training corpus that powers ChatGPT, Claude, and Perplexity answers. Verify your robots.txt with the AI bot allowlist before any large content push.