SEO & AI Search

robots.txt

Quick definition

A file at the site root telling crawlers which paths they may or may not access.

What it actually means

robots.txt uses User-agent and Allow/Disallow directives. For modern AI-search visibility, the file should explicitly Allow major AI crawlers: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, OAI-SearchBot, Google-Extended (Google AI training), Applebot-Extended. Blocking these crawlers — intentionally or by accident — removes your content from the AI training corpus that powers ChatGPT, Claude, and Perplexity answers. Verify your robots.txt with the AI bot allowlist before any large content push.

Related terms

llms.txt

A proposed standard file at /llms.txt that gives AI agents a curated, condensed view of your most important content.

AI Search

Search delivered by AI tools like ChatGPT, Claude, Perplexity, Google AI Overviews, and Bing Copilot. Cites sources directly in answers.

Ready to see what your data is telling you?

Book a free strategy call. We'll analyze your current setup, identify the biggest opportunities, and show you exactly where the revenue is hiding.

Schedule a Free Strategy Call