AI Crawler Guide — manage GPTBot, ClaudeBot and others
The guide explains differences between AI crawlers and helps prepare appropriate robots.txt rules.
Crawler overview
- GPTBot and other OpenAI crawlers
- ClaudeBot, Perplexity and Google-Extended
- Differences between training and content retrieval
Configuration
- Choose allowed and disallowed bots
- Generate robots.txt rules
- Review the resulting configuration before publishing