About Robots.txt Generator
What It Does
The Robots.txt Generator creates a robots.txt file that tells search engine crawlers which parts of your website they can and cannot access. Place the generated file at your domain's root to control crawling behavior.
Features
- Quick presets — allow all crawlers, block all, block AI training bots, or use WordPress defaults
- Multiple user-agent groups — set different rules for Googlebot, Bingbot, GPTBot, etc.
- Allow and Disallow rules — control access to specific paths
- Crawl-delay — optional delay between crawler requests (supported by Bing, Yandex)
- Sitemap reference — point crawlers to your XML sitemap
- Download — save as a ready-to-upload robots.txt file
Common Use Cases
- Block AI bots (GPTBot, CCBot, ClaudeBot) from training on your content
- Hide admin, staging, or private directories from search results
- Allow search engines while blocking scrapers
- WordPress sites: block wp-admin, wp-includes, and feed paths
Privacy
Everything runs in your browser. No data is sent to any server.