Robots.txt Generator
Build a robots.txt file visually — add allow/disallow rules, block AI scrapers, set a sitemap URL, and preview the output in real time. Your data never leaves your device.
Quick Presets
Global Settings
Add Rule
1 rule
User-agent: *Disallow
robots.txt preview
All rules are generated locally in your browser. Nothing is uploaded or stored.
Frequently Asked Questions
- Is the Robots.txt Generator free?
- Yes, 100% free. No account, no limits, and no sign-up required.
- Are my settings safe?
- All processing runs in your browser. Nothing is uploaded to any server — your robots.txt is generated entirely on your device.
- What does robots.txt do?
- The robots.txt file tells search engine crawlers and bots which pages or sections of your site they are allowed or not allowed to crawl. It is placed at the root of your domain (e.g. https://example.com/robots.txt).
- How do I block AI scrapers with robots.txt?
- Use the 'Block AI Scrapers' preset, which adds Disallow rules for GPTBot, CCBot, Claude-Web, anthropic-ai, Google-Extended, and cohere-ai. These are the main bots used by AI companies to train their models.
- What is crawl-delay in robots.txt?
- Crawl-delay tells bots how many seconds to wait between consecutive requests to your server. It helps reduce server load. Note: Googlebot ignores this directive — use Google Search Console to control Googlebot's crawl rate instead.
- Does it work on mobile?
- Yes, the tool works on all modern browsers including mobile Safari and Chrome on iOS and Android.