SEO Generators
Robots.txt Generator
Build a compliant robots.txt with per-user-agent rules, sitemap reference and crawl-delay — copy or download ready to deploy.
Allow crawling (uncheck to block everything)
Output
User-agent: * Disallow:
About the Robots.txt Generator
robots.txt is the first file a crawler requests, and a single misplaced slash can deindex an entire site. Disallow: / blocks everything; Disallow: /admin blocks one directory.
This generator builds a syntactically valid file from per-user-agent rules, adds your sitemap reference, and lets you download it ready for deployment.
How to use the Robots.txt Generator
- Start with a User-agent: * group unless you need different rules per crawler.
- Add Disallow lines for paths that should not be crawled: admin areas, internal search, cart and checkout, faceted filter URLs.
- Add the absolute URL of your sitemap — the Sitemap directive requires a full https:// URL, not a path.
- Save the output as robots.txt in your site root and confirm it loads at yourdomain.com/robots.txt.