Robots.txt Generator
Generate robots.txt rules for crawlers
Fill the user-agent, allow/disallow rules, and sitemap first to generate robots.txt; validation rules are available in Advanced mode.
The full guide also includes pitfalls, worked examples, snippets, FAQs, and related tools for checking results or troubleshooting.
About this tool
Draft one robots.txt user-agent group with one Allow or Disallow path per line and optional Sitemap and Crawl-delay lines. Paths without a leading slash produce warnings but remain in the output. The HTTPS sitemap option warns about HTTP; it does not block it. Deploy the reviewed text at the intended host’s /robots.txt. This tool does not test a live crawler or protect private content.
Production Snippets
A crawl rule is not a removal request
text
User-agent: *
Disallow: /drafts/
Sitemap: https://example.com/sitemap.xml
Crawlers can read this path rule from /robots.txt. To read a noindex on /drafts/, they must be allowed to fetch that page.Suggested Workflow
Frequently Asked Questions
Does Disallow keep a URL out of search or protect a private page?
No. It asks cooperating crawlers not to fetch the path. A blocked URL can still be known or indexed from links. Protect private content with authentication or access controls.
How should noindex work with robots.txt?
A crawler must be able to fetch the page to read its meta noindex or X-Robots-Tag. Blocking that URL in robots.txt can prevent the crawler from seeing noindex.
What do the path and sitemap fields accept?
Enter one path per line, including any deliberate * or $ pattern. The tool does not evaluate rule precedence or matching. Sitemap should be an absolute URL; multiple user-agent groups and multiple sitemap fields are not generated by this form.
Does Google honor Crawl-delay?
Googlebot does not support Crawl-delay. Other crawlers vary. When enabled, this tool converts a positive numeric input to a whole number by rounding down; it does not change a crawler’s actual request rate.
Keep browsing