Robots.txt Generator
Generate a clean robots.txt to guide search crawlers.
Free · No sign-up · Works in your browser
Use * for all crawlers.
robots.txt
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Sitemap: https://example.com/sitemap.xmlAbout the Robots.txt Generator
Free robots.txt generator — create a valid robots.txt with allow/disallow rules, crawl-delay, and your sitemap reference.
How to use the Robots.txt Generator
- 1Set the user-agent (use * for all crawlers).
- 2Add the paths you want to disallow, and any allow exceptions.
- 3Add your sitemap URL.
- 4Copy the result and upload it as robots.txt at your site root.
Frequently asked questions
What is a robots.txt file?
robots.txt is a file at your domain root that tells crawlers which paths they may or may not request. It guides crawling, not indexing.
Does Disallow remove a page from Google?
No. Disallow blocks crawling but a blocked URL can still be indexed if linked elsewhere. Use a noindex meta tag to keep pages out of the index.