Robots.txt Generator
Generate a clean robots.txt to guide search crawlers.
Use * for all crawlers.
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Sitemap: https://example.com/sitemap.xmlAbout the Robots.txt Generator
Create a correct robots.txt file free, without memorizing the syntax. Choose which crawlers the rules apply to, add Disallow paths (like /admin or /cart) and Allow exceptions, reference your sitemap, and copy a valid file ready to upload to your site root. A well-formed robots.txt keeps crawl budget focused on the pages you want in Google — and one syntax mistake can hide your whole site, so let the generator get it right.
How to use the Robots.txt Generator
- 1Set the user-agent (use * for all crawlers).
- 2Add the paths you want to disallow, and any allow exceptions.
- 3Add your sitemap URL.
- 4Copy the result and upload it as robots.txt at your site root.
Frequently asked questions
What is a robots.txt file?
robots.txt is a file at your domain root that tells crawlers which paths they may or may not request. It guides crawling, not indexing.
Does Disallow remove a page from Google?
No. Disallow blocks crawling but a blocked URL can still be indexed if linked elsewhere. Use a noindex meta tag to keep pages out of the index.
Where does robots.txt go?
At your domain root — yoursite.com/robots.txt. Crawlers only look there; a robots.txt in a subfolder is ignored.