Robots.txt Generator
Generate a clean robots.txt to guide search crawlers.
Sign in to use this tool
Create a free account to unlock every tool — plus a 14-day free trial of HQ Pro, no card required.
About the Robots.txt Generator
Create a correct robots.txt file free, without memorizing the syntax. Choose which crawlers the rules apply to, add Disallow paths (like /admin or /cart) and Allow exceptions, reference your sitemap, and copy a valid file ready to upload to your site root. A well-formed robots.txt keeps crawl budget focused on the pages you want in Google — and one syntax mistake can hide your whole site, so let the generator get it right.
How to use the Robots.txt Generator
- 01Set the user-agent (use * for all crawlers).
- 02Add the paths you want to disallow, and any allow exceptions.
- 03Add your sitemap URL.
- 04Copy the result and upload it as robots.txt at your site root.
Frequently asked questions
What is a robots.txt file?
robots.txt is a file at your domain root that tells crawlers which paths they may or may not request. It guides crawling, not indexing.
Does Disallow remove a page from Google?
No. Disallow blocks crawling but a blocked URL can still be indexed if linked elsewhere. Use a noindex meta tag to keep pages out of the index.
Where does robots.txt go?
At your domain root — yoursite.com/robots.txt. Crawlers only look there; a robots.txt in a subfolder is ignored.