Robots.txt Generator
Robots.txt generator
A robots.txt file at your site root tells crawlers what they may access. User-agent names the crawler, Disallow/Allow control paths, and a Sitemap line points to your XML sitemap. Configure it below and copy the result to /robots.txt. (An empty Disallow allows everything.)
robots.txt
Frequently asked questions
What is a robots.txt file?
A file at your site root telling crawlers which paths they may crawl — via User-agent, Disallow/Allow, and a Sitemap line.
How do I allow all crawlers?
"User-agent: *" then an empty "Disallow:". Empty Disallow blocks nothing. Add a Sitemap line to help discovery.
Does robots.txt stop indexing?
Not reliably — Disallow stops crawling, but a blocked page can still be indexed if linked elsewhere. Use a noindex meta tag to keep a page out of results.
Place the output at yourdomain.com/robots.txt. Test it in Google Search Console's robots.txt report before relying on it.