robots.txt Generator

Build a robots.txt without accidentally hiding the whole site.

What you provide

Ignored by Google. Honoured by some other crawlers.

Result

robots.txt with 3 disallow rules

robots.txt controls crawling, not indexing, and the difference matters more than it sounds. A page blocked here can still appear in results if other sites link to it, because the crawler is told not to fetch it but is not told to forget it exists. To keep a page out of the index, let it be crawled and serve a noindex directive. Blocking it here actually prevents that directive from ever being seen. The file must sit at the root of the domain, applies only to that host and scheme, and is a public document that anyone can read.

Save as /robots.txt at the site root
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search
Diagnostic telemetry
Mode
Open with exceptions
Disallow rules
3
Allow rules
0
Sitemap declared
no
Model training crawlers
not blocked
Lines
4

What this cannot tell you

  • Generates a file from your choices. It does not upload anything or check what your site currently serves.
  • robots.txt is a request, not an access control. Well-behaved crawlers honour it and nothing stops anyone else, so it must never be used to protect anything private.

Take this with you

robots.txt Generator on RunTheTests — free browser-based seo tools, no sign-up.