RunTheTests
/

robots.txt Generator

Build a robots.txt without accidentally hiding the whole site.

What you provide

Ignored by Google. Honoured by some other crawlers.

Result

robots.txt with 3 disallow rules

robots.txt controls crawling, not indexing, and the difference matters more than it sounds. A page blocked here can still appear in results if other sites link to it, because the crawler is told not to fetch it but is not told to forget it exists. To keep a page out of the index, let it be crawled and serve a noindex directive. Blocking it here actually prevents that directive from ever being seen. The file must sit at the root of the domain, applies only to that host and scheme, and is a public document that anyone can read.

Save as /robots.txt at the site root
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search
Diagnostic telemetry
Mode
Open with exceptions
Disallow rules
3
Allow rules
0
Sitemap declared
no
Model training crawlers
not blocked
Lines
4

What this cannot tell you

  • Generates a file from your choices. It does not upload anything or check what your site currently serves.
  • robots.txt is a request, not an access control. Well-behaved crawlers honour it and nothing stops anyone else, so it must never be used to protect anything private.

Take this with you

How this calculation works

The generator writes a standard robots.txt with a wildcard user-agent group, your allow and disallow paths, and a sitemap declaration. Paths are validated to start with a slash, since everything in this file is relative to the site root. The option to block model training crawlers adds a separate group for each of the named agents, because they only honour a rule addressed to them by name rather than the wildcard group.

What the results mean

Disallow
Asks crawlers not to fetch matching paths. It controls crawling only, and a disallowed page can still appear in results if other sites link to it.
Allow
Carves an exception out of a broader disallow. The more specific rule wins, so allowing a subfolder inside a blocked folder works as expected.
Sitemap
Tells crawlers where your sitemap lives. It applies site-wide regardless of which user-agent group it appears in.

Common problems and fixes

The site vanished from search results after a launch
Check for a Disallow: / left over from staging. This is the single most common cause, and it happens when a staging configuration is deployed to production. Remove the rule, then request re-crawling of the important pages rather than waiting, since recovery is otherwise slow.
A page is blocked here but still shows in results
That is robots.txt working as designed. Blocking crawling does not remove a page from the index, and a page with external links can be listed from those links alone, usually without a description. To remove it properly, allow crawling and serve a noindex directive, or return a 404 or 410. Blocking the crawl actively prevents the noindex from ever being read.

Frequently asked questions

Does robots.txt keep a page private?

Not at all. The file is public, anyone can read it, and it names exactly the paths you would rather nobody looked at. Compliant crawlers respect it and nothing else has to. Anything genuinely private needs authentication.

Where does the file have to live?

At the root of the host, as /robots.txt, and it applies only to that exact host and scheme. A file on example.com does not cover shop.example.com, and each subdomain needs its own.

Should I block model training crawlers?

It depends on what your content is for and how it earns its keep. Blocking them means your material is less likely to be used for training, and also less likely to be cited in assistant answers that increasingly send traffic. The named agents honour a rule addressed specifically to them, which is why they appear as separate groups rather than under the wildcard.

Put this on your own site

Free to embed, no attribution required beyond the source link the frame carries itself. It runs entirely in your visitor's browser, sets no cookies and loads no third-party script.

Embed code
<iframe src="https://runthetests.com/embed/robots-txt-generator/" width="100%" height="560" style="border:1px solid #e5e5e5;border-radius:8px" title="robots.txt Generator" loading="lazy"></iframe>

Preview it at https://runthetests.com/embed/robots-txt-generator/. Embedded pages are marked noindex, so yours stays the canonical copy — not this one.

More in SEO