RunTheTests
/

Meta Robots & Indexability Checker

Decode every robots directive on a page and find the one keeping it out of search.

The page to audit

The page is fetched once from a Cloudflare edge and its markup analysed here in your browser.

How this check works

The page is fetched once and both places a robots directive can live are read: the meta tag in the markup and the X-Robots-Tag response header. The header is the one people forget. It never appears in the page source, view-source shows nothing wrong, and yet it carries exactly the same authority — a noindex in either place is enough, and when the two disagree the more restrictive one wins. Alongside the directives, the check reads the signals that decide how an indexable page is presented: the canonical link, the title, the description and the heading structure. What it deliberately does not do is fetch robots.txt, because robots.txt answers a different question. Disallow stops a crawler from requesting the page; it says nothing about whether the address may appear in results, which is why a blocked page can still surface as a bare URL with no snippet.

What the results mean

noindex
The page asks to be left out of search results entirely. Deliberate on a staging site or a thank-you page; a silent catastrophe when a template ships it to production.
nofollow (as a robots value)
Asks engines not to follow any link on the page. Distinct from rel="nofollow" on a single anchor — this one applies to every link at once.
noarchive / nosnippet / max-snippet
Presentation controls, not indexing controls. The page still appears; these limit the cached copy or how much text the result may quote.
X-Robots-Tag
A robots directive sent as an HTTP response header instead of markup. Common on PDFs and files that have no HTML to carry a meta tag — and on pages where a server rule was applied wider than intended.
No directive at all
The default is index, follow. An absent robots tag is not a problem to fix; adding one that says index changes nothing.

Common problems and fixes

The page is blocked in robots.txt but still shows up in Google
Working as designed. Disallow prevents crawling, not indexing — the address can still be listed from links alone, as a bare URL. To remove a page from results it needs a crawlable noindex, which means unblocking it in robots.txt so the directive can actually be read.
View-source shows no noindex, but the page will not index
Check the X-Robots-Tag response header — this tool reads it for exactly that case. A server or CDN rule can attach noindex to whole path patterns, and it is invisible in the markup.
We removed the noindex weeks ago and the page is still missing
The directive only takes effect when the page is next crawled, and recrawl of an unimportant-looking page can take weeks. Request indexing in Search Console to shortcut the wait, and confirm the removal actually reached production rather than staying on staging.
Search Console says crawled, currently not indexed — but there is no directive
Then indexing is permitted and the engine is declining anyway, which is a quality judgement rather than a technical block. Thin, near-duplicate or doorway-like pages sit in this state indefinitely. No robots directive causes it and none will fix it.

Frequently asked questions

What is the difference between noindex and robots.txt disallow?

They act at different stages. Disallow in robots.txt stops the crawler requesting the page at all; noindex lets it crawl but asks it not to list the page in results. Combining them is the classic mistake: once the page is blocked from crawling, the noindex can never be read, so the address may linger in results as a bare URL. To de-index a page, let it be crawled and serve noindex.

Does a noindex page pass link value?

Initially yes, but not forever. Once a page has carried noindex long enough, search engines stop crawling it and treat its links as they would nofollow. A page meant to accumulate and pass authority should not be noindexed, and a page meant to be excluded should not be relied on for internal linking.

How do I put a robots directive on a PDF or an image?

Files without HTML cannot carry a meta tag, which is precisely what the X-Robots-Tag header is for. It is set in the server or CDN configuration for the file's path, and this checker reads it from the response the same way a crawler does.

My page has no robots tag at all — should I add one?

No. The absence of a directive means index, follow; that is the default and stating it explicitly changes nothing. Robots tags are for saying no. The only reason to add one is to restrict something.

More in SEO