How to find noindex pages on your site

What does a noindex directive actually do to a page?

A noindex directive tells a search engine to crawl the page, read every word, and then keep it out of the results. It isn't an error, a penalty, or a crawl failure — it's an instruction being obeyed exactly as written. The directive lives in one of two places: a robots meta tag inside the HTML head, or an X-Robots-Tag in the HTTP response header. The two are completely independent of each other. Strip the meta tag out of your template and a header-based noindex keeps working precisely as before, which is why so many "I removed it and nothing changed" support tickets turn out to be header problems all along.

Here's the part that trips people up: robots.txt and noindex pull in opposite directions. A Disallow rule stops the crawl, so the crawler never sees the noindex sitting inside the page — and the URL can linger in results as a bare link with no description underneath. For a noindex to do its job, the page has to stay crawlable. The most common accidental source is a leftover from staging: a site-wide rule copied to production, a WordPress "Discourage search engines" checkbox that survived launch day, or a CDN rule adding a header to every response on a path nobody remembers configuring.

Plenty of pages deserve a noindex: thank-you pages, internal search results, cart and checkout steps, filtered listings, thin tag archives, and author pages on a one-author blog. My honest take is that most small sites carry too few noindex tags rather than too many — they let hundreds of near-empty archive URLs dilute the topical authority they worked hard to build. The danger was never the deliberate noindex. It's the accidental one sitting on a service page or a cornerstone article, quietly costing you rankings on Google and citations inside AI answers for months before anybody thinks to look.

A noindex directive tells a search engine to crawl the page, read every word, and then keep it out of the results. It isn't an error, a penalty, or a crawl failure — it's an instruction being obeyed exactly as written. The directive lives in one of two places: a robots meta tag inside the HTML head, or an X-Robots-Tag in the HTTP response header. The two are completely independent of each other. Strip the meta tag out of your template and a header-based noindex keeps working precisely as before, which is why so many "I removed it and nothing changed" support tickets turn out to be header problems all along.

How do I find noindex pages in Google Search Console?

Google Search Console hands you the list for free: open your property, click Pages under Indexing, scroll down to "Why pages aren't indexed," and click the row labelled Excluded by 'noindex' tag. That opens a detail view with the affected URLs and an export button for CSV or Sheets. This is the authoritative answer to one very specific question — which pages Googlebot crawled and was then told to skip. It reflects what Google actually saw on its most recent crawl, not what your CMS believes it published, and those two versions of reality disagree far more often than anyone expects.

Two settings change what you're actually looking at. The toggle at the top switches between All known pages and All submitted pages; submitted means URLs sitting in your sitemaps, and that's the higher-signal view, because anything you deliberately put in a sitemap and then noindexed is a straight contradiction worth resolving today. Remember too that the report lags — it's built from crawl data that can be several days old — and the URL examples are capped, so a large site sees a sample rather than the complete set. Fix the pattern behind the rows, not just the dozen URLs on screen.

To learn where the directive came from, paste a URL into the inspection bar at the top of Search Console. The Page indexing panel spells it out in plain language: "Indexing allowed? No: 'noindex' detected in 'robots' meta tag," or "detected in 'X-Robots-Tag' http header." That single line saves hours of guesswork, because it tells your developer which layer to open — the template or the server. Once you've made a change, run Test Live URL to see the current response instead of the cached crawl, then use Request Indexing to push the page back into the queue.

How do I check whether one specific page is noindexed?

For a single page, view the source and search it. Press Ctrl+U (Cmd+Option+U on a Mac) to open the raw HTML, then Ctrl+F for "noindex". If a robots meta tag is present, you'll see something like content="noindex, follow" sitting in the head, usually near the title and canonical tags. Finding nothing proves less than most people assume, though — it rules out only one of the two hiding places, and only within the HTML the server sent before any JavaScript ran. Two more checks are needed before you can honestly declare a page indexable.

Next, the response header. Open DevTools, switch to the Network tab, reload the page, click the very first row — the document request itself — and read the Response Headers for X-Robots-Tag. From a terminal, curl -I https://example.com/page/ does the same job in a single line and is easy to run across a list of URLs. This check matters most for files that can't hold a meta tag at all: PDFs, images, XML feeds. If a whole folder of downloadable guides has quietly disappeared from search, a header rule applied to that path is almost always the culprit.

Finally, compare the raw source against the rendered DOM. The Elements panel in DevTools shows the page after scripts have run, so a tag injected by a tag manager or a client-side framework appears there and nowhere else — view-source alone will never reveal it. A browser extension that surfaces robots directives condenses all three checks into one glance and earns its place if you audit pages weekly. Whatever you use, test the live production URL over HTTPS with the exact trailing slash and parameters, because a staging subdomain and a redirect target can return entirely different headers.

How do I find every noindex page across the whole site?

Crawl your own site and filter the results to noindex. A desktop crawler walks your internal links the way a bot does, records the robots meta tag and the X-Robots-Tag for every URL it touches, and lets you filter the Directives report to Noindex and export the whole lot to a spreadsheet. This is the only method that surfaces pages Google hasn't crawled yet — brand-new sections, recently shipped templates, entire subfolders that never made it into the index because the directive was baked in from day one and nobody ever saw a ranking drop to investigate.

The workflow itself takes minutes. Enter your root URL, start the crawl, open the Directives tab, choose the Noindex filter, and read the Meta Robots and X-Robots-Tag columns side by side so you can tell which layer produced each one. Free tiers typically cap a crawl at a few hundred URLs, so on a larger site start from a section URL rather than the homepage, or run several targeted crawls. Turn on JavaScript rendering if your site is a single-page app. And crawl your sitemap in list mode too, since an orphaned page with no internal links never shows up in a link-following crawl.

The site: operator is your thirty-second smoke test. Search site:yourdomain.com/blog/ and eyeball whether the result count lands anywhere near the number of posts you've actually published; a section that should hold 200 URLs and shows eleven has a template-level problem waiting for you. Treat that number as a rough estimate rather than a measurement — Google has said for years it's approximate, and it fluctuates between searches. It won't tell you why a page is missing, either, because a noindex, a canonical and a robots.txt block all look identical from the outside. Use it to spot the fire, then use the other methods to locate it.

What should I do once I find a page that's wrongly noindexed?

First decide whether the page belongs in the index at all. If it does, remove the directive at its source rather than papering over it, then verify with a live test and request indexing. Source means the layer that generates the tag: a per-post setting in your SEO plugin, a bulk rule applied to a post type or taxonomy, a hardcoded line in a theme template, a server configuration block, or a CDN rule writing the header on the way out. Flipping the post-level setting achieves nothing if the header is being added upstream of your CMS entirely.

Verify properly, and verify twice. Reload the live URL, confirm the meta tag is gone from both the raw source and the rendered DOM, and confirm curl -I no longer returns an X-Robots-Tag. Then run Test Live URL in Search Console until it reports that indexing is allowed. Recovery isn't instant: expect days for an important page and weeks for the long tail. You can speed it along by linking to the page from somewhere already crawled frequently — your homepage, a hub page, a recent post — and by resubmitting the sitemap. Rankings usually return close to where they were, though not always.

If the page genuinely shouldn't be indexed, leave the tag exactly where it is — and resist adding a robots.txt block on top, or the directive can never be read. The durable fix is turning this into a routine: an indexability check before anything publishes, plus a monthly crawl of the full domain. That's part of what we automate at Artiql — every article we research, write and publish to your own domain ships crawlable and indexable, and you can see how a page reads to answer engines with our GEO score check. Organic content compounds for years; a stray noindex is one of the few mistakes that zeroes it out completely. Book a quick demo and we'll look at your site together.

Frequently asked questions