The worst news in an indexing report is never written there as a number. It is where the number is missing.
On 21 August 2026 we were looking at Search Console for our own site. Indexed: 262 pages. Not indexed: 529. The first instinct with a report like that is to attack the biggest pile. The biggest pile looked like this:
| Reason | Pages |
|---|---|
| excluded by noindex tag | 420 |
| blocked by robots.txt | 25 |
| discovered, currently not indexed | 37 |
| crawled, currently not indexed | 24 |
| not found (404) | 11 |
| page with redirect | 10 |
| alternate page with canonical | 2 |
Four hundred and twenty pages with noindex. It looks like a disaster demanding immediate action. In fact it is the healthiest line in the whole report: the site closed these addresses itself, the crawler saw the instruction and obeyed. The system works as designed.
The problem sat two rows below.
Thirty-seven addresses the crawler never reached
Under «discovered, currently not indexed» sat the About page and the portfolio page. Both are in the main menu. Google knew they existed and had never once spent a request on them.
Meanwhile it was diligently walking 420 addresses marked «do not index».
That is the real find in the report. Not «lots of noindex», but the distribution of attention: the crawler discovered the closed pages earlier and judged them more worth fetching than two pages linked from every page of the site.
Where it gets them from
The answer was in a single file. We have a utility page — a service map, a full list of everything on the site. It carries 296 links and weighs 887 KB.
For a crawler that is the most convenient entry point: one page showing everything. It crawls it, takes hundreds of addresses away and starts walking them — including the ones closed from indexing. Crawling is spent, the index does not grow.
Importantly, the page itself is not a mistake. The mistake is assuming that «let the crawler see everything» is free. Every link you expose is a request for crawling, and the crawler fulfils those in its own order, not yours.
A sitemap that contradicted its own pages
Four days earlier, on 17 August, we found something worse than uneven distribution: four pages sat in the sitemap and served noindex at the same time.
Those are directly contradictory instructions. The sitemap says «here is an address, please crawl it». The page says «do not index me». In Search Console such an address settles into the errors, and rightly so — the site is contradicting itself.
The cause turned out to be mundane and instructive. We have a rule: a section with fewer than three pages is not indexed — too thin to be useful. A reasonable rule. But the threshold «three» was written as a constant inside the page's code, and the sitemap build knew nothing about it and added everything.
The rule left behind: any indexing threshold must be one value, shared by the page and by the sitemap. Two copies of one constant in different files will drift apart — the only question is when. And they will drift silently: nothing breaks, tests pass, and the error surfaces weeks later in a search engine's report.
Eleven pages that no longer exist
A separate line in the report is the 404s. Eleven addresses that used to work.
Some of them are the result of ordinary tidying. We were removing numeric suffixes from a few addresses: …/consent-mode-v2-4 became …/consent-mode-v2. The new address works; checking the old one slipped our minds. A few more pages moved to another section — again with no redirect from the old place.
The rule here is easy to state and easy to forget: what you have to test is not that the new address opens but that the old one leads to it. The first is obvious and tests itself. The second goes unnoticed until you look at the report.
The second group of 404s is more interesting. These are addresses of the form /en/… written with Ukrainian slugs — pages that once existed in every language and later remained only in Ukrainian. Fifty-two of our two hundred and something pages have a single language version: there is no translation, and we chose not to publish a machine one. How language versions break from the inside has a breakdown of its own. Google remembers the old addresses and will keep visiting them for a long time.
How to look at the same thing on your site
1. Open the indexing report in full. Search Console, «Page indexing». Look not at the headline number but at the list of reasons, and read it from the bottom up — from the small categories to the large ones. The most important thing is almost always in «discovered, currently not indexed»: those are pages the crawler knows about and ignores.
2. Reconcile the sitemap with the state of the pages themselves. The simplest manual check:
curl -s https://your-site.com/sitemap.xml \
| grep -o '<loc>[^<]*' | sed 's/<loc>//' | head -50 \
| while read u; do
echo "$(curl -s "$u" | grep -c 'noindex') $u"
done
Any 1 in the first column is a page contradicting its own sitemap.
If there are many addresses, this is worth automating. We keep a script in the project that walks every address from every sitemap disguised as Googlebot and checks four things: response code, a robots.txt block, noindex in meta tags and in the X-Robots-Tag header, plus canonical. That last point about the header is not a detail: noindex can be delivered in the server response, and you will not see it in the page source at all.
3. Find your catalogue page. The one from which everything is visible. Count the links on it and see how many lead to addresses you never intended to index.
4. Test the old addresses, not the new ones. Make a list of everything you renamed or moved over the past year and walk the old addresses. Each should answer 301, not 404.
Three traps that cost us time
This is the part usually missing from articles, and it is the most useful.
Do not limit the search to the first kilobytes of a page. The first version of our script read 60 KB and reported two pages with no canonical. The canonical was there — it just sat after a large block of structured data. A false alarm produced by optimizing the measurement.
Do not measure everything at once in many threads. Sweeping three hundred addresses in sixteen threads with a twenty-second timeout gave us three false alarms in a row: the heaviest pages return 630–670 KB and do not finish in time, and truncated HTML reads as «page with no canonical and with noindex». Either fewer threads or a longer timeout.
A 502 during a sweep is often not a site defect. If you have just shipped an update, the container may have been restarting at that exact moment. Re-check before opening a ticket.
What all three share: the measuring instrument is wrong more often than the thing it measures. Before believing an alarming result, check the suspect address on its own, by hand.
Honest limits
If your site has fifty pages, none of the above applies to you. Google states plainly in its documentation that managing crawling is not a concern for most sites: the crawler comes often enough and the budget is sufficient. The topic switches on at a few hundred addresses and becomes important in the tens of thousands — that is, primarily in large catalogues.
Fixing the distribution of crawling adds no rankings and brings no traffic by itself. It does one thing: it removes the reasons why a page capable of working never enters the index at all. What happens next is still decided by what the page says and whether anyone is looking for it.
One more thing worth stating plainly: a page with noindex is fine. Utility sections, empty categories and filter duplicates should be closed. What is wrong is not that there are many of them, but that the crawler spends on them the time it owed to something else. That difference is visible only in the report — and only if you read past the first number.
