Indexing a large catalog of an online store is work on ensuring that each of the tens of thousands of pages is accessible to the bot, is not duplicated, and is rendered quickly. It begins not with a site map, but with a count: how many items are actually in the catalog and how many of them the search engine receives as a list. These two numbers rarely match. At the gear store we measured, the discrepancy between the sitemap and the standard search was 0.4% — and it's those percentages that show where the catalog is leaking.
we calculate the catalog, rebuild the site map, remove repetitions, add depth to the bypass
after a free audit
Price
30
Guarantee
days after project sign-off
we calculate the catalog, rebuild the site map, remove repetitions, add depth to the bypass
What are we doing?
15–45
The term of the first cycle
working days, then a monthly installment
54,302
The largest catalog in its own dimension
products and 635 categories
59,189
The largest site map in size
addresses: 18 subcards for 3,000 and another for 1,321
413.8
Speed by 54 thousand items
ms to the first byte - volume is not a sentence
number of pages in the index: this is the decision of the search engine
What we do not promise
free indexing slice, 3-5 business days
Before the estimate
Process steps
Transparent stages with approval at every step
Total duration:15–45 days
1
A slice of indexing status
3-5 working days
We count the catalog using two methods, we analyze the map by submaps, we look at repeats, language branches, update dates and response time. Everything with the date of measurement - there will be something to compare with later.
2
Comparison with what the search engine sees
2-4 working days
Search Console data is next to our external count. The main thing becomes visible here: how many addresses you offer to the bot and how many of them actually reached the index.
3
Reconstruction of the site map
3-8 working days
Index file, reasonably sized submaps, last modified working dates, all language branches, map declaration in robots.txt. One entry per address, no exceptions.
4
Duplication cleaning and canonicalization
4-12 working days
Duplication patterns—different category paths, filter parameters, language prefixes—are covered by rules. We warn you in advance: the number of pages in the index will decrease for some time, and this is expected.
5
Depth reach and speed
3-10 working days
Relinking to deep nodes, cache of categories, card return under bypass. We measure on real addresses from the end of the directory, not on the main one, because the main one is almost always fast.
6
Monthly slice
monthly, from launch
Five numbers once a month: addresses in the map, unique addresses, pages in the index, time of appearance of a new product, response time. Catches a broken regen before it eats a block.
Technologies & integrations
What we build on and what it connects to
Stack
sitemap-index from submaps: format limit — 50,000 addresses per file, but cut earlier, 3,000–3,500
lastmod: only works as long as the dates actually change - a frozen date is worse than no date at all
bypassing the map with a script: actually counts the addresses; on tens of thousands of addresses, it is done slowly, so as not to put the site down
standard CMS search as the second method: gives a number where the search gives a full output, and is silent where it does not
Search Console: the only source of the number of indexed; data with download delay and limitation
Screaming Frog: shows the depth of clicks; does not see pages to which there are no links
structured data Product and Offer: change the appearance of the snippet, not items
nginx and category cache: keep yield under bypass but hide slow requests instead of treating them
Integrations
Google Search Console
Bing Webmaster Tools
Google Merchant Center
GA4
Cloudflare
What's included
Complete list of work and what you get as a result
Calculating the directory by two independent methods — sitemap traversal and regular CMS search — with a resolution of the discrepancy: the size of the directory ceases to be a matter of trust in the admin report
Sitemap as an index of submaps of several thousand addresses: the file is checked by eyes, quickly generated and updated in parts
Last Modified Dates That Really Change: A date set blindly by the generator invalidates the entire file
All language branches in the map — or a separate file for each language so that the second version is not left out of the list
Cleanup of repetitions: one address - one record, so that the detour goes to new pages, and not to the tenth repetition of a known
Canonical URLs for Filters and Sorts - options stop multiplying copies of the same category
Relinking based on depth: the path to the card from the end of the directory should be several clicks, not just a line in the site map
Directory throughput under traversal load - category cache and card response time measured on real deep addresses, not root
Structured data Product, Offer and breadcrumbs on cards: it affects the appearance of the snippet, but not the position
Monthly slice of five numbers: addresses in the map, unique addresses, pages in the index, time of appearance of a new product, response time
When this service isn't right
What's not included — so there are no surprises at delivery
Writing descriptions for thousands of cards is a separate job with a separate budget
Guarantee of the number of pages in the index: the decision is made by the search engine, not the contractor
Completion of the catalog and expansion of the assortment
Advertising and buying traffic for the period until the directory reaches the index
Rewriting the engine: if the CMS does not know how to render the correct map, this is a development task
Who it's for
Situations where this service delivers results
Scenario 1 of 4
The catalog is growing faster than the search engine bypasses it
New positions are added every week, and last year's cut lives on in the search. The reason is usually not in the budget: the bot goes in circles to the same addresses, because the site map gives them several times, and new pages lie without any internal link. First, we measure how much repetition eats up the bypass, and only then we talk about the rest.
We'll review your situation in a free audit
Scenario 2 of 4
Tens of thousands of items and one map file for everything
The format allows 50,000 addresses in a file, but a file at the limit is inconvenient for everyone: it takes a long time to generate, it is difficult to check, it is not updated in parts. In the largest of the catalogs we measured, the map is assembled from 19 submaps — 18 for 3,000 addresses and one for 1,321. Such files are read by eyes and regenerated in pieces.
Scenario 3 of 4
The catalog is doubled in languages
Two language branches are not two catalogs in the work, but a double list of addresses and a separate reconciliation. In the equipment store for 54,302 products, the card holds 110,096 addresses precisely because of the second language. There we also found that the language codes in the markup were set incorrectly: both branches exist, but they are declared to the search engine as one language.
Scenario 4 of 4
The site map is formally available, but the bot does not receive a list of addresses
A state we've seen from the outside more than once: /sitemap.xml returns code 200 with an empty body. The "address opens" check does not catch it - you need to look at the body of the response. There are 5,561 items in the professional cosmetics store: the catalog is live, the file is empty, and the bot cannot find out about the products from anywhere.
Work on indexing vs. "let's put a map generation module"
How this option differs from the alternative
Site map box moduleOur approach
Directory sizetaken from the admin's report: as much as in the database, as much in the filewe calculate by two methods and show the difference between them
Repeatthe file includes all category paths, so the product is included several timesone entry per address, duplicate patterns are closed by rules
Language versionsoften only the default languageall branches in the map or a separate file per language, language codes are reconciled
Checking the resultlooking at the response code: 200 with an empty body is considered a working cardwe read the body of the answer and count the records
What's nextthe file lives until the generator breaks — and no one knows about itfive figures every month and analysis of changes
A free slice of your directory's indexing status
In a catalog of tens of thousands of items, the most difficult question is simple: how many of them are generally available to the search engine. The answer can be obtained from the outside - and we do this before talking about money, because it is this figure that determines the amount of work.
What we measure
Directory size by two methodsWe count on the site map and cross-check with standard search, then show the discrepancy. In the equipment store, it was 0.4% — and it is this difference that indicates a hidden problem.
Site map structureIs it divided into sub-maps, how many addresses are in each, does the file not rest against the format limits.
RepeatHow many entries per unique address. In a large directory, this is the main consumer of the bypass.
Catalog return speedAnswer time on the card and in the category. A reference from the measurement: a catalog with 54,302 products gives the first byte in 413.8 ms — the size does not oblige the site to be slow.
Language branchesIs the catalog duplicated by languages, are all versions present in the map, and are the language codes declared correctly?
Dates of last modificationAre they actually updating or have they frozen since the last generation failure.
What you get
The size of the directory calculated by two independent methods, with the difference between them.
State of the site map: submaps, addresses, repeats, dates.
A list of what prevents the bot from going deep into the directory, in order of impact.
Conversation for 40 minutes on the document.
Timeline: 3-5 working days
Why is it free
Because this is measured work, and without numbers, both parties are trading blindly. We count your catalog ourselves, from the outside, and show exactly how we counted - the method is verified.
What's next
Next is an estimate by stages with calendar dates. We warn honestly: the site map does not guarantee indexing, and 55 thousand addresses in it do not mean 55 thousand pages in the index.
Short form: your contact and site URL
Did not find your case?
Describe how it works on your side — we will tell you whether “Indexing a large directory” fits and what it means in your situation. No brief and no call: one question, one answer.
Contract, act and 30-day warranty
Every project gets a written contract: scope, deadlines, amount, acceptance procedure. After delivery — act and invoice, then 30 calendar days of warranty.
Sole proprietor & bank transfer
The contractor is a registered sole proprietor. Payment by invoice with closing documents.
Rights & access — yours
Code, design and materials transfer to you after full payment. Domain, hosting, repository and analytics are registered to you.
Client portal instead of email chains
During the project you get access to a portal: contracts, invoices, acts and project status in one place.
European clients
Among our work — projects for Norway, Bulgaria, Moldova and Spain.
Verifiable numbers
Every case in the portfolio comes with a link to a live site and a technical measurement.
Audit first, then pricing
There is no price list on the site intentionally: the scope of the same work differs multiples between clients.
We say "no" when unsure
If the task isn't ours or the deadline is unrealistic — we tell you upfront.
What affects the price
Why two seemingly identical tasks are priced differently
The actual size of the directoryNot the one in the admin, but the one visible from the outside. The difference between 5,561 items and 54,302 is a difference in the number of map files, traversal time, and reconciliation volume, not a percentage of the same job.
How many repetitions per addressRepetitions are not done piecemeal: they fall into patterns, and one rule is written on the pattern. But first these patterns need to be found, and this is a traversal of the entire map, not a sample of ten addresses.
Card generator statusThe file is built in seconds - we work with the existing one. It gives an empty body or assembles in tens of seconds - first we fix the generator itself, and this is a separate line of the estimate.
The number of language branchesEach branch doubles the list of addresses and adds a check: whether all versions are in the map, whether the language codes are correctly declared. In a store with 54,302 products, it is the second language that gives 110,096 addresses in the file.
Recoil speed under bypassThe directory responsible for 413.8 ms, the bot bypasses as calmly as a small site. A catalog with an 840.5ms response must be accelerated first, otherwise the traversal will limit itself.
Access to Search ConsoleThere is access — we can see how many pages are actually in the index, and we work by the number. There is none - we work on what is visible from the outside, and part of the result remains unverified.
Cases
Tasks and results in numbers — all metrics measured by us
online store of tactical equipment, catalog of more than 50,000 items
Task
Confirm the real size of the directory and check how it is presented to the search engine.
Solution
Bypassing the index file of the site map with a selective check of submaps — the first, sixteenth, thirty-first, and last; cross-counting by full-time search of 100 items per page.
Result
54,302 products and 635 categories. The map is assembled from 32 commodity files: the checked submaps contain exactly 3,500 addresses each, the last one — 104; a total of 110,096 addresses in two languages. Cross-validation by search yielded 54,501–54,600 positions, a discrepancy with the map of 0.4%. First response 413.8 ms, full download 511.1 ms, nine types of structured data. A defect was found: the language codes in the markup are set incorrectly - two branches exist, but they are declared as one language. Measured on 07/31/2026.
online store of garden equipment and spare parts, the largest site map in the selection
Task
Count the spare parts catalog that does not fit into one sitemap file.
Solution
We actually went through the submaps, page by page, checked the map announcement in robots.txt and took the response time on the most difficult pages.
Result
55,321 commodity addresses — 18 subcards of 3,000 and 1,321 in the nineteenth — plus 3,868 categorical: at least 59,189 addresses. The map is explicitly declared in robots.txt, a markup of six types of structured data. The first response of 840.5 ms with 454.5 KB of markup is the slowest result of the batch and the first point of work. Separately, the stack age is visible: PHP 7.3.33, not supported since December 2021. Measured on 07/31/2026.
online store of professional cosmetics, catalog of over 5,000 items
Task
Find out why a catalog of this size is almost not represented in search.
Solution
Counting the directory by regular CMS search, checking the body of the /sitemap.xml response and the composition of the counters on the main page.
Result
5,561 products in the live catalog — 55 search pages for 100 items and 61 for the last one. At the same time, /sitemap.xml returns code 200 with a body of zero bytes: the search engine does not receive a list of addresses at all. One language version with no language markup, no analytics counter on master, first response 801.2ms with 153.8KB of markup. Measured on 07/31/2026.
What we need from you
We can't start without this — best to prepare in advance
1Site address and sitemap address - this is enough to make the first cut without any accesses.
2Access to Search Console: no one, including us, can see the real number of indexed pages without it.
3Access to CMS or your developer to change map generation, give updates and canonical addresses.
4Understanding how often the catalog is updated and how many items are added per week: the frequency of regeneration depends on this.
5The decision of what to do with product variants — separate addresses for size and color or parameters of one card. This is a business decision, not a technical one.
6One person on your side with the right to approve changes in the address structure.
If something is missing — let us know, we'll help you gather it or do it as a separate task.
FAQ
Most frequently asked questions — with concrete answers
How do you calculate the size of the catalog - from my words?
No, by two independent methods. The first is a walkthrough of the site map with a count of addresses in each submap. The second is a standard CMS search with an empty query and the maximum number of items per page. Then we show the discrepancy. In the tactical equipment store, the map returned 54,302 items, the search returned 54,501–54,600 items: a discrepancy of 0.4%. This difference is interesting, because it shows how many products live in the catalog, but did not make it to the list for the bot.
Is a large catalog necessarily slow?
No. A catalog of 54,302 products renders the first byte in 413.8 ms and complete HTML in 511.1 ms. A catalog for 5,561 items in the same sample — 801.2 ms. Pulls down the template, module set, and stack age, not the number of rows in the database. The slowest directory measured has PHP 7.3.33 under the hood, a version that was deprecated in December 2021.
How many addresses to put in one map file?
The format limit is 50,000 addresses per file, but you should split it earlier. In the measured directories, submaps are made for 3,000 and 3,500 addresses, and this is a convenient size: the file is quickly assembled, it can be opened and counted by eye, and when a part of the directory is changed, only the necessary piece has to be regenerated. Our recommendation is to keep the submap within three to four thousand addresses, even when the format allows ten times more.
Will all products be included in the index?
No, and it would be untrue to promise such a thing. A sitemap is a request to be crawled, not a guarantee of indexation, Google writes about it directly. A card without description, without distinction from neighboring cards and without demand may remain outside the index during any technical work. Our part: to make every page accessible, non-duplicated, and quickly delivered. The decision about the index is left to the search engine.
Our sitemap opens, so everything is fine?
It doesn't mean It is not the response code that should be checked, but the body. In the store of professional cosmetics, /sitemap.xml returns a code of 200 and zero bytes: the browser shows an empty page, monitoring considers the site to be working, and the bot does not receive any address from the directory for 5,561 items. This is the cheapest breakdown of all and the most expensive in terms of consequences, because it goes unnoticed for months.
Does it make sense to index every position of a large directory?
Not always. When the addresses are entered at the level of size and color, the number of pages grows many times without an increase in demand: a hundred variants of one model is a hundred almost identical pages competing with each other. It is often more correct to keep the model in the index, and leave the options with the card parameters. The decision here is made by the business, because it is about the assortment; we only show what it costs in the bypass.
How often to check indexing status?
In the big catalog - every month. The status changes by itself: products are added, old ones disappear, after updating the CMS, the regeneration of the map breaks. Then the file remains in place, and the dates in it freeze, and outwardly everything looks normal. A five-figure slice once a month takes little time and catches this one before it eats a quarter.
After cleaning the duplicates, will the number of pages in the index drop?
Most likely yes, and this is the expected course. When one card is accessible in three ways, three weak pages lived in the index instead of one strong one. After cleaning, one thing remains: the number of "pages in the index" decreases, and the traversal stops being spent on repetitions. Therefore, we fix it to the works and show it next to the number of unique addresses - otherwise the normal course of events is read as a deterioration.
In response, how many items are in the catalog according to two counting methods, how many addresses are in the map, how many of them are duplicates, and what prevents the bot from reaching the depth. If the catalog is small, we will say so in the first email and we will not sell extra.
From measured cases54,302 products and 635 categories
There is no price list on the site on purpose: the same work differs several times over between two clients, and a “from” figure explains nothing in that case. First a free audit — we count your pages, duplicates and speed — then we name the sum and the deadline and fix both in the contract.