Analyzing server logs for SEO: what the search bot is actually doing on your site
Analyzing server logs for SEO is the analysis of the access log, where each access to the site is recorded: address, response code, time, size, agent string. It answers a question that no other tool does: where the bot actually went, not where you invited it with a sitemap. Further, it can be seen from this what part of the bypass is eaten up by transitions and errors, which sections the bot did not reach in a month and at what time the server responds the slowest. Before work, we check for free whether your magazines are suitable: they live for a day on some hostings.
layout of the tour by sections, response codes and hours of the day
after a free audit
Price
30
Guarantee
days after project sign-off
layout of the tour by sections, response codes and hours of the day
What's out
week; the moon gives a picture you can believe
Minimum data period
7-18
The term of analysis
working days from receipt of magazines
free magazine eligibility check, 1-2 working days
Before the estimate
a reverse DNS query rather than trusting the agent string
Checking bots
items and traffic growth: the logs explain the bot, not the demand
What we do not promise
Free check if there is anything to analyze
The access log is the only place where you can see the actions of the bot, not our assumptions about them. But first you need to make sure that the logs are kept, kept for the right time and contain the right fields. This inspection costs nothing and saves both parties frustration.
What we measure
Are access logs kept?On some hostings, they are turned off or stored for a day - then there is nothing to analyze, and it is better to find out before paying.
Storage depthFor what period are the data available. Meaningful conclusions need at least a week, preferably a month.
Composition of fieldsDoes the records include the address, response code, size, and agent line. Without the response code, half of the conclusions are impossible, without the processing time - the speed section.
The presence of an intermediate layerA cache or network service in front of the site can provide part of the requests themselves. Then not everything reaches the server logs, and proxy logs are also needed.
Is the real address of the request visible?Behind the proxy in the address field is the proxy itself. If the real address is not hidden in the header, the bot check will give zero real addresses - this should be known before the start.
Data volumeHow many lines per day. This determines how long the parsing will take and whether a sample is required instead of a solid pass.
What you get
Answer whether your logs are suitable for analysis and for what period.
A list of what will have to be turned on or adjusted before work.
A short conversation about which issues the analysis will be able to close and which will not.
Timeline: 1-2 working days
Why is it free
Because on the part of hosting companies, the answer will be "there are no magazines", and it would be unfair to take money for this. Better to spend the day and be honest.
What's next
If the magazines are suitable, we name the cost of analysis and the term. If not, we'll show you what can be extracted from Search Console and external crawl statistics instead.
Short form: your contact and site URL
Contract, act and 30-day warranty
Every project gets a written contract: scope, deadlines, amount, acceptance procedure. After delivery — act and invoice, then 30 calendar days of warranty.
Sole proprietor & bank transfer
The contractor is a registered sole proprietor. Payment by invoice with closing documents.
Rights & access — yours
Code, design and materials transfer to you after full payment. Domain, hosting, repository and analytics are registered to you.
Client portal instead of email chains
During the project you get access to a portal: contracts, invoices, acts and project status in one place.
European clients
Among our work — projects for Norway, Bulgaria, Moldova and Spain.
Verifiable numbers
Every case in the portfolio comes with a link to a live site and a technical measurement.
Audit first, then pricing
There is no price list on the site intentionally: the scope of the same work differs multiples between clients.
We say "no" when unsure
If the task isn't ours or the deadline is unrealistic — we tell you upfront.
What affects the price
Why two seemingly identical tasks are priced differently
Volume of magazinesOne hundred thousand lines are read in an evening. Tens of millions is already a sample, a separate pass through unique addresses and verification of bots in batches, because checking each line separately is pointless in terms of time.
Data periodThe week shows daily cycles. The moon also shows the week and lets you see what has changed after your own edits. A longer period means more data and more work to compile.
Record formatThe standard format of nginx or Apache is read immediately. A format with permuted fields, multiple servers behind a balancer, or a log without a response code means separate format parsing before analysis begins.
Intermediate layersWhen a cache or a network service is in front of the site, the picture is divided into two sources: the server does not see part of the requests at all. Erection has to be done by hand, and this is a noticeable part of the work.
Site size and complexityIn a catalog with a clean structure, the layout by sections is done by the rule of half a line. Where addresses are generated by category paths, filter parameters, and language prefixes simultaneously, each type must be described separately.
Is there a Search ConsoleWith access, we check the logs with the official scan statistics and see the discrepancy. Without access, one source remains - there will be conclusions, but without a second support.
Who it's for
Situations where this service delivers results
Scenario 1 of 5
The catalog is updated, and new cards are not visible in the search for a long time
The question is not how many addresses are in your sitemap, but how many of them the bot actually invited in a month. The magazine gives both lists: what was done ten times and what was not done at all. The external dimension shows only the first side - in the shop of the professional tool, 25,537 entries of the map were led to 6,857 different addresses.
We'll review your situation in a free audit
Scenario 2 of 5
The traffic was slow, and no report explained exactly when it started
The log contains the date and time of each request. You can see the day when the bot stopped entering the section; it can be seen that the addresses began to give 301 instead of 200; it can be seen that the response time has increased, and the frequency of the bypass has followed. This is not proof of the cause of the fall, but it is the only place where the event has an exact date.
Scenario 3 of 5
The site is large and the server responds slowly
In the measured garden equipment store, the sitemap announces 59,189 addresses with a first response time of 840.5 ms. The math is simple: if the bot went one query at a time, a complete traversal would take about fourteen hours. This is our calculation, not a measurement: the bot walks parallel and unevenly, and it is precisely this unevenness that the log shows.
Scenario 4 of 5
They recently changed their address structure or moved
After migration, the bot goes to the old addresses for months - they are in its memory. The log shows how many bypasses go to redirects, whether chains of two or three transitions have been assembled, and which old addresses still bring requests.
Scenario 5 of 5
The hosting complains about the load and it is not clear who is creating it
Half of the "search bots" in raw statistics are price parsers and vulnerability scanners with a fake agent string. We divide the flow by checking the origin of the request and then it is clear who it makes sense to limit and who cannot be touched.
What's included
Complete list of work and what you get as a result
We isolate search bot hits from the general flow and check the origin of each address - then you work with real crawl numbers, and not with the sum of requests from all parsers in the world
We break down the tour by sections of the site: how many requests reach the directory, how many go to service addresses and filters that should not have entered the index at all
We calculate the distribution by response codes - it becomes clear what share of the visit is spent on transitions, errors and inaccessible pages instead of products
We make a list of the addresses that the bot requests most often and compare it with what is a priority for you: usually the top of the list is not what you promote
We compile a reverse list: important pages that the bot did not reach even once during the observation period - this is the queue for relinking and the site map
We measure the response time of the server specifically to the bot's requests and break it down by hours of the day: if the site is twice as slow at night, you will know about it with a number
Checking if your own traversal rules are being followed: Closed partitions that actually scan are a regular finding
We compare the picture from the logs with the Search Console crawl statistics and with your sitemap - the three sources rarely match, and the discrepancy is a diagnosis in itself
We give a written report with numbers, samples of raw lines for each conclusion and a list of works - with a division into what is done per hour and what the developer needs
When this service isn't right
What's not included — so there are no surprises at delivery
Implementing the fixes we find are separate jobs, and we list them instead of adding them silently
Analyze live visitor behavior: This is what analytics are for, not server logs
Configuring log collection on hosting where they are disabled is discussed separately
Working with visitors' personal data: we need bot requests, and people's lines can be anonymized before transmission
Process steps
Transparent stages with approval at every step
Total duration:7–18 days
1
Checking and retrieving logs
1-2 working days
Let's see what you have stored, for what period and in what format. Here we agree on the method of transfer: archive for the period or access for reading. If something is missing, we say before the start of work, not in the report.
2
Parsing the format and filtering out someone else's stream
1-3 working days
We bring the records to one table, discard the static and requests of people, and leave the requests of bots. Each address is checked for origin: the fake agent line is filtered out here, otherwise all subsequent arithmetic is incorrect.
3
Detour map
2-5 working days
Layout by sections, answer codes, frequency and hours of the day. Separately - the top of the most frequently requested addresses and a list of priority pages, which are not in the magazine at all.
4
Reconciliation with sitemap and Search Console
1-3 working days
Let's compare the three lists: what you announced, what the bot asked for, and what the official scan statistics show. Differences between them are half of the report's conclusions.
5
Report and conversation on it
2-5 working days
Written document: numbers, samples of raw lines for each conclusion, a list of works divided into quick and those that require a developer. Then an hour of talking so you can ask a question to a person, not a file.
Did not find your case?
Describe how it works on your side — we will tell you whether “Analysis of server logs” fits and what it means in your situation. No brief and no call: one question, one answer.
Cases
Tasks and results in numbers — all metrics measured by us
online store of professional tools, catalog of more than 6,000 items — our external technical dimension
Task
Understand where the traversal goes without having logs: there was no access to the server, so we looked from the side the bot is looking from.
Solution
Touring the site map with a separate count of records and unique addresses, measuring response time, checking language versions.
Result
25,537 entries in the site map yielded 6,857 unique addresses — 6,486 product cards and 361 categories, that is, each product was announced an average of 3.9 times under different channels. The map itself is generated in 34.2 seconds: the bot that requests it waits the same amount. All addresses in it lead to the Russian-language branch, although the site gives Ukrainian by default. Measured on 07/31/2026. What the method didn't show was how many of the 25,537 addresses the bot actually queried. This is visible only in the magazine.
underwear retail store, catalog of over 110,000 items — our external technical dimension
Task
Estimate what proportion of declared addresses actually make sense to crawl before talking about the crawling budget.
Solution
A sample of 150 random product addresses of the site map with a check of the response code of each, counting repetitions in product sub-cards, measuring the response time of the main and card.
Result
50 addresses out of 150 (33%) returned a 301, and all 50 led to a different position - a different size or color. At the scale of the directory, this is about 37,000 addresses, each of which costs a bot request and adds nothing to the index. There are 3,774 duplicate entries in the commodity subcards, individual addresses are repeated up to 55 times. The main one gives the first response in 1,544.6 ms versus 391.4 ms on the card — and it is given without caching. Measured on 07/31/2026.
garden equipment and spare parts store, a catalog of about 55,000 items — our external technical dimension
Task
Calculate whether it is realistic to bypass the declared volume at the actual speed of the server.
Solution
Recalculation of submaps (18 files of 3,000 addresses plus 1,321 in the nineteenth, separately 3,868 categories), verification of the Sitemap directive in robots.txt, response time measurements.
Result
59,189 addresses in the sitemap with the first response of 840.5 ms — the slowest server of the batch, which is expected for 55,000 products. A sequential traversal of this volume at this rate would take about 14 hours of continuous queries; this is our arithmetic, not the measurement, and it is this that the log replaces with the actual frequency numbers. The map is declared in robots.txt at a non-standard address - otherwise the bot would search for it blindly. Measured on 07/31/2026.
Technologies & integrations
What we build on and what it connects to
Stack
The nginx / Apache access logs are the only source where every request is; but only the one that reached the server
Reverse DNS query with direct verification — filters out fakes; on large volumes, it goes in batches to unique addresses
GoAccess — quick summary of a large file; the layout by sections of the catalog still has to be done separately
Own processing scripts - count exactly what is needed; it is impossible to check the result without raw data, so we give the samples together with the report
Search Console, crawl statistics - official reconciliation; one search engine, summary and delay in several days
robots.txt — we check whether the rules are followed; bypassing does not remove the address from the index
Proxy logs are needed where a network service is in front of the site and some requests do not reach the server
Integrations
Google Search Console
Bing Webmaster Tools
Cloudflare
hosting panel
Server Logs vs. Scan Statistics Report
How this option differs from the alternative
Only crawl statistics in Search ConsoleOur approach
Data unitgrouped summary by file type and destinationseparate request: address, code, time, size, agent
What bots are visibleonly google botseveryone, including parsers under someone else's name
Freshnessdata with a delay of several daysthe record appears at the time of the request
Depth backabout three months, and nothing needs to be adjusted for ithow long your server stores—often 7-30 days
What can be proventhat the general circulation went down in some periodthat a specific address gave an error on a specific day
The price of accessfree and immediately - that's why we always start with this reportlogs, access and parsing are required
What we need from you
We can't start without this — best to prepare in advance
1Access logs for a period of one week, better than one month: archive or read access. Rows of live visitors can be anonymized before transmission - we need bot requests.
2Description of intermediate layers: cache, network service, balancer, several servers. Without it, we will analyze a part of the picture, considering it as a whole.
3Read access to Search Console - to check official crawl statistics.
4Sitemap URL and current robots.txt: What the bot actually did is compared to these.
5List of sections that are a priority for you. Otherwise, "important pages" will have to be determined by us, and you will receive a report on other people's priorities.
6Dates of the last changes on the site: migration, change of address structure, moving to another server. You can see them in the magazine, but it's faster when you know what to look for.
If something is missing — let us know, we'll help you gather it or do it as a separate task.
FAQ
Most frequently asked questions — with concrete answers
How is this different from the Crawl Stats report in Search Console?
Detailing and coverage. Google's report groups requests by file type and destination and shows a summary with a delay of several days: it shows that the bypass has failed, but not where exactly. In the log, each request is a separate line — address, response code, time, size, agent. Therefore, the question "why did it bypass" the report closes at the level of the trend, and the magazine - at the level of specific addresses that began to give 301 or 500 of such and such a number. The second difference: Search Console only shows Google bots. Bing and parsers that go by someone else's name do not exist in it. We start with the report anyway - it is free and gives a reference for three months ago.
I don't have access to the logs - where do I start?
From checking whether they are being conducted at all. On some hostings, the logs are disabled by default, on some they are stored for a day and are deleted. We do this check for free in one or two days. If there are no logs, some of the conclusions can be obtained by other methods: Search Console crawl statistics plus an external sitemap crawl with a check of response codes. That's how all three cases on this page are done — and that's why each one has a paragraph about what this method didn't show. It is a weaker instrument and we call it weaker.
How do you tell a real bot from a fake?
The agent line is faked with a single line of code, so we don't take it seriously. The check goes through a reverse DNS query to the address and a direct query back: the address must resolve to the official name of the search engine, and this name must resolve back to the same address. One return request is not enough, because the record is under the control of the owner of the address. Separately, there is a technical detail on which independent attempts stumble: if a network service is in front of the site, the address of the service itself, and not of the person who came, is in the address field in the log. As long as the real address isn't slipped through the header and read from it, the check will honestly show zero real bots on the entire array.
What is usually found in magazines?
The most frequent finding is one: the detour goes to the wrong place. Requests are spent on service addresses, on filter combinations, on pages that have not been around for a long time, and on transitions. How large-scale this story is, you can see even from the outside: in the measured lingerie store, out of a sample of 150 random sitemap addresses, 50 returned 301, and on the scale of the directory, this is about 37 thousand addresses, for each of which the bot pays a request. The second most frequent find is a section that is not in the magazine at all: the pages are advertised and linked, and the bot has not reached them even once in a month. The third is the nighttime response time, which is twice as bad as the daytime response time, and a simultaneous drop in the bypass frequency.
For what period are the data required and how to store them in the future?
At least a week to see daily cycles. The moon gives a picture that can already be believed, especially when the catalog is updated unevenly. Less than a week is guesswork backed by numbers, and that's not how we work. A tip to do today regardless of whether we do anything together: enable 30 days of log storage with rotation and archiving. It costs nothing but a few gigabytes of disk space. The worst situation is when the traffic dropped yesterday, and the magazines live for a day: there is no way to establish the reason.
Is it possible to understand why the traffic has dropped from the logs?
Sometimes it does, and that's the most valuable scenario. The log shows the date when the bot stopped entering the section; it can be seen that part of the addresses began to give an error; it can be seen that the response time has increased, and the frequency of traversal has followed it. But the line is hard: the log explains bot behavior, not human behavior. If traffic has dropped due to a season, a change in demand, or a new competitor in publishing, it will not be in the magazines. Therefore, we tackle this issue only in conjunction with analytics and Search Console.
Isn't it easier to just block extra bots and not deal with it?
This is the most common advice and the easiest way to make things worse. Blocking according to the frequency of requests hits the real search engine first - it just happens more often than all others. Blocking by agent string doesn't work against those who forge that string, which is exactly the ones you wanted to stop. Our recommendation: first look in the log and see who took how much, then limit by address - by the origin of the request, with a list of allowed search engines. And check separately if your load protection is blocking any of them already.
Logs weigh gigabytes - how to transfer them?
Compressed archive for the required period, in parts. Text logs compress well: a monthly file of a few gigabytes usually turns into hundreds of megabytes. If it is inconvenient to transfer, we take access for reading and take the necessary things ourselves. We do not need personal data: the address of a human visitor can be anonymized or truncated before transmission, the conclusions about bots are not affected. What should be kept unchanged is the request address, response code, time, size, and agent string.
How long does it take and what do you hand over?
7-18 working days from the moment we receive the logs: 1-2 days for verification and receipt, 1-3 days for format analysis and screening of foreign traffic, 2-5 crawl map, 1-3 sitemap and search console reconciliation, 2-5 report and conversation. On hand - a written document with numbers, samples of raw lines for each output and a list of tasks, divided into what is done by the setup in an hour and what needs a developer. Implementation is not included in this work.
Write what kind of hosting you have and whether access logs are kept.
In one or two days, we will answer whether your data is suitable for analysis, for what period and what needs to be adjusted. If there is nothing to analyze, let's say so in the very first letter and show what can be obtained without magazines.
From measured cases25,537 entries in the site map yielded 6,857 unique addresses
There is no price list on the site on purpose: the same work differs several times over between two clients, and a “from” figure explains nothing in that case. First a free audit — we count your pages, duplicates and speed — then we name the sum and the deadline and fix both in the contract.