Four alerts that never existed: monitoring that lived only in the docs

On 5 September 2026 we decided to check not the site itself but the thing that was supposed to tell us the site had gone down. Our internal documentation had carried a line since 17 January 2026: four alert rules, checked every 30 seconds. Error rate above 5% for five minutes, response time (p95) longer than 2 seconds for 10 minutes, an exhausted database connection pool, a day without a single brief. It sounded like protection against everything that can happen to an agency site.

We opened Grafana, where those rules were supposed to live. There were zero rules there.

This article is about how alerts can exist on paper only, why a line like that survives for months, and how to find out in roughly half an hour whether your site has even one alert that actually reaches a human.

What we found on 5 September

There turned out to be three defects, and each one on its own made those four rules useless.

The rules were not in the system. The file describing them did sit on the server, but no program read it: the configuration of the metrics collection system had no reference to it, and a separate service to send notifications out did not exist at all. The file just sat there.

The rules had nothing to look at. All four were written against site metrics: the number of HTTP requests, their duration, errors, the state of the database. These metrics are declared in the code, which means they have a name and a description. But the pages of the site never call the function that was supposed to record them. We checked this once more on 17 September: the metrics store held not a single data series for requests, for errors, or for the database. Even if someone had hooked up the file with the rules, they would have been comparing emptiness against a threshold.

There was no delivery either. The only notification channel in Grafana was the standard email receiver with a placeholder address. It exists in every fresh installation and sends nowhere.

So from mid-January to 5 September at the very least, we could only have noticed the site going down by opening it ourselves — or from a person who failed to use it.

Why nobody noticed

Because a missing alert looks exactly like a quiet day. When everything works, the real rules and the imaginary ones are equally silent. The difference shows up at the moment of the failure, and that is precisely when nobody is reading the documentation.

The second reason is how the line itself was written. It was specific: rule names, thresholds, the check interval. Text like that raises no doubts, because it resembles a description of work already done. In reality it was a plan written in the form of a report.

And third: the dashboards were there. Panels with the familiar headings about request rate and response time stood in Grafana, and the mere fact of their existence created a feeling that monitoring existed. The charts on those panels were empty, but an empty chart is easy to read as “nothing is happening right now.”

What we put in their place

That same day we started with the server rather than the site, because it was the server that had already gone down on us. On 2 August 2026 the shared server ran out of memory: processes started being killed off, and the machine had to be rebooted. We learned about that not from a system but after the fact.

So on 5 September we got collectors for the metrics of the server itself and of the containers, a separate dashboard, and four rules that this time do exist in Grafana and send messages to the owner's Telegram:

• load average above 8 (on eight cores) for 10 minutes;
• less than 1 GB of available memory out of 15 for 5 minutes;
• the root disk more than 90% full for 15 minutes;
• any metrics source, the site included, not responding for 5 minutes.

Each rule carries a hint in the text of the message: where to look first and what has already happened. The rule about memory states outright that this is exactly how the failure of 2 August began.

We rewrote the documentation to match what actually exists, and kept the old line about the four rules together with an explanation that they never existed. Otherwise someone would find it in the history and believe it again.

What the first 12 days showed

On 17 September all four rules were “inactive” and “healthy”: the system evaluates them, there are no evaluation errors, the condition is not met.

To work out whether the rules are silent because everything is fine or because they do not fire, we looked at the metrics themselves over the 12 days from the rules going live on 5 September to 17 September.

• Load reached 19.04 at individual moments, but the highest value that held for a full 10 minutes in a row was 6.79. It never reached the threshold of 8.
• Available memory dropped to a low of 4.58 GB. That is a long way from the 1 GB threshold.
• The disk climbed to 93.05%, which is above the threshold. But never for long: across all 12 days there were 16 one-minute samples above 90%, and the highest value that held for all 15 minutes in a row was 88.13%.
• The site as a metrics source failed to respond once, and that was a single one-minute sample. The collectors for the server and the containers did not miss one.

The conclusion cuts both ways. The rules were justified in staying silent: no condition held for its whole window of time. But there has been no real firing yet either, so the path from a breached condition to a message on a phone is still not confirmed by a live case.

And the case of the disk shows the limit of any threshold with a hold time: short spikes to 93% are something this rule does not see by design. We configured it that way deliberately, so that temporary files do not wake anyone up. But if the disk ever fills up completely within those same few minutes, the alert will arrive after the problem.

How to check your own site

You do not need to understand code for this. What you need is access to the monitoring system and half an hour.

1. Count the rules that exist in the system, not in a document. In Grafana that is the Alerting → Alert rules section, in Uptime Kuma the list of monitors, in a hosting panel the notifications section. Write down each one: what it checks, what the threshold is, how long the breach has to hold. If the list is empty or shorter than you were told, start here.
2. Open each rule and look at what it watches. If a rule counts errors on the site, open that same query on its own and make sure it has any data at all for the past week. A rule over an empty metric is not silent in the same way every time: in Grafana each rule has its own setting for the case of missing data, and that setting decides whether you see the familiar “OK” or a separate “No data” state that is easy to overlook in the list. In three of our server rules it is set to “No data,” while in the rule about an unreachable target it goes straight to alerting. An empty metric will catch a breakdown in neither variant, so look at the data under the query, not at the color of the status.
3. Check where the notifications go. Look at the list of delivery channels: email, Telegram, SMS. An address along the lines of example@email.com, or a channel created by a person who no longer works with you, means there is no delivery.
4. Send a test notification and wait for it on your phone. In Grafana the Test button is in the settings of every channel. Only a message you have seen counts, not a green “sent” label.
5. Ask yourself three questions: if the site starts returning a 500 error, who finds out about it and when; if the disk runs out of space; if the emails with leads stop going out. Each of them should have a rule name and a person the rule reaches. The answer “we will see it in the statistics” means “nobody.”

What we have not solved

Our new rules watch the server and whether the site responds. They do not watch whether the site works correctly.

We still cannot see 500 errors on an individual page, because the request metrics this whole story started with have stayed empty. A failed email delivery does not produce a notification either. On 8 September 2026 two emails did not go out, the log recorded it, and we found those entries by hand when we opened the log ourselves. We went through the email story in detail separately: the form says “thank you” and the email never leaves.

So on 5 September we closed the crudest part: a server failure will no longer slip past us. A quiet breakdown inside the site still will. Our next step is exactly that: to record request errors and failed emails and to put rules over them with the same delivery channel.

If you need someone to set up this kind of watch for your site and to verify that the messages really do arrive, that is uptime monitoring work. Responding to the failures themselves and to the state of the server is part of server administration.

The rule we took away for ourselves: an alert exists once it has reached a human at least once. Everything else, a line in the documentation included, is only an intention.