On 17 August 2026 we did to LEO Chat what usually only happens to a production system by accident: loaded it with 300 requests per second and, in the middle of the test, started forcibly killing processes — the engine, the NATS queue, the Redis cache. Not one by one, gently, but with SIGKILL, which gives a program no chance to shut down cleanly.
This article is about why we did that to our own product and what the result actually showed — including what it does not prove.
A question rarely checked before a release
Most chat bot tests check whether the bot answers correctly. Less often do they check what happens when part of the system goes down mid-operation — which is exactly what happens in production: a container restart during a deploy, a network glitch, memory exhaustion. The question is not "will a component fail" but "what happens to the customer's message that was being processed at that exact moment."
For LEO Chat this question is particularly sensitive because part of the scenarios are order placement and payment confirmation. A lost message here is not an inconvenience — it is an order nobody saw, or worse, a payment confirmation that never reached the customer.
What exactly was tested
The tool was k6, a load-testing tool that generates a stream of requests to a defined profile. The target was 300 requests per second, which for a chat bot means simulating several hundred simultaneous conversations with messages arriving one after another.
Alongside the load, the test forcibly killed key components of the stack: the message-processing engine itself, the NATS queue (through which messages pass between services), and Redis (cache and part of the session state). The killing was not a graceful shutdown but SIGKILL: the process disappears instantly, with no chance to finish writing anything unfinished.
The metric measured: how many messages were lost (sent but never acknowledged) and how many were duplicated (the same message processed twice — for example, an order confirmed twice instead of once).
The result
0 lost and 0 duplicated confirmed messages.
This means: despite forcibly killing the engine, the queue and the cache under a load of 300 requests per second, not a single confirmed message disappeared, and none was processed twice. Technically this rests on the NATS queue storing messages persistently (not only in memory) and having a processing acknowledgment mechanism — a service that crashed before acknowledging a message does not remove it from the queue forever; it waits for a retry.
What this result does not prove
This is a load test on the local full stack — the same service configuration as production, but not production itself, not with real customer traffic and not under real network latency between data centres. Production load can reveal things a local test cannot: specific network delays, resource contention with other services on the same server, behaviour under sustained rather than one-off load.
It is also not a test of 100% availability: while a component was being forcibly killed, some requests experienced a delayed response while the system recovered — the queue did not lose messages, but it did not process them instantly at the moment of failure either. For the customer this means a slower reply for a few seconds, not a lost conversation.
Why this matters at all for a chat bot
Most chat bot product pages talk about answer accuracy and speed. Behaviour during a failure — rarely, because it is an uncomfortable topic: showing that a system went down, even in a controlled test, means admitting it can go down. We consider it more honest to show this limit with a date and a number than to stay silent about it or promise "100% uptime" backed by nothing.
The same approach — measuring our own limits, not only our strengths — we already applied to LEO Chat's answer accuracy: 89.7% on a reference set of questions, with a direct statement that the rest goes to an operator. The chaos test is the same logic applied to infrastructure instead of answer quality.
What to check with your chat bot vendor
If you are choosing a chat bot for a store, message-queue reliability rarely makes it onto the list of questions — yet it is exactly what determines whether a payment confirmation gets lost during a routine server restart at your vendor. Three concrete questions worth asking: are messages stored persistently until processing is acknowledged, is there a retry mechanism after a component failure, and was a real-load test run with a date and a number — not "we tested it," but exactly how many requests and what result.
We publish our own measurements precisely because "trust us" is not an argument here: a verifiable number with a date is an argument, a promise is not.
