Observability is just paranoia with graphs

Every line of code I have ever written has, at some point, surprised me. So I instrument everything. Not because I think it will fail, but because the parts that “obviously cannot fail” are usually the parts that fail in ways that make you question whether computers are real.

This is not a treatise on OpenTelemetry. I am not going to explain distributed tracing theory using diagrams that look like subway maps designed by a sleep-deprived architect. This is a field guide for the small project. The side project. The thing you built in a weekend and deployed for “just a few users,” which now somehow processes actual money and sends push notifications to people at 2AM. The kind of system where your monitoring strategy is usually: check logs, stare at logs, add more logs, become log.

The first time I understood observability

Years ago, I had a tiny service that sent transactional emails. Very simple architecture: API receives request, queue job, worker sends email. Three moving parts. Basically impossible to break. Naturally, it broke in the dumbest way imaginable. Emails would randomly stop sending for exactly 11 minutes. Not 10. Not 12. Eleven. Then everything resumed like nothing happened. CPU fine. Memory fine. Database fine. Queue fine. Logs completely useless. At one point I genuinely considered cosmic radiation.

After two days of debugging, the culprit turned out to be a database connection leak. One error path in the worker wasn’t returning connections to the pool. Under normal traffic, everything looked healthy. During small traffic spikes, the pool slowly filled up until every available connection was exhausted. New jobs would block waiting for a free connection, retries piled up, and email delivery quietly stalled. Eventually, idle connections timed out and were released back to the pool. The backlog drained, emails started sending again, and the system behaved as if nothing had ever happened.

”I’ll add monitoring later”

This is the biggest lie developers tell themselves after “I’ll clean this up later,” “temporary workaround,” “this should scale fine,” and “Safari probably supports it.” The problem is observability feels unnecessary right until the exact moment you desperately need it. And that moment usually happens during deployment, during sleep, during a demo, or immediately after you told someone “it’s stable now.”

Small systems especially create false confidence because everything feels local and understandable. You think you know every part of this app. Then suddenly one API route takes 14 seconds only in production, one customer uploads a file that destroys your thumbnail service, Redis reconnects itself into another dimension, or a cron job runs twice because time itself is fake. And now you are SSH’ing into a server muttering things like “why is nginx sweating.”

Logs are not observability

Logs are evidence. Observability is context. A log tells you “payment failed.” Observability tells you which customer, after which deployment, on which server, after what sequence of requests, with what latency spike, while what database query was dying internally. A surprising amount of backend engineering is just building systems that can explain themselves before you have to interrogate them manually.

Because the worst bugs are not crashes. Crashes are merciful. The worst bugs are partial failures, slowdowns, retries, race conditions, “works locally,” “only happens sometimes,” and “cannot reproduce.” Which is software engineer language for “may God help us all.”

Instrument the weird stuff

Everyone instruments API requests. Few people instrument queue depth, retry counts, websocket reconnects, cache hit ratios, third-party latency, background jobs, external provider timeouts, and how long PDF generation takes on the one VPS held together by optimism. The weird stuff is where the interesting failures live. I once added tracing to a video processing pipeline mostly because I was bored. Turns out FFmpeg slowed down because videos were variable frame rate. Computers are incredible machines built entirely out of betrayal.

My rule now

If something would be annoying to debug at 3AM, instrument it now. That is the entire philosophy. Not enterprise-grade observability. Not twelve dashboards. Not “AI-powered telemetry.” Just: future me does not deserve suffering. That means request IDs everywhere, timing external calls, structured logs, basic tracing, metrics for anything asynchronous, and alerts for things that silently stop working.

You do not need a NASA control room. You just need enough information so that when production catches fire, you can at least identify which fire.

Small systems deserve observability too

People treat observability like it only matters at scale. But honestly, tiny systems benefit the most. Because in a small project there is no SRE team, there is no ops engineer, there is only you. You are developer, DevOps, QA, support, incident response, and occasionally therapist. And when something breaks, you do not open a war room. You open fifteen browser tabs and start guessing.

Good observability reduces guessing. That is the real value. Not pretty graphs. Not dashboards nobody checks. Not screenshots for architecture Twitter. Just reducing the amount of time you spend staring into logs trying to determine whether the problem is your code, the database, Cloudflare, DNS, Docker, timezone handling, or the moon entering retrograde.

Final thought

I used to think observability was something you added once a system became serious. Now I think the opposite. A system becomes serious the moment you cannot comfortably explain what it is doing. And every system reaches that point much faster than you expect.