Revision · Part 5: Observability

Part 5 fixes the fact that a service running in Kubernetes and scaling under load is also completely opaque. Observability is the discipline of making a running system explain itself, and it rests on three pillars — logs, metrics, and traces — which Rust’s tracing ecosystem gives you from one instrumentation.

What this part covered

• Structured logging, not println! — events with fields and levels, and a span per request that carries context (method, path, user, trace_id) into every log line beneath it, answering “what happened, exactly?”
• Metrics and the four golden signals — a Prometheus /metrics endpoint exposing latency, traffic, errors, and saturation, plus quill-specific counters (posts created, tokens generated, cache hits), charted in Grafana; answers “how much, how fast, right now?”
• Distributed tracing — propagate a trace id from the Astro frontend through the web binary, the worker, Postgres, Redis, and the Claude call, so one slow request becomes one readable waterfall; answers “where did this request spend its time?”
• Instrument once, get three views — a single #[instrument] span feeds logs (structured context), metrics (timing), and traces (a node in the waterfall), so you annotate the code once and the three pillars fall out of it.

The takeaway

Part 5’s answer to the thread is that the system has to be able to tell you when it’s unhealthy — before a user does. Observability turns “it’s slow, I guess?” into “the p99 of the draft endpoint tripled at 14:03 and every slow request spent its time in the Claude call, correlating with a rate-limit retry storm.” That’s the difference between guessing and operating. Next, Part 6 uses this visibility to measure and make quill fast.