Part 5 · Observability

Your service is now running in Kubernetes, scaling under load — and completely opaque. When a user says “the editor was slow at 2 p.m.,” you have no way to answer why. Part 5 fixes that. Observability is the discipline of making a running system explain itself, and it rests on three pillars: logs (what happened), metrics (how much, how fast, aggregated), and traces (the path of one request through every service). Rust’s tracing ecosystem gives you all three from one instrumentation.

What you’ll build

   request ─►[ span: http ]─►[ span: db ]─►[ span: claude ]─►[ span: cache ]
                 │               │              │                │
                 ▼               ▼              ▼                ▼
   logs ── structured JSON lines (level, target, fields, trace_id)
   metrics ─ /metrics  →  Prometheus  →  Grafana dashboards
   traces ── OpenTelemetry  →  one waterfall per request, across services

• Structured logging with tracing: not println!, but events with fields, levels, and a span per request that carries context (method, path, user, trace_id) into every log line beneath it.
• Metrics: a Prometheus /metrics endpoint exposing the four golden signals — latency, traffic, errors, saturation — plus quill-specific counters (posts created, tokens generated, cache hits).
• Distributed tracing: propagate a trace id from the Astro frontend through the web binary, the worker, Postgres, Redis, and the Claude call, so one slow request is one readable waterfall.

The three pillars, and what each answers

Pillar	Question it answers	quill example
Logs	What happened, exactly?	“post 42 update failed: unique_violation on slug”
Metrics	How much, how fast, right now?	p99 latency of POST /draft over the last hour
Traces	Where did this one request spend its time?	1.9 s draft = 40 ms db + 1.8 s Claude + 60 ms render

The chapters

Ch	Title	You add	The idea it teaches
1	Structured logging	tracing + spans, JSON output	events vs prints, spans as context, log levels that mean something
2	Metrics	/metrics, golden signals, Grafana	counters/gauges/histograms, RED/USE, what to actually chart
3	Distributed tracing	OpenTelemetry, trace propagation	one request across services, finding the slow hop

Instrument once, get three views

The reason to build on tracing rather than bolt on three separate libraries: a single #[instrument] span feeds logs (as structured context), metrics (as timing), and traces (as a node in the waterfall). You annotate the code once; the three pillars fall out of the same instrumentation.

The thread

Once it compiles, what does it take to run for real users? Part 5’s answer: the system has to be able to tell you when it’s unhealthy — before a user does. Observability is what turns “it’s slow, I guess?” into “the p99 of the draft endpoint tripled at 14:03 and every slow request spent its time in the Claude call, which correlates with a rate-limit retry storm.” That’s the difference between guessing and operating.

Next: Part 6 · Make It Fast →