obs / field notes
> Cheat sheet

Thinking Like an Observability Engineer

Seven mental models that separate people who watch dashboards from people who can answer any question about a running system.

> Premise

The core shift

Monitoring asks

Is it up?

Known-unknowns. You decided in advance what to watch.

Observability asks

Can I answer a question I haven't thought of yet?

Unknown-unknowns. The data is rich enough to explore at 3am.

> Signals

The three pillars

  • Metrics

    Tell you that something happened and how much.

  • Logs

    Tell you what exactly happened.

  • Traces

    Tell you where in the request path it happened.

> Models

The mental models

  1. 1. Start from the question, not the signal

    Instrument for the questions you'll need to answer later.

    • Attach the dimensions you'll want to slice by: tenant, version, region, request type.
    • If a dimension isn't on the telemetry, you can't ask about it during an incident.

    Ask yourself: "What will I need to filter by when this breaks at 3am?"

  2. 2. Symptoms first, causes second

    Page on what users feel. Everything else is context.

    • Alert on latency, errors, saturation, correctness: SLO-style.
    • "Disk 85%" with no user impact isn't a page. Move it to a dashboard.
    • A symptom alert with no diagnostics attached is a dead end.

    Ask yourself: "Would a user notice this right now?"

  3. 3. Every signal is a lie until correlated

    One pillar gives you a hint. Three linked pillars give you an answer.

    • Move fast: metric spike → exemplar / trace ID → the log line for that span.
    • Shared IDs everywhere: trace_id, service, version, instance.
    • No shared IDs = three silos, not observability.

    Ask yourself: "Can I jump from this graph to the exact request?"

  4. 4. Interrogate the pipeline, not just the app

    A flat line can mean "healthy" or "the collector died."

    • Check scrape gaps, sampling, aggregation windows, clock skew, retention rollups.
    • Monitor the monitoring: up, scrape duration, ingest lag, series churn.

    Ask yourself: "What would this look like if the telemetry itself were broken?"

  5. 5. Cost is a design dimension

    Cardinality, volume and retention are trade-offs, not chores.

    • Know what each label costs and which question it buys you.
    • Never put unbounded values in labels: user IDs, UUIDs, full URLs.
    • Sample traces, aggregate logs, keep raw metrics short and rollups long.

    Ask yourself: "Is this label worth the series it creates?"

  6. 6. Postmortems should end in a query

    Every incident should leave the system easier to see.

    • Build the one alert or panel that would have caught it earlier, not another generic dashboard.
    • Save the query that found root cause. It's tomorrow's runbook.

    Ask yourself: "What would have shown this 20 minutes sooner?"

  7. 7. Distributions and rates, not averages

    p50 hides everything interesting. Always ask "compared to what?"

    • Default lens: rate() over a window plus histogram quantiles (p95 / p99).
    • Compare to a baseline: same time yesterday, previous deploy, a sibling pod.
    • Counters only go up, so look at their rate. Gauges go both ways, so look at their value.
    histogram_quantile(0.99, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))

    Ask yourself: "Which users are in the tail, and what do they have in common?"

> Incident

When something is on fire: the triage loop

  1. What changed?

    Deploys, config, traffic shape, upstream dependencies.

  2. Who is affected?

    Slice by tenant, region, version until it isolates.

  3. Where in the path?

    Find one slow or failing trace and follow the span.

  4. Is the data lying?

    Verify the collector before trusting the graph.

  5. Mitigate first

    Rollback, scale, shed load. Root-cause after.

> Reference

Quick reference

Do

  • Alert on SLOs and user-facing symptoms.
  • Propagate trace_id through every hop and log line.
  • Tag telemetry with deploy version so "what changed?" is one query.
  • Keep a dashboard for the pipeline itself.
  • Look at p95 / p99 and rates by default.

Don't

  • Page on causes (CPU, disk) with no user impact.
  • Put unbounded values in metric labels.
  • Trust a graph before checking the collector is alive.
  • Reach for averages when the problem is in the tail.
  • Close a postmortem without shipping a query or alert.
> Go deeper

Learn more: docs & tools

Google SRE

The why: SLOs, alerting philosophy and incident practice from the people who coined the term.

Prometheus · PromLabs

The how: the data model, PromQL and tooling that most metrics stacks speak.

VictoriaMetrics

At scale: high-cardinality storage, long retention and a PromQL-compatible query language.