What are the three pillars of observability — logs, metrics, and traces — and how do they complement each other?

#Monitoring tells you something is broken. Observability lets you ask why, about a system you've never seen fail this way before.

The old model of "watching" a system was a curated set of dashboards and alerts built in advance for failure modes someone already anticipated. Observability is the opposite bet: instrument the system richly enough, in three complementary ways, that when it fails in a way nobody anticipated, you can still ask arbitrary questions of it and get answers — without shipping new code first. The three pillars — logs, metrics, and traces — aren't three tools competing for the same job. Each one answers a different question, and an incident usually walks through all three in order.

#Metrics: is something wrong, and how bad

A metric is a numeric time series: a counter (http_requests_total, only ever increases), a gauge (queue_depth, moves up and down), or a histogram (http_request_duration_seconds, buckets of how long things took). Metrics are cheap to store because they're pre-aggregated — a Prometheus counter takes the same space whether it represents 10 requests or 10 million — which is exactly what makes them good for long retention, dashboards, and alerting on SLOs ("p99 latency for /checkout exceeded 500ms for 5 minutes"). What they can't tell you is which request was slow, or why — aggregation is the whole point, and aggregation destroys per-request detail on the way in.

#Traces: where in the system the time actually went

A trace represents one request's journey across every service it touched, as a tree of spans — each span records one unit of work (a handler, a database call, a downstream RPC) with a start time, a duration, and a parent span it's nested under. Reconstructing that tree tells you exactly where a slow request spent its time: 4ms in the gateway, 6ms creating an order, 290ms waiting on a downstream payment call that itself spent most of its time waiting on a card processor. Traces are what make distributed latency debugging tractable at all — without them, "the checkout endpoint is slow" gives you no idea which of the six services it touches is actually the culprit. The cost is volume: capturing a full trace for every single request doesn't scale past a certain size, so production systems sample (keep a percentage of traces, or specifically keep every trace that errored or was unusually slow via tail-based sampling).

#Logs: what exactly happened, in detail

A log is a discrete, timestamped event with arbitrary detail attached — a stack trace, a request payload, a user ID, a specific error message. Logs are the highest-cardinality, highest-detail signal, and the most expensive to store and query at volume, which is why they're rarely where you start an investigation and almost always where you finish it, once metrics and traces have narrowed the search down to "this specific request, in this specific service, at this specific time."

#One request, three artifacts, at every hop

The signal that ties all three together isn't a tool, it's a single shared identifier — a trace_id — generated at the edge and propagated to every downstream call via a header (the W3C traceparent standard). OpenTelemetry is the vendor-neutral instrumentation standard that emits all three signal types with that identifier already attached, which is why a modern stack treats "add observability" as one integration, not three.

Every hop in that single request emits all three signal types simultaneously: a log line with the shared trace_id as a structured field, a metric increment feeding the aggregate dashboards, and a span that becomes part of one assembled trace once collected. None of them is "the real one" — they're three views of the same event, at three different levels of detail and cost.

#Correlating all three during an actual incident

This is the realistic shape of almost every incident: a metric-based alert tells you that something regressed and roughly how bad, because metrics are cheap enough to watch continuously and alert on. You then pull up traces for that service and time window to see where the request spent its time — which hop is actually slow, not just that "checkout" as a whole is slow. Once you've isolated a representative slow or failing trace, you take its trace_id and pull the exact log lines for that one request, across every service it touched, to see the actual error message, payload, or stack trace that explains why. Skipping straight to logs without narrowing via metrics and traces first means grepping an ocean of log lines with no idea which ones are even relevant.

#Comparing the three pillars

🧠 Check your understanding
0/3 · 0/3 answered
1. An alert fires because p99 latency for an endpoint breached its SLO. What is the most efficient next step, before diving into logs?
Add a new metric and wait for it to collect more dataRestart every service in the request pathImmediately grep production logs for the word 'error' across all servicesQuery traces for that endpoint and time window to localize which downstream hop is actually slow
2. Why is putting `user_id` as a Prometheus metric label a common production mistake?
It creates a separate time series per distinct user_id value, causing a cardinality explosion that most metrics backends can't handleuser_id is not a valid metric nameMetrics only support numeric labelsPrometheus cannot store strings as labels
3. What is the mechanism that lets a log line, a metric increment, and a trace span from the same request all be correlated together during an incident?
All three pillars are actually the same underlying data structureThey are stored in the same physical database automaticallyCorrelation requires manually cross-referencing timestamps onlyA shared trace_id generated at the edge and propagated via a header (e.g. W3C traceparent) to every downstream service and included in log fields