You're standing up a brand-new service — design its observability stack from scratch

#Why this question is a favorite at the senior level

"Design the observability for a new service" is a system-design question wearing an operations costume. It tests whether you think about production readiness before the first incident, whether you know the difference between collecting data and being able to act on it at 3 a.m., and whether you can reason about cost and signal-to-noise trade-offs rather than just naming every tool you've heard of. A strong answer walks through logging, metrics, tracing, and alerting as one connected system — not four separate purchases — and explicitly ties the whole thing back to what an on-call engineer actually does when paged.

The three classic pillars of observability are logs, metrics, and traces. A senior answer adds a fourth, often-overlooked layer that turns raw signals into something a human can act on: alerting built on SLOs and error budgets, plus the on-call tooling that connects everything. Here's how to build all of it, in order, for a brand-new service.

#Step 1: Structured logging, from the very first commit

Logging is the foundation because it's the cheapest to get right early and the most expensive to retrofit. Every log line the service emits should be structured JSON with a consistent schema: timestamp, level, service, environment, and — critically — trace_id and span_id injected by the tracing instrumentation. This single field, trace_id, is what lets you pivot? from "I see an error in the logs" to "show me every span of the exact request that produced this error," which is the single most useful cross-pillar link in the entire stack.

Log level discipline matters more than most teams admit: DEBUG for local development, INFO for meaningful business events (not every function entry/exit), WARN for recoverable anomalies, ERROR for failures that need attention. A service that logs everything at INFO makes the aggregation pipeline expensive and the signal impossible to find in the noise. Logs should also never carry PII or secrets in plaintext — a leaked API key in a log line is a security incident, not just a hygiene issue.

#Step 2: Metrics using the RED and USE methods

Metrics answer "is the system healthy, in aggregate, right now" cheaply and at high resolution. Instrument the service with a Prometheus client library (or OpenTelemetry Metrics, which can export to Prometheus or a vendor backend) and expose a /metrics endpoint for scraping. Two frameworks cover almost everything you need:

• RED (for services): Rate of requests, Errors, Duration (latency, ideally as a histogram so you can compute p50/p95/p99, not just an average that hides tail latency).
• USE (for resources): Utilization, Saturation, Errors — CPU, memory, connection pool, queue depth.

These map directly onto the four golden signals from Google's SRE book: latency, traffic, errors, and saturation. Build one Grafana dashboard per service using exactly these four signals as its top row, before any custom business metric gets added below — a dashboard that leads with fifteen bespoke metrics and buries latency and error rate is a dashboard nobody can use during an incident.

#Step 3: Distributed tracing to see across service boundaries

Once a request crosses more than one service, metrics and logs alone can't show you the shape of the call graph. Instrument the service with an OpenTelemetry SDK, which propagates trace context via the W3C traceparent header across every outbound HTTP/gRPC call, so a single trace_id ties together every span from every service the request touched. Spans export to an OpenTelemetry Collector, which fans out to a tracing backend — Jaeger, Grafana Tempo, or a vendor APM.

At any real scale, you cannot record every span of every request forever — the storage and processing cost is prohibitive. This is where sampling matters: head-based sampling decides whether to keep a trace at the very start (cheap, but you might discard the one trace you needed); tail-based sampling buffers the whole trace and decides at the end, which lets you deliberately keep 100% of traces that errored or ran slow while sampling down the boring, fast, successful ones. Tail-based sampling is more operationally complex (it needs a buffering collector) but gives you the traces that actually matter for debugging, not just a random sample of mostly-healthy traffic.

#Step 4: Alerting on SLOs and error budgets, not static thresholds

This is the step that separates a senior design from a junior one. Static threshold alerts ("page if CPU > 80%") are noisy and don't reflect user impact. The better approach: define a Service Level Indicator (SLI) — e.g., the proportion of requests completed successfully within a latency target — and a Service Level Objective (SLO) on top of it, like "99.9% of requests succeed within 300ms, measured over a rolling 30-day window." The gap between 100% and the SLO (0.1% here) is your error budget — the amount of "badness" you're allowed to spend before you've broken your promise to users.

Alert on burn rate, not on the raw SLI: how fast is the service consuming its error budget, over multiple time windows (e.g., a fast 5-minute window to catch a sudden outage, and a slower 1-hour window to catch a persistent, lower-grade degradation). A single-window static alert either fires on tiny blips that self-resolve or misses a slow-burning problem for hours. Multi-window, multi-burn-rate alerting pages only when the budget is actually at risk, which directly reduces alert fatigue.

Route alerts through Alertmanager (Prometheus) or a vendor equivalent, then to PagerDuty or Opsgenie for on-call scheduling, escalation policies, and deduplication. Every page should link to a runbook — if a human has to improvise a response from scratch at 3 a.m., the alert wasn't ready to ship.

#Tying it all together for on-call

The point of this diagram isn't the tool names — swap Datadog?, New Relic, or Elastic for any node and the shape doesn't change. The point is the correlation path: a page fires because an SLO burn-rate alert crossed a threshold, the on-call engineer opens Grafana and sees which golden signal moved, drills from the metric into the specific slow or failing traces sharing that time window, and from a trace's trace_id jumps straight into the exact log lines from every service that request touched — no manual timestamp-matching across three different UIs. That correlation path, built in from day one via trace_id propagation, is the difference between a 10-minute diagnosis and a 90-minute one.

#Cost and operational trade-offs worth naming out loud

An interviewer will respect an answer that proactively raises trade-offs rather than presenting an unlimited-budget design:

• Cardinality control on metrics. A Prometheus label with unbounded values (user ID, raw URL with path params) causes cardinality explosion and can take down the metrics backend itself — labels must stay low-cardinality (route template, status code class, not the raw path).
• Sampling strategy for traces. 100% trace retention is rarely affordable past a certain request volume; tail-based sampling biased toward errors and slow requests gets most of the debugging value at a fraction of the cost.
• Log volume and retention tiers. Hot storage for recent logs, cheaper cold/archive storage for compliance-driven long retention, and disciplined log levels to avoid paying to store noise.
• Alert fatigue is a design failure, not a tuning problem. If on-call routinely acknowledges and dismisses pages without acting, the SLO thresholds or burn-rate windows are wrong, not the humans.

🧠 Check your understanding
0/3 · 0/3 answered
1. A team defines an SLO of "99.9% of requests succeed within 300ms over a rolling 30-day window." What does the remaining 0.1% represent?
The error budget — the amount of acceptable failure the team can spend before breaking its commitment to usersA rounding artifact with no operational meaningThe percentage of traffic that should be sampled for tracingThe maximum allowed CPU utilization on the service's hosts
2. Why is tail-based sampling often preferred over head-based sampling for distributed tracing in a cost-constrained system?
Tail-based sampling is simpler to operate because it requires no bufferingTail-based sampling decides after the full trace completes, so it can deliberately retain traces that errored or ran slow while discarding routine successful onesTail-based sampling eliminates the need for a trace_id entirelyHead-based sampling only works with Prometheus, while tail-based sampling works with any backend
3. Which combination best represents the RED method for monitoring a service (as opposed to the USE method for a resource)?
Retries, Escalations, DiagnosticsReliability, Efficiency, DurabilityRate, Errors, DurationRequests, Events, Downtime