How does distributed tracing work — what's a trace, a span, and how does context propagate across service calls?

#The question tracing answers

A request comes into your API gateway, hits the order service, which calls the customer service and the inventory service, and the inventory service calls a database and a caching layer. The whole thing takes 340ms and the customer says it feels slow. Logs from five different services, each with its own timestamps and no shared identifier, will not tell you which of those seven hops accounted for the 340ms. Distributed tracing exists to answer exactly that question: where, across a chain of service calls, did the time actually go.

#Trace and span, precisely

A trace represents one request's entire journey through the system — every service it touched, from the moment it entered at the edge to the moment the final response left. A trace is not a single record; it's a tree (or more generally a directed graph) built out of many smaller records.

A span is one of those records: a single unit of work with a name, a start time, a duration, a status (ok or error), a set of key-value attributes, and optionally timestamped events within it (comparable to structured log lines attached to that unit of work). Every span carries a span_id unique to itself and a parent_span_id pointing at the span that caused it — except the very first span in a trace, the root span, which has no parent. Every span in the same trace shares one trace_id.

Reconstructing that tree from trace_id plus each span's parent_span_id is exactly how a trace viewer draws the familiar waterfall diagram, and it's immediately obvious from this one where the 180ms went: not the network hops, not the customer-service branch, but a 95ms database query buried three levels down inside inventory-service.

#Getting the trace ID across a process boundary

Within one process, propagating context is comparatively easy — a language runtime's async-local-storage or thread-local mechanism can carry "the currently active span" through nested async calls without threading a parameter through every function signature. The hard part is crossing a process boundary, because the callee has no shared memory with the caller at all. The only channel available is whatever the two processes are already using to talk — HTTP headers, gRPC metadata, or a message's headers on a queue.

This is standardized as the W3C Trace Context specification. Before a service makes an outbound call, it serializes the active trace and span IDs into a traceparent header of the form 00-<trace-id>-<parent-span-id>-<flags> — for example 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. The receiving service parses that header, and any span it creates while handling the request is registered as a child of the span ID it just received, even though the two services have never shared any memory and may not even be written in the same language.

#Instrumenting it: OpenTelemetry

OpenTelemetry (OTel) is the vendor-neutral standard for producing this data: an API and SDK for creating spans and propagating context, plus exporters that ship the finished spans to a backend like Jaeger, Tempo, Zipkin, or a commercial APM. Auto-instrumentation libraries hook into common frameworks — HTTP servers and clients, popular database drivers — so a large share of spans are created without you writing anything. Manual instrumentation is for the spans that matter to your business logic specifically.

Wrap long lines
import { trace, context, propagation, SpanStatusCode, SpanKind } from "@opentelemetry/api";

const tracer = trace.getTracer("order-service", "1.0.0");

async function getOrderSummary(orderId: string) {
  return tracer.startActiveSpan("get-order-summary", async (span) => {
    span.setAttribute("order.id", orderId);

    try {
      const customer = await callDownstream("customer-service", "/customers/" + orderId);
      const inventory = await callDownstream("inventory-service", "/reservations/" + orderId);

      span.setStatus({ code: SpanStatusCode.OK });
      return { customer, inventory };
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR, message: (err as Error).message });
      throw err;
    } finally {
      span.end();
    }
  });
}

async function callDownstream(serviceName: string, path: string) {
  return tracer.startActiveSpan(
    "http-call:" + serviceName,
    { kind: SpanKind.CLIENT },
    async (span) => {
      const headers: Record<string, string> = {};
      // Serializes the active trace/span IDs into the outgoing "traceparent"
      // header so the callee's spans attach as children of this one.
      propagation.inject(context.active(), headers);

      try {
        const res = await fetch("https://" + serviceName + path, { headers });
        span.setAttribute("http.status_code", res.status);
        return await res.json();
      } finally {
        span.end();
      }
    }
  );
}

startActiveSpan both creates the span and makes it the "current" one in the execution context, which is what lets a nested call to callDownstream automatically create its span as a child of get-order-summary without any explicit parent reference — the context propagation machinery handles that part in-process. Crossing to the next service is the one place you need to act explicitly: propagation.inject is what actually writes the traceparent header.

#Nobody traces every request

At meaningful scale, recording and exporting a span for every single request is often too expensive — in storage cost and in the CPU overhead of the instrumentation itself. Sampling decides which traces get kept. Head-based sampling makes the decision at the very start of a trace, before anything is known about how it will turn out — typically a random percentage, like "keep 1% of traces," decided at the root span and propagated down so the whole trace is sampled consistently. It's cheap and simple, but a rare, expensive, or failing request has the same small chance of being kept as a routine one, so teams usually combine it with a rule like "always keep traces that end in an error," decided locally per service. Tail-based sampling defers the decision until the entire trace has been assembled, which requires a collector to buffer all of a trace's spans in memory until it completes — but it lets you keep every trace that was slow or errored regardless of the random sample rate, at the cost of running that buffering infrastructure.