How does distributed tracing sampling actually work — why can't you trace 100% of requests at scale?

#Why not just trace everything

A single trace of one request through a modern microservice architecture can easily contain dozens of spans — one per service hop, one per database call, one per outbound HTTP call, one per queue publish. Each span carries a handful of key-value attributes, timestamps, and a reference to its parent. That's not free: it costs CPU to create and serialize, network bandwidth to export from every process to a collector, ingestion capacity on the collector itself, and storage in the tracing backend (Jaeger, Tempo, Honeycomb, X-Ray) — and unlike a metric, which stays a fixed size no matter how much traffic you have, tracing data grows linearly with request volume and with how deep your call graph is.

At meaningful production scale — tens of thousands of requests per second across dozens of services — tracing every single request would mean exporting and storing an enormous, mostly redundant volume of data: the overwhelming majority of requests are fast and successful, and a trace of a fast, successful request looks almost identical to the last ten thousand fast, successful traces. You're paying full price to keep data that adds almost no new information. Sampling exists to spend that budget on the traces that are actually informative instead.

#Head-based sampling: decide first, ask questions never

Head-based sampling makes the keep-or-drop decision at the very start of a trace — typically at the edge service or the client SDK that creates the root span — and propagates that decision to every downstream service via the trace context (the sampled flag inside the W3C traceparent header, or an equivalent in whatever propagation format you use). Every service downstream just respects the flag it received; it doesn't make its own independent decision.

The simplest policy is probabilistic: sample X% of traces, chosen via a deterministic hash of the trace ID (not a fresh random roll per service — that would make different services in the same trace disagree about whether it's sampled). This is why it scales so well: no coordination between services is required, there's no buffering, and the decision is made once, cheaply, before any real work happens.

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

# Sample 5% of newly created (root) traces; any trace that already
# arrived with a sampled=true flag from an upstream caller is kept
# regardless, via ParentBased's default respect-the-parent behavior.
sampler = ParentBased(root=TraceIdRatioBased(0.05))
provider = TracerProvider(sampler=sampler)

The problem with head-based sampling is that the decision is made before anyone knows whether the trace will be interesting. A flat 5% sample overwhelmingly captures the same boring, fast, successful requests that make up 95%+ of your traffic, and is statistically very likely to miss the rare error or the one request that took 8 seconds instead of 80 milliseconds — precisely the traces you'd actually want when debugging an incident.

#Tail-based sampling: decide after you've seen the whole story

Tail-based sampling defers the decision until the entire trace has finished — after every span from every service has arrived. A dedicated layer, typically the OpenTelemetry Collector's tail_sampling processor, buffers all spans belonging to a trace ID for a configurable wait window, then evaluates policies against the complete trace: always keep it if it contains an error, always keep it if total latency exceeded some threshold, and otherwise fall back to a low probabilistic rate for the boring remainder.

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    expected_new_traces_per_sec: 2000
    policies:
      - name: always-sample-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: always-sample-slow-requests
        type: latency
        latency:
          threshold_ms: 500
      - name: baseline-probabilistic
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

exporters:
  otlp:
    endpoint: tempo:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [otlp]

This gets you far higher-fidelity traces exactly where you need them — errors and slow requests are essentially always kept, instead of surviving only by chance — at a real infrastructural cost: every span, from every service, must first reach the collector and sit in memory for the entire decision_wait window before a decision is made, which means the collector layer needs enough memory to buffer the full trace volume for that window, and if a service pool load-balances traces across multiple collector instances, it must do so consistently by trace ID (not round-robin per span), or the same trace's spans end up split across collectors that each only see a partial picture.

#What a sampled trace looks like next to a dropped one

Note that in head-based sampling, spans for a dropped trace are still created locally — the instrumentation doesn't know the trace is unsampled until it checks the propagated flag — they're just never exported, which is where the actual cost saving happens (network and storage, not the marginal CPU of building the span object itself).

#Choosing between them in practice

In real production systems these aren't mutually exclusive: a common pattern is head-based sampling at the SDK to cap how much data even reaches the collector fleet (so the collector isn't itself overwhelmed), with tail-based policies applied on what does arrive, so errors and slow requests are captured near-exhaustively while the ordinary case is thinned out.