What are the main metric types — counters, gauges, histograms — and when do you use each?

#Three shapes of data, one instrumentation library

Every modern metrics system — Prometheus, and everything that speaks its data model (Datadog?, Grafana Mimir, Amazon Managed Prometheus, VictoriaMetrics) — is built around a small number of metric types, and picking the right one is not a stylistic choice. Each type carries different mathematical guarantees, and using the wrong one produces numbers that are technically emitted but practically meaningless. The three you'll be asked about in almost any backend interview are counters, gauges, and histograms.

#Counter — a number that only goes up

A counter is a cumulative value that can only increase (or reset to zero on process restart). It exists to answer "how many of X happened," never "how many of X exist right now." http_requests_total, errors_total, bytes_sent_total are canonical counters — you never decrement a counter, and if your instrumentation library lets you, that's a sign you actually wanted a gauge.

You almost never query a counter's raw value; you query its rate of change over a window, because the absolute value ("14,502,991 requests since the process started three weeks ago") is rarely interesting on its own. In PromQL that's rate(http_requests_total[5m]) — requests per second, averaged over the last five minutes. Rate calculation is also why counters are designed to survive process restarts gracefully: a counter reset (the value drops because the process restarted) is a known, handled case in rate(), so a deploy doesn't produce a nonsensical negative rate.

#Gauge — a number that goes up and down

A gauge is a snapshot value at the moment it was read: it can go up, down, or stay flat, and represents "how much of X exists right now," not "how many of X happened." queue_depth, active_connections, memory_usage_bytes, goroutines_running, temperature_celsius are gauges. There is no meaningful "rate of a gauge" the way there is for a counter — you read it directly, you can average it over time, and you can alert on thresholds ("gauge above 90 for 5 minutes").

The single most common metrics bug in real codebases is using a gauge for something that should be a counter (or vice versa): a "total errors" gauge that some code path resets to zero on deploy silently discards history and makes rate()-style dashboards impossible; a "queue depth" counter that only ever increments hides the fact that the queue is actually draining.

#Histogram — the shape of a distribution

A histogram answers a question neither counter nor gauge can: not "how many" or "what's the current value," but "what does the distribution of values look like" — almost always used for latency, request size, or payload size. A histogram buckets observations into configurable ranges (le="0.1", le="0.5", le="1", le="+Inf", meaning "less than or equal to") and, under the hood, exposes each bucket as a counter of how many observations fell at or below that boundary, plus a _sum (total of all observed values) and _count (total number of observations).

The payoff is that histograms recorded on different instances of the same service can be summed together server-side and then have quantiles computed after aggregation — histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) gives you a fleet-wide p99 latency. This is precisely what a summary metric type (Prometheus's other latency type, which computes quantiles client-side per instance) cannot do: client-side quantiles are mathematically impossible to average correctly across instances, because the p99 of an average is not the average of p99s. That's the reason histograms, not summaries, are the default recommendation almost everywhere latency needs to be aggregated fleet-wide.

Bucket boundaries matter enormously and are usually the part people get wrong: too few buckets, or boundaries clustered in the wrong range (e.g. all buckets under 10ms for an API whose real latency lives between 50ms and 2s) and histogram_quantile produces an inaccurate, linearly-interpolated estimate no matter how much traffic you have.

#Instrumenting all three

Here's a real, runnable example using Python's prometheus_client, the reference implementation of Prometheus's client-side API — the same three constructors exist, with the same semantics, in every language's client library (prometheus/client_golang, Micrometer for Java, prom-client for Node).

from prometheus_client import Counter, Gauge, Histogram, start_http_server
import random
import time

http_requests_total = Counter(
    "http_requests_total",
    "Total HTTP requests processed",
    ["method", "path", "status"],
)

inflight_requests = Gauge(
    "http_requests_inflight",
    "Number of HTTP requests currently being processed",
)

request_latency_seconds = Histogram(
    "http_request_duration_seconds",
    "HTTP request latency in seconds",
    ["path"],
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5),
)


def handle(method: str, path: str) -> None:
    inflight_requests.inc()
    start = time.perf_counter()
    status = "200"
    try:
        time.sleep(random.uniform(0.01, 0.3))  # simulated business logic
    except Exception:
        status = "500"
        raise
    finally:
        elapsed = time.perf_counter() - start
        request_latency_seconds.labels(path=path).observe(elapsed)
        http_requests_total.labels(method=method, path=path, status=status).inc()
        inflight_requests.dec()


if __name__ == "__main__":
    start_http_server(8000)  # exposes /metrics for Prometheus to scrape
    while True:
        handle("GET", "/checkout")

Notice the shape of each call: .inc() and nothing else for the counter (never .dec()), .inc() and .dec() for the gauge (it moves both ways), and .observe(value) for the histogram, which is the only one of the three that records an actual measured value rather than just "one more happened."

#Choosing the right type, at a glance

🧠 Check your understanding
0/3 · 0/3 answered
1. Which metric type is the right choice for 'number of requests currently being processed by this service'?
CounterGaugeHistogramSummary
2. Why is histogram_quantile() computed server-side across aggregated buckets generally preferred over a client-side summary's quantiles for fleet-wide latency dashboards?
Percentiles computed per-instance cannot be correctly averaged across instances, but histogram buckets can be summed before computing the percentileSummaries are deprecated in Prometheus and no longer supportedHistograms don't require defining bucket boundaries in advanceHistograms use less memory on the client than summaries
3. What is the main risk of adding a raw, unbounded value like user_id as a metric label?
It has no real downside as long as the app scales horizontallyIt causes cardinality explosion, creating a separate time series per unique value and potentially overwhelming the metrics backendPrometheus will silently drop the metricIt makes the metric name too long