How does a log aggregation architecture like ELK or Grafana Loki actually work end to end?

#Why logs can't just live on the box anymore

In a distributed system, a single user request might touch six services running on containers that get rescheduled, restarted, or destroyed within minutes. If each service only writes logs to its own local disk, debugging turns into archaeology: you have to figure out which node the container happened to land on, hope it hasn't been evicted yet, and exec in before the evidence disappears. Centralized log aggregation solves this by shipping every log line off the host in real time, so the entire fleet's logs become searchable from one place, correlated by request ID, service name, or trace ID.

This is one of the most common system-design-adjacent interview questions in backend and platform roles, because it tests whether you understand why each stage of a logging pipeline exists — not just that "Kibana shows logs."

#The five stages every architecture shares

Whether you're running the classic ELK stack (Elasticsearch, Logstash, Kibana) or the more modern Grafana Loki stack (Promtail or Fluent Bit, Loki, Grafana), the pipeline breaks down into the same five conceptual stages:

1. Collection. A lightweight agent runs on every host — usually as a Kubernetes? DaemonSet, one per node — and tails container stdout/stderr or log files on disk. Common agents: Filebeat, Fluentd, Fluent Bit, Promtail. The agent enriches each line with metadata the application never had to think about: pod name, namespace, container ID, node name, labels.
2. Transport / buffering. Shipped logs land in a buffer — often Kafka, sometimes Redis — before they reach the processing tier. This absorbs traffic spikes and protects the pipeline: if Elasticsearch or Loki goes down for maintenance, logs queue up in Kafka instead of being dropped or backing up into the application itself.
3. Processing. This is where the two architectures diverge the most (see below). Logstash applies grok patterns and filters to turn unstructured text into structured fields. Loki instead extracts only a small set of labels and leaves the log line's body untouched.
4. Storage & indexing. The processed data lands in a purpose-built store: Elasticsearch's inverted index for ELK, or Loki's chunk store (usually S3/GCS-backed) plus a lightweight label index.
5. Query & visualization. Kibana or Grafana lets you filter by field/label, run full-text or LogQL/DSL queries, build dashboards, and — critically — jump from a metric spike straight into the logs that explain it.

#ELK vs. Loki: two fundamentally different bets

The most interesting part of this question is explaining why Loki exists when Elasticsearch already does full-text log search. The answer is cost and operational weight at scale.

Loki's design philosophy is explicitly "index like Prometheus, store like S3": it deliberately gives up some query flexibility to keep the index tiny, which is what lets it scale to terabytes of logs per day without an Elasticsearch-sized cluster bill.

#Collection patterns in Kubernetes

Two patterns dominate. A DaemonSet runs one log shipper per node and reads every container's stdout/stderr from the node's container runtime log directory — this is the default, most resource-efficient approach for the common case. A sidecar runs one shipper per pod, useful when an application writes to a file instead of stdout, or needs per-app parsing before anything leaves the pod. Sidecars cost more (one extra container per pod) so they're the exception, not the rule.

#Retention, tiering, and why structured logging matters

Nobody keeps every log line hot forever — it's too expensive. Elasticsearch's Index Lifecycle Management (ILM) rolls indices through hot (fast SSD, actively written), warm (cheaper, read-only), and cold/frozen tiers, eventually deleting or archiving. Loki applies retention per log stream directly against its object-storage-backed chunks, which is naturally cheaper since cold data is already sitting in S3-class storage.

The single highest-leverage decision a team makes here happens upstream of any of this infrastructure: whether the application emits structured JSON logs instead of freeform text. A JSON log line with explicit timestamp, level, service, trace_id, and message fields needs no fragile grok/regex parsing step at all — the processing stage becomes a straight field mapping. Freeform text logs force Logstash (or a Loki pipeline stage) to guess field boundaries with regular expressions that break the moment someone changes a log message's wording.

#What this means in an interview

If asked this question, don't just name tools — narrate the pipeline: collection agent, buffer, processing/parsing, index/storage, query layer, and explain the ELK-vs-Loki trade-off in terms of what gets indexed, because that single design decision explains almost every downstream difference in cost, query power, and operational complexity.