How do health checks and readiness/liveness probes actually work, and why are they different?

#Two different questions, two different probes

"Is this instance broken and should be restarted?" and "is this instance currently able to serve traffic?" sound similar but demand opposite actions, which is exactly why Kubernetes? (and most orchestration platforms) expose them as two separate probes rather than one. A liveness probe answers the first question: if it fails repeatedly, the orchestrator kills and restarts the container, on the assumption that the process is stuck in a way only a restart fixes (a deadlock, a hung thread pool, a memory leak past the point of recovery). A readiness probe answers the second: if it fails, the orchestrator simply stops routing new traffic to that pod (removes it from the Service's endpoint list) without touching the container at all — the process keeps running, and traffic resumes automatically the moment the probe passes again. A third, less universally known one, the startup probe, exists purely to give slow-starting applications (a JVM warming up, a large cache to preload) a grace period before liveness checks even begin, so a legitimately-still-booting app isn't killed for failing a liveness check it was never going to pass yet.

#Why conflating them causes crash loops

The single most common mistake — and a great interview trap — is wiring the liveness probe to check downstream dependencies (a database connection, a downstream API). It seems reasonable ("if I can't reach the database, something's wrong!"), but it's precisely backwards: if the database has a brief blip, every pod in the deployment fails its liveness check simultaneously, the orchestrator restarts all of them at once, and now you've turned a transient database hiccup into a self-inflicted, fleet-wide outage — the pods come back up, immediately fail liveness again because the database still hasn't recovered, and you're in a crash-loop that makes the original problem worse and harder to diagnose. The database being briefly unreachable is a readiness concern (stop sending traffic here until it's back) not a liveness concern (this process itself is not broken, it just can't currently reach something external).

#Kubernetes configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-service
spec:
  replicas: 4
  selector:
    matchLabels:
      app: checkout-service
  template:
    metadata:
      labels:
        app: checkout-service
    spec:
      containers:
        - name: checkout-service
          image: registry.example.com/checkout-service:1.14.2
          ports:
            - containerPort: 8080
          startupProbe:
            httpGet:
              path: /healthz
              port: 8080
            failureThreshold: 30   # allow up to 30 * 5s = 150s to boot
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 0   # startupProbe already covered boot time
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3      # 3 consecutive misses -> restart
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8080
            initialDelaySeconds: 0
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 2       # 2 consecutive misses -> drop from Service
            successThreshold: 1       # 1 success -> add back to Service

Notice the two probes point at different endpoints — /healthz and /readyz — on purpose; sharing one endpoint for both is how teams accidentally reintroduce the crash-loop problem above.

#The endpoint implementations

package main

import (
	"encoding/json"
	"net/http"
	"sync/atomic"
)

// draining is flipped to true during graceful shutdown so readiness
// fails immediately and the load balancer stops sending new traffic,
// while in-flight requests finish normally.
var draining int32

func healthzHandler(w http.ResponseWriter, r *http.Request) {
	// Liveness: only check that THIS process's core loop is responsive.
	// No downstream calls here, deliberately -- a slow dependency must
	// never be able to get this pod killed.
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(map[string]string{"status": "alive"})
}

func readyzHandler(w http.ResponseWriter, r *http.Request) {
	if atomic.LoadInt32(&draining) == 1 {
		http.Error(w, "draining", http.StatusServiceUnavailable)
		return
	}

	if err := pingDatabase(); err != nil {
		http.Error(w, "database unreachable: "+err.Error(), http.StatusServiceUnavailable)
		return
	}

	if err := pingDownstreamPaymentsAPI(); err != nil {
		http.Error(w, "payments API unreachable: "+err.Error(), http.StatusServiceUnavailable)
		return
	}

	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(map[string]string{"status": "ready"})
}

func pingDatabase() error {
	// A cheap connectivity check against the connection pool,
	// not a full query -- e.g. db.PingContext(ctx) with a short timeout.
	return nil
}

func pingDownstreamPaymentsAPI() error {
	// A lightweight health check against a critical hard dependency.
	return nil
}

func main() {
	http.HandleFunc("/healthz", healthzHandler)
	http.HandleFunc("/readyz", readyzHandler)
	http.ListenAndServe(":8080", nil)
}

The draining flag is worth calling out on its own: on SIGTERM, a well-behaved service flips it to true immediately, which makes /readyz start failing right away so the load balancer stops routing new requests, while the process keeps running long enough to finish requests already in flight before actually exiting — this is what makes rolling deploys avoid dropped requests, and it has nothing to do with the liveness probe at all.

🧠 Check your understanding
0/2 · 0/2 answered
1. Why is it a mistake to make a liveness probe check the database connection?
Liveness probes cannot make network calls at allA brief database outage would cause every pod to fail liveness simultaneously, triggering a fleet-wide restart that turns a transient dependency issue into a self-inflicted crash loopLiveness probes run less frequently than readiness probes, so the check would be staleDatabase checks are always too slow to complete within a timeout
2. What is the practical effect of a readiness probe failing, as opposed to a liveness probe failing?
Both trigger an identical container restartThe pod is removed from the Service's routable endpoints without restarting the container, and is added back automatically once the probe passes againReadiness failures immediately terminate the entire DeploymentReadiness failures are ignored unless liveness also fails