How do you handle a hot key or hot partition that's overwhelming a single shard?

#What a hot key actually is

A hot key (or hot partition) is what happens when your sharding scheme is theoretically balanced — hash-based, evenly distributed, textbook-correct — but real-world traffic isn't uniform. One viral product page, one celebrity's follower-count, one Black Friday SKU, or one globally-shared feature flag gets accessed or updated orders of magnitude more than any other key, and every request for it lands on the exact same physical shard because that's what deterministic hashing guarantees. The rest? of your cluster can be sitting at 5% CPU while that one shard is pegged at 100% and timing out — adding more nodes to the cluster does nothing, because the hash function still sends every request for that key to the same place.

#Detecting the problem first

Before reaching for a fix, confirm it's actually a hot-key problem rather than general undersizing: look for one partition/shard with dramatically higher CPU, request latency, or throttling errors than its siblings while the rest of the cluster is idle. Most managed data stores expose per-partition metrics for exactly this reason (DynamoDB's per-partition throttling metrics, Redis Cluster's per-slot stats, Kafka's per-partition consumer lag). If every shard is uniformly hot, you have a capacity problem, not a hot-key problem, and the fix is different (more shards, not smarter keys).

#Fix 1: salt the key to spread writes across shards

The standard technique is key salting: instead of one logical key, maintain N physical keys, and route each write to a randomly (or round-robin) chosen one. Reads become a scatter-gather that sums across all N. This trades read-side complexity for write-side scalability — usually an excellent trade, since hot keys are almost always hot on the write side (view counters, like counts, trending scores).

import random
import redis

NUM_SHARDS = 10

class ShardedCounter:
    """Spreads a single hot counter across NUM_SHARDS physical keys so that
    no single Redis key (and therefore no single hash slot / partition)
    absorbs all the write traffic."""

    def __init__(self, client: redis.Redis, base_key: str, shards: int = NUM_SHARDS):
        self.client = client
        self.base_key = base_key
        self.shards = shards

    def _shard_key(self, shard: int) -> str:
        return f"{self.base_key}:shard:{shard}"

    def increment(self, amount: int = 1) -> None:
        # Pick a random shard on every write so load spreads evenly
        # across all NUM_SHARDS keys instead of hammering one.
        shard = random.randint(0, self.shards - 1)
        self.client.incrby(self._shard_key(shard), amount)

    def get_total(self) -> int:
        # Scatter-gather: read every shard and sum. This is the cost we
        # pay on the read side for cheap, distributed writes.
        pipe = self.client.pipeline()
        for shard in range(self.shards):
            pipe.get(self._shard_key(shard))
        values = pipe.execute()
        return sum(int(v) for v in values if v is not None)

#Fix 2: cache the aggregate to bound read cost

Scatter-gather isn't free — summing 10 (or 100) shards on every single read multiplies your read load by the shard count. If the aggregate doesn't need to be perfectly fresh (a view counter that's 2 seconds stale is fine; a bank balance is not), cache the summed total for a short TTL so most reads are served from a single cache hit instead of a full scatter-gather.

CACHE_TTL_SECONDS = 2

class CachedShardedCounter(ShardedCounter):
    """Bounds the cost of get_total()'s scatter-gather by caching the
    aggregated result for a short window. A hot key is read far more
    often than it needs a perfectly fresh value."""

    def __init__(self, client: redis.Redis, base_key: str, shards: int = NUM_SHARDS):
        super().__init__(client, base_key, shards)
        self._cache_key = f"{base_key}:total_cache"

    def get_total(self) -> int:
        cached = self.client.get(self._cache_key)
        if cached is not None:
            return int(cached)
        total = super().get_total()
        self.client.set(self._cache_key, total, ex=CACHE_TTL_SECONDS)
        return total

#Fix 3: aggregate locally before you ever hit the shard

The cheapest write is the one you never send. For counters especially, application servers can accumulate increments in local memory for a short window (say, 500ms-1s) and flush a single batched increment instead of one round trip per event. If a product page gets 10,000 view events per second across 50 app servers, local pre-aggregation can turn that into 50 requests per second hitting the counter (one per server per flush interval) instead of 10,000 — often enough on its own to make salting unnecessary.

#Fix 4: bake the shard into the partition key itself

Systems like DynamoDB don't give you a separate "shard picker" — the partition key is the sharding mechanism. The pattern is the same idea applied at the schema level: append a random or hashed suffix directly to the partition key so what used to be one hot partition key becomes several physical partitions.

import random

SUFFIX_RANGE = 10

def write_partition_key(product_id: str) -> str:
    """DynamoDB-style: bake the shard suffix into the partition key itself
    so the hot item's writes land on SUFFIX_RANGE distinct physical
    partitions instead of one."""
    suffix = random.randint(0, SUFFIX_RANGE - 1)
    return f"PRODUCT#{product_id}#SHARD#{suffix}"

def read_all_partitions(product_id: str) -> list[str]:
    return [f"PRODUCT#{product_id}#SHARD#{i}" for i in range(SUFFIX_RANGE)]

A BatchGetItem (or the equivalent bulk read in your store of choice) against read_all_partitions(product_id) replaces the single hot GetItem with a fan-out read across evenly-loaded partitions.

🧠 Check your understanding
0/3 · 0/3 answered
1. Why doesn't adding more read replicas fix a hot partition caused by a high write volume to a single key?
Read replicas actively slow down write throughputReplicas only exist for backup purposes and serve no trafficHot partitions only ever affect read traffic, never writesWrites still have to go through the same primary/leader partition regardless of how many read replicas exist
2. In the ShardedCounter pattern, what is the main cost being traded for write scalability?
Reads become more expensive because get_total() must scatter-gather across all shards and sum themWrites become slower because of the extra hashing stepIncreased storage cost only, with no effect on readsThe pattern eliminates the need for caching entirely
3. Why is random suffixing that isn't actually uniform (e.g., sharding by day of week) still risky?
It is only a problem for SQL databases, not NoSQLA low-cardinality or skewed suffix just reproduces the same hot-partition problem at a smaller number of new partitionsIt has no risk, any suffix strategy fully solves hot-key problemsIt changes the read consistency model of the database