What is negative caching (caching ‘not found’ results), and why does it matter for protecting your backend?

#The Blind Spot in Ordinary Caching

Most caching strategies are built around one implicit assumption: the thing being looked up exists. A cache-aside read path is simple — check the cache, and on a miss, go to the database, get the row, and write it back into the cache for next time. That pattern protects the database beautifully for popular, existing keys. It does nothing at all for keys that do not exist, because there is nothing to cache. Every single lookup for user:99999999 (a deleted account, a typo'd ID, or an attacker probing sequential IDs) sails straight past the cache and lands on the database, every time, forever.

This gap has a name in distributed-systems circles: cache penetration. It shows up in a few very ordinary ways — a mobile client that keeps retrying a request for a resource that was deleted server-side, a public API endpoint that gets hit by bots enumerating IDs looking for something valid, or simply a bug where a foreign key points at a row that no longer exists. None of these are exotic; they are the default behavior of a system that only caches "yes."

#Negative Caching: Remembering "No" Too

Negative caching closes the gap by caching the absence of a result, not just its presence. When a lookup comes back empty, you write a small sentinel value into the cache — instead of leaving that slot empty — with a TTL that is deliberately much shorter than the TTL you use for real data. The next request for the same missing key gets its answer from the cache layer and never touches the database at all.

The TTL asymmetry matters. If a "not found" result lived for an hour, a user who just signed up would appear not to exist to your own backend for up to an hour, purely because someone (possibly the client itself, prefetching) looked them up half a second before the signup transaction committed. A short negative TTL — seconds to a couple of minutes — bounds that staleness window to something a support ticket will never be filed about, while still absorbing the bulk of a hammering pattern, since repeated lookups typically arrive in a tight burst rather than spread out over hours.

#Implementation

Wrap long lines
const redis = require("./redisClient"); // ioredis instance
const db = require("./db");

const NEGATIVE_SENTINEL = "__NOT_FOUND__";
const POSITIVE_TTL_SECONDS = 3600; // 1 hour — real records change rarely
const NEGATIVE_TTL_SECONDS = 30;   // 30 seconds — bounds staleness after a create

async function getUserById(userId) {
  const cacheKey = `user:${userId}`;
  const cached = await redis.get(cacheKey);

  if (cached === NEGATIVE_SENTINEL) {
    // Negative cache hit — we already know this ID has nothing behind it.
    return null;
  }
  if (cached !== null) {
    return JSON.parse(cached);
  }

  // True miss: neither a positive nor a negative cache entry exists yet.
  const user = await db.query("SELECT id, name, email FROM users WHERE id = $1", [userId]);

  if (!user) {
    await redis.set(cacheKey, NEGATIVE_SENTINEL, "EX", NEGATIVE_TTL_SECONDS);
    return null;
  }

  await redis.set(cacheKey, JSON.stringify(user), "EX", POSITIVE_TTL_SECONDS);
  return user;
}

// Invalidate the negative entry the moment a matching record is created,
// so a just-created user is never forced to wait out the negative TTL.
async function onUserCreated(user) {
  await db.insert("users", user);
  await redis.del(`user:${user.id}`);
}

Three states now exist where before there were two: a positive hit (return the cached row), a negative hit (return "not found" without touching the database), and a true miss (go to the database, then populate whichever cache entry the result deserves). The sentinel has to be something that can never collide with a real serialized value — a fixed string outside your normal serialization format, a dedicated key prefix, or even a separate Redis key space are all valid choices; what matters is being able to tell "empty because negative" apart from "empty because nothing was ever cached."

#Where This Actually Matters

• Enumeration attacks. An attacker walking through /api/orders/1, /api/orders/2, … looking for one that exists gets absorbed by the negative cache almost entirely — only the first hit on each ID reaches the database.
• Broken references. A product_id that was deleted but is still referenced by old cart rows, wishlists, or search index entries generates repeated lookups for a row that will never come back; negative caching stops that from becoming background load.
• Retry storms. A client that retries a 404 aggressively (bad backoff logic) no longer generates one database query per retry.
• Public autocomplete / lookup endpoints. These are the most exposed to random or malicious input, and the most likely to see near-uniform random keys that are all misses.

🧠 Check your understanding
0/2 · 0/2 answered
1. Why does a negative cache entry typically use a much shorter TTL than a positive one?
Shorter TTLs make GET commands run fasterTo bound how long a newly created record can incorrectly appear as 'not found'Redis expires string values faster than JSON valuesNegative entries take more memory to store
2. What problem does negative caching primarily protect against?
Repeated database lookups for keys that don't existSlow network links between app servers and the cacheCache eviction under memory pressureSerialization overhead for large objects