Design a web crawler that can scale to billions of pages

#Start With the Constraints, Not the Components

Before drawing any boxes, get three numbers straight, because they dictate every design decision downstream: how many pages you need to have crawled (say 5 billion), how fresh they need to stay (news sites recrawled hourly, static brochure sites recrawled monthly), and how polite you must be (no more than roughly one request every 1–2 seconds to any single host, robots.txt honored). Crawling 5 billion pages a month is roughly 2,000 fetches per second sustained, spread across however many hosts exist on the open web — and that fetch rate has to be achieved without concentrating more than a couple of requests per second on any individual site, which is why "distribute the work" and "throttle per host" pull in opposite directions and the whole system is built around reconciling them.

#The URL Frontier: the Heart of the Design

The frontier is not a single queue — a single FIFO would either starve low-priority hosts or let one host dominate the fetch pool. The standard design (going back to Mercator-style crawlers) is two layers:

1. Front queues — one per priority tier — rank URLs by a freshness/importance score (how often the page changes, how many other pages link to it). New and recrawled URLs are dropped into the front queue matching their priority.
2. Back queues — one per host — guarantee politeness. A separate router pulls from the front queues (biased toward higher priority) and routes each URL into the back queue for its host. A fetcher only ever pulls from a back queue whose host is not "in flight" and whose per-host cooldown has elapsed.

This split is what lets the crawler be greedy about priority globally while being strictly polite locally, per host.

#Politeness Is Not Optional

Robots.txt is fetched once per host, cached with a TTL, and consulted before every fetch to that host — both for disallowed paths and for an explicit Crawl-delay. Even when a site specifies no delay, the crawler enforces a sane floor (commonly 1–2 seconds between requests to the same host) and backs off exponentially on 429 or 5xx responses. Consistent hashing of the hostname across the fetcher fleet is what makes this enforceable without a distributed lock: every URL for example.com is routed to the same node (or small set of nodes), so per-host rate limiting is a local, in-memory counter rather than a distributed coordination problem.

#Deduplication Happens at Two Different Levels

• URL-level dedup stops the same URL from being enqueued twice. At billions of URLs, a hash set of every seen URL is too large to keep fully in memory — this is the canonical production use case for the Bloom filter from the previous lesson: a "definitely not seen" answer skips re-enqueuing for free, and the rare false positive just means a page is very occasionally under-crawled, which is an acceptable trade at this scale.
• Content-level dedup catches pages that are reachable via different URLs but are effectively the same document — mirrors, tracking-parameter variants, boilerplate template pages. Exact hashing (SHA-256 of the body) only catches byte-identical duplicates; SimHash or MinHash instead produce a fingerprint that stays similar for near-identical documents, so pages differing only by a timestamp, an ad slot, or a session token still collapse into one entry in the index.

URL normalization has to happen before either dedup step: lowercasing the host, stripping default ports, sorting or dropping tracking query parameters, and resolving . / .. segments — otherwise the same page under four cosmetically different URLs looks like four different pages.

#Fault Tolerance and Fetcher Failure

Work is handed out as leases, not permanent assignments: a fetcher checks out a batch of URLs from its back queue with a lease timeout. If the fetcher crashes or hangs, the lease expires and another fetcher in the fleet picks the same URLs back up — no central coordinator has to notice the crash and manually reassign work. URLs that fail repeatedly (DNS failures, TLS errors, persistent 5xx) are shunted to a dead-letter queue with capped retries and exponential backoff, rather than being retried forever and burning frontier capacity.

#Storage and Freshness Scheduling

Raw HTML lands in blob storage, keyed by a content hash, with a separate metadata store (URL, fetch timestamp, HTTP status, content hash, outbound links) in a wide-column or document store built for high write throughput. The link graph derived from those outbound links feeds both freshness scheduling and importance ranking: pages that change frequently (observed by diffing successive crawls) get requeued sooner, and pages with high in-link counts (a cheap proxy for PageRank-style importance) get a priority boost? in the front queue so the crawl budget is spent where it matters most.