How do you choose a concurrency model for a given workload — CPU-bound vs. I/O-bound?

#Start by classifying the workload, not the technology

The single most common mistake in this space is picking a concurrency model because it is fashionable or because the team already knows it, instead of because it fits how the work actually spends its time. Every unit of work a backend does falls somewhere on a spectrum between two extremes:

• I/O-bound: the task spends most of its wall-clock time waiting — on a database round trip, an HTTP call to another service, a disk read, a DNS lookup. The CPU is idle during that wait; the limiting resource is how many things you can have "in flight" at once, not how much computation any one core can do.
• CPU-bound: the task spends most of its wall-clock time actually computing — encoding video, compressing a file, running a cryptographic hash, evaluating a machine learning model, rendering a PDF. The limiting resource is genuinely how many cores you have, because no amount of clever scheduling makes a core do two multiplications at once.

Most real backend services are a mix, and the practical move is to identify which parts of the request path are which — the JSON-serializing, database-querying, cache-checking parts of a typical CRUD endpoint are overwhelmingly I/O-bound; a thumbnail-generation or report-rendering endpoint sitting next to it is CPU-bound — rather than assuming the whole service is one or the other.

#The three models

Thread-per-request. Every incoming request is handed a dedicated thread (an OS thread in classic Java servlet containers like Tomcat, or a green thread in something like Ruby's Puma) that runs the handler top-to-bottom as ordinary, synchronous, blocking code — a database call just blocks that thread until it returns. The code reads exactly like a script, with no callbacks or futures to reason about, which is the model's biggest advantage: stack traces are linear, debugging is straightforward, and there is no risk of accidentally blocking a shared event loop. Its ceiling is set by how expensive threads are in that runtime: OS threads typically reserve a fixed stack (often ~1MB on the JVM by default) and cost real memory and scheduler overhead just sitting there blocked, so tens of thousands of concurrent slow connections can exhaust memory or the OS's thread limits long before CPU becomes the bottleneck — the classic C10K problem.

Event loop (single-threaded async). One thread (or a small, fixed number of them) runs a loop that dispatches whichever I/O operation has become ready — Node.js, Python's asyncio, and Nginx's worker processes all follow this shape. Nothing is allowed to block that thread; instead of calling a blocking read, code registers a callback or awaits a promise/future, and the loop moves on to service other ready work while the kernel handles the actual I/O asynchronously. Because a blocked-on-I/O "connection" costs almost nothing while it waits (no dedicated thread, no dedicated stack), a single event loop can hold open tens of thousands of concurrent connections cheaply — its ceiling is memory per connection object, not per-thread cost. Its sharp edge is exactly the flip side of that efficiency: any CPU-bound work executed on the loop's thread — a large JSON parse, a synchronous crypto operation, a tight loop — blocks every other pending request on that thread for as long as it runs, because there is no preemption.

Worker pool (thread pool or process pool for real parallel compute). A fixed or bounded set of workers actually execute in parallel across cores — an OS thread pool in a runtime with true parallel threads (Java, Go, Rust, C#, Kotlin/JVM), or separate OS processes in a runtime with a global interpreter lock (Python's multiprocessing, or multiple single-threaded worker processes behind a load balancer, which is how Gunicorn or Puma-in-clustered-mode gets multi-core use out of Python or Ruby). This is the only model of the three that gives genuine CPU parallelism, because it maps work onto multiple independently-scheduled execution contexts rather than multiplexing it onto one. The trade-off is coordination cost: moving work into a worker pool from a request-handling thread means some form of hand-off (a queue, an IPC channel, serialized messages between processes), and pool size beyond the core count buys nothing for pure computation — it just adds context-switch overhead, mirroring the same math that governs CPU-bound thread pool sizing generally.

A fourth shape is worth naming because it increasingly blurs the first two: M:N green threads / coroutines (Go's goroutines, Kotlin coroutines, Erlang/Elixir processes). Code is written in the synchronous, linear style of thread-per-request, but the runtime multiplexes a very large number of these lightweight units onto a small pool of OS threads, parking a coroutine instead of its underlying OS thread when it blocks on I/O — giving the event loop's connection-cost efficiency with the debugging ergonomics of synchronous code, and additionally spreading across multiple OS threads (Go's GOMAXPROCS) to get real parallel execution for CPU-bound segments too, which neither a pure thread-per-request nor a pure single-threaded event loop model does on its own.

#Comparison at a glance

#The decision framework

1. Measure, don't guess. Determine, per code path, whether time is dominated by waiting or computing.
2. Pure I/O-bound, need very high concurrency (tens of thousands of simultaneous connections, e.g. a chat gateway, a proxy, an API gateway): reach for an event loop or M:N coroutines. Thread-per-request will hit thread/memory limits far sooner for the same connection count.
3. Pure CPU-bound: parallelism is capped by core count no matter what model you pick, so size a worker pool to roughly the core count and use real parallel threads if your runtime has them (Java, Go, Rust, C#) or separate processes if it has a GIL-like constraint (Python, and historically Ruby's MRI).
4. Mixed, which is the common case for a typical web backend (I/O-bound most of the time, with occasional CPU-bound spikes like report generation, image processing, or password hashing): keep the request-serving path on whichever I/O-oriented model you've chosen, and explicitly offload the CPU-bound chunks to a separate worker pool or job queue so a slow computation never blocks the connections handling everything else. This is the single most common concurrency bug in production event-loop services: someone runs a synchronous, CPU-heavy operation directly in a request handler and it stalls every other in-flight request on that process.
5. Weigh operational simplicity against the scaling ceiling you actually need. Thread-per-request is the easiest model to write, debug, and onboard new engineers onto — pick it when your realistic concurrency needs (hundreds to a few thousand connections) do not require squeezing out every bit of per-connection efficiency. Reach for event loops or M:N coroutines only when you genuinely need the higher ceiling, since they demand more discipline (never block the shared thread) in exchange for that ceiling.

🧠 Check your understanding
0/2 · 0/2 answered
1. A backend service handles two kinds of requests: fetching user profiles (a single fast database lookup) and generating PDF invoices (heavy in-process rendering). Both currently run on the same single-threaded event loop. Users report that profile fetches become slow whenever many invoices are generated at once. What is the root cause and the right fix?
Profile fetches should be switched to thread-per-request insteadThe event loop is being blocked by the CPU-bound PDF rendering; move rendering to a separate worker pool or processThe event loop needs more OS threads to handle more connectionsThe database connection pool is too small; increase it
2. A service needs to hold open roughly 50,000 concurrent long-lived connections that are idle almost all the time (e.g. a notification/streaming gateway), with negligible CPU work per connection. Which model is best suited, and why?
Thread-per-request, because linear code is easier to debug at scaleAn event loop or M:N coroutine model, because each idle connection costs little beyond memory for its state, unlike a dedicated OS thread per connectionA worker pool sized to the core count, because more workers means more concurrencyIncreasing terminationGracePeriodSeconds so idle connections do not time out