How do you handle large file uploads reliably — chunked and resumable uploads?

#Why "Just Upload the File" Breaks at Scale

A single PUT of a 4 GB video file over a flaky mobile connection is a bet that the connection survives for the entire transfer, with no partial credit for a byte of it. Drop at 95% and the naive implementation restarts at 0%. The fix is to stop treating the upload as one atomic operation and instead treat it as a session with state: split the file into chunks, upload chunks independently, and give the client a way to ask "what do you already have?" so it only ever sends what's missing.

This is exactly the shape of Amazon S3's multipart upload API, Google's resumable upload protocol, and the open tus protocol — different wire formats, identical underlying idea.

#The Upload Session Lifecycle

The critical property: the client never has to guess where it left off. It asks the server, gets back the authoritative list of received chunk indices, and resumes from the first gap.

#Implementation

Below is a real (simplified but functioning) resumable upload service in Node/Express. Every design decision maps back to a reliability requirement:

• Chunk index + checksum, not just bytes — a retried PUT for the same index is idempotent?: it overwrites, it doesn't duplicate.
• A session that can be interrogated — GET /uploads/:id returns exactly what's been persisted, which is what makes resuming possible at all.
• Assembly only happens once all chunks are present — partial completion is rejected explicitly rather than silently producing a truncated file.

Wrap long lines
const express = require('express');
const crypto = require('crypto');
const fs = require('fs');
const path = require('path');

const app = express();
const UPLOAD_DIR = './uploads';
// uploadId -> { fileName, chunkSize, totalChunks, receivedChunks: Set<number> }
const sessions = new Map();

app.use(express.json());

// 1. Create a session before any bytes are sent.
app.post('/uploads', (req, res) => {
  const { fileName, totalSize, chunkSize } = req.body;
  const uploadId = crypto.randomUUID();
  const totalChunks = Math.ceil(totalSize / chunkSize);

  fs.mkdirSync(path.join(UPLOAD_DIR, uploadId), { recursive: true });
  sessions.set(uploadId, {
    fileName,
    chunkSize,
    totalChunks,
    receivedChunks: new Set(),
    createdAt: Date.now(),
  });

  res.status(201).json({ uploadId, totalChunks });
});

// 2. Resume support: tell the client exactly what we already have.
app.get('/uploads/:id', (req, res) => {
  const session = sessions.get(req.params.id);
  if (!session) return res.status(404).json({ error: 'unknown upload session' });

  res.json({
    receivedChunks: [...session.receivedChunks].sort((a, b) => a - b),
    totalChunks: session.totalChunks,
  });
});

// 3. Upload a single chunk. Safe to retry: same index just overwrites.
app.put(
  '/uploads/:id/chunks/:index',
  express.raw({ type: '*/*', limit: '20mb' }),
  (req, res) => {
    const session = sessions.get(req.params.id);
    if (!session) return res.status(404).json({ error: 'unknown upload session' });

    const index = Number(req.params.index);
    const expectedChecksum = req.header('X-Chunk-Checksum');
    const actualChecksum = crypto.createHash('sha256').update(req.body).digest('hex');

    if (expectedChecksum && expectedChecksum !== actualChecksum) {
      return res.status(409).json({ error: 'checksum mismatch, resend this chunk' });
    }

    const chunkPath = path.join(UPLOAD_DIR, req.params.id, String(index));
    fs.writeFileSync(chunkPath, req.body);
    session.receivedChunks.add(index);

    res.json({ received: index, remaining: session.totalChunks - session.receivedChunks.size });
  }
);

// 4. Finalize: only assemble once every chunk has actually arrived.
app.post('/uploads/:id/complete', (req, res) => {
  const session = sessions.get(req.params.id);
  if (!session) return res.status(404).json({ error: 'unknown upload session' });

  const missing = [];
  for (let i = 0; i < session.totalChunks; i++) {
    if (!session.receivedChunks.has(i)) missing.push(i);
  }
  if (missing.length > 0) {
    return res.status(400).json({ error: 'upload incomplete', missing });
  }

  const finalPath = path.join(UPLOAD_DIR, session.fileName);
  const output = fs.createWriteStream(finalPath);
  for (let i = 0; i < session.totalChunks; i++) {
    const chunkPath = path.join(UPLOAD_DIR, req.params.id, String(i));
    output.write(fs.readFileSync(chunkPath));
  }
  output.end(() => {
    sessions.delete(req.params.id);
    res.json({ status: 'complete', path: finalPath });
  });
});

app.listen(3000);

And the client-side resume logic — the piece that actually makes this reliable in practice:

Wrap long lines
async function uploadFile(file, chunkSize) {
  chunkSize = chunkSize || 5 * 1024 * 1024;
  const totalChunks = Math.ceil(file.size / chunkSize);

  let uploadId = localStorage.getItem('upload:' + file.name);
  let startChunk = 0;

  if (uploadId) {
    // Resuming: ask the server what it already has.
    const status = await fetch('/uploads/' + uploadId).then((r) => r.json());
    startChunk = status.receivedChunks.length; // naive: assumes contiguous receipt
  } else {
    const session = await fetch('/uploads', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ fileName: file.name, totalSize: file.size, chunkSize: chunkSize }),
    }).then((r) => r.json());
    uploadId = session.uploadId;
    localStorage.setItem('upload:' + file.name, uploadId);
  }

  for (let i = startChunk; i < totalChunks; i++) {
    const chunk = file.slice(i * chunkSize, (i + 1) * chunkSize);
    const buffer = await chunk.arrayBuffer();
    const checksum = await sha256Hex(buffer);

    await fetch('/uploads/' + uploadId + '/chunks/' + i, {
      method: 'PUT',
      headers: { 'X-Chunk-Checksum': checksum },
      body: buffer,
    });
  }

  await fetch('/uploads/' + uploadId + '/complete', { method: 'POST' });
  localStorage.removeItem('upload:' + file.name);
}

#Cleaning Up Abandoned Sessions

Not every upload session finishes. A user closes the tab at chunk 3 of 200 and never comes back. Left unattended, the chunk directories for those sessions accumulate forever. Give every session a TTL (a background sweeper deletes sessions with no activity for, say, 24 hours) and reclaim the partial chunks on disk along with the session metadata. This mirrors the reservation-TTL pattern that shows up again later in this module for abandoned shopping carts.

🧠 Check your understanding
0/2 · 0/2 answered
1. What makes a chunked upload resumable, specifically?
Chunks are uploaded over UDP instead of TCPChunks are compressed before sendingThe server exposes an endpoint that reports exactly which chunks it has already receivedThe client uses HTTP/2
2. Why should the PUT for a given chunk index be idempotent (safe to retry)?
To reduce the number of HTTP headersTo avoid using checksumsBecause a client can't always tell whether a previous attempt for that chunk actually succeeded before the connection droppedBecause idempotency is required by the HTTP spec for all PUT requests