Why Your Node.js API Falls Apart Under Load

The symptom

Traffic ticks up 20%. P95 was fine at 80ms. Now half the requests sit in 'waiting for connection' until they time out, the client retries, and you have a retry storm on a pool that was never sized for this. The process isn't 'down.' It's busy doing the wrong work: blocking the loop, holding sockets, and amplifying load.

Why it happens

Node is single-threaded for JS. JSON.parse on a fat payload, a sync crypto call, a tight loop in a mapper — that stalls everyone. Separately, pg / mysql pools default to a small max. Under load, acquire() queues. Timeouts fire, clients retry, the queue grows. There is no backpressure at the HTTP edge, so the database never gets a chance to catch up. Adding instances without a queue just multiplies the same mistake.

const pool = createPool({ max: 10 }); // 'default' until it isn't

async function handleRequest(req) {
  const conn = await pool.acquire(); // queues forever under load
  try {
    return await conn.query(req.sql);
  } finally {
    pool.release(conn);
  }
}

How to fix it

Cap the pool explicitly and fail fast with 503 + Retry-After instead of waiting 30s. Put a queue or a concurrency limiter in front of the expensive path. Move CPU-heavy work off the loop (worker threads, a job consumer). Load-test the pool at 2× expected peak before you call it production. The metric that matters is wait time for a connection, not 'CPU looks fine.'

Takeaway

Frameworks don't save you from unbounded waits. Name the scarce resource (loop time, pool slots, downstream QPS), put a limit on it, and make the limit visible. That's the whole performance job.