Your System isn't Slow. It's Waiting.

Over the past year I rebuilt the throughput characteristics of two services on the critical path of a financial ledger: one that writes transactions into the general ledger, and one that posts trades into it. Volume grew roughly an order of magnitude over that period, and runs that once finished comfortably were finishing after midnight.

Almost none of the gains came from writing faster code. They came from working out what the system was waiting on. Here are the five principles that produced every measurable win, and the numbers they produced.

1. Instrument before you optimize

The most expensive habit in performance work is shipping a change and calling it faster because it feels faster. Build the dashboard before the fix. If the metric doesn't exist yet, that's the first commit, not the last.

Every optimization below was benchmarked against a comparable production run: same workload shape, closest available day either side. One example, a scheduled reconciliation sweep:

The first row is the one that matters. The faster run also carried 8.5% more volume, which makes every figure beneath it a lower bound rather than a like-for-like comparison. I only know that because the instrumentation was already there. Without it I would have under-claimed the result and had nothing to say when someone asked how I knew.

2. Look for barriers before slow code

This was the highest-leverage pattern of the year, and I found the same defect in two unrelated services.

One was a scheduled sweep that collected up to 100 units of work, dispatched them concurrently, then awaited all of them before starting the next cycle. The other polled 10 messages off a queue and waited for all 10 to finish before polling again.

In both cases a single slow item stalled every other item in its cohort, and the cost is worse than it looks. Utilization collapses toward 1/n. With ten slots and one item two orders of magnitude slower than its neighbours, nine slots sit idle for nearly the entire window.

A cohort barrier converts nine working slots into nine idle ones. Replacing it with a producer/consumer pipeline reclaims them.

The fix was identical both times: a producer and a consumer with a bounded channel between them. The producer refills as the consumer drains, and a concurrency permit is acquired per work item instead of per cohort. Nothing waits on a neighbour.

The same shape shows up in messaging. Waiting on an acknowledgement per message serializes 20,000 round trips the broker was happy to pipeline. Queue the sends, flush once, resolve the futures afterward.

3. Horizontal scaling has a prerequisite

Two Kafka consumers in the service that writes transactions into the ledger were statically scaled: one vertically scaled instance for the primary consumer, a fixed replica count for the one handling write confirmations, regardless of consumer lag or traffic.

Autoscaling is the obvious answer. The reason it hadn't been done was not configuration. The primary consumer also ran the service's scheduled jobs, so scaling it to N replicas would have run every cron job N times. The real first step was extracting scheduled work into a service of its own. Only then could the consumer be treated as what it actually was, a pure Kafka consumer, and scaled on consumer-group lag.

Capacity tracking backlog on a 1.6M-request run. Container sizing unchanged; horizontal scaling is the only variable.

Two things worth taking from this. Autoscaling was blocked by coupling, not by config: the scaling policy took a day, the refactor that made it safe was the project. And change one variable at a time. I deliberately left container sizing alone, because if I had resized pods in the same release and throughput regressed, nothing would have told me which change caused it.

4. Match the primitive to the workload

The service that posts trades was already concurrent before I touched it; it fanned work out across platform threads. That is why the barrier stayed hidden. "We're already parallel" sends you looking somewhere else.

But the work is almost entirely IO: writes to a database, writes to the ledger. Platform threads are expensive and finite, and parking thousands of them on socket reads spends the most constrained resource in the JVM on nothing at all. Virtual threads are nearly free for this shape. The split that works:

The old fan-out wasn't insufficiently concurrent. It was concurrent with the wrong primitive, behind a barrier.

5. Judge a win by cost per unit of work

The easiest way to make a pipeline faster is to hand it more connections, a bigger pool, a longer timeout, and sometimes that is exactly the right move. Doing more work genuinely costs more. A consumer processing four times the volume on four replicas is not cheating, and the autoscaling above spends more resources by design.

The axis that matters is resource cost per unit of work. When throughput rises and per-unit cost falls, the structure changed. When a limit rises and per-unit cost stays flat, you bought time. That is a legitimate purchase, but it isn't a fix, and the trend that forced it is still moving.

The second kind is easy to spot in hindsight. Before this work, a transaction timeout had been quadrupled from 5 to 20 minutes purely to stop large batches from failing. Nothing about the cost of a batch changed; the failure simply moved out of view until batches grew again.

The results I trust most are the ones where per-unit cost fell far enough to show up as an absolute decrease. Peak database connections fell from 450 to 42 while the run got faster and carried more volume. Memory halved while batch sizes doubled. A ~40x throughput gain used no additional connections at all.

So when a change costs more of something, the question isn't whether the number went up. It's whether you bought more work or bought quiet.

What it comes down to

Ask what a system is waiting on before asking how to make it faster. That reordering accounts for nearly every number above.

Two habits keep the answers honest. Benchmark against a comparable production run and say so when the volumes differ, which turns your number into a lower bound instead of a guess. And judge the result by resource cost per unit of work rather than absolute usage: spending more to do more is scaling, spending more to stay level is deferral.

...

Interested in working at Wealthsimple? Check out the open roles on our team today.

​