Shadow traffic before customer traffic

A core worker-policy service at Clipboard Health depended on a webhook synchronization path with seven or more hops. Measured application delivery was approximately 66%. Nothing paged. Every hop worked most of the time; the product simply lost about a third of its updates, quietly, at the end of the chain.

I designed and led the replacement: database change streams published through an event bridge to an idempotent consumer. This post is about the part of the migration that did the most work — running the new path in shadow before any customer depended on it.

Why tests weren’t enough

The replacement passed its tests. For a delivery system, that is a low bar: tests exercise the traffic you thought to imagine, and delivery systems fail on the traffic you didn’t. The legacy path’s ~66% wasn’t one broken component — it was the compound interest of many mostly-working hops. Confidence built that path. I wanted evidence for the new one.

So before cutover, the new path ran in shadow mode against real production traffic at progressively larger levels — doing all of its work, with none of its effects visible to customers.

What shadow traffic caught

Shadow mode exposed two correctness defects that were silently dropping roughly 10,000 events in three hours. Silent is the operative word: no errors, no dead letters, just events that never arrived — the same failure mode the legacy path had normalized for years, reproduced by its replacement before it ever touched a customer.

Fixing those defects raised shadow delivery from approximately 66% to 90% — and, as important, turned the remaining gap into something we could enumerate instead of guess at.

A ramp is a sequence of claims

The live cutover then advanced through a 1%, 10%, 30%, 50%, and 100% ramp with immediate rollback available at every stage. Each stage is a claim about blast radius: if we are wrong, we are wrong for this many workers, and we can undo it now. The goal is a boring cutover — the interesting failures should already have happened in shadow.

During the completed production ramp, the path applied approximately 286,000 worker events per day with zero failed events and a clean dead-letter queue. Consumer p99 held around 15 milliseconds. The seven-plus-hop legacy path was removed, along with roughly 9,300 daily update messages.

What I’d reuse

Measure delivery before improving it. The ~66% figure did more to justify the project than any architecture argument — and it defined what done meant.

Prefer evidence to confidence. Shadow traffic converts “should work” into “does work, at this traffic shape, today.” It found in hours what review had missed for years.

Make rollback boring. A ramp with an always-armed rollback turns a migration from one bet into a sequence of small, reversible claims. Boring cutovers are the ones you get to write about.