A critical path that silently lost data
A core worker-policy service depended on a webhook synchronization path with seven or more hops, repeated data fetching, and incomplete delivery.
Measured application delivery was approximately 66%. Correctness and operability—not raw throughput—were therefore the first constraints to solve.
- shadow
- 1%
- 10%
- 30%
- 50%
- 100%
One event, one accountable delivery path
I designed the event-driven path and led its implementation: database change streams published through an event bridge to an idempotent consumer, with explicit backfill and rollback controls.
The relay became a reusable library rather than an application-specific bridge, leaving a platform primitive for later event-driven migrations.
Shadow traffic before customer traffic
The new path ran in shadow mode at progressively larger traffic levels before cutover. That exposed two correctness defects silently dropping roughly 10,000 events in three hours.
Fixing those defects raised shadow delivery from approximately 66% to 90%. The live path then advanced through a 1%, 10%, 30%, 50%, and 100% ramp with immediate rollback available.
The measured production result
During the completed production ramp, the path applied approximately 286,000 worker events per day with zero failed events and a clean dead-letter queue.
Consumer p99 remained around 15 milliseconds, the obsolete seven-plus-hop path was removed, and approximately 9,300 daily update messages were eliminated.