Md Abdullahil Shafi
← All case studies

TestReach · 2022 — present

Reliability: designing for failure at millions of events a day

Retries, exponential backoff, queue fairness and a webhook platform redesign — and a ~60% drop in processing failures.

ReliabilityEvent-driven architectureSQSWebhooks

Context

The platform is event-driven at its core: AWS queueing services, asynchronous processing, and an outbound webhook platform that delivers assessment events to customer systems. At assessment scale, even small failure rates are large absolute numbers.

The problem

Failed processing and failed deliveries did not fail politely. Naive retries amplified load exactly when the system was struggling, one busy tenant could starve others through shared queues, and every team handled failure differently — so failures surfaced as inconsistent, hard-to-reason-about behavior.

What I did

  • Generalized queue-based processing with retries and exponential backoff as the standard pattern, instead of per-team ad hoc handling.
  • Introduced queue fairness so a single high-volume tenant cannot starve everyone else.
  • Redesigned the webhook platform around the same failure model: bounded retries, backoff, and observable delivery state.
  • Reduced producer-side failures at the source with the type-safe ingestion library.

What changed

The failure model in one picture — fail, back off, recover:

delivery failsretry with exponential backoffqueues stay fair across tenantsmessage recovered

Exponential backoff: delay doubles with each retry1s2s4s8s16sretry 1retry 2retry 3retry 4retry 5delay doubles each retry — failing fast, backing off, recovering

Result: ~60% fewer event-processing failures.

The result

Event-processing failures dropped by roughly 60%, and behavior under examination-schedule spikes became predictable rather than dramatic. The same failure model now applies across queues and webhooks, which makes new integrations boring — the highest compliment an asynchronous system can get.

~60%
fewer event-processing failures
Exponential
backoff as the default retry curve
Per-tenant
queue fairness under load

What I would do differently

Dead-letter handling and replay tooling should be part of the first queue, not a follow-up. Reliability work is cheapest when the failure model is designed in from the start; retrofitting observability and replay onto live queues cost more than building them upfront would have.