Md Abdullahil Shafi
← All case studies

TestReach · 2022 — present

DataLake: a type-safe ingestion library as leverage

One platform and one library that turned data ingestion from a recurring engineering project into a straightforward integration.

Data engineeringType safetyDeveloper experienceDistributed systems

Context

The platform runs many backend microservices, and every one of them needs to emit events into the central data platform. Demand is spiky by nature — examination schedules drive sharp, predictable-but-large surges — and the data feeds reporting and analytics downstream.

The problem

Each service that integrated with the lake implemented its own ingestion path. The effort repeated, and the behavior diverged: different retry logic, different error handling, different type assumptions. At millions of events per day, small inconsistencies become real operational noise.

What I did

  • Led the design of the DataLake ingestion platform and its service-integration model.
  • Designed a companion type-safe ingestion library for backend engineers, with minimal integration effort as the explicit design goal.
  • Made consistent behavior — retries, batching, error semantics — the library’s job, not each service’s.
  • Kept the producer contract typed end to end, so breaking changes surface at compile time rather than in the queue at 2 a.m.

What changed

The shape of the leverage — many services, one typed path into the lake:

Many microservices, one type-safe path into the data lakemicroservicemicroservicemicroservicemany producerstype-safe ingestion libraryminimal integration effort · typedconsistent behaviorqueueretries · backoffDataLake~2–2.5M events/daydemand spikes from examination schedules are absorbed before the lake

The result

The platform processes roughly 2–2.5 million ingestion events per day and absorbs the examination-schedule spikes without manual intervention. Ingestion went from a recurring engineering task to a relatively straightforward integration, and behavior is consistent across services. It is a strong example of developer leverage: a small engineering investment that improves productivity across many engineers — and one of the projects I am most proud of.

~2–2.5M
ingestion events per day
1 library
instead of N bespoke ingestion paths
Compile time
where producer contract breaks surface

What I would do differently

The library should have shipped together with the platform. The first integrating services predate it, and they paid exactly the inconsistency cost the library later removed — migrating them was work the library’s existence could have avoided. I would also invest in producer-contract conformance tests earlier.