# DataLake: a type-safe ingestion library as leverage

> One platform and one library that turned data ingestion from a recurring engineering project into a straightforward integration.

TestReach · 2022 — present

## Context

The platform runs many backend microservices, and every one of them needs to emit events into the central data platform. Demand is spiky by nature — examination schedules drive sharp, predictable-but-large surges — and the data feeds reporting and analytics downstream.

## The problem

Each service that integrated with the lake implemented its own ingestion path. The effort repeated, and the behavior diverged: different retry logic, different error handling, different type assumptions. At millions of events per day, small inconsistencies become real operational noise.

## What I did

- Led the design of the DataLake ingestion platform and its service-integration model.
- Designed a companion type-safe ingestion library for backend engineers, with minimal integration effort as the explicit design goal.
- Made consistent behavior — retries, batching, error semantics — the library’s job, not each service’s.
- Kept the producer contract typed end to end, so breaking changes surface at compile time rather than in the queue at 2 a.m.

## What changed

The shape of the leverage — many services, one typed path into the lake:

## The result

The platform processes roughly 2–2.5 million ingestion events per day and absorbs the examination-schedule spikes without manual intervention. Ingestion went from a recurring engineering task to a relatively straightforward integration, and behavior is consistent across services. It is a strong example of developer leverage: a small engineering investment that improves productivity across many engineers — and one of the projects I am most proud of.

- **~2–2.5M** — ingestion events per day
- **1 library** — instead of N bespoke ingestion paths
- **Compile time** — where producer contract breaks surface

## What I would do differently

The library should have shipped together with the platform. The first integrating services predate it, and they paid exactly the inconsistency cost the library later removed — migrating them was work the library’s existence could have avoided. I would also invest in producer-contract conformance tests earlier.