Order Tracking — Event-Driven Ingestion with Kafka

Mon, 01 Jan 0001 00:00:00 +0000

Order tracking looks simple until the ingestion path has to accept bursts of updates without making request latency depend on database work, while still applying events for each order in the correct sequence. The original version of this project processed tracking updates synchronously from HTTP into persistence. That was easy to reason about, but it coupled request handling to downstream processing and made independent scaling difficult.

The constraints mattered more than the choice of broker:

Snapshot Ingestion — Resilient Coordination with PostgreSQL

Mon, 01 Jan 0001 00:00:00 +0000

A snapshot provider is an awkward system boundary. The payload can be large, the network can fail, malformed records can appear halfway through the document, and a processing worker can disappear after claiming work. None of those failure modes should become /search latency or availability problems.

This project separates that unreliable ingestion boundary from the stable read path. A fetch worker streams the provider XML into durable staging rows, process workers claim completed work in PostgreSQL, and the API serves canonical data from PostgreSQL only. The interesting decision is not that PostgreSQL can coordinate the pipeline. It is deciding that, for this workload, introducing Kafka would add more operational machinery than the problem currently earns.

Projects on Enrique Goberna

Order Tracking — Event-Driven Ingestion with Kafka

Snapshot Ingestion — Resilient Coordination with PostgreSQL