Metadata
- Author: Whatnot Engineering
- Full Title:: Scaling Our Data Stack With Kafka and Real-Time Stream Processing
- Category:: 🗞️Articles
- Document Tags:: Streaming architecture for real time analytics
- URL:: https://medium.com/whatnot-engineering/scaling-our-data-stack-with-kafka-and-real-time-stream-processing-56554dcbb0fc
- Finished date:: 2023-02-11
Highlights
the main requirement for this system was to decouple how features store their data from how they report data (View Highlight)
Testing event producers
we’d run a local Kafka cluster in CI/CD using Docker. (View Highlight)
Schemas
There’s a spectrum here — one extreme chooses a single schema for all event types and the other a bespoke schema for each individual event type. We landed somewhere in the middle — we looked at our use cases and came up with a few event schemas that covered the majority of the event types (View Highlight)