Real-time events produced at DoorDash use an in-house framework, Iguazu, that embeds custom serialization metadata which is used for generic and dynamic stream processing. While this makes consuming these events easy for our microservice applications, this provides a challenge when attempting to consume these events more generally.
In this talk we’ll walk through how we’ve taken advantage of the highly extensible framework within Apache Pinot to develop a custom StreamMessageDecoder implementation incorporating the Iguazu consumer library. By leveraging this approach we are able to seamlessly integrate with the existing real-time ecosystem at DoorDash to power real-time analytics for any team or product at DoorDash.
