Agenda

Until 2019, the Data Engineering team primarily built analytical data assets on our data lake to serve the needs of our customers. The cadence of this varied from hours to days for the different pipelines. This inconsistency led to missed business opportunities and a less-than-ideal customer experience. For instance, The customer executive couldn't pitch for a product upsell/cross-sell during a call. Stale marketing emails were sent to the consumers.

We realized that our Source of Record (SOR) applications were diverse and there was a need to build an operational layer with real-time data aggregation or master data record for key Atlassian entities such as Person, Company, Entitlements, etc to integrate them into core data-entities for Atlassian.

To rectify this, we embarked on a journey to build out data micro-services to integrate data from source systems, external 3P applications such as clearbit, etc to set up a real-time, highly available, and consistent operational store using technologies like AWS SQS, Kinesis, DynamoDB, ElasticSearch and event processing via microservice. We constructed a robust, layer-based framework that was capable of handling multiple events concurrently. It was designed to select the most critical event based on priority or weightage, thereby facilitating the construction of a single, comprehensive record.

The operational layer then feeds into our analytical data lake. It not only catered to our analytical requirements but also addressed the operational needs of the rest of the company, enabling efficient data application services.

The presentation will focus on the practical implementation of the design, and development of real-time systems, including the challenges we have solved.