Registration Open, 2nd Floor

Cost to register: $799 Apache Pinot is a high-performance database engineered to serve analytical queries with extremely high concurrency, boasting latencies as low as tens of milliseconds. It excels at ingesting streaming data from sources like Apache Kafka and is optimized for real-time, user-facing analytics applications. In this full-day training, we will explore the architectures of Apache Kafka, Apache Flink, and Apache Pinot. We will run local clusters of each system, studying the role each plays in a real-time analytics pipeline. We’ll begin by ingesting static data into Pinot and querying it. Not content to stop there, we’ll add a streaming data source in Kafka, and ingest that into Pinot as well, showing how both data sources can work together to enrich an application. We’ll then examine which analytics operations belong in the analytics data store (Pinot) and which ones should be computed before ingestion. We’ll implement those operations in Flink. Having put all three technologies to use on your own in hands-on exercises, you’ll leave prepared to begin exploring the three of them together for your own real-time, user-facing analytics applications. LEARNING OBJECTIVES At the successful completion of this training, you will be able to: - List the essential components of Pinot, Kafka, and Flink - Explain the architecture of Apache Pinot and its integration with Apache Kafka and Apache Flink - Form an opinion about the proper role of Kafka, Flink, and Pinot in a real-time analytics stack - Implement basic stream processing tasks with Apache Flink - Create a table in Pinot, including schema definition and table configuration - Ingest batch data into an offline table and streaming data from a Kafka topic into a real-time table - Use the Pinot UI to monitor and observe your Pinot cluster PREREQUISITES To participate in this workshop, you will need the following: - Docker Desktop: We will use Docker to run Pinot, Kafka, and Flink locally. If you need to install it, please download Docker Desktop and follow the instructions to install it at https://www.docker.com/get-started/ - Resources: Pinot works well in Docker but is not designed as a desktop solution. Running it locally requires a minimum of 8GB of Memory and 10GB of disk space

Head of Developer Advocacy

Senior Technical Product Manager

Uncorking Analytics with Apache Kafka®, Apache Flink, and Apache Pinot®

Breakfast, Market Street Foyer

In this talk, Kishore Gopalakrishna, the creator of Apache Pinot and CEO of StarTree, reveals critical distinctions between mission-critical analytics workloads and their non-mission critical counterparts. As companies increasingly rely on data-driven insights to steer their strategies, the urgency of real-time analytics solutions becomes evident. Gopalakrishna will explain why certain data operations demand immediate attention and why conventional data warehousing solutions may fall short. Drawing from his expertise in developing Apache Pinot, he highlights the essential features required to meet the demands of mission-critical analytics, and will showcase a framework for easily identifying mission-critical workloads that significantly impact internal &amp; external customer experience. Join us to gain a deeper understanding of why mission-critical data can't afford to wait, and how cutting-edge solutions like Apache Pinot are reshaping the analytics landscape forever.

Co-Founder & CEO

Mission Critical Data Can't Wait: It's Time for Real-Time

Real-time analytics industry leaders will join us on stage for a 30-minute discussion of their recent experiences in the space. We'll discuss what drove them to adopt real-time analytics technologies, the business outcomes they were able to achieve as a result, the experiences of building applications and services in an emerging technology category, and how they see the category unfolding in the immediate future. This panel will provide you with a roadmap to success in the real-time systems you're building now and in the future.

VP, Developer Relations

Director of Engineering

VP, Product Engineering

Chief Product Officer

Real-Time Analytics Panel, Moderated by Tim Berglund

Morning Break, Market Street Foyer

Visit all sponsors to be entered in our raffle!

Expo, Market Street Foyer

In this tech talk we would like to present an overview of how, at Uber, we leverage Pinot as a datasketch store for ML model monitoring use case. We also would like to highlight the power of Apache Pinot's aggregation system and capabilities of recently introduced datasketch functions. Problem and solution: Machine Learning life cycle operates with large-scale data across different storage systems, both online and offline. ML practitioners need to monitor and debug the data quality and distribution for each data set and also cross-compare different data sources. However, it is challenging to have a monitoring application that directly works with raw data from different systems. As a solution, we do data profiling for those datasets, and store the profiling results as data sketches inside Pinot. We leverage Pinot to achieve scalable storage and low-latency queries for data sketches so that we can enable both continuous ML monitoring and adhoc UI based debugging experience. To enable this, we integrated Apache Datasketches into Pinot, particularly focusing on KLL, CPC and Frequent Items sketches, and it clearly shows performance advantages compared to alternative storage solutions like Druid and MySQL. Join us for a deep dive into optimizing real-time analytics in ML applications with Apache Pinot.

Senior Software Engineer

Software Engineer

Real-Time ML Model Monitoring with Datasketches and Apache Pinot at Uber

Stripe has been using Apache Pinot for the past 2 years to store financial data to power both merchant-facing products and critical internal tools. Our use cases require supporting low latency queries with up to 99.99% query availability and large clusters up to petabytes in size. This talk will go over some of the operational lessons learned from running our Pinot clusters in production. We'll go over our cluster topology strategy and key configs we use to maintain our SLAs and keep our clusters stable.

Operating Apache Pinot at Scale

In today's hyper-connected digital landscape, real-time API usage analysis and billing have become paramount. With the proliferation of APIs at the heart of modern applications and services, gaining real-time insights into their use is often mission-critical as it depends on critical factors such as latency, throughput, freshness, and the correctness of generated insights. This talk discusses the architecture of a real-time API usage analytics system composed of Redpanda, Apache Flink, and Apache Pinot. Redpanda, as a scalable streaming data platform, enables high-volume, low-latency API volume data ingestion from API gateways in real time. The ingested data is streamed through Flink for streaming ETL, enabling operations like joins, aggregations, and transformations, feeding the final output of the pipeline to Apache Pinot for serving analytics at scale. In addition to that, we will use Flink for rate-limiting incoming API requests. Having such a system enables businesses to make informed, agile decisions, addressing performance bottlenecks, security threats, and resource allocation issues in the API infrastructure promptly. Also, it ensures seamless user experiences by identifying usage patterns, issues, and opportunities as they occur, proactively enhancing the quality of your product. Whether you're a data engineer, an application developer, or an architect, this talk promises to equip you with the knowledge and tools to build a real-time analytics solution that empowers your organization with the insights it needs to stay ahead in a data-driven world.

Senior Developer Advocate

Transforming Your APIs into Business Gold—Architecting a Real-Time API Usage Analytics Platform

In the dynamic realm of IoT, real-time analytics plays a pivotal role in making data-driven decisions and responding swiftly to changing conditions. I will demonstrate how Apache Pulsar's native MQTT support can be easily combined with Apache Pinot to provide real-time analytics on IoT data. This session goes beyond the theoretical by providing a hands-on code demo, where we will walk through the implementation of an IoT analytics pipeline that will serve as a template for setting up their own real-time IoT analytics solution.

Developer Advocate

Empowering Real-Time IoT Analytics with Apache Pulsar and Apache Pinot

Lunch, Regency Ballroom II

At DoorDash we process 6 billion messages day across 2500 Real-Time pipelines in total. Our pipelines ingest data from mobile devices and internal services, stream-process the data with Apache Flink and Kafka before writing to Datalake(s). We use Trino, Pinot and other tools to query from Datalake(s). Along the way, we have built a rich set of automation tools to maintain the lifecycle of these pipelines from provisioning to clean-up. The lessons learnt as our business and data-needs scale guide us in both improving our pipelines and the related-automation. We hope to share our learnings with the broader Stream Processing community.

Software Engineer in Data Platform

Automating Real-Time Pipelines at Scale in DoorDash

Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero.

Beyond Tiered Storage: Serverless Kafka with No Local Disks

In this talk, we'll describe our motivation and journey of re-architecting the StarTree Cloud platform, powered by Apache Pinot. The talk will be centered around how we introduced the concept of &quot;workspace&quot; — a logical entity to provide isolation on a single cluster with shared infrastructure. We'll walk through how we architected everything with workspace-first thinking throughout the stack, from the onboarding app, to data management and ingestion, to the Pinot cluster management. We'll also dive into integral security issues like our RBAC implementation.

Founding Engineer

Head of Product

Re-Architecting Our SaaS Infrastructure for Better Workload Isolation

Sometimes you really do need your dashboards to be faster. At Slack, we needed to deliver data at speed, having recently adopted Tableau as our next-generation visualization tool. Not long after, users reported issues with query latency, limitations on row count, and inability to use custom SQL. We solved this by introducing Apache Pinot as a data source for Tableau, replacing the Trino connector for several dashboard use cases. Only two months after this internal transition, we have over a terabyte of data in Pinot powering internal dashboards. And StarTree recently open-sourced the Pinot Tableau connector, removing the need for Slack to mature our home-grown connector. This drastically reduces the time required to integrate Pinot with Tableau while increasing the data quality level of the integration. This talk covers our zero-to-one path to power Tableau dashboards with Pinot data.

Driving Internal Analytics with Pinot and Tableau

Afternoon Break, Market Street Foyer

In this presentation, we'll discuss our adoption of Apache Pinot to enhance real-time, user-facing analytics for Web3 applications. We'll explore how Pinot was integrated into our analytics stack and the specific use cases it now empowers, from handling massive data volumes to delivering instant query responses. The focus will be on the technical strategies and insights that enabled us to leverage Pinot's capabilities to meet the scalability and performance demands of the Web3 ecosystem.

Co-Founder & CTO

Co-Founder

Getting User-Facing Analytics Right for Web3 at Scale: A Real-Time Story with Apache Pinot

Streaming data is rapidly becoming a key component in modern applications, and Apache Kafka has emerged as a popular and powerful platform for managing and processing these data streams. However, as the volume and complexity of streaming data continue to grow, it becomes increasingly critical to have efficient and effective ways of querying and analyzing this data This is where query engines like Apache Flink, Trino, Timeplus Proton, Materialize, and ksqlDB come in. These powerful tools offer flexible and scalable ways of processing and analyzing streaming data in real-time, enabling users to extract valuable insights from their data streams. In this talk, we will introduce the audience to the world of querying streaming data on Kafka/Pulsar with SQL, compare and contrast the features and capabilities of each of these tools, and provide an in-depth analysis of their respective Pros and Cons. We will also discuss the best practices and scenarios where each tool is most effective. In conclusion, query engines like Apache Flink, Trino, ksqlDB, Timeplus Proton, RisingWave, Materialize and are useful tools in processing and analyzing streaming data. With their ability to extract valuable insights from real-time data streams, these tools are a valuable asset for modern data-driven applications. Coffee elements are added to the talk, to help audience to compare different OSS offerings.

Cofounder and Head of Product

Query Real-Time Data using SQL, a Guide for Coffee Lovers

ETL is like twinkies and cockroaches... it will be around a lot longer than any of us, never dying, always adapting. Batch ETL processing has been the de-facto standard for over 20 years for moving data from OLTP to OLAP databases, but now, with more and more analytic applications and data products requiring fresher data, some of our favorite tricks of the ETL trade have needed to adapt to the streaming world. In this talk, we will look at how the physics of data processing change in the real-time world and what are some of the best practices to address those changes.

Founding Engineer and Product Lead

Old Dogs, New Tricks—Best Practices for Moving from Batch to Real-Time ETL

Sponsored by StarTree and ConfluentOn the rooftop pool deck of the Signia by Hilton, we’ll celebrate the triumph of real-time analytics over near-real-time analytics. This will be a fun networking event you’ll never forget! Please wear your badge for entry.

Summit Party, the Pool Deck

Jay Kreps, Founder and CEO of Confluent, will present his vision of the unification of the operational and analytical worlds with data streams.

Streams Forever: Unifying the Operational and Analytical Worlds

Moloco has been leaning heavily on batch pipelines, powered by Airflow, Dataflow, and BigQuery on GCP to run hourly or daily batch pipelines. This session goes through the story of how Moloco introduced streaming, based on the use cases the real-time platform covers, and different phases through which it went through to build its real-time data platform.

How We Used Pinot to Power Adtech Analytics Dashboards

Real-time OLAP databases such as Apache Pinot excel at accommodating on-the-fly analytical queries, while streaming databases like KsqlDB and RisingWave are designed to perform incremental computation over streaming data. Are these two types of systems friends or foes? Can we achieve the best of both worlds? In this presentation, I'll delve into the distinctions between stream processing and OLAP and why blending their strengths isn't a straightforward task. We'll also explore the inherent design differences between streaming databases and real-time OLAP databases, and showcase how Apache Pinot and RisingWave can work together in production settings.

Founder & CEO

Bridging Stream Processing & Real-time OLAP: Can We Have It All?

Query-time JOINs are a new capability leveraged by many user-facing real-time analytics nowadays. With Apache Pinot's new Multi-Stage Query Engine, we are now offering users a wider range of possible ways to gain insights from their data. In this talk we will delve into the world of JOINs, discuss various optimizations we added in Apache Pinot for running fast and efficient JOINs, and highlight the best use cases for query-time JOINs.

Multi-Stage Joins—Industry's First Multi-Stage Approach to Expand Use Cases for Apache Pinot

In this talk, we will share the journey of OLAP at Uber and our consolidation over the years from multiple technologies including Elasticsearch, AresDB, MemSQL, Clickhouse into a single platform built on top of Apache Pinot. We will also share our learnings around open-source technology adoption as well as the development/operation strategies.

Principal Engineer

Evolution of OLAP at Uber

In the digital-first world, the efficiency and performance of APIs play a pivotal role in ensuring seamless interactions between different software applications. This is especially crucial in the on-demand delivery sector, where DoorDash has positioned itself as a market leader. The Developer Insights team at DoorDash has been instrumental in providing invaluable analytics and insights to DoorDash's API partners, ensuring that their interactions with the DoorDash Drive delivery platform are optimized for performance and reliability. In this session, we will delve into an in-depth case study of how the Developer Insights team utilizes Apache Pinot for real-time analytics and StarTree ThirdEye for anomaly detection to serve DoorDash's API partners, including a major player, Mx. With an impressive array of 12 alerts, each capitalizing on the query enumeration feature, the team successfully supports over 550 Mx partners, ensuring they are immediately notified of any anomalies.

Product Lead

Revolutionizing API Partnerships: Real-Time Anomaly Detection at DoorDash

The migration of data-intensive analytics applications to cloud-native environments promises enhanced scalability and flexibility but introduces complex cost models that pose new challenges to traditional optimization strategies. While on-premises setups focused on speed, cloud deployments require a more nuanced approach, factoring in cloud storage operations costs, which can escalate rapidly in real-world scenarios. In a presentation, Hope Wang and Chunxu Tang will analyze these challenges through a case study on Presto's large deployment. They will show their findings of unexpected cost implications with standard I/O optimizations like table scans, filters, and broadcast joins when implemented in cloud environments. They will also highlight the need for a paradigm shift in optimizing data-intensive applications for the cloud and advocate for developing new I/O strategies, balancing performance and costs while tailored to cloud ecosystems' unique demands. By attending this session, you will: • Understand the complexities and cost challenges of optimizing data-intensive applications in cloud-native environments. • Gain insights from an in-depth case study on the economic implications of traditional I/O optimizations in cloud deployments. • Learn about the need for a paradigm shift towards developing I/O strategies that are both performance-efficient and cost-effective in the cloud.

Staff Research Scientist

A Case Study in API Cost of Running Presto in the Cloud at Scale

Observability data has become some of the most common types of data being generated and stored within the software industry. At the most basic level we have time series and log data, which are used not just for observability but also in business analytics and customer behavior data mining. Shortly after metrics and logs, trace data also became heavily used to provide deeper insights into applications and distributed systems. In this presentation, we will survey the observability ecosystem and common architectures to ingest, store, query and visualize metrics, logs and traces. We will then cover how we expanded Pinot's feature set to allow it to become an observability storage and analytics engine that can power observability platforms such as Grafana.

Building an Observability Platform Using Apache Pinot

Streaming pipelines have become more and more difficult to build and operationalize as complexity increases. In this talk, you will learn how Snowflake is simplifying streaming architecture to enable users to work with streaming data easily through Snowpipe Streaming and Dynamic Tables. Snowpipe streaming is designed for rowsets with variable arrival frequency. Calling the streaming API prompts low-latency loads of streaming data rows using the Snowflake Ingest SDK and your own managed application code. The session includes a live demo of the following: - Record audio from the laptop's default microphone. - Do speech to text transcription - Pipes the words from the transcript to Snowflake using Snowpipe Streaming. - Create a Dynamic Table that aggregates words and their counts. - Use an Ads table that refers to the aforementioned Dynamic Table to determine what ads to show to the user.

Yell at the Laptop and It Might Decide to Show You the Right Ad: Building a Voice-to-Text Streaming Pipeline

In today's fast-paced world of software development, new changes (both in code and infrastructure) are being released at breakneck pace. At Uber, we roll out ~11,000 changes every week and it's important for us to have a way to quickly be able to identify and resolve issues caused by these changes. A delay in detecting issues can create a number of issues including impacts to: user experience, our ability to facilitate transactions on the platform, company revenue, and the overall trust and confidence of our users. At Uber, we have built a system called &quot;Healthline&quot; to help with our Mean Time To Detect (MTTD) and Mean Time To Resolve (MTTR) issues and to avoid potential outages and large-scale user impacts. Due to our ability to detect the issues in real time, this has become the go-to tool for release managers to observe the impact of canary release and decide whether to proceed further or to rollback. In this talk we will be sharing details on how we are leveraging Apache Pinot™ to achieve this in real time at Uber scale.

Engineering Manager

Real-Time Analytics for Mobile App Crashes

Until 2019, the Data Engineering team primarily built analytical data assets on our data lake to serve the needs of our customers. The cadence of this varied from hours to days for the different pipelines. This inconsistency led to missed business opportunities and a less-than-ideal customer experience. For instance, The customer executive couldn't pitch for a product upsell/cross-sell during a call. Stale marketing emails were sent to the consumers. We realized that our Source of Record (SOR) applications were diverse and there was a need to build an operational layer with real-time data aggregation or master data record for key Atlassian entities such as Person, Company, Entitlements, etc to integrate them into core data-entities for Atlassian. To rectify this, we embarked on a journey to build out data micro-services to integrate data from source systems, external 3P applications such as clearbit, etc to set up a real-time, highly available, and consistent operational store using technologies like AWS SQS, Kinesis, DynamoDB, ElasticSearch and event processing via microservice. We constructed a robust, layer-based framework that was capable of handling multiple events concurrently. It was designed to select the most critical event based on priority or weightage, thereby facilitating the construction of a single, comprehensive record. The operational layer then feeds into our analytical data lake. It not only catered to our analytical requirements but also addressed the operational needs of the rest of the company, enabling efficient data application services. The presentation will focus on the practical implementation of the design, and development of real-time systems, including the challenges we have solved.

Data Enthusiast

Senior Data Engineer

Micro-Service Architecture for Real-Time Data Processing at Atlassian

The StarTree community is made up of people from all over the globe, spanning different experiences, languages, cultures, beliefs, experience levels, and more. We welcome and embrace this diversity and want this environment to be a warm, safe, respectful, generous, and welcoming experience for everyone. With that, we created a general code of conduct.

Behaviors that reinforce these values contribute to a positive environment, and include, but not limited to:

Be welcoming. We strive to be a community that welcomes and supports people of all backgrounds and identities. This includes, but is not limited to, members of any race, ethnicity, culture, national origin, color, immigration status, social and economic class, educational level, sex, sexual orientation, gender identity and expression, age, physical appearance, family status, technological or professional choices, academic discipline, religion, mental ability, and physical ability.

Acknowledging time and effort. This community is composed of volunteers. This is not a “paid” support forum. Expressing gratitude to those who help you goes a long way.

Be respectful of differing viewpoints and experiences. Not all of us will agree all the time, but disagreement is no excuse for poor behavior or poor manners. We might all experience some frustration now and then, but we cannot allow that frustration to turn into a personal attack. It’s important to remember that a community where people feel uncomfortable or threatened is not a productive one.

Be careful in the words that you choose. Be kind to others. Do not insult or put down other community members. Harassment and other exclusionary behavior are not acceptable. This includes, but is not limited to:

Violent threats or violent language directed against another person

Posting sexually explicit or violent material

Posting (or threatening to post) other people’s personally identifying information (“doxing”)

Personal insults, especially those using racist or sexist terms 

Advocating for, or encouraging, any of the above behavior

Repeated harassment of others. In general, if someone asks you to stop, then stop

No soliciting. This community is built for collaborative support and encouragement, not a place for sales or solicitation. Please keep any sales-related efforts out of this community. Event promotions are welcome.

Code of conduct violations reduce the value of the community for everyone, and we take them seriously. If you believe someone is violating the code of conduct, please let us know.

To report an incident, email us at 

. While attending Real-Time Analytics Summit, you can also report an incident at the StarTree community desk.

If a participant engages in behavior that violates this code of conduct, the StarTree community Code of Conduct team may take any action they deem appropriate, including warning the offender or expulsion from the community and community events with no refund of event tickets.

This code of conduct was written with inspiration from our friends at 

RTA Summit Code of Conduct