Back To Schedule
Friday, November 15 • 3:40pm - 4:10pm
Serverless Event-Driven Data Pipeline Platform

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Airflow has become the defacto data pipeline platform in many companies.
Airflow was designed to run static, slow-moving workflows on a fixed schedule, and it is a great tool for that purpose. However, users often get into trouble by forcing their use cases to fit into Airflow’s model.
A few examples that Airflow can not satisfy in a first-class way includes:
- Complex DAGs leak application code into Pipeline
- DAGs which need to be run off-schedule or with no schedule at all
- DAGs that run concurrently with the same start time - DAGs with complicated branching logic - DAGs with many fast tasks
- DAGs which rely on the exchange of data - Parametrized DAGs

In this talk we present a brand new Serverless Event-Driven Pipeline Platform written in Scala that addresses all the problems above

avatar for Rahul Chitturi

Rahul Chitturi

Principal Software Engineer, Coatue
avatar for Neelabh Gupta

Neelabh Gupta

Software Engineer, Coatue Management
Full-stack web development, TypeScript, React, Scala, Python

Friday November 15, 2019 3:40pm - 4:10pm PST