Back To Schedule
Friday, November 15 • 1:00pm - 1:30pm
Moonshot Spark: serverless with GraalVM

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Can Apache Spark slip its earthly bounds and go serverless, clusterless? Popular cloud services are becoming more capable. AWS Lamba now runs three times longer, Fargate has become less expensive. GraalVM can reduce resource usage while improving cold start times. Consider how to handle small bursts of work. Would a standalone container suit best? If cold startup times weren't such an issue! What about a "mission control" model, where a long-running Spark driver dispatches work to ephemeral executors? What you gain in flexibility and convenience means concessions in performance. Chinning up with GraalVM native image helps. Shuffle is still problematic. Which experimental shuffle manager is best suited to the outer reaches of the cloud? There's not a practical use case for larger workflows - yet. But let's use this moonshot as a lens to magnify cloud performance issues. Explore how these solutions could apply to services you already use.

avatar for Rose Toomey

Rose Toomey

Software Engineer, Coatue Management
Big data, Spark, Scala, fintech, ETL pipelines, digital assets, and object allocation. I love a performance mystery.

Friday November 15, 2019 1:00pm - 1:30pm PST