In this talk, we will share our experiences on building Metaflow, a Python library that empowers data scientists at Netflix to prototype, build, deploy, and operate end-to-end machine learning solutions. We started building Metaflow at Netflix to provide a solid foundation for hundreds of internal ML use cases, from classical statistical analysis to large-scale applications of deep learning. Metaflow is designed with a human-centric mindset: instead of reinventing the wheel for large-scale computing or machine learning, we integrate existing solutions into a delightfully consistent and easy-to-use package. This talk focuses on our philosophy towards Machine Learning infrastructure and dives into the internals of Metaflow; it will highlight lessons that we have learned in building a Python library that needs to be robust, performant, and flexible enough to solve a large set of complex real-world business problems related to machine learning. This talk is for you if you want to learn how to develop systems for big data and ML in Python.
