Scale By the Bay 2019

As the deep-learning revolution matures, there is ever-growing demand for bigger datasets, larger models and more compute infrastructure. What is the role of algorithmic design in this?  I will show several ways to infuse structure into deep networks to overcome these limitations, viz., through tensors, graphs, physical laws, and simulations. Tensorized neural networks lead to large rates of compression while improving on generalization and robustness. In order to speed up multi-node model training, I will demonstrate how simple gradient compression (SignSGD) leads to communication savings while preserving accuracy. Thus, with better algorithmic design, it is possible to obtain “free lunches” and obtain better efficiency in ML.