Collaborative Research: CCSS: Learning to Optimize: From New Algorithms to New Theory

Solving machine learning (ML) problems requires efficient and scalable optimization algorithms. State-of-the-art general purpose algorithms often need to compute a large number of iterations and hence have limited applicability to real-time applications. To circumvent this shortcoming, learning to optimize (L2O) methods aim to learn a shorter (i.e., faster) optimization path over a task distribution at meta-training, based on the tasks’ common structures and a more global view of their geometries, and then apply the learned optimizer to new similar tasks at meta-testing.

Despite extensive empirical success, the existing L2O methods perform well mainly on optimization tasks with similar structures, but likely perform poorly on out-of-distribution tasks. Furthermore, there has been little theory understanding the convergence and generalization of L2O algorithms. Thus, the proposed program will design novel L2O approaches, so that the trained optimizer can generalize to a broad range of practical tasks, particularly out-of-distribution tasks, and will have guaranteed convergence and generalization performance in L2O training and testing.

Specifically, the proposed program will design new L2O approaches with both generalizability to out-of-distribution tasks and safeguarded feature for guaranteed worst-case convergence, will develop a theoretical framework for analyzing the convergence rate for L2O meta-training, and will provide comprehensive characterization of the generalization performance for L2O meta-testing. The new algorithms and theory will be evaluated over applications of on-device model adaptation in internet-of-things (IoT) systems, sparse recovery for images and wireless signals, and algorithmic adaptation in reconfiguration of communication systems.

The project is anticipated to significantly mature the field of L2O, and provide training opportunities for a diverse group of students at the new intersection of optimization, machine learning, signal processing, and data science.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.