AF: Small: Optimal algorithms and new models for statistical estimation

Data is becoming both more valuable, yet harder to effectively analyze. Across many areas of statistics and machine learning, one desires algorithms that 1) give more accurate predictions, 2) require less data to operate, and 3) are robust to the wide array of errors that show up in real data. This project focuses on addressing this challenge in two archetypal settings.

First, this project seeks to develop new algorithms and analysis tools for one of the most useful and versatile primitives of statistics: estimating the mean of a probability distribution, given data from it. Second, it seeks to develop new models that identify, unify, and hope to resolve the challenges of dealing with data that come from a non-ideal sampling process.

These problems lie at the intersection of statistics, machine learning, and computer science. More broadly, this project will help build a pipeline of future researchers, by creating new pathways for undergraduates towards research and to graduate school, and by exploring with computer science students the role of research, the thought process of research, the mechanics of research, and the impact of research.

In more detail, the first target of this project is the fundamental problem of "mean estimation", often considered the most important classical estimation problem in statistics: given samples from a probability distribution in one or more dimensions, estimate its mean as accurately and robustly as possible. Surprisingly, there are many important settings in which good solutions to this basic question remain to be found.

This project will focus primarily on the setting of high-dimensional data, and will aim to develop algorithms that are efficient, accurate, and flexible. The second area of focus is the challenge of making accurate statistical inferences from data, despite the data originating from a non-uniform, biased sample. This project develops new frameworks for how to leverage insights about the data-collection process to improve the accuracy of statistics, a challenge that has been grappled with by many fields including computer graphics, econometrics, sociology, and statistical physics.

The techniques developed to tackle these problems will reveal new algorithms and subtle probabilistic phenomena that will inform the next generation of solutions to the challenges of effectively using valuable data.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.