RUI: Predictive models with Incomplete and Fragmented Observations, and New Advances in Virtual Re-sampling for Big Data

A major focus of this project is on the development of new procedures to carry out statistical modeling, prediction, and inference in the presence of missing data. Incomplete, missing, censored, and partially observed data are prevalent in many areas of medical sciences, engineering, economics and social sciences, which can in turn complicate the task of prediction and inference in data-driven decision-making processes.

The investigator will study and explore the effectiveness of several new methods for handling missing values in complex data structures without imposing unrealistic or unnecessarily stringent conditions on the underlying mechanisms that cause the absence of information. Another major aim of this research project is to develop efficient data re-sampling methods to alleviate the formidable computational cost of computer-intensive statistical methods in big-data scenarios, where the data analyst must deal with, and sort through, massive amounts of data.

The advent of such efficient methods is timely as the wave of ultra-large datasets has taken over many data-analytic initiatives in medicine, agriculture, and environmental protection. Additionally, this project embraces research experiences for graduate and undergraduate students, many of whom will then be persuaded to move on to further studies and research careers in STEM disciplines.

This research project deals with two broad classes of problems related to predictive models and inference. The first part focuses on selected topics in predictive models such as regression and classification for a number of nonstandard realistic setups. Specifically, the investigator will develop several local-averaging-type regression estimators in general metric spaces for incomplete and fractionally observed data with applications to statistical classification and the related problem of unsupervised machine learning.

The aim is to carry out a rigorous study of the convergence properties of these estimators in various norms which is necessary for correct prediction and inference. In particular, this project will study and develop new exponential performance bounds for the Lp norms of the proposed estimators. The problem of bandwidth estimation for incomplete and fragmented functional data will also be studied; this is particularly important as the optimal bandwidth minimizing quantities such as the MISE or ISE is not necessarily optimal in classification.

The second part of this research plan considers new objectives in virtual re-sampling as a method to reduce the formidable computational cost of big-data bootstrap in a number of important and challenging problems, while still retaining the benefits of bootstrap methodology. In particular, the investigator will develop virtual re-sampling strategies to (i) approximate the distribution of several refined higher criticism statistics for multiple testing problems in big-data scenarios, and (ii) to speed up the logarithmically slow rates of convergence of important functionals of density and regression estimators in two-sample problems such as those based on deconvolution density estimators and their sup-functionals for errors-in-variables models in big-data scenarios.

To achieve the objectives under (i) and (ii), the investigator will use adaptations of the methodologies used in the strong approximations of bootstrap empirical processes in the literature.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.