Distance-Based Analysis for Complex High-Dimensional Data

Throughout the course of the twentieth century, distances have played a significant role in important areas of statistics, which include classification, clustering, discriminant analysis, multidimensional scaling, sampling, spatial statistics, scoring rules, and kernel methods in machine learning. Distances are also central to the definition of divergence measures, relative entropy and information gain, some of which are fundamental to the concept of C.R.

Rao's quadratic entropy and the analysis of diversity in ecology and other areas of science. Yet, at present there are significant gaps in our knowledge and in the emerging statistical literature on the use of distance-based tests and analyses for complex high dimensional data. One example is analysis of similarity which is among the most cited and most widely used distance-based statistical methods but is limited by an absence of relevant mathematical knowledge.

This research will derive new mathematical knowledge on various distance-based statistical methods, and apply this for providing answers to important scientific questions arising in a number of disciplines in forestry, ecology and marine science, such as: (1) how biodiversity changes in tropical forests? (2) how taxonomic and functional profiles of bacterial communities change with environmental conditions in different oceanic regions? The project establishes collaborations among several disciplines and between two US academic institutions and provides research and training to graduate and undergraduate students.

The project develops a new body of knowledge on distance-based statistical methods and computation for analyzing complex, high dimensional data that arise in the form of compositions, trees, graphs, or networks. The distances considered here are all non-Euclidean -- either non-metric dissimilarities that do not satisfy any triangular inequalities or just discrete numbers -- but they all arise from conditionally positive definite kernels.

Examples of distances include the squared Euclidean distance, the Bray-Curtis dissimilarity, the Jensen-Shannon distance, Unifrac or the Kantorovich-Rubinstein metric, the Aitchison distance, the edit distance, various graph kernel and spectral distances, and other distances based on optimal transport problems. Specifically, the project advances the mathematical theory and computation of exact distribution-free two and multi-sample runs tests, change points, and other related problems by counting runs along the shortest Hamiltonian path (or loop) of the pooled sample of data points.

The project also considers analysis of similarity and related distance-based rank tests and derives new mathematical results that allow us to pursue more advanced statistical analyses. The project contributes to: (i) a deeper analysis of biodiversity in tropical forest; (ii) an investigation of how taxonomic and functional profiles of prokaryotic communities change with environmental conditions in different oceanic regions; (iii) a study of the variability of composition of rare earth elements in deep-sea muds of the Pacific Ocean; and (iv) an understanding of the relationship of intertidal communities in the Oregon coast with respect to upwelling and nutrient delivery.

The project integrates mathematics research, science and education and will provide opportunities for dissertation work for graduate students.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.