RI: Medium: Probabilistic Box Embeddings

Artificial intelligence (AI) and machine learning are revolutionizing the pace of progress in science, biomedicine, healthcare, business, economics, and the national defense. A foundational technical choice in AI and machine learning is that of representation. Before a machine can reason over data, that data must be represented in a way that enables parameters to be learned, and useful inferences to be made.

The choice of representation has profound implications for the method’s capabilities and safety. This project explores an new alternative fundamental representation that is expected to provide better expressivity, interpretability, uncertainty characterization, and robustness, thereby laying groundwork which has the potential to provide in future representational foundations advantageous to AI safety and commonsense reasoning.

The fundamental representation for data and concepts in nearly all machine learning, including neural networks, is the vector: a point in d-dimensional space. Vectors conveniently support symmetric distance calculation, semantic neighborhoods, and geometric reasoning. For example, learned vectors representing "eagle," "bird," and "fly" may designate points that are close to each other, indicating that they are semantically closely related.

However, there are intriguing reasons to consider representations based not on points, but rather regions––regions of varying breadth and overlap, able to capture (like Venn diagrams) that "bird" is a broader concept than "eagle" and "all eagles are birds" and "some but not all birds fly." This project focuses on machine learning research in a new learnable representation called box embeddings, d-dimensional hyperrectangles, which are closed under intersection, can represent arbitrary directed acyclic graphs, define regions whose volume is easily calculated, and can precisely and compactly represent large joint probability distributions. The research will address foundational open research questions concerning (1) fundamentals such as expressivity, regularization, and alternative geometric spaces; (2) relation to graphical models, having already shown that boxes have interestingly different strengths; and (3) deep learning with boxes.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.