EAGER: Developing data and evaluation methods to assess the generality and robustness of AI systems for abstraction and analogy-making

The ability of humans to make conceptual abstractions and analogies is at the root of many of our most important cognitive capabilities, such as learning new concepts from small numbers of examples, flexibly adapting our prior knowledge and experience to new situations, and communicating our knowledge to others. While AI has made dramatic progress over the last decade in areas such as vision, natural language processing, and robotics, current AI systems almost entirely lack the ability to form humanlike abstractions and analogies.

The lack of such abilities is in part responsible for the lack of robustness in current AI systems, as well as their difficulties with extrapolating what they have learned to diverse situations. While there have been many efforts in past AI research on this topic, each individual effort has generally focused on a specific problem domain, without careful evaluation of the AI system’s robustness within its domain or its generality across different domains.

In this project we will promote progress in AI by creating a web-based platform that offers a diverse set of abstraction and analogy-making challenges for the research community as well as new evaluation methods that test for generality and robustness within and across different challenge domains. We will use our platform to evaluate selected existing AI approaches and to measure human performance on our challenges in order to compare with AI systems’ performance.

Our work will contribute to the AI research community by spurring new approaches and evaluation methods for abstraction and analogy-making in machines, and will contribute more broadly via the development of methods for robust and generalizable AI systems.

Our specific research plan is to (1) curate an initial suite of idealized challenge domains inspired by Hofstadter’s letter-string analogies, Raven’s progressive matrices, Bongard problems, and Chollet’s Abstraction and Reasoning Corpus; (2) develop evaluation methods along dimensions such as robustness to variations on a particular concept, generality across domains, and scalability to more complex instances of a problem; (3) evaluate selected AI methods for abstraction and analogy using our evaluation methods; and (4) measure human benchmarks on our challenge suite using paid participants on the Amazon Mechanical Turk platform. At the end of the project period, we will have demonstrated the utility and promise of our challenge problems and evaluations, and will have gained insight into their limitations.

This work will set the stage for future efforts on expanding our challenge suite, improving our evaluation metrics, and developing and evaluating novel AI approaches to abstraction and analogy.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.