Collaborative Research: CompCog: Adversarial Collaborative Research on Intuitive Physical Reasoning

People are able to reason about the world in amazingly complex ways, yet we consider these capacities part of simple “common sense,” generally shared across individuals and cultures. We toss and catch balls, stack dishes in the sink, and pour a morning cup of coffee with almost no effort. Yet the cognitive systems that support these capabilities are not well understood; even our most advanced attempts to reverse engineer them in robots fall short of human-level efficiency or flexibility.

This grant was designed as an “adversarial collaboration” to bring together scientists from two different sides of a critical debate about the nature of human physical reasoning abilities. One theory (championed by the MIT PIs) suggests that this physical reasoning is based on a cognitive system that allows people to simulate what might happen next, similar to how physics engines for video games are used to predict what will happen next in those scenes.

While this theory has provided many successful explanations of human behavior, including making precise predictions about how people think Jenga towers will fall, or where they think balls flying through the air will land, another growing body of research (led by the NYU PIs) has demonstrated many instances where the simulation theory cannot adequately describe what people do, but where simpler and approximate “rules-of-thumb” (even inaccurate ones) can. Because human physical reasoning is unlikely to be purely simulation or purely based on simplified rules, a team of experts from both sides of this debate will be crucial for advancing our understanding of the cognitive processes that underlie these reasoning capabilities.

Towards reconciling these views, this grant advances the idea that consideration of known human limitations -- e.g., in memory or attention -- can explain the processes that people use when reasoning about the physical world. The goal is to integrate these constraints into a more complete theory of human reasoning that can account for both our failures and our successes in comprehending the physical world.

True understanding of these processes will require “reverse engineering” human cognition and perception by designing computational models with similar limitations and capabilities to people. These scientific models may provide insight for researchers in AI and robotics who are interested in designing systems that interact with the world like people, including self-driving cars or the control of prosthetic limbs.

Furthermore, exploring how people learn and reason about physics may provide new approaches for physics education. Finally, studying and modeling these facets of physical reasoning will require developing extensible tools, which will be released as open-source software to open up the research into human physical reasoning to a wider set of scientists.

This project studies and proposes to resolve tensions between theories of human physical reasoning that suggest that it is based on relatively accurate simulatable mental models, and those that suggest it is based on heuristics and other qualitative forms of reasoning. The research includes experiments related to those that have been used to demonstrate simulation theory, but modified to induce shortcuts in physical reasoning in two broadly different ways.

Aim 1 experiments consider scenarios that are expected to run into human resource limitations, either in attention, memory, or time – for instance, asking people to predict the stability of complex towers of blocks with too many pieces to track individually. Aim 2 experiments consider scenarios that could be reasoned about with simulation, but could more easily be reasoned about with simple rules or heuristics – for instance, studying how people use rules like “the heavier side will tip over” when judging which direction a balance beam stacked with objects will fall.

Human behavior in these experiments is examined for deviations from pure simulation theory in line with the expected resource limitations (e.g., using rules, focusing on a subset of objects, or representing objects more coarsely), and computational models are developed to explain this behavior. These models are designed around the framework of “resource-rational” cognition, which suggests that people deploy limited cognitive resources in a way that efficiently solves the problems they encounter.

The behavioral results and models together allow investigation into (a) whether and when people’s physical reasoning is constrained by resource limitations, and (b) the types of shortcuts people take to circumvent these limitations. Performing this research requires developing an integrated software suite for designing experiments and modeling across a wide variety of physical scenarios.

Designing these integrated packages typically requires a large set of technologies -- physics simulators, graphics engines, computational modeling methods -- that are outside the reach of most psychologists, which in turn limits research into human physical reasoning. The PIs are in a unique position to contribute here because their laboratories are focused on computational models of psychology and they have an extensive track record of developing open-source software used by multiple research groups worldwide.

The software suite used in this grant is designed to be open-sourced and shared with the broader research community to facilitate further research into human physical reasoning without requiring extensive knowledge of the underlying technologies.

This work was supported by SBE/BCS Perception, Action, and Cognition, EHR Core Research (ECR), and CISE/IIS Robust Intelligence.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.