A test of a novel non-probabilistic model of 3D cue integration

Despite at least a century of scientific investigation, it is still not entirely understood how the human brain constructs a perception of three-dimensional (3D) objects and space from the 2D information reaching the eye through light rays. It has been known for a long time that this ability is attained on the basis of a variety of different visual signals (that the brain interprets) called “depth cues”, which the brain then combines together to derive the 3D structure of the scene.

Examples of depth cues can be observed in paintings where linear perspective, shading, and even simple contours depict a 3D world on the flat canvas just as they do on the flat human retinas while people observe real 3D objects. However, how these cues encode 3D information and how the different cues are combined to generate our stable coherent perception of 3D visual space and objects remains poorly understood.

The prevailing theory in the scientific literature (Bayesian probabilistic inference theory) postulates that the brain derives a 3D structure of the world by determining how likely a particular 3D structure is given the information on the retina. Implementing this model requires a host of assumptions, such as that depth cues deliver “noisy” estimates of 3D parameters that are still, on average, accurate.

However, a number of common and important observations cannot be fully explained by the Bayesian model and cast doubt on the critical computational assumptions of the model. Moreover, the Bayesian model struggles to explain important differences in the “quality” of 3-dimensionality that we perceive between viewing the real world and artificial situations such as pictorial images, or virtual or augmented reality (VR, AR).

This project tests a new theory (the Intrinsic Constrained theory) that makes an entirely different and simpler set of assumptions compared to the Bayesian theory, but that can predict a wider range of perceptual phenomena. The project integrates the IC theory with a recently proposed theory that postulates that the visual system does not generate a single encoding of 3D space, but two distinct encodings, one which is relevant to understanding the scene (i.e., 3D object shape and layout) and the other that underlies visually guided movements like reaching for and grasping an object.

The latter encoding is claimed to underlie the special subjective experience of 3-dimensionality that is most obvious while viewing stereoscopic images (e.g., 3D movies). This project will show that the IC model can efficiently incorporate the claims of two distinct representation of 3D space. In doing so, it is able to provide a better explanation of a range of fundamental aspects of our perception of 3-dimensionality that are challenging to explain with the prevailing model, including those required for a better understanding of factors important for developing 3D technology (e.g., VR and AR).

This award supports empirical research that tests a computational model arising from the Intrinsic Constraint theory of cue integration against the prevailing computational model belonging to the Bayesian framework. Specifically, the new model challenges three main assumptions of the Bayesian model that will be tested by experiments in distinct work packages: (1) that depth cues on average provide veridical (accurate) 3D estimates; (2) that these estimates are stochastic and that their probability distributions are encoded by the visual system; and (3) that the process of cue integration results in a single encoding of 3D structure.

Instead, the Intrinsic Constraint model predicts that cue estimates are biased, deterministic, and that cue integration results in two distinct encodings of 3D structure. Using state-of-the-art visual display and motion tracking apparatus that allow both psychophysical and psychomotor response measurements, the investigators are conducting a comprehensive set of experiments aimed at critically testing two theories (Bayesian and IC).

The first work package establishes which theory better explains 3D perception based on single or combined depth cues. The second work package establishes if depth cues provide stochastic estimates as proposed by the Bayesian theory or if cues provide deterministic noise as proposed by the IC model, with noise in 3D estimates due to extraneous experimental factors.

The third work package shows how differences in the subjective experience of 3-dimensionality is linked to the efficacy of the 3D encoding underlying guidance of movement, but not that underlying perceptual judgements.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.