The Origin of Crossmodal Correspondences in Learning and Experience

What we take to be our reality relies on the seamless integration of all the sensory input we continuously receive from our environment. How does multisensory integration shape the way I experience the world around us?

I will investigate crossmodal correspondences as a proxy to understand how our multisensory, subjective experience is rooted in learning from our lifelong exposure to the world around us.

The hypothesis is that crossmodal correspondence, for example between blue and low pitch, are due to their common associations with things in the world, such as the sea or the sky.

I will test a computational model that quantitively predicts crossmodal correspondences based on objects and materials from our natural environment (ecological model). Then, I examine how ecological associations drive the development of crossmodal correspondence during childhood.

In a fourth approach, I consider immersing participants in a virtual reality and make the learn new crossmodal associations, e.g., between orientation and sound. Then, I test how manipulations of those associations affect their immersion in the virtual environment.

With the help of two experts in artificial intelligence, I will train a generative adversarial network to learn natural associations between vision and sound from urban environments and the Equatorian jungle, and test whether the trained network predicts human crossmodal associations.

Results of this project will establish how implicit, unconscious learning of regularities in our environment produces automatic associations across modality and shapes our rich, multimodal experience of the world around us.