Semantic alignment of modal auditory category and multimodal lexical-semantic networks

The natural world is filled with information that spans all of our senses. When we walk through a park we not only see a rich visual world, we also hear sounds ranging from birds tweeting to the conversations taking place around us. All of this information has meaning, informing us about the structure of our current world, how it is likely to immediately change, and how we might best respond adaptively to it.

Sounds are an important source of information for daily life. Sounds alert us to danger, they allow us to navigate to targets that cannot be seen, and they help us to know where we are in the environment. To use this information, the brain must both correctly identify sounds and integrate them with other sources of information to support ongoing cognitive functioning.

The proposed work seeks to understand how the brain extracts meaning from sound and how the process is integrated with those used to extract meaning from other senses, primarily vision. The project also aims to provide advanced training to both graduate students and post-doctoral scholars in cutting-edge empirical, analytical, and modelling approaches.

In detail, the project aims to test an hypothesis about how meaning is extracted from sounds using fMRI. The auditory semantic alignment hypothesis proposes that a set of cortical areas accomplish this extraction through a dual-specialization for auditory processing and particular categories of sounds. These areas are proposed to extract particular types of information from certain categories of sound and pass that information onto a broader semantic network that is amodal, integrating over the senses.

Evidence for a similar system has been observed in visual processing. If a homologous structure is observed for auditory processing, it would suggest a general mechanism for extracting meaning from external stimuli. The proposed work has the potential to significantly improve our understanding of how our cortical responses support our perception and adaptation to a dynamic natural world.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.