Identifying mechanisms of multisensory memory using virtual reality and fMRI

Project Summary Forming and retrieving memories for objects and events is fundamental to human experience. Complementary mechanisms within the medial temporal lobe (MTL) support the formation and retrieval of memories for objects, such that the perirhinal cortex supports memory for features of individual items, and the hippocampus supports

recollection of the object within the context it was originally encountered. Although most research on object memory has been conducted using visual stimuli alone, a small body of research has shown that visual objects encoded along with their characteristic sound (e.g., a dog and a bark) are better remembered later on than

objects encoded only visually, or with a meaningless or non-characteristic sound. Despite this evidence and the ubiquity of multimodal stimuli within natural environments, our knowledge of the extent of multisensory influences on memory and its underlying mechanisms is very limited. My recent behavioral work showed that

the benefits of multisensory encoding are based on improved recollection of the context in which those objects were encoded, suggesting that multisensory processing uniquely engages mechanisms that bind these objects to surrounding information at encoding. However, it remains unclear what features of the encoding context are

better retrieved, and what neural mechanisms are modulated by multisensory processing to allow for this benefit. The proposed research addresses this gap in knowledge by collecting the critical evidence to determine the facets of memory for items and surrounding events that are influenced by multisensory

processing, and the brain activity patterns and regions that are involved in retrieving such information. We hypothesize that multisensory processing at encoding improves object memory by increasing the likelihood that the object and context will be bound into an episodic memory to support later recognition of that object,

rather than by enhancing memory for the individual object itself. Further, we predict that patterns of activation within the hippocampus during retrieval of multimodal objects will also carry information about the environment in which these items were encoded. To rigorously test the type of memory that multisensory processing

impacts, there is a need to assess memory for unimodal and multimodal objects that are embedded within rich, naturalistic spatiotemporal context. To this end, we have developed an immersive, naturalistic encoding task within virtual reality (VR) environments, which contain controlled but animated visual and audiovisual objects.

This novel approach will allow us to assess whether multisensory processing influences memory for individual objects alone or if these objects are more readily bound to their surroundings to support memory for events within context (Aim 1). Further, analyses of fMRI data will be used to determine which regions of the MTL and

cortex are specifically involved in memories of multisensory objects seen in context (Aim 2). The overarching goal of this project is to advance our knowledge of how episodic memories are formed during naturalistic experiences and this work will contribute to our ability to explain and predict real-world behaviors.