CRII: HCC: RUI: Developing personal audio archives and technology-mediated listening practices for everyday sonic thinking

This project will develop tools that enable people to record personally meaningful sounds in their everyday lives and to use these recordings to help people think about and reflect on the sounds in their lives. With the widespread availability of smartphones, many people are able to document important moments from their personal lives with digital photos, which they can creatively remix and share.

Yet these digital photo archives leave out the many moments that unfold via sound, such as the laughter of a friend, or the sound of a loved one cooking a favorite meal. Further, the personal sounds people do collect are often embedded within video files, which focus attention on visual information, and tools for working with audio primarily support music or speech, ignoring the wide range of other meaningful sounds people encounter in daily life.

By developing tools that can help people who have no special training to document and interact with the sounds they encounter in their daily lives, this project seeks to empower people to learn new skills for listening as a means to reflect on the role of sounds, and technology, in their lives.

To investigate the untapped possibilities of digital listening in everyday experience, this project aims to develop new technologies for personal digital audio archives that can support sonic thinking and close listening as reflective practices. The project will follow a research-through-design process that incorporates course-based research and participatory design methods.

The project will be conducted in four phases. In phase 1, the research team will develop prompts and activities for people to digitally record sounds in real-world contexts and use these sounds creatively and analytically. In phase 2, undergraduate students will perform the activities developed in phase 1, including practicing sound design and audio data science with their personal audio data, in a reflective and social classroom setting.

In phase 3, under the guidance of the researcher, students in a human-computer interaction research and design practicum course will analyze the data from phase 2 and use it to develop functional prototypes for recording and using everyday audio data. In phase 4, through a long-term diary study, the researchers will deploy the prototypes from phase 3 to understand how they impact sonic thinking skills and support critical reflection on data and technology.

This project will contribute novel interfaces and technologies for recording sounds in daily life. It will also explore new behavioral practices for people to engage with sound in meaningful ways through acts of creation and remixing, building knowledge about how people might use digital listening to reflect on the role of technology in their lives.

By developing new, digitally-mediated listening practices, this research extends existing audio data science techniques into unexplored personal and social contexts, creating new research directions in the field of human-in-the-loop machine listening.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.