Excellence in Research: Incorporating Attention into Computational Auditory Scene Analysis Using Spectral Clustering with Focal Templates

Humans display an uncanny ability to focus on a sound of interest, even in the presence of interfering sounds or noise. For instance, a child’s voice may be immediately recognized and reacted-to in the context of parental protection, a bird-watcher may recognize and follow the call of a house finch, or business associates may discuss the latest events over lunch in a crowded diner.

In each of these examples, a person directs his or her attention to the sound source of interest. Once attention is directed, a human can continue to focus on that sound, even to the degree that he or she may not perceive (or is able to passively ignore) other sounds. Computational listening systems, at present, are unable to replicate that ability.

If such ability was possible, computational listening systems could contribute to an array of application domains. Current hearing aid technology is woefully inadequate in group settings, where noise often overwhelms the listener and the listener is unable to compensate. Those who suffer from hearing loss feel socially isolated, leading to a lower quality of life.

Performance of hearing aids, and the lives of those who rely on them, would improve drastically if these devices could automatically adapt to isolate and focus on salient sound sources. Effective acoustic monitoring could be deployed to safety-related applications, such as automatically detecting slurred speech from someone suffering a stroke, or to alert the deaf to important loudspeaker announcements in a public place.

Autonomous devices (robots, drones, etc.) could employ enhanced listening techniques to guide their movements or actions, potentially better serving or protecting the public. The ability to focus on a particular instrument within a musical ensemble could lead to improved automatic music transcription systems and enhanced tools for performance analysis and training.

The applications of computational listening systems exhibiting perceptually-based attention are boundless.

This research will investigate the use of focal templates to incorporate attention into computational auditory scene analysis (CASA). CASA attempts to reproduce (via computational algorithms and systems) the ability of humans to perceive an acoustic scene given only auditory input, and the underlying methods of CASA are modeled upon psychological principles of perception.

A focal template may be considered a type of dynamic time-frequency filter that passes only those auditory elements conforming to a pattern of interest. Sound events of interest (if present in the audio) will be detected using spectral clustering, which has emerged as a useful technique for grouping "like" elements within a set; here, the goal is to group auditory elements that pass through the focal template.

This work will: 1) implement a model of attention into CASA through use of spectral clustering with focal templates, 2) measure the impact of incorporating attention on the performance of CASA systems in isolating particular sounds of interest, and 3) determine effective methods to develop focal templates while considering how to scale to the "general" listening case. This project includes development of an academic minor in "Sound and Music Computing", a unique educational opportunity to prepare students to contribute to the research while integrating across STEM and non-STEM disciplines.

An outcome of this work is to build partnerships and strengthen the Computational Research on Music & Audio Team of Interdisciplinary Collaborators (CRoMA-TIC). Through these activities, the project will: 1) advance the state of the art in CASA, 2) foster collaborative relationships within this field, 3) develop a well-recognized area of expertise and research capacity at Lincoln University, and 4) develop a pipeline for undergraduate students leading to related careers or graduate study.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.