Collaborative Research: RI: Medium: Robust Perception through End-User Adaptation

For an intelligent system (such as a robot or a self-driving car) to enter end-users' daily lives in a safe and reliable way, the system must generalize beyond the development laboratory to uncontrolled environments where it will be deployed. For instance, a self-driving car may be built and optimized for sunny weather but may be driven by the user in icy or snowy conditions.

The state of the art in machine learning and perception cannot generalize or adapt to this sheer diversity of deployment scenarios. This project seeks to address this challenge by leveraging the fact that people are creatures of habit and tend to use their devices consistently and repeatedly in specific ways (for example, they drive their cars repeatedly over a small set of routes between their home, office, and the marketplace).

Such repetitive usage provides ample opportunities for the intelligent system to adapt itself to the end-user's specific circumstances, no matter how challenging or different they are. This project builds upon this insight to design robust perceptual systems that will adapt to a diverse array of real-world challenging settings, including self-driving cars in different driving locations and various time and weather conditions.

Guaranteeing that an intelligent system can operate reliably across such diverse settings is necessary to unlock the societal benefits that researchers in machine learning, computer vision, and robotics are striving to achieve. Beyond the research community, the project will contribute to education by training undergraduate and graduate students and by outreach to high-school students through workshops and summer programs, especially to benefit underrepresented minorities.

This research project investigates the design and development of robust perceptual systems through adaptation, by exploiting a specific and well-known property of end-users: Humans are creatures of habit and tend to operate devices in specific ways and environments repeatedly and consistently. For example, most people drive their cars primarily along the same routes every day.

In particular, the investigators explore three key ideas: (1) adapting the perceptual system by recording sensory input during usage, generating highly reliable pseudo-label annotations that incorporate physical constraints and cross-sensor consistency, and fine-tuning the system while it is offline via dual-task co-adaptation; (2) personalizing the system through repetition, by aligning playbacks over time to leverage deep neural networks' ability to memorize and by augmenting data for diverse settings through label propagation across recordings; (3) verifying adaptation by developing methods to detect and remove noisy labels using learning dynamics and active user verification. These three research aims will be complemented by a comprehensive evaluation plan to include multiple existing self-driving data sets, a newly collected data set by the team of investigators that captures diverse environments along a repeated route, and navigation in home robot scenarios.

This research effort towards a much larger, more challenging adaptation problem will open the door to novel solutions in the intersection of computer vision, machine learning, and robotics, including but not limited to reasoning about physics, modeling the relationships between rich perceptual tasks, adapting to changing output distributions, and leveraging patterns in the provenance of the data itself.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.