Collaborative Research: CNS Core: Small: Towards Unsupervised Learning on Resource Constrained Edge Devices with Novel Statistical Contrastive Learning Scheme

Deep learning models have been deployed in an increasing number of edge and mobile devices to power various tasks in our life, from personal assistance in smartphones and augmented reality (AR)/mixed reality (XR) glasses to healthcare robotics. One drawback of existing deployment, however, is that neural networks do not adapt to different users and application domains, nor do they evolve when new unseen data stream in once trained in the cloud and deployed in the devices.

Existing on-device training schemes all require manual data labeling, which can be very expensive or challenging once deployed on devices due to strong requirements on expert knowledge, data privacy, communication cost, or latency. Therefore, it is more practical and useful for on-device learning models to be able to learn from new streaming data in-situ with as few labels as possible, in a resource-constrained environment.

This project aims to lay the technological foundation for unsupervised on-device learning framework, in which the on-device deep learning models can continuously learn visual representations with minimal human intervention. Three tasks will be carried out to achieve efficient computation and memory utilization, as well as high learning speed and accuracy while overcoming the non-independent and identically distributed (non-IID) issue in streaming data.

This project will be evaluated with real systems and applications with industry collaborators Misty Robotics and Facebook on target applications including robotics, augmented reality (AR) and mixed reality (XR).

The success of this project will lead to higher accuracy for machine learning-powered devices and a better user experience for everyone. More importantly, this project will enhance the fairness of AI by improving the inference performance for minorities under-represented in the data collection process, through continuous personalization on new incoming data.

It will also enable learning capability for devices deployed in remote areas such that they can quickly adapt to new environments, which will drastically benefit various consumer, business, scientific and national security applications such as battlefield scouting and outer space exploration. The education impacts of the proposed research include the integration of various educational activities based on the resources available to the two PIs such as DAC System Design Contest; outreach for local K-12 students through Pitt’s Investing Now summer school and ND’s CS curriculum for K-12 students in Indiana; undergraduate research with emphasis on minority participation, and course integration of the research outcomes.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.