Collaborative Research: CCSS: Continuous Facial Sensing and 3D Reconstruction via Single-ear Wearable Biosensors

Facial landmark tracking and 3D reconstruction are popular and well-studied fields in the intersection of computer vision, graphics, and machine learning. Despite their countless applications such as human-computer interaction, facial expressions analysis, and emotion recognition, existing camera-based solutions require users to be confined to a particular location and face a camera at all times without occlusions.

This highly constrained setting prevents them from being deployed in many emerging application scenarios, in which users are likely to engage in three-dimensional body/head movements. This project aims to provide a new form of single-ear biosensing system that can unobtrusively, continuously, and reliably sense the entire facial and eye movements, track major facial landmarks, and further render 3D facial animations via cross-modal transfer learning.

The research outcome of this project will push the limits of ear-worn biosensing to enable rich sensing capabilities that are currently infeasible, such as camera-free facial landmark tracking, and real-time 3D facial reconstruction, etc. Relying on the learning model studied in this project, the project team is building two representative applications, i.e., facial sensing for mobile virtual reality (VR)/augmented reality (AR), and speech enhancement using the reconstructed facial landmark dynamics.

The project will substantially advance the wearable and biosensing techniques as well as transfer learning across multiple sensing modalities.

The project is bridging the gap between the anatomical and muscular knowledge of the human face and electrical and computational modeling techniques to develop analytical models, hardware, and software libraries for sensing face-based physiological signals. In particular, the project team is building a low-power low-noise circuit to sense the entire facial muscle activities using single-ear biosensors.

The team is also developing a compressing algorithm that activates the sensing and communication components only when facial changes are detected, which can significantly increase the battery lifetime and reduce the computational cost of the wearable system. Moreover, to enable camera-free 3D facial reconstruction, the team is developing a cross-modal learning model that consists of a visual facial landmark detection network and a biosignal network, in which knowledge embodied in the vision model can be transferred to the biosignal domain during training.

To further enhance the model’s robustness, the team is integrating the third modality (i.e., inertial sensors) into the cross-modal learning model and exploring domain adaptation and continual learning techniques. Additionally, the team is exploring model compression and acceleration techniques to enable the on-device deployment on existing head-worn devices such as VR/AR headsets

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.