Collaborative Research: CISE-ANR: RI: Small: 4DShape: 4D Human Shape Analysis and Synthesis

This U.S.-French joint research project addresses the growing interest in modeling, analyzing, and generating 3D shapes and movements of human bodies and faces. Advances in scanning technology, 3D mesh-extraction algorithms, computer vision, and hardware-accelerated computer graphics have enabled access to large-scale datasets of human body representations.

However, while artificial intelligence and machine learning have achieved remarkable success in processing image data, working with 3D shapes in the form of meshes presents unique challenges that often degrade performance in common computer vision and graphics tasks. To overcome these challenges, this project aims to integrate rigorous shape analysis concepts into the design of geometric deep learning models.

These models will directly process raw 3D surface scans, independent of acquisition methods, to develop robust algorithmic pipelines for key problems in human body and face analysis. Applications of this work include single-object data representation and reconstruction, body motion generation, facial expression retrieval, and automatic animation. Beyond its implications for augmented and virtual reality, the project will train graduate students and strengthen collaboration between investigators from four institutions in the U.S. and France.

The research focuses on developing efficient deep learning architectures that incorporate fundamental shape invariances into machine-learning pipelines. It consists of three key thrusts: (1) Invariant 3D-to-3D Registration and Reconstruction: This thrust will develop a framework adapted to human body shapes and face scans, combining mathematical shape analysis concepts with advancements in latent space and auto-encoder models in computer graphics; (2) Extension to Time-Dynamic (4D) Data: Building on static 3D data, this thrust will extend methods to dynamic 4D data (3D plus time) for motion analysis and generation.

The approach will involve constructing a non-linear structure in the human shape latent space, using a blend of data-driven techniques and physically motivated elastic deformation energies. This will allow accurate modeling of the complex nature of real-life human body motions and deformations; and (3) Prompt-to-Shape Learning: This thrust will focus on mapping prompts, such as text or voice recordings, to shape spaces.

Through these efforts, the project seeks to advance the state of geometric deep learning and its applications while fostering international collaboration and academic training.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.