Implicit Neural Representations for Facial Animation

Neural Radiance Fields (NeRFs) are a type of implicit neural representation that uses neural networks to model complex visual data without explicitly defining their parameters. In recent years, they have emerged as a leading solution in 3D graphics and asset generation. Their growing prominence extends beyond static 3D modelling and is increasingly embraced by the animation and virtual reality (VR) communities, primarily due to their ability to balance high compressibility and exceptional reconstruction accuracy.

NeRFs hold substantial promise in a variety of applications, and one fascinating domain is the field of Facial Animation. Despite their merits, NeRFs come with their set of challenges and we aim to achieve the following research objectives:

1. Improvement in Explainability and Controllability: NeRFs operate as "black box" models and their opacity makes it difficult to understand and control the specific physical attributes of the output. In the context of facial animation, this could lead to suboptimal control over the appearance and emotions conveyed by animated avatars. To this end, we will develop disentangled representations of facial identity and expressions to improve controllability.

2. Integration of Speech: Achieving multimodal synchronization between visual and auditory cues is a challenging and ongoing endeavour. We will develop models that will not only capture the visual dynamics of facial expressions but also synchronize them with the phonetic elements of speech.

3. Watermarking: NeRFs and similar generative models have raised concerns about their potential misuse, especially in the creation of "deepfake" content. Deepfakes can be used to manipulate images and videos to a high degree of realism, posing risks to privacy, security, and misinformation. As a response, we aim to develop robust watermarking and authentication techniques that are crucial to detect and mitigate "deepfake" content.

Animated human avatars have a wide array of applications - rich VR experiences, teleconferencing, gaming, and online education. The integration of NeRF-based facial animation would elevate the level of realism, resulting in an immersive VR environment. Moreover, the data-driven approach would streamline workflows by eliminating the need for artists to spend hours on manual avatar design.

This would not only enhance the quality of digital experiences but also accelerate creative processes, making NeRFs an invaluable asset for artists, developers, and users alike.

Our research on developing neural representations for facial animation directly aligns with EPSRC's interest in "Image and Vision Computing." In recent years, generative models have become the technological "holy grail" for industry and academia alike and EPSRC has been funding various research projects under this theme across the country, counting on the massive potential of AI models.

By advancing the state-of-the-art in NeRFs for facial animation, our research would expand the boundaries of 3D vision and graphics. The outcomes of this research promise enhanced visual realism and multimodal interactivity, with broad implications for the future of human-computer interaction and digital communication, fully reflecting EPSRC's commitment to advancing innovations in the field.