Emote VR Voicer

AI systems are increasingly able to detect a speaker's emotions, leading to a new affective channel that can be explored in art. The controls available in standard Virtual Reality (VR) can be supplemented with speech recognition, natural language processing, and sentiment analysis. We embody this potential in the frontend of a new audiovisual interface, which would translate detected meanings of utterances to the morphing of abstract 3D animated shapes, enabling a radical new aesthetic experience.

VR provides a relatively new medium in artistic practice, with under-explored input possibilities such as voice and body movement. As a controller, the voice increases access, allowing people with mobility problems to interact with the software. Our voice is also deeply connected to our general sense of wellbeing. People may be shy at first to use their voice (Harma 2011), but once they get into the flow, using the voice creatively can be remarkably freeing (Clift, 2012).

Live audio analysis provides a potent control mechanism to drive graphical generation, enabling new feedback networks between output and voice input. This was previously demonstrated in the AudioVirtualizer, a VR interface that generates aesthetically sophisticated visuals in response to a selection of musical audio files or live microphone input (van 't Klooster and Collins, 2021).

Evaluation of this interface on members of the general public and improvisers showed that users were most intrigued and encouraged by voice interaction with mapped visuals. This was further explored in the VR art game VRoar (2023) where interaction takes place through voice and controller input. The proposed project builds on these prototypes by making a new intelligently responsive VR app incorporating speech recognition and meaning classification.

Such AI facilities are already available in the public domain through code libraries but haven't yet been integrated in artistic projects. Such an interface would be widely accessible and not need prior instruction. This project allows us to answer the following research questions:

How can we develop a realtime VR app that understands the meaning of the spoken/sung word and maps semantically to aesthetically rewarding graphics?

Which mappings and designs work best for an interactive VR app that aims to encourage people to play with their voice, and increase wellbeing? The benefits would be:

- Engaging a wider range of users by stimulating people to create interesting animations in an immersive VR environment. The app could be used at home for fun or together with a voice specialist to improve vocal performance.

- To work with live video projection for musicians, poets or theatre makers in a performance context or as an art installation (with some modifications).

- The final app aims to improve the emotional wellbeing of the user by providing a creative and fun experience exploring the voice, or physical and emotional wellbeing by helping voice recovery. - Making new contributions to the field, and advance careers of the team members involved in the project.