AI-Enabled XR

Purpose and goal:

This project aims to combine the concept pf XR training simulators with large language models (LLMs) to create a more dynamic and responsive training environment. The goal is to develop a working prototype that integrates these technologies, enhancing language-based interactions in XR. By incorporating the latest advancements in multi-modal LLMs, the project seeks to move beyond basic text-based interactions and explore new, innovative applications in XR environments.

Expected results and effects:

We aim to establish a consortium of collaborators through the organisers´ network to develop XR training simulators. By project end, we expect to produce a working prototype of a basic XR training scenario with an LLM-based character for user interaction. Additionally, we will generate a detailed report outlining further development steps, key application areas, challenges of language-specific training (e.g., Swedish), and potential ethical considerations.

Approach and implementation:

This project emerged from 2023 workshops and inquiries exploring XR and AI, highlighting the need for a dynamic, responsive VR simulator enabled by LLM technology. These simulators could help individuals gain equitable education, overcome anxiety, or develop skills through continuous practice. We will involve representatives from our established EU and US networks, ensuring the expertise needed to develop the project and providing a strong foundation for future cross-country collaborations.