EAGER: TaskDCL: Adapting Mixed Reality Training Programs to Real-World Scenes to enhance Human-AI Teaming in Emergency Responses

This EArly-concept Grant for Exploratory Research (EAGER) project funds research that intends to speed up the development of mixed reality and artificial intelligence technologies to help first responders, aiming to reduce training-related risks and casualties. The research team will work with the Fairfax Fire and Rescue Department to explore how AI can be used in mixed reality tools to improve training and effectiveness for first responders.

By adding virtual elements like fires, hazards, firefighters, robots, and people in need of rescue to real-life scenes, these mixed reality scenarios help first responders practice handling real-world challenges through interactive training. The project will also involve a postdoctoral researcher and undergraduate students, including those from underrepresented groups in science and technology fields. The team will share their findings at conferences focused on mixed reality and training.

This EAGER project offers a novel interdisciplinary research perspective by integrating concepts and techniques from mixed reality, artificial intelligence, human-computer interaction, and movement science to advance first responder training. The goal is to devise a novel optimization-based generative framework for adapting mixed reality training scenarios to real scenes, which will offer ample training opportunities for first responders to practice, accomplishing different first-responder tasks (e.g., firefighting, search and rescue) via human-artificial intelligence collaboration enabled by mixed reality headsets.

To carry out the research, the team will first investigate how artificial intelligence techniques could be integrated with mixed reality devices to provide first response assistance. The team will then devise a generative framework based on optimization techniques for adapting mixed reality training scenarios to real scenes. The team will conduct user studies to evaluate the performance gain brought about by the advanced mixed reality interfaces and the synthesized training scenarios.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.