CAREER: Structured High-Agency Interactive Narratives for Virtual Environments

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).

Narratives are fundamental to the way we think, communicate, and learn. Virtual environments such as training simulations invite the user to play the role of one character in a narrative, while the system controls all the other non-player characters and the environment. Interactive narratives are effective tools for teaching people how to perform a task and educating people about important topics, but writing interactive narratives is challenging.

Most are manually written, ensuring a nice structure but limiting their scope because every choice must be imagined in advance. Some environments are realistic simulations that give users freedom to do a wide variety of actions, but then it is hard for the designer to guarantee the narratives have the necessary content. This project will use artificial intelligence planning algorithms to create narratives at run time in games and training simulations.

Planned narratives can have the structure of a hand-written story and the freedom of a simulation. This project will explore fast algorithms for generating narratives as well as models of what users remember and expect. Over the project duration, the research team will develop virtual environments (such as a virtual reality de-escalation training simulation for police officers) that evolves from a role-playing exercise between two people to a fully automated virtual environment where the artificial intelligence personalizes the interactive narrative for each player.

This project frames interactive narratives as an improvisational exercise between a player who controls one character and an experience manager who controls all the other elements of a virtual environment. These partners communicate their beliefs, intentions, memories, and expectations via the actions they choose to take, which is a noisy channel that requires inference for understanding.

This project operationalizes the interactive narrative creation as a Mutual Implicit Question Answering (MIQA) process. Each action taken by one participant causes their partner to implicitly ask questions about why they took that action and/or implicitly answers questions that were raised earlier. The better one partner can answer questions raised by the other, the closer they are to mutual understanding.

MIQA combines research on multi-agent artificial intelligence planning, cognitive models of memory and expectations, and procedures from automated question answering to represent both partners and how well they understand one another. Participants in an interactive virtual environment will do paired exercises where one will act as player and the other as experience manager.

During this exercise, both partners will answer questions about their actions and about their perceptions of their partner’s actions. They will also report on their perceptions of the structure of the narrative and the player agency. These exercises will begin as person-to-person exercises but will evolve into person-to-intelligent-agent exercises through data gathering and model refinement.

The research hypothesis is that the intelligent agent will be able to provide high-agency, structured interactive narratives that approach the quality of those created with a human partner. This hypothesis will be evaluated using a Turing test: can the player identify whether the interactive narration is controlled by a human or an intelligent agent.

These exercises will take place in a virtual environment called Camelot, but the research team will simultaneously implement the same procedures into an ongoing virtual reality training simulation that is being built in consultation with police officer training experts to teach best practices for de-escalating potentially dangerous situations.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.