CAREER: Digital Twins of Surgical Environments for Situational Awareness and Immersive Simulation

Developing new tools and procedures for robot-assisted surgery is difficult and expensive. This is because of the need for special-purpose tools and environments for building and testing prototype surgical robots, even before clinical testing can begin. This project seeks to address these problems through developing “digital twins” of image-guided surgery.

The high level idea is to develop virtual reality simulations and computational models that closely mimic real patients, surgical tools, and operating environments. These simulations will help designers correct possible problems with robot and procedure designs early on; this in turn will allow faster, cheaper design cycles that will make robot-assisted surgery design more practical and widely available.

The simulations can also be used to support real operations; room-scale models of the operating environment will support algorithms to monitor the robotic systems for safer surgeries, as well as provide new opportunities for educating both surgeons and the general public about ways technology is used to advance surgery.

The technical aims of this project are divided into two thrusts: In the first thrust, the team will develop innovative methods that enable the creation of virtual immersive and interactive simulation environments that precisely recreate scenarios of the real world. To build these so-called digital twins, the team will create computer vision and machine learning techniques to create, update, and leverage digital twins of the surgical environment from multi-modal measurements.

The second thrust will use digital twin models of unstructured, human-in-the-loop environments to explore how these immersive simulation models may be used to accelerate the design of intelligent surgical systems and provide complementary situational awareness to enhance decision making.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.