I-Corps: An offline surgical telementoring platform that guides the surgeon via an augmented reality headset

The broader impact of this I-Corps project is the development of a software and hardware platform that aims to address the gap in lack of surgical care expertise in rural and austere environments. Usually, most surgeries are delegated to large urban medical centers. When evacuation is not possible and immediate treatment is imperative, surgeons and medics rely on written guides or video teleconferencing seeking for assistance.

Lack of expert surgeons and a stable internet connection make the current mentoring practices ineffective. The platform is lightweight, portable, interactive, and standalone, enabling effective health care in rural areas. The proposed technology also may be helpful in training surgeons in using new tools, techniques, and surgical procedures without expensive and bulky simulators.

This I-Corps project is based on the development of software applications and hardware for telementoring with augmented reality and artificial intelligence (AI). The proposed infrastructure includes a hardware platform that relies on cameras, see-through displays and depth sensors to increase the quality of collaboration between mentor and trainee. The platform is designed to enhance the mentor and trainee sense of co-presence through an augmented visual channel, which has been shown to lead to measurable improvements in the trainee’s surgical performance.

The goal is to develop a technology to complement or substitute the expert mentors when needed. The platform uses a suite of applications and hardware that includes an AI module that allows telementoring of generalist surgeons to perform surgeries through an augmented reality headset. The AI agent monitors the surgery state and predicts the next instruction, which is conveyed verbally to the mentee.

The system is a portable, standalone design that may be used in rural and austere environments, where there is no expert supervision and a lack of a stable internet connection.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.