CRII: RI: RUI: Generating Haptics in Telerobotics through Perception Complementarities during Physical Distancing

Haptic feedback, the sense of touch and awareness of movement, is a fundamental sensory pathway for human everyday life and plays a particularly essential role in object manipulation. In a physically distanced or miniaturized world, telepresence using robot proxies can assist humans with remote tasks, yet an ongoing issue is the limited haptic capabilities due to the lack of high-fidelity, low-cost haptic interfaces or sensors.

As a plethora of information exists in the digital age, mostly through images or video streams, the ability to interpret the sense of touch from vision enables new opportunities. This project explores software solutions that can estimate contact force/torque from visual data and train robots to replicate force sensitive soft body manipulation tasks.

This software agent will be trained to transmit real-time force/torque estimation without the need for haptic sensors. Promising application domains include providing surgeons with vision-based haptic feedback during robot-assisted minimally invasive surgery, short term telepresence demand for emergencies or disaster response, and teleoperating a companion robot to perform a haptically enabled virtual hug or a remote handshake.

The goal is to create a novel software solution that helps humans stay connected, complete remote tasks through intelligent touch estimation, and lower the barrier to entry for haptic teleoperation by reducing accessibility and hardware requirements.

This project leverages preliminary endeavors in vision-based force estimation in robot assisted minimally invasive surgery (RMIS). Meanwhile, the technology will evaluate the accuracy of the artificial force, analyze added benefits or limitations of the artificial haptic information, explore the sim-to-real transfer learning capabilities of the proposed framework through Variational Autoencoder-Generative Adversarial Networks (VAE-GANs) and prioritize cross-robot support by ensuring software compatibility with Robot Operating System (ROS), Collaborative Robotics Toolkit (CRTK) and Asynchronous Multi-Body Framework (AMBF) for dynamic simulation and visualization.

The wide spectrum of applications ensure significant potential of this forward looking research to impact telerobotics in soft object manipulation.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.