Mid-Air Touch Technology for Accelerated Digital Human Integration and Enhanced Multisensory Experiences

In this FLF proposal, we expand the advances of mid-air haptic technology through the lens of multisensory experiences by applying the principles of human-computer interaction (HCI). Traditional user interfaces typically employ physical touch, e.g., pressing buttons and taping on touchscreens. Mid-air interaction technology replaces "taping on screens" with hand-gesture recognition algorithms and holographic touch.

Cameras track your hands to activate actions on a computer system from a distance while phased ultrasonic arrays deliver tactile sensations to your palms and fingertips in mid-air. This unique combination enables novel interaction paradigms previously only seen in sci-fi movies (e.g., Minority Report). We can now interact with computers, digital objects and with each other in immersive 3D environments that we can not only see and hear, but can also touch and feel, without the need to wear or hold any additional controllers.

To date, innovation in mid-air technologies has been focussed on hardware and software development to advance engineering methods related to accuracy, recognition and rendering. However, very little is known about how these technologies influence human behaviour and therefore it is unknown how to exploit human perception to improve interactivity and help society.

This fellowship is at the forefront of enabling the evolution of those novel and innovative interaction paradigms through the specific lens of multisensory experiences not just taking a leap from physical to mid-air touch interactions, but also integrating it with other sensory modalities (vision, audio, smell). Through studying the cross-sensory associations and integration of mid-air touch into multisensory experiences, this fellowship aspires to create more emotionally engaging and compelling digital experiences where the human senses are as important as in the real world.

Then, taking advantage of the effect of multisensory stimuli on human decision-making, we take a unique perspective on understanding the impact of such novel mid-air interfaces on societal responsibility. In particular, we focus on improving the users' sense of agency (SoA) when interacting with autonomous systems. Today, intelligent systems involve increased automation and make many decisions for us (cars, smart homes, robots), which tend to reduce the SoA i.e., the feeling of being in control of one's actions.

By understanding the effects of mid-air interactions enhanced by multisensory experiences on the SoA, we aim to design and develop more responsive systems that provide the user with a feeling of being in control and strengthen responsible interactions.

As a result of this project, we ambition to help create a future where the holographic display in autonomous vehicles can be touched and felt, and that this tactile dimension generated by focused ultrasound patterns is designed to promote a sense of trust towards the autonomous system, and a future where we can have a 3D video call with friends and family whilst being able to communicate our affection by sending tactile sensations generated by a beam of focused ultrasound and even perceive smells from their environment.

This FLF will challenge existing interaction paradigms in HCI through a systematic understanding of the role of mid-air haptics from a multisensory perspective, aiming to design digital interactions where you can see, hear, touch and smell, and this multisensory experience helps to share agency between users and technology. Our approach will break from the conventional studies in mid-air technology by fostering a new and inclusive ecosystem of multidisciplinary research around mid-air technology involving psychology, neuroscience and HCI so that the impact of these technology is not limited to hardware and software, but also provides a positive impact on society with respect an accelerated digitisation of human experiences.