CAREER: Foundations of IoT-Supported Mobile Augmented Reality

Mobile augmented reality, which integrates computer-generated virtual objects with real-world environments around a user in real time, has the potential to bring to the users, vibrant and immersive representations of important information at the right time and the right place within the physical world. Yet existing mobile augmented reality systems often make mistakes in identifying the objects present in the real world, thus generating incorrect virtual objects for the users.

They also make mistakes in placing virtual objects within the real world, making them appear as shifting, oscillating, or jumping around. This project will address such environmental awareness limitations by taking advantage of Internet of Things (IoT) sensors and controllers located in the vicinity of mobile augmented reality devices. Many IoT devices, such as smart lights, cameras, and displays, are already pervasive in smart homes, offices, and other environments.

This project will improve mobile augmented reality environmental awareness by using these IoT devices as sources of additional information about the real world, and as means for partially controlling the state of the world. The project will make mobile augmented reality more reliable, paving way for its applications in health-critical and safety-critical domains, such as medicine and transportation.

The project will develop multi-disciplinary research skills in a diverse cohort of graduate, undergraduate, and high school students. The findings of this project will be showcased as interactive demonstrations at K-12-oriented events, with the particular aim of attracting women and underrepresented minorities to science and engineering.

This project will lay the foundations for studies of IoT-supported mobile augmented reality by developing techniques for designing, implementing, and evaluating it, and by demonstrating IoT-supported methods that improve the performance of core mobile augmented reality algorithms. The goals of this research are divided into three thrusts. The first thrust will develop an IoT-supported mobile augmented reality platform, a game engine-based IoT-supported augmented reality emulator, and new IoT-supported mobile augmented reality applications.

The second thrust will examine and quantify the failures in mobile augmented reality semantic awareness, and will develop algorithms that address these failures by making use of the IoT camera-provided availability of multiple views of the same physical space and IoT-based smart light and shade controls. The third thrust will examine and quantify failures in mobile augmented reality spatial awareness, develop machine learning-based techniques to predict whether the failures are likely to occur in a given environment, and demonstrate algorithms that improve spatial awareness by controlling the state of smart lights, shades, and displays available in the environment.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.