CNS Core: Small: A Split Software Architecture for Enabling High-Quality Mixed Reality on Commodity Mobile Devices

By blending the physical and digital worlds into a programmed experience, Mixed Reality (MR) allows users to visualize and interact with digital information such as 3D overlays and real-time data and has important applications in many societal domains including education, remote working, military training, and health care such as tele-medicine. Despite the tremendous potential of the MR technology, the MR solutions available in today’s market are either enterprise-grade which are costly or consumer-grade which can only support low-quality MR content which leads to poor user experience.

The high cost and/or low-quality of current enterprise-grade and consumer-grade MR solutions lead to a fundamental “content-adoption” dilemma faced by the MR industry: the lack of MR content has limited the market penetration of custom-made MR headsets, and the low market penetration of MR headsets in turn has hindered the development of MR content.

This NSF CSR project proposal will develop key technologies to enable high-quality MR on commodity mobile devices like smartphones, etc.., viewed by a simple see-through head-mount devices (HMD) with a high-resolution camera for input and a projector for output such as Nreal Light glasses. Such technologies will transform millions of smartphones (equipped with the above inexpensive HMDs) into ubiquitous MR devices and in doing so help the MR industry to overcome the “content-adoption” dilemma and pave the way for wide adoption of the MR technology and its many important applications.

This project aims to create the first split software architecture that enables high-quality MR applications to run on commodity mobile devices; the capability to jointly optimize offloading multiple Deep Neural Network (DNN)-based tasks constituting a complex, resource-intensive application such as MR over the bandwidth-limited and time-varying wireless network; the capability to jointly schedule multiple DNN-based tasks of resource-intensive applications such as MR to efficiently share all local resources such as the CPU, GPU, and other processors such as NPU on emerging mobile devices; and the capability to support high-quality multi-player MR on commodity mobile devices by scaling the split software architecture across multiple mobile devices to efficiently share the limited global resources such as the wireless network and the edge cloud.

The proposed research will have lasting impact on knowledge discovery, the computer industry, and the society. Technically, this work anticipates having far-reaching impacts outside the area of supporting AR/VR/MR on commodity smartphones by developing general edge-assisted software architectures for enabling the class of latency-sensitive 5G/6G applications on current and future mobile computing platforms such as smart glasses.

Developing the proposed technologies for MR have the potential to fundamentally overcome the “deployment-content” dilemma faced by the industry as well as fostering the proliferation and wide adoption of MR technologies and its many societal applications. The importance of this work will be further heightened by making smartphones an important enabler of accessing information and new technologies like AR/VR/MR for people in both developed and developing countries and hence being an important tool in overcoming the “digital divide".

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.