SHF/FET: 2.5D/3D Heterogeneous Integration of In-Pixel and Near-Pixel Compute Chiplets

Personal computers and mobile phones have been the primary platforms for interaction with the digital world, pervading every corner of the modern society in the past few decades. Augmented Reality/Virtual Reality (AR/VR) is regarded as the general human-oriented computing platform for the next decade. In the smart headset, eye tracking provides crucial information on human perception, and it is a key pillar that supports many AR/VR applications such as displays, user interfaces, and foveated rendering.

Hence, real-time processing of eye tracking workloads is critical, but modern eye tracking algorithms are much slower than the desired specifications (over an order of magnitude slower). Eye movements including saccade dynamics are one of the fastest movements in human body. Therefore, to accurately track eye movements, we need to perform image segmentation tasks at a very high frame rate (>1000 frames per second, FPS).

However, this involves a large amount of data movement between the image sensor and the processor that performs the segmentation tasks. In the current practice, integration of frontend complementary-metal-oxide-semiconductor (CMOS) image sensor and backend microprocessor is slow and inefficient as they are generally placed in separate packages. To efficiently perform high frame rate processing, in-pixel and near-pixel compute paradigms have been proposed to reduce the latency and energy consumption from the costly data movements to provide >100× reduction in communication latency by placing compute close to the source of data.

The outcome of this award will have synergies with other national efforts to revamp domestic semiconductor research and development under the CHIPS and Science Act. Besides the AR/VR applications, tracking fast eye movements could be useful in biomedical applications such as early diagnosis of Alzheimer’s and Parkinson’s diseases. The objective of the research and education integration is to train the next generation of workforce with domain expertise and interdisciplinary skills in the broad area of semiconductor device, integrated circuit and advanced packaging.

This award aims to advance the software-hardware co-design for in-pixel and near-pixel compute for eye tracking in AR/VR applications. A multi-mode image sensor that supports eye tracking is proposed, featuring a successive frame differencing method for the event map generation with in-pixel compute chiplet. It will be capable of providing both an event map at a higher frame rate as well as a full resolution image at a lower frame rate.

Moreover, the near-pixel compute chiplet will run eye tracking inference with dual-mode deep neural networks for both high accuracy and high frame rate. The proposed research activities also include exploring the fundamental device technologies such as characterizing the amorphous oxide semiconductor transistor for pixel read-out circuitry and exploring the advanced packaging techniques for system-level heterogeneous integration.

Silicon tape-outs are planned to validate the in-pixel and near-pixel compute chiplets with a pathway for heterogeneous integration on a silicon interposer.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.