CAREER: On the Fairness of Light Transport for Unbiased Low-level Vision

Does the physics of light disadvantage certain skin types? The visual appearance of human skin is remarkably diverse. Anatomical variations in not just shades of lightness, but also oiliness, texture, and thickness all influence how images will appear.

Such anatomical variations are linked to demographics and may explain why artificial intelligence (AI) pipelines disadvantage certain skin types (e.g., darker skin, older skin, female vs male skin). Existing efforts in mitigating bias focus on biases at higher-layers of the AI stack. In contrast, this research award studies bias from the bottom-up by probing how light interacts with skin to form images.

In doing so, it is possible to not only identify, but also correct for physics-based bias in imaging. Today, physics-based bias leads to performance gaps in everyday imaging systems, such as facial identification or medical imaging. If successful, the research award can make these systems fairer.

The project integrates the research with education and outreaches to middle and high school students.

This research award rests on three pillars. The first pillar seeks to qualitatively and mathematically identify how images vary based on skin variations. The investigative team will focus on variations in skin tone, thickness, wrinkles, and oiliness.

The demographic link to these variations will also be studied in collaboration with physicians enabling computer graphics renderings to be cross-checked with real, human subject data. The second pillar aims to assess how these variations in image appearance subsequently affect downstream image processing and AI pipelines. The investigative team intends to study, for instance, how photometric stereo- a widely used technique to obtain 3D shape - can trace its source of performance bias to variations in skin.

The third pillar synthesizes these insights to create novel computational imaging systems that resist bias due to skin variations. It is quite possible that in seeking to design a fair imaging system, the overall performance may be affected. In the event there is such a Pareto tradeoff between fairness and performance, the investigative team will design a flexible system that can sample multiple points on the Pareto curve.

The overall choice - of which tradeoff point is optimal - lies with societal and community goals. In summary, this award hopes to set a unique foundation for analyzing bias in computational imaging.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.