CAREER: Differentiable Programming for Visual Computing

Deep neural networks are a modern machine learning method that is known to produce excellent results for processing visual data such as images and 3D content. However, deep neural networks have some known limitations. They usually do not model the underlying physical process (e.g., light transport or dynamics) directly, they require significant computational resources for training and inference, and they are difficult to debug and control.

On the other hand, while classical visual computing algorithms that explicitly model the formation of visual data (e.g., how a camera captures a picture or how objects move physically) suffer less from these issues, they often do not apply as broadly as modern machine learning methods because they do not learn from a large amount of experience. This research will bridge the gap between the two approaches by creating classical visual computing algorithms that are differentiable.

That is, the functioning of these algorithms depends smoothly on a set of internal parameters that can be tuned automatically using deep learning approaches. The project will optimize these domain-specific differentiable visual computing programs using data to get the best of both worlds. Project outcomes will have broad impact in applications such as enabling self-driving cars to make better decisions, training robots to interact with the environment using physical information, creating more realistic virtual worlds, designing buildings with better lighting, designing physical objects with desired appearance and functionality, and allowing movie artists to create better film shots.

The systems developed through this research will be incorporated into new programming courses and tutorials, and the PI is committed to working with early career scholar programs to promote participation in visual computing and differentiable programming.

This project pursues a synergistic plan that includes the design of differentiable programming systems, algorithms, and applications. To these ends, it will be necessary to adapt domain-specific algorithms and compute their derivatives in correct and efficient manners, to design new visual computing algorithms that leverage both data-driven priors and domain-specific knowledge, and to parameterize the problem for optimization to avoid local minima and satisfy constraints.

Neither traditional automatic differentiation nor modern deep learning systems address these challenges. The algorithms, systems, and applications will evolve together to help each other. Concretely, this project will develop differentiable programming languages that can properly handle discontinuities, and automatically optimize code performance to efficiently process millions or billions of pixels, particles, or triangles by exploiting structured sparsity in visual computing programs.

It will also develop new domain-specific differentiable visual computing algorithms with improved efficiency and accuracy in image processing and physical simulation, by retaining the structures of classical algorithms while replacing hand-built heuristic components of the algorithm with data-driven elements.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.