SHF: Small: Practical Mutation Analysis for Quality Assurance of Deep Learning Systems

Deep learning in Artificial Intelligence (AI) relies on deep neural networks (DNNs) to learn complex patterns from data. As DNNs become increasingly integrated into critical applications, ensuring their accuracy and robustness is essential. While testing is a widely used quality assurance method, strong performance on test data does not necessarily indicate that a DNN is robust or generalizable.

In recent years, real-world failures--ranging from algorithmic bias to life-threatening errors--have highlighted the need for systematic test quality assessment. Mutation analysis, a technique originally developed for traditional software testing, has emerged as a promising approach for evaluating DNN test data quality. However, all forms of DNN mutation analysis remain computationally expensive, limiting their practical adoption.

This project proposes novel techniques to increase mutation testing performance through approximation techniques based on Fast Fourier Transforms (FFTs). By making DNN mutation analysis more efficient, this research will enhance the reliability of deep learning systems while significantly reducing the computational burden of quality assurance.

This project introduces several techniques for accelerating both model-level and source-level DNN mutation analysis, making it a more practical quality assurance method. Model-level mutation analysis involves creating small variations, or mutants, of a trained DNN by modifying its internal structure and observing how these changes affect its behavior.

Source-level mutation analysis, on the other hand, mutates the code or training data used to build the DNN before retraining it from scratch. Both approaches require running a large number of mutants, making them computationally intensive. To address this challenge, the project will explore Fourier analysis, a mathematical technique that transforms data into frequency components, as a way to efficiently compare model behaviors and compress mutation-related computations.

Additionally, methods such as mutant grouping--which clusters similar mutants to reduce redundant computations--and memorization--which stores and reuses previously computed results--will be adapted from traditional software testing to deep learning. For source-level mutation analysis, the project will investigate techniques based on training data selection (i.e., identifying the most informative data points), data distillation (i.e., compressing large datasets into smaller, high-quality subsets), and active learning (i.e., prioritizing the most uncertain or impactful data points for retraining).

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.