CAREER: Enhanced Reliability and Efficiency of Software Regression Testing in the Presence of Flaky Tests

Software is usually developed in a continuous development and integration process that incorporates incremental changes leading to successive releases of the software, where each release undergoes rigorous software testing to check whether recent code changes had broken existing functionalities. This process, known as regression testing, is widely used in software development practice.

A major problem in the generation of test cases is the presence of flaky tests: tests that non-deterministically pass or fail on the same version of the code. Failures from flaky tests can mislead developers about their recent changes, waste developers’ time, and reduce developers’ trust in software testing. Many software development organizations have reported that flaky tests are one of their biggest problems, because they confound assurance goals.

This project aims to improve the reliability and efficiency of regression testing in the presence of flaky tests. It will produce tools that aim to be efficient and effective at resolving the inherent nondeterminism. The work focuses on (1) reducing the cost of flaky-test detection and debugging techniques by predicting important test properties, (2) developing new techniques to predict flakiness-related properties, (3) speeding up and reducing the resources needed by regression testing, (4) developing new techniques to systematically detect flaky tests, and (5) reducing the flakiness in Android user interface testing.

The project will also produce curriculum for education and training on the topic of programming in the face of nondeterminism, and will work with industry to transfer technology.

The research on flakiness will move from the typical, black-box approaches to a new level for detecting, debugging, and fixing through novel, white-box and learning-based approaches. The approach will use static and dynamic analyses to compute state pollution, which may affect test flakiness based on the test execution order. The work involves combinatorial design theory to improve the efficiency of order-dependent test detection.

Special attention will be paid to flaky tests in graphical user interfaces using record-and-replay and test input generation. Test coverage computations, which can be used to predict whether a code change will affect the test's output, will use a machine learning approach. The work will result in tool implementations and large-scale evaluations in open source and proprietary environments.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.