SHF: Small: Knowledge, Methodologies, and Tool-support for Combating Technical Debt in Machine Learning Systems

Machine Learning (ML) models utilize algorithms to recognize patterns, draw conclusions from existing data, and apply those to new data. A classic application is image processing, where a model may learn to recognize particular images through training with many sample images. ML systems are software systems that incorporate ML models to make decisions.

Because they treat example data as "code" and because they consist of many subsystems that support the learning, ML systems exhibit not only problems typically found in classical software but also unique problems not found in traditional software. A fundamental problem in ML systems is technical debt, where software developers trade off short-term gains in bug fixes and new software features for long-term quality.

Such debt makes changing ML systems complex and error-prone, which can negatively impact system effectiveness and, ultimately, how Artificial Intelligence (AI) is woven into the fabric of society. This project will improve ML systems' long-term reliability and evolvability, positively impacting computer vision, autonomous driving, medicine, and outliers identification.

Tools developed as a result of this project are also expected to democratize the AI workforce, as they will assist data scientists and software engineers of varying proficiency in evolving and maintaining reliable and accurate (industrial, general) ML systems. Such tools can contribute to a diverse, globally competitive STEM workforce and increase US economic competitiveness.

This project will promote software engineering concepts early in machine learning practice by augmenting and creating community college-level curricula. Dissemination will occur through publicly distributing datasets, papers, open-source software, and Open Educational Resources.

Because ML systems are predisposed to technical debt, modifying them is difficult, expensive, and potentially error- and omission-prone. Notably, there is a critical knowledge gap concerning the extent to which (i) data and multiple model interactions influence technical debt, (ii) Object-Oriented Programming (OOP) can be used to reduce technical debt by encapsulating model code representing subtle ML algorithm differences, and (iii) data can be abstracted and refactored (semantics-preserving, source-to-source program transformation) alongside code to combat technical debt in real-world ML systems.

Without adequate insight into the challenges of, methodologies for, and tools to alleviate ML-specific technical debt, ML systems may be fallible and challenging to maintain long-term. This project will contribute new methodologies and automated refactorings for ML systems that improve their long-term reliability and evolvability. First, (multiple model) code and data will be mined for (manual) refactorings alleviating (ML-specific) technical debt.

Then, methodologies and novel refactorings at both the code and data levels for automatically (i) utilizing more object-orientation in model code and (ii) simplifying (multiple) complex (interacting) models will be formulated. Finally, a novel research infrastructure that (i) automatically extracts refactoring preconditions from API documentation, (ii) evaluates refactorings, and (iii) integrates refactoring identification capabilities with popular refactoring miners will be designed.

This contribution is significant because it fills the void of techniques, methodologies, and tools for effectively developing---and evolving long-lived---trustworthy (general) ML systems.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.