NSF Convergence Accelerator Track H: Determining Community Needs for Accessibility Tools that Facilitate Programming Education and Workforce Readiness for Persons with Disabilities

One billion people, 15% of the world’s population, have experienced a disability. Disabilities present major barriers to entering the labor force; approximately 80% of persons with disabilities (PWDs) were excluded from the 2021 labor force. Meanwhile, programming jobs continue to grow explosively but are largely inaccessible to PWDs.

Standard programming interfaces–-screens, keyboards, mice–-are difficult to operate for many PWDs with physical challenges. This project supports individuals with physical disabilities that result in barriers to learning and engaging in programming, blocking access to the widely available, lucrative, and upwardly mobile technology workforce. The work will develop a means of ameliorating negative labor outcomes faced by PWDs by developing and evaluating prototypes of multimodal interfaces (e.g., speech, eye tracking, pedals) that enable PWDs to learn, practice, and utilize programming skills.

The project will also develop a path for PWDs to train for—and enter—the programming workforce, thereby bridging the career gap that blocks most PWDs from such career opportunities. Impacts of this project include: 1) increased representation of PWDs in STEM jobs, 2) increased economic and personal well-being for PWDs, 3) improved economic competitiveness of the U.S., and 4) enhanced infrastructure for research and education.

Enabling PWDs access to programming skills and employment will produce an influx of PWDs in the technology sector, increase diversity in STEM, and lead to sustained employment and economic well-being for PWDs. Since programming jobs can often be done remotely, this work will remove transportation barriers for PWDs, especially those who use wheelchairs.

As job stability is a primary concern for many PWDs, this will lead to sustained improvements in quality of life. Furthermore, increasing the workforce of people who can program will improve US economic competitiveness while increasing the diversity of the workforce. Finally, developing effective, user-friendly, and sustainable ways of interfacing with computers will also be useful in K-12 and in research nationwide.

It will allow for early programming education and computer use in K-12 settings for children with physical disabilities, and for aging seniors who experience loss of dexterity and fine motor control.

This project will produce multiple technical advances and contributions, including: 1) a large corpus of insights about the relevant PWD subpopulation needs for engaging in learning and practicing workforce-ready programming; 2) co-designed personalizable prototype interfaces specifically designed to be affordable, accessible, and work across platforms; 3) machine learning models that will enable personalization of the developed tools to meet individual user needs; and 4) an inclusive evaluation framework informed by the advisory board and community partners during the co-design sessions. This work combines input devices beyond keyboards and mice to create an off-the-shelf, personalizable, multimodal input interface for teaching and enabling workforce-ready programming for at least one identified subpopulation of users with similar physical abilities.

The prototypes explore combinations of input modalities processed by multimodal, personalized machine learning models to translate inputs to output keystrokes and cursor activity. Prototypes will leverage large-scale pretrained language models to guide the translation of user input to code output and will use a single input modality with a focused output space (i.e., writing Python functions, operators, and program-specific variables); multiple prototypes will be created and tested with the relevant PWD populations.

The corpus of data from each modality will be used to identify user sets with similar abilities through clustering techniques, while a centralized learning backbone can be fine-tuned per user population using low-parameter fine-tuning approaches such as Transformer-based Adapters. These technical advances have the potential to significantly expand access to the technology labor force for the PWD community, thereby profoundly impacting the technological workforce opportunities of individuals with physical sensorimotor disabilities.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.