Exploring Data by Voice: Making Data Exploration Accessible for Blind and Low-Vision Learners Using AI

Data science has become essential in modern society, with growing career opportunities and widespread adoption in educational curricula. However, blind and low-vision (BLV) students are significantly underserved in this field, often lacking the tools necessary for meaningful engagement with data. This three-year project, a collaboration between the Concord Consortium and Perkins School for the Blind, addresses the critical need for accessible data science tools in K-12 education.

Leveraging a cutting-edge Large Language Model (LLM) from generative AI technologies, and partners’ expertise in educational technology and BLV learning innovations, the project team will create a multimodal data exploration environment. By enabling BLV students to interact with data through voice commands, sonification, and AI-generated audible descriptions, researchers aim to transform the educational experience and broaden participation in STEM.

The project team will research and develop an AI-powered agent embedded in the NSF funded Common Online Data Analysis Platform (CODAP), a free, open source, data analysis application designed to engage students in data exploration. The AI-powered agent will provide the interface between the user, the generative AI model, and CODAP. It will interpret BLV users’ voice commands to perform data transformations, generate data representations, facilitate non-sequential navigation and exploration of data representations, and provide verbal and sonified descriptions of data representations.

The project will employ an iterative development process that includes co-design sessions with BLV users and testing with experienced accessibility researchers, and will investigate two research questions: (1) In what ways can generative AI-based technologies be leveraged to facilitate accessible interaction with data for BLV users? (2) What effect does the availability of interactive and generative technologies have on BLV students’ ability to engage with and make meaning of datasets? The research team will develop automated tests measuring LLM responses for faithfulness, answer relevance, and context relevance.

User interaction with the AI-powered agent will be logged. Student screencast recordings, and transcripts of prototype testing by the accessibility experts and students will be triangulated with the logged data and analyzed using both deductive and inductive codes. The output of the project includes the web-based, AI-powered agent embedded in CODAP.

Source code and LLM training materials including prompts, retrieval data and fine-tuning data, will be made publicly available in GitHub repositories. Research findings and products will be disseminated at conferences and in journals on accessibility, AI and learning sciences research.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.