Exploring AI-augmented mobile augmented reality for precollege culturally-relevant science learning simulations

This project aims to investigate a model for AI-supported mobile Augmented Reality (AR), by creating novel, spoken language interfaces to allow students to create and manipulate culturally relevant educational simulations that run on smartphones. The main technological challenge it is attempting to solve is improving the usability of AR simulations in classroom settings, and the main learning challenge it is addressing is the need to provide learners and teachers with learning experiences that align with Next Generation Science Standards (NGSS) for scientific modeling.

The proposed work brings together computer scientists, learning scientists, and science educators to co-design a suite of interactive AR “laboratories,” aligned with 9th grade Biology NGSS content, which learners will be able to manipulate via voice commands.

The AR laboratories will be designed to incorporate culturally sustaining pedagogy by selecting culturally relevant lessons and guiding questions, incorporating cultural cues into the software design, and making use of formative and summative assessments which allow learners to apply their knowledge to culturally meaningful contexts. A Large Language Model (LLM) will be used to convert the learner commands into system actions in two ways: constrained (by mapping to a set of predefined laboratory-specific tasks) and more open-ended (by generating custom content for the laboratory experience, like text, images, or 3D models).

Curricular materials will also be developed using a backwards-design process that creates educational experiences to suit the targeted learning goals. The research will use a mixed methods approach in two different school districts across several design iterations, culminating in an assessment that will contrast the experiences against traditional paper lessons.

The research questions will explore (1) science teachers’ perceptions of the teaching and learning benefits of AI-enhanced science software, (2) student learning outcomes and perceptions when using AI-enhanced science software, and (3) how student learning when using AI-enhanced science software compares to traditional paper lessons. These questions will be addressed via semi-structured teacher interviews to detect challenges and gauge their perceptions of how the AR experiences influenced student learning, normalized gain scores computed from pre- and post-tests to measure student learning, semi-structured student interviews to capture learner perspectives, and discourse analysis to contrast the type of small group discussions that emerge in regular lessons versus the novel AR-enhanced lessons.

This work will help explore the as-yet-underexplored question of how generative AI can be used to support STEM learning in classrooms, developing design guidelines and implications for creating such experiences. It will also provide the developed laboratories and curriculum as freeware to be used by science teachers, and the open-source infrastructure for creating AI-enhanced AR learning experiences to be used by future researchers.

This project is funded by the Research on Innovative Technologies for Enhanced Learning (RITEL) program that supports early-stage exploratory research in emerging technologies for teaching and learning.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.