CAREER: Developing New Scientific Instruments for Classroom Observation: A Multi-modal Machine Learning Approach

This project will harness artificial intelligence (AI) to improve both the quality of classroom teaching and the precision of educational research by providing teachers and scientists with new methods of observing the inter-personal dynamics between teachers, students, and their peers. Decades of research have demonstrated that the quality and quantity of interactions between teachers and students can have a major impact on student engagement, attitudes toward learning, and downstream academic and socio-emotional outcomes.

Despite the progress that has been made in studying classroom interactions and their impact on students' learning, as well as in developing effective interventions to help teachers teach better, the status quo of educational measurement is currently a significant roadblock to further progress in both educational research and teacher training. Specific problems with contemporary methods include ignoring the possibly different classroom experiences of individual students and minority subgroups, and providing only limited actionable feedback for teachers.

This project will depart from standard observation protocols, which typically describe the "average" classroom experience of the "average" learner, and instead focus on characterizing over time the fine-grained experiences of every student in the classroom. The scientific instruments developed during this project will also be used to help teachers to identify potential biases when interacting with particular students in their classes.

To achieve these goals, the team will make advances in multi-modal (vision, speech, natural language) machine learning to devise new architectures that analyze videos of school classrooms and perceive fine-grained interactions. The envisioned AI systems will (1) identify who is interacting with whom, when, and how in a school classroom; (2) find the key events during a teaching session that are most important for teacher feedback; and (3) summarize interactions for each student along different dimensions to find students who need more attention and to uncover possible bias.

Based on these new analyses, the team will develop (4) predictive models to estimate socioemotional and academic outcomes outcomes. Finally, the team will (5) devise new teacher training experiences that help teachers to perceive classroom dynamics more accurately. The project will result in contributions to the fields of computer vision, speech analysis, machine learning, and education, and will offer new insights into automatic speaker diarization, person tracking, sentiment analysis, and classroom observation analysis.

The scientific and educational agendas provide opportunities for inter-disciplinary training of research assistants; they will also enable and benefit from collaboration between the research team and teachers in both Massachusetts and Virginia.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.