CAREER: Fair assessments: Examining Cultural Familiarity to Decrease Bias in Engineering Classroom Assessments

Test scores are used to inform who gets to pursue and stay in engineering education at every juncture, pre-college through graduate school. The persistent achievement discrepancies between students of Color and White undergraduate engineering students are commonly blamed on students or their previous education rather than acknowledging that the tests themselves or testing process could be biased against test takers.

Factors such as differing cultures, educational access, and test taking strategies all contribute to the test taking experience and can result in tests that are inherently easier or harder for different groups of students. Very little is known about how biases contribute to the problem of discrepancies in achievement between students of Color and their White peers in engineering.

The research on test fairness in engineering education has been limited by the low representation of students of Color in engineering. This CAREER project will apply an innovative research approach for small sample sizes that will advance understanding about the role of cultural familiarity within the context of concept inventories. By answering the research questions, this CAREER project will build a firm foundation for a life-long leadership position in research on fair assessment practices and educational activities to train instructors in increasing the fairness of their classroom tests.

This CAREER project will inform the field of engineering education by identifying how engineering education instructors and researchers can increase the fairness of assessments. Through its integrated research and education plans, the work will improve decision-making in engineering education and achievement for students of Color. Because there are so few students of Color in engineering, traditional approaches to studying assessment bias cannot be used.

This project will use an explanatory sequential mixed-methods approach which integrates studies of item difficulty and discrimination for each group of students separately and then will conduct interviews to explain the quantitative findings and build theory. The new approaches and methods developed will lead to significant strides in solving the problem of unfair assessment in engineering education.

The research questions will investigate: To what extent do items in these three assessments demonstrate adequate functioning for students of Color? To what extent do score discrepancies remain once inadequate items are removed? What are patterns of cultural familiarity and content of problematic items and items that show acceptable functioning?

How do students of Color experience testing in concept inventories? In the education plan, findings will be applied to develop and disseminate educational materials for three primary groups of learners: (1) future engineering faculty, (2) current engineering faculty, and (3) minority- and women-in-engineering program staff. The findings will benefit society directly from the research outcomes, activities related to conducting the research in this project, and educational activities.

The first research question provides important information concerning the extent to which unfairness in assessments contributes to disparities in achievement. The second and third research questions lead to principles for fair assessments that will be widely shared with engineering instructors and future faculty (via a graduate course and partnerships with national engineering education organizations); these will improve fairness for thousands of students across the country.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.