Research: Faculty Assessment Mental Models in Engineering Education

The process of forming engineers is an iterative one that requires feedback to indicate developmental progress and identify areas for improvement. A primary source of feedback comes from assessment, which can play many roles in engineering education: a signal to students of what they do and do not understand about a concept; feedback to instructors about students’ conceptual understanding as well as what may or may not be working regarding their own teaching approaches; and information to administrators and prospective employers evaluating students’ abilities.

Although assessments function as a linchpin in the formation of engineers, it is unclear how faculty members - i.e., the individuals typically designing and implementing these assessments - think about this pertinent signaling mechanism. Because faculty members often have autonomy in making course decisions, understanding how they think about assessments is essential to establish the foundation on future efforts in promoting diverse and improved assessment approaches in engineering education.

To better understand how faculty think about and make decisions on assessment, we have designed a three-phase study that uses interviews, surveys, and natural language processing techniques to gather extensive data from a diverse sample of faculty who will undoubtedly have diverse views on students and assessment. The outcomes of this study will include characterizing faculty mental models of assessment and how those models inform instructional decisions.

In developing these outcomes, we will also identify potential biases, misconceptions, and problematic, systemic patterns in assessment implementation. The knowledge generated through this project will inform better faculty training and policies to advance this vital area in the formation of engineers.

To characterize faculty mental models and how they inform decision-making regarding assessment, we will engage in a three-phased, multi-method study. Drawing on exploratory interviews, Phase 1 will map the landscape of mental models that faculty members might have related to assessment. Phase 2 will draw on the initial interviews and add experience sampling methods to expand this mapping to connect those mental models with decisions made by faculty related to assessment over the course of a semester.

Phase 3 will use a survey to expand the study sample in order to support inferential statements about the population of engineering faculty members in US engineering education ecosystems more broadly with regard to mental models and decisions related to assessment. This study will make several important contributions regarding intellectual merit. First, our study will identify a range of mental models that faculty engage in assessment-related decision-making, providing a view of the current state in engineering education.

Second, building on these findings, we will illuminate the connections between these central instructional decisions and deeper perspectives that faculty members wield. Third, from a methodological perspective, we will implement ground-breaking methods that combine experience sampling methods (ESM) and natural language processing (NLP) and provide a model for other researchers in engineering education to do the same.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.