Research: Defining and Assessing Engineering Judgment

Judgment is critical to the field of engineering, including design and data analysis. The Accreditation Board for Engineering and Technology (ABET) calls for an undergraduate engineering student to be able to apply engineering judgment upon graduation. However, when asked what is engineering judgment, engineers, including faculty, are often not able to readily respond.

Thus, although judgment is a common term, it tends to be an elusive notion. Consequently, the assessment and measurement of engineering judgment to ensure its formation is currently limited due to lack of a holistic, agreed-upon, concrete definition for it. In addition to not assessing judgment specifically, educators generally do not provide explicit instruction on judgment as part of the formation of the engineering student.

Given this absence of direct instruction and assessment, research is needed to create educational materials and tools to support the formation of all engineering students. This research will develop these materials and tools based on an expert, consensus-driven definition for engineering judgment, which will be generated as a first step in the research.

This study will significantly impact the engineering field – engineering curriculums, students, educators, ABET accreditation, and industry stakeholders, ultimately strengthening society through well-engineered products, systems, and services. Purposefully educating and assessing aspiring engineers relative to judgment is the major impact of this work, as this has not occurred to any widespread nor consistent degree to this point.

This exploratory research will first determine an expert-driven definition for engineering judgment using a consensus-based Delphi panel study. The expert panel will be comprised of a diverse group of professionals and include faculty who have investigated and published on various aspects of engineering judgment, ABET program evaluators, and instructors of design and data analysis courses across multiple engineering disciplines.

The first research question is as follows: What is a holistic, expert-driven definition for engineering judgment? The agreed-upon definition will be used to develop assessment tools for judgment and test and apply them in the context of three courses covering competencies highly relevant to engineering judgment – design and data analysis. Written reflection by students will be investigated as a method to assess judgment, with reflection questions to be posed after team engineering projects.

Using a coding scheme informed by the Delphi study, a qualitative content analysis of the reflections will be conducted to extract evidence of judgment. Evidence of judgment will also be extracted from student interviews, surveys, and project deliverables using the coding scheme and a related rubric (also to be developed), allowing validation of results from various sources.

Knowledge from the Delphi study and assessment process will directly lead to development of educational modules for explicit instruction on engineering judgment for use by educators and students. By considering demographic variables such as gender and ethnicity as part of the assessment process, the development of educational modules applicable to all students will be promoted.

The assessment data will further address the following questions: 1) What breadth of engineering judgment competencies do students exhibit in their reflections, interviews, survey responses, and projects, and 2) What are students’ understandings of engineering judgment, and how can we support them in developing this ABET outcome, including consideration of potential differences by demographic variables? To promote adoption of the educational modules and assessment tools and disseminate the findings widely, a professional website will be employed, and we will have a presence at national conferences and collaborating engineering schools to create awareness through workshops and presentations.

This work will start the needed conversation across engineering about the elusive (and critical) notion of engineering judgment in pursuit of assessment and instructional resources to support the formation of today’s engineer.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.