CAREER: Belief Formation in Verification Strategies and the Evaluation of Verification Evidence

In systems engineering, verification is the process of building confidence in the proper functioning of a system. Although it is a key activity in engineering, the fundamentals of how engineers utilize objective data to inform inherently subjective judgments about the system during verification is not well understood. This Faculty Early Career Development Program (CAREER) award supports fundamental research into this process and education efforts geared towards improving understanding about and skills for verification activities.

The research focus is on modeling the information structures by which engineers build confidence in a system's performance. This knowledge can lead to improved verification strategies and more rigorous ways to evaluate verification data. Hence, the research activities can bring about a potentially transformative paradigm for the practice of systems engineering.

Improved verification strategies benefit society by enabling efficient system development with a more robust verification coverage, which may lead to increased safety and efficacy of commercial products and public services and greater overall economic competitiveness. The innovative education paradigm introduced in this project embeds verification into regular engineering assignments to prepare students to exercise sound judgement and be effective decision makers in engineering.

Middle school students are engaged with a computer game that teaches conceptual diversity beyond calculations. By realizing that no single answer exists for an engineering problem, students embrace the potential diversity in engineering solutions and focus on understanding how prior knowledge, mental models, preferences, and team member attitudes shape the meaningfulness of their solutions.

The research objective of this CAREER project is to study the formation of engineers' beliefs during systems engineering verification activities and to identify patterns and heuristics that better enable engineers to capture their beliefs in information artifacts. Explanatory models of how engineers' beliefs form will be established by combining Bayesian analyses, traditional and dynamic belief graphs, and decision theory with empirical observations of professional engineers and information artifacts (e.g., verification plans and reports).

Cognitive mapping will be used to elicit belief structures, Bayesian analysis will be employed to model engineers' beliefs, and Credal sets will be used to account for differing engineers' beliefs. The educational objective of this project is to help students recognize and embrace the subjective (yet non-arbitrary) nature of correctness in engineering problems.

Novel instructional approaches to two key populations for engineering will be implemented and assessed. A digital game to model real-world contexts as math word problems will be used to seek and capture the conceptual diversity that emerges when middle school students cooperate while solving problems. Undergraduate students will be exposed to open-ended problems that foster a solution-verification iterative approach to engineering in several courses from their freshmen to senior years.

Overall, this project will contribute to determining the effects of engineers' beliefs and belief structures on verification strategies and the evaluation of verification evidence.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.