Research Initiation: Exploring Metacognitive Strategies with External Resources for Engineering Courses in the Age of Generative AI

The internet provides vast resources to aid undergraduate engineering students in their studies. In addition to the course-supplied materials, which often go underutilized, students tend to use three major types of external resources: (1) solution sets (whether provided by the publisher, instructor, or homework support service); (2) supplemental videos; and now (3) generative artificial intelligence (AI) tools.

AI tools are quickly growing in popularity and possess unparalleled capabilities compared to what has been seen in the past from educational products. Still, they come with significant shortcomings that demand new skills and approaches to be optimally employed. For example, although modern chatbots exhibit an impressive ability to solve a wide range of problems across standardized tests like the LSAT and SAT and converse about advanced topics at just about any level, they exhibit undesirable behavior that can have negative impacts on the learning process.

In particular, ChatGPT and similar tools have been described as having the capacity to lie; the phenomenon of a large language model “lying” by producing plausible but incorrect information is often called a hallucination. Considering the often authoritative and human-like text that chatbots produce, students must be able to identify accurate and useful information independently within the outputs.

Although a considerable amount of work has been done to understand what students are using generative AI tools for, we have much less information on how they interact with them – in other words, how students prompt tools like ChatGPT – and how they evaluate the outputs the tools provide. This study will focus on how undergraduate engineering students utilize external resources and analyze the strategies employed to achieve the maximum benefit of generative AI tools.

The study will help educators better teach students how to interact with generative AI tools to improve their learning. Additionally, this study will serve a training opportunity for the PI to develop educational research tools under the mentorship of the Co-PI.

With the advent of generative AI, there is considerable excitement about the progress toward the grand challenge of personalized learning for all students. However, there is still a gap in the literature regarding how students engage with ChatGPT and similar tools to supplement their learning, especially from the perspective of metacognition. Thus, we will address the following research questions: RQ1a) How do engineering students use external resources for problem-solving assistance in their coursework?

RQ1b) Why do engineering students engage with specific external resources over others, such as ChatGPT, with respect to the resource’s perceived ease of use and usefulness? RQ2) What metacognitive strategies do students employ when collaboratively problem-solving with external resources, especially generative AI? Additionally, this study will serve a training opportunity for the PI to develop educational research tools under the mentorship of the Co-PI.

The project will be an explanatory sequential mixed methods design consisting of a quantitative phase followed by a qualitative phase. The first phase, a survey grounded in the principles of metacognition and the Technology Acceptance Model will be administered at the University of Cincinnati to understand student adoption and continued use of external resources to complete coursework.

Students will then be sampled in a stratified fashion to contextualize the survey findings with think-aloud interviews concerning their practices when engaging with these outside resources, which addresses the third research question. In the think-aloud interviews, students will be tasked with a novel design challenge that they will be instructed to solve using ChatGPT as a co-designer.

By directly investigating how students interact with generative AI tools while problem-solving, the project will provide insights to educators on designing technologically supported learning experiences that cater to diverse student groups.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.