Doctoral Dissertation Research in DRMS: Connecting Artificial Intelligence Literacy and Human-AI Decision Making Outcomes in Organizational Hiring

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).

Organizations are increasingly incorporating artificial intelligence (AI)-based technologies into decision-making processes. For example, hiring teams may use AI-based tools that analyze application data, such as resumes and interview recordings, to provide hiring recommendations. AI-based algorithms can process large amounts of data and generate recommendations that inform decisions that used to be made solely by human decision-makers.

Despite the widespread use of AI-based technologies, everyday non-expert users of these technologies may not have sufficient knowledge about AI, or AI literacy, to make fair decisions using AI-based recommendations. This Doctoral Dissertation Research Improvement Grant (DDRIG) examines whether receiving recommendations from an AI-based source impacts outcomes when decision-makers vary in their AI literacy.

This project has two primary purposes: 1) to develop a measure for people's AI literacy within the context of decision-making, and 2) to test this measure in a hiring scenario experiment. This research supports NSF’s mission to promote the progress of science by contributing tools that support further research on human-AI interaction, a subject becoming more relevant as organizations continue to introduce new AI-based technologies into the workplace.

This work helps people, regardless of technical background, become better informed and thoughtful human-AI decision-makers. The findings from this research advance a validated measure for AI literacy that researchers, individuals, and organizations can use to measure individuals' general understanding of AI-based technologies and identify potential gaps in knowledge that can impact how they use AI-based information to make decisions.

This research also informs the development of educational resources that help job seekers navigate the AI-based hiring process.

The research entails two phases. Phase I involves a scale development effort that uses past research and pilot interview data to develop and test scale items for an AI literacy measure. To determine whether people's AI literacy plays a significant role in how people use AI-based information to make hiring recommendations, phase II involves a hiring scenario experiment.

In the experiment, participants evaluate a mock job application and use input from a secondary evaluation to decide whether the applicant should move forward in the hiring process, and participants' AI literacy (as measured by the scale developed in phase I) is used as a control variable. The results from this investigation may help various stakeholders better understand how people's understanding of AI influences decision-making outcomes.

The findings have the potential to contribute to work on AI training and education at large. Furthermore, the experiment is the first application of the AI literacy measure that catalyzes future research exploring the relationship between AI literacy and human-AI decision-making.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.