Mixed-method evaluation of implementing artificial intelligence in chest diagnostics for lung disease

Background

Artificial Intelligence (AI) refers to advanced technology that can perform complex tasks linked with human intelligence. AI has been used to support radiology in several clinical settings, including lung cancer detection and diagnosis, and evidence suggests that AI can contribute to accurate diagnosis, reduce errors, and improve efficiency. However, there is limited evidence on implementation and use of AI in real-world settings, including staff experiences, patient and carer experience, effectiveness, and costs.

In June 2023, NHS England announced the Artificial Intelligence Diagnostic Fund (AIDF), which is funding 11 networks of NHS Trusts across England to implement AI for chest diagnostics in 2024. Aims and objectives

Our evaluation is the first phase of a planned two-phase evaluation. Our findings will both inform a Phase 2 evaluation and/or any future longer-term evaluations.

We will evaluate early deployment and implementation of AI for chest diagnostics as part of AIDF, to explore factors influencing implementation, and identify settings and data sources for a potential phase 2 evaluation and/or future longer-term evaluations. Our research questions are: 1. How can we best collect patient and public perceptions of using AI diagnostic tools in clinical practice?

2. How can services best measure the impact of AI deployment on patients and the clinical pathway?

3. What are the key cost components of AI tools for chest diagnostics that are necessary for an economic evaluation of the AI diagnostic tools in clinical practice? 4. How are AI tools for chest diagnostics procured, deployed and implemented at network and trust levels?

5. What are stakeholder experiences (staff and AI suppliers) of the use of AI in chest diagnostics and associated care pathways?

6. Which factors influence implementation at network and trust levels? (including contextual factors and implications for EDI) Methods

This will be a rapid, mixed-method evaluation of early deployment and implementation of AI for chest diagnostics, to be conducted over 10 months to inform a potential phase 2 evaluation and/or future longer-term evaluations. We will conduct a rapid scoping review followed by stakeholder consultation discussions (RQ1-3).

We will combine qualitative, quantitative, and health economic perspectives (RQ2-6). We will engage with network leadership of 11 networks, and we will conduct 3-4 in-depth case studies. We will use stakeholder interviews, non-participant observations of oversight meetings, and analysis of relevant planning and progress documents, to: i) analyse deployment and implementation at network and trust levels including influential factors and stakeholder experiences and perspectives on early implementation (RQ4-6), ii) identify relevant outcomes, available data, and network/trust capability to collect and analyse these data, and provide advice on effective data use (RQ2) and iii) map chest diagnostic pathways, identify key costs and available data sources and tools, and explore whether it is possible to estimate costs related to different clinical pathways, including AI and non-AI pathways (RQ3).

Patient and Public Involvement and Engagement (PPIE)

Patients and the public have been and will continue to be central to this study. Our team includes the RSET PPIE co-lead and four public members with an interest in chest diagnostics. All five attend team meetings and have supported the planning and writing of this protocol (e.g., commenting on drafts and contributing to planning discussions).

They will be involved in writing any recruitment documents and research tools. They will support our analysis (e.g., helping to interpret findings) and any outputs we produce (e.g., writing papers and presentations).

Additionally, we held a PPIE workshop for members of the public with experiences of and interest in these services. Attendees supported our proposed approach and shaped the focus of our plans. All PPIE involvement activities will be compensated in line with INVOLVE payment guidance.

The study will have an Advisory Group (see Section 11.3), which will include a range of stakeholders with relevant expertise, including patients, carers, and/or representatives of relevant charities. Timelines for delivery · February 2024: Protocol drafted, shared with peer reviewers and NIHR; developed research tools

· February 2024: Finalise protocol in light of peer review and NIHR feedback; obtain ethical and local permissions. · February-July 2024: Scoping review and stakeholder consultation workshops (workstream 1)

· March (after approvals)-August 2024: data collection, rolling analysis, integration, and formative feedback (workstream 2-5) · July-November 2024: complete project; share summative Phase 1 findings. Anticipated impact and dissemination

We will share formative lessons on factors influencing implementation and potential ways to address challenges. We will also share summative lessons on how delivery, impact, and patient and public experiences of services might be monitored and evaluated.

Dissemination methods will be discussed and agreed with stakeholders. We propose to share regular updates at national and network level established weekly meetings (e.g., the AIDF weekly network meeting and drop-in session), other meetings where staff from trusts are present, and via the NHS Futures platform. We will also share findings through academic and professional-focused journal articles and conferences.

We will produce accessible summaries of our findings, which may include slide-sets, blogs, and animations.

Through these, we anticipate addressing important gaps in the evidence base highlighted by the NICE evidence generation plan for AI in radiotherapy (published September 2023)1 and influencing how implementation and impact of AI for chest diagnostics are monitored and evaluated at national, network, and Trust/service levels. We will also help shape the approaches taken in phase 2 and/or future evaluations, which will provide further important insights on progress and impact of AI for chest diagnostics.