PREDICT: Pragmatic Recalibration and Evaluation of Drift In Clinical Tools

Research question Can existing techniques be used to detect and repair temporal drift in clinical prediction models, to minimise risk of patient harm?

Background There is a push for the increased use of prediction models to be used throughout the NHS to improve patient outcomes, support clinical decision making, and improve service efficiency. However, current regulations do not enforce any testing after model deployment.

This can result in temporal drift – the phenomenon where model errors increase over time due to changes in population demographics, clinical coding systems, and other circumstances. Temporal drift will be increasingly problematic as more prediction models are deployed within the NHS. There is potential to cause patient harm via inappropriate decision-making if this is not addressed.

For example, QRISK2 was developed using READ codes, and was therefore impacted by the shift from READ to SNOMED-CT within primary care software systems. This led to a discussion around the potential removal of QRISK2 from the GP software systems SystmOne and EMIS.

Both software providers and the MHRA have expressed their strong support for the PREDICT project to investigate temporal drift before it becomes a major problem.

Aim Compare different approaches for the detection and repair of temporal drift within healthcare prediction models and open a dialogue with policy makers on how best to design guidance to mitigate risks.

Methods PREDICT consists of three inter-linked work-packages: WP1: Test different drift detection/update strategies using simulated data. Build a unified framework to compare strategies (statistical process control, Bayesian models, regular sampling).

Experiment with the speed at which temporal drift of various magnitudes can be detected and potentially repaired, when we can control the true amount of drift. WP2: Real-world clinical decision support.

Apply the WP1 framework to existing clinical decision support tools (QRISK2 and eFI+) within the Connected Bradford dataset. WP3: Dissemination and stakeholder activity.

Develop a video highlighting the patient experience of prediction models being used in clinical care and concerns regarding temporal drift. Co-develop guidance on the presentation of prediction tools to patients to ensure meaningful engagement. Explore further funding opportunities with TPP to implement the framework within their OpenSAFELY platform.

Run a stakeholder workshop to discuss policy implications and high-impact publication inviting participants from NHS England, MHRA, NICE, ICBs, TPP, EMIS, funding bodies, and researchers.

Impact and Dissemination PREDICT will generate impact by highlighting the importance of temporal drift, and uniting the relevant decision-makers (e.g. MHRA, NHS England, TPP, EMIS) in discussions on how to approach the problem.

A high-impact publication summarising the workshop discussions will be co-authored and published in a high impact medical journal, such as the BMJ.

Over the medium and long-term, PREDICT will jump-start regulatory change to address the growing issue of temporal drift.

This will be achieved via obtaining further funding in collaboration with TPP to implement our framework on OpenSAFELY for researchers to use in the future.

We see this having an enormous positive impact on patient care and service provision as the use of clinical prediction models becomes increasingly commonplace.