Improving the evaluation of mental health digital health interventions in randomised controlled trials by understanding and incorporating user engagement

Background Development of digital health interventions for mental health is rapidly increasing, over 10,000 apps now available. They overcome traditional barriers, offering individuals a confidential way to get the support needed. However, with so many available it's an onerous task choosing the right one.

This is made worse as most have no evidence on their efficacy. Research has started to catch up, but many aspects are still unknown. One of these components, essential in mental health, is engagement. Digital health interventions have high attrition rates, but little is known of what the right engagement level is.

The problem is challenging as these interventions are multifaceted like complex interventions but with an added dimension of unconstrained access.

Individuals involved in project development highlighted that understanding engagement and its impact on outcomes as a priority.

This can be achieved using statistical methods already available in a randomised evaluation, but a definition of engagement is required. Use of these methods has increased recently in other complex intervention studies where engagement is well defined. This research will provide the foundations for defining engagement for digital health interventions.

Research questions How to define engagement levels for mental health digital health interventions? How should trial analysis of digital health interventions adjust for the effective engagement level?

Aims and objectives: Determine methods used to summarise engagement and incorporate this in clinical trial results Define engagement levels using indicators in two case studies; EFFIP and BRIGHTEN Define the effective engagement level for each case study and re-analyse the outcomes Develop a methodological framework for researchers to incorporate engagement in trials of other digital health interventions Methods and timelines for delivery: Outlined below as four work packages: Work Package 1: A systematic review of randomised controlled trials (RCTs) of mental health digital health interventions (month 1 to 9) Work Package 2: Defined engagement levels using indicators, e.g. pageviews, in case studies using Principle Component Analysis (month 10 to 18) Work package 3: Define the effective engagement level and re-analyse case studies using complier average causal effect analysis and evaluate using simulations (month 19 to 27) Work Package 4: Methodological framework combining learnings from previous aims (month 28 to 36) The case studies: EFFIP Study - an RCT comparing COPe-support, a web-based multi-component virtual learning environment for carers of those with psychosis against an attention control.

BRIGHTEN Study - An RCT comparing two apps to treat depression against an information control app.

Impact and dissemination: I will address outstanding clinically important questions regarding how to define engagement to digital health interventions and the impact to outcomes in the trial analysis, an important issue raised by patients involved in project development.

Beyond this, it provides essential methodology for trialists to evaluate other mental health digital health interventions, leading to better guidance and helping patients to make informed treatment choices. I will disseminate to researchers and statisticians at seminars, conferences and in open access journals.

For an effective impact I will target three audiences; researchers in mental health, trialists and intervention developers.