Development of new methodology to frame and address the impact of poor-quality studies on meta-analysis

Background Meta-analysis of randomised clinical trial is used to summarise the effects of healthcare interventions.

Since trials are not conducted with equal rigour, an assessment of the quality of RCTs in a meta-analysis is recommended.

However, identifying potentially biased studies, and even subsequently downgrading review recommendations, is only a partial solution to the problem.

A method is required that produces meta-analysed estimates of treatment effects that takes into account the variable study quality of the primary studies whilst being both accessible and generalisable.

Previous work has identified some counter-intuitive properties of meta-analysis models regarding the disproportionate impact they give to some studies on the meta-analysis.

This happens when a study artificially inflates the between-study heterogeneity, whilst making a limited contribution to the pooled effect size meaning they “do more harm than good”.

Aims and objectives Further develop and evaluate accessible approaches to alleviate the negative impact of low-quality studies in meta-analysis Promote the best performing methods through detailed illustrative examples and an interactive online app Methods An extensive simulation study will be conducted in which biased studies are added to meta-analysis datasets, initially, in order to understand how they impact on numerous results including the pooled estimate and heterogeneity.

Then several methods for dealing with variable study will be applied to the simulated meta-analytic datasets. These methods will all be simple to implement and focused around excluding a proportion of the evidence.

Methods of excluding studies, based on relative and absolute sample size, study power, a studies results or a studies impact on the results of the meta-analysis, will be considered.

Methods will be evaluated, compared to standard meta-analysis, based on the following criteria: a) reduction of bias in the pooled effect size; b) increase in precision of the pooled effect size; and c) reduction of bias in the estimated heterogeneity. We will take the best performing methods and use them to re-analyse existing meta-analysis datasets.

The implications of applying the methods on any subsequent trials in the same field will also be considered.

An online interactive app will be developed that includes results of this project and allows users to implement the recommended methods.

We will work with our diverse team of public contributors to make sure their ideas and views are taken into account throughout the project to ensure that we ask questions that are important to patients/public and guarantee reviews are accessible to those affected and to the decision makers. Timelines for delivery The project will be of 13 months duration from October 2024 – October 2025.

Anticipated impact and dissemination We aim to develop an important tool for meta-analysis that will have broad applicability and become recommended in key systematic review guidelines.

In order to maximise the impact of the work we will draw upon the experience and creativity of a diverse group of experienced public contributors, visual story telling agency, the PPI-SMART group and interactive tools, to ensure our findings are accessible and disseminated in ways appropriate to the diverse range of stakeholders.