Wasted research when systematic reviews fail to provide a complete and up-to-date evidence synthesis: the example of lung cancer

For many conditions, multiple competing treatments are available, many of which have been assessed in randomized trials [1]. Clinicians and patients who are making medical decisions need to know which treatment works best among all treatments for the condition of interest. They increasingly turn to systematic reviews and meta-analyses for current evidence-based assessments of the relative benefits and harms of treatments.

To decide the best treatment for a patient, clinicians and patients need a comprehensive, up-to-date synthesis of evidence for all treatments available for a given condition [2–4]. This synthesis could be provided by considering the whole set of conventional meta-analyses on all treatment comparisons or a network meta-analysis [5].

However, systematic reviews as currently performed may fail to meet clinicians’ and patients’ needs [6]. Systematic reviews and meta-analyses are insufficiently informative if they do not cover all alternative treatments or do not include all available current evidence. In fact, most meta-analyses have a narrow scope and focus on specific treatments [7]. Moreover, many meta-analyses become quickly out-of-date because clinically important evidence can accumulate rapidly, but updating a systematic review can be as costly and time-consuming as the original review [8, 9]. This failure to rigorously synthesize the totality of relevant evidence may have a detrimental effect on treatment decisions and future research planning.

The exponential growth in publications of randomized trials, especially in oncology, increases clinicians’ and patients’ need for broad meta-analyses encompassing all the evidence for all competing treatments [10]. Lung cancer, in particular, remains the fifth leading cause of disability-adjusted life years in developed countries and represents a key area of current therapeutic innovation [11]. With recent progresses in therapeutics, the number of patients with advanced non-small cell lung cancer (NSCLC) who receive second-line treatments is increasing, but which second-line treatment to recommend is unclear.

We used the example of NSCLC to quantify the waste of research related to systematic reviews failing to provide a complete and up-to-date synthesis of evidence over time.

We first used a comprehensive strategy to repeatedly identify all randomized trials, with published and unpublished results, and all systematic reviews of second-line treatments for advanced NSCLC available up to the end of each year from 2009 to 2015. Second, we sequentially assessed the amount of randomized evidence that was covered by systematic reviews collectively: for the years 2009 to 2015, we assessed the articles published up to December 31 of each of those years for proportion of treatments, treatment comparisons, trials, and patients covered by systematic reviews on this topic, with comparison to the total randomized evidence available at each time.

In this study, we assessed whether the whole set of conventional meta-analyses on pairwise treatment comparisons would allow for meeting clinicians’ and patients’ needs, to provide a comprehensive, up-to-date synthesis of evidence for all treatments. Our comparison of the amount of randomized evidence covered by systematic reviews and all randomized trials available for inclusion revealed a substantial waste related to the failure of systematic reviews to accumulate evidence scientifically: the evidence covered by existing systematic reviews on the topic was always substantially incomplete, with 40 % or more of treatments, treatment comparisons, and trials missing.

All meta-analyses on the same topic evaluating only a small fragment of the evidence has consequences for patient care and research planning. We need to identify the treatments with harmful effects and also treatments with side effects that have no advantage as compared with alternative treatments. As well, we need to be able to identify the treatment or group of treatments that works best. The relative beneficial effect of a treatment may be missed if some treatment comparisons of interest are not covered by systematic reviews. In our example, no systematic review encompassed all available treatments. Moreover, encompassing all the evidence for all treatments may have important implications for planning subsequent trials and helping prioritize future research to improve the evidence base [46]. In fact, based on the synthesis results and the geometry of the network of evidence, one could design a trial of treatments infrequently compared or a trial of the best potential treatment [47, 48].

Several reasons explain why the 29 systematic reviews did not cover all the randomized evidence. First, the trials may have addressed narrow and focused questions as compared to each review’s selection criteria (patients, interventions, and comparators). However, all trials we identified would have been eligible for inclusion in at least one systematic review; missing trials were not excluded from systematic reviews because they were not eligible.

Second, many meta-analyses become quickly out-of-date, sometimes by the time they are published. In fact, the systematic identification of trials is complex and time-consuming. In our case study, only one of 29 reviews was an update of a previous review and a large proportion of treatment comparisons covered by systematic reviews was out-of-date. This updating issue may be improved by the automation, as much as possible, of the whole trial search and selection process. These automated technologies would alleviate the burden on systematic reviewers. Third, another reason for the failure of systematic reviews to accumulate all available evidence is inadequate search methods for unpublished trials. In our case study, among the 15 trials (20 %) with unpublished results, only three (20 %) were included in systematic reviews. Among the 29 systematic reviews, 79 % could be considered at high risk of missing trials that would have met the inclusion criteria because they did not report duplicate study selection and data extraction, comprehensive literature search, or searching for reports regardless of publication type. Automated processes, such as meta-search engines, could systematically cover sources such as industry and non-industry trial registries and results databases to identify posted results and improve the evidence synthesis [49, 50]. Beyond the example at hand, the current way of conducting systematic reviews explains why they are inherently at risk of providing a fragmented, out-of-date panorama of the evidence for all treatments.

A broader scope on all evidence available for all treatments of a given condition is naturally provided by a systematic review with network meta-analysis, which allows for examining the totality of the randomized evidence using trial networks [51, 52]. Although the production of network meta-analyses follows an exponential growth, their number is still relatively limited [53]. With the increasing development of new treatments, examining networks of randomized trials is essential, and multiple treatment comparisons cannot be avoided [54]. Conventional meta-analysis focuses on comparisons of two treatments only. However, network meta-analysis allows for comparing all treatments to each other with direct and indirect comparisons, even if randomized trials are not available for some treatment comparisons. In our example, the available direct evidence informed 5 % of all possible comparisons between the 47 treatments. Ideally, the scoping exercise to define the criteria for considering trials in the systematic review should include all alternative treatments for the target condition [55]. It is still possible that a network meta-analysis will selectively choose treatments to include in the network, and excluding treatments may affect estimated treatment effects [56]. In our study, we identified two network meta-analyses; neither covered the whole evidence available at their time of analysis and considered a very restrictive network instead. Finally, network meta-analysis offers more flexibility to assess individual treatments. It offers the opportunity to overcome a common issue in standard meta-analysis whereby different treatments and sometimes different types of treatments are lumped together. In our example, erlotinib and gefitinib were frequently lumped together in EGFR tyrosine kinase inhibitors and docetaxel and pemetrexed were lumped together in monochemotherapy. This type of evidence lumping does not provide the relevant information for clinicians and patients who want to know which specific treatments work the best.

The paradox is that the set of systematic reviews fail to cover all the evidence, diminishing the value of research to clinicians by missing important comparisons. In fact, the production of meta-analyses has been substantially and rapidly increasing worldwide [57, 58]. In parallel, the evidence covered by systematic reviews features many gaps, and multiple overlapping meta-analyses on the same topic are common [59, 60].

As an alternative to classical systematic reviews performed at one point in time, a new theoretical framework is “living systematic reviews”, defined as high-quality online summaries of health research updated as new research becomes available [61]. We propose to push further the shift towards a new paradigm by switching: 1) from a series of standard meta-analyses focused on specific treatments (many treatments being not considered) to a single network meta-analysis covering all treatments; and 2) from meta-analyses performed at a given time and frequently out-of-date to a cumulative network meta-analysis systematically updated as soon as the results of a new trial become available, an approach to synthesis we call “live cumulative network meta-analysis”. In Fig. 5, we show the methodological steps we propose for live cumulative network meta-analysis.

Another challenge would be to consider all outcomes that are important or critical to patients for decision making in these live cumulative network meta-analyses [69, 70]. In our case study, we included trials regardless of reported outcomes; in a sensitivity analysis, we excluded trials that did not report treatment effects on overall survival or progression-free survival. However, there are other patient-important outcomes, in particular to measure the symptom burden of the disease and the quality of life of patients. More generally, it will be crucial to consider networks of trials according to the reporting of the different patient-important outcomes. In fact, the geometry of the network of trials could vary across outcomes because of differential reporting of outcomes (e.g., efficacy and safety outcomes) across drugs and trials.

Nonetheless, embracing the perspective of networks of trials of all alternative treatments for each condition, and in particular developing live cumulative network meta-analyses, could greatly benefit various stakeholders, including physicians, patients, and also guideline developers, funders, and decision-makers [71]. Networks of trials and their synthesis through network meta-analysis could increase the value of research when treatment recommendations are based on an exhaustive up-to-date network of randomized evidence [72]. Guideline developers and other decision-makers may further benefit from network meta-analyses if these implement recent developments to rate the quality of the body of evidence supporting treatment effect estimates for all patient-important outcomes (e.g. GRADE Summary of Findings tables) and rankings from network meta-analysis. For instance, in the field of medical therapies for open angle glaucoma, Li et al. recently showed that if a network meta-analysis had been conducted earlier, prostaglandins could have been shown as the most effective class in lowering intraocular pressure 7 years ahead of the guideline recommendation [73]. Moreover, biomedical research funding agencies could improve the prioritization of research proposals if they had access to a mapping of existing (and ongoing) trials evaluating all available treatments for a specific condition at the time a new trial is planned. Visualizing the network of trials and identifying which new trial maximizes the information can help stakeholders assess gaps in evidence and choose the next treatment comparison or trial that needs to be prioritized.

To our best knowledge, our study is the first to highlight the substantial waste associated with the failure of systematic reviews to accumulate evidence across all treatments (for the same disease). Moreover, our methodology based on a series of systematic overviews and networks of randomized evidence is novel and could be replicated in other fields. Our study has several limitations. First, we examined only one topic, second-line treatments for advanced NSCLC. However, our results should be generalizable, because the scope of meta-analyses is frequently narrow, with 81 % of standard meta-analyses that do not include all treatments and 43 % that cover only specific treatments [7]. Contrary to Haidich and colleagues, who assessed the evidence at the level of each systematic review, we assessed the cumulative evidence covered by all systematic reviews on a topic with a “bird’s-eye view”. Second, our gold standard included trials that meta-analysts may have considered ineligible for inclusion in their systematic reviews. However, we performed several sensitivity analyses and results were consistent. Third, we excluded trials in which the control group received different chemotherapy drugs at the discretion of the investigators. Such trials could also contribute to the synthesis in a network meta-analysis with a class effect model [74]; however, they do not allow for assessing the relative merits of the specific drugs. Similarly, we excluded randomized trials and systematic reviews comparing two different administration schemes because our focus was the comparison of alternative treatments against each other. Nonetheless, our analysis could be extended to such randomized trials and systematic reviews. In such cases, each relevant node would have several subnodes that relate to different administration schemes. Fourth, we may have missed some systematic reviews and randomized trials because of reporting bias, but we tried to minimize this possibility by an exhaustive search covering conference abstracts and registries. Fifth, we started our analysis for the year 2009; this pre-specified year was somehow arbitrary and we acknowledge that this starting point could have been earlier, considering that the first systematic review was published in April 2001; the second one was published in December 2005. However, there was no or little randomized evidence available between competing treatments at that time. Finally, we did not perform any outcome data synthesis at this stage. One may ask if covering all the randomized evidence available would have led to clinically important differences for the 10 % to 17 % of treatment comparisons partially covered by systematic reviews. In the framework we are promoting, a network meta-analysis would allow for estimating all treatment comparisons. Adding up to 40 % of missing evidence (about 8,000 patients) to the network would likely lead to clinically important differences, in particular for treatment comparisons partially covered by systematic reviews.

We illustrated that systematic reviews of a given condition provide a fragmented, out-of-date panorama of the evidence for all treatments. Embracing the perspective of networks of trials of all alternative treatments for each condition can have important consequences and should be adopted more generally. The waste of research associated with the failure to accumulate evidence could be reduced by the development of live cumulative network meta-analyses.