Wasted research when systematic reviews fail to provide a complete and up-to-date evidence synthesis: the example of lung cancer

Abstract

Background

Multiple treatments are frequently available for a given condition, and clinicians and patients need a comprehensive, up-to-date synthesis of evidence for all competing treatments. We aimed to quantify the waste of research related to the failure of systematic reviews to provide a complete and up-to-date evidence synthesis over time.

Methods

We performed a series of systematic overviews and networks of randomized trials assessing the gap between evidence covered by systematic reviews and available trials of second-line treatments for advanced non-small cell lung cancer. We searched the Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, MEDLINE, EMBASE, and other resources sequentially by year from 2009 to March 2, 2015. We sequentially compared the amount of evidence missing from systematic reviews to the randomized evidence available for inclusion each year. We constructed cumulative networks of randomized evidence over time and evaluated the proportion of trials, patients, treatments, and treatment comparisons not covered by systematic reviews on December 31 each year from 2009 to 2015.

Results

We identified 77 trials (28,636 patients) assessing 47 treatments with 54 comparisons and 29 systematic reviews (13 published after 2013). From 2009 to 2015, the evidence covered by existing systematic reviews was consistently incomplete: 45 % to 70 % of trials; 30 % to 58 % of patients; 40 % to 66 % of treatments; and 38 % to 71 % of comparisons were missing. In the cumulative networks of randomized evidence, 10 % to 17 % of treatment comparisons were partially covered by systematic reviews and 55 % to 85 % were partially or not covered.

Conclusions

We illustrate how systematic reviews of a given condition provide a fragmented, out-of-date panorama of the evidence for all treatments. This waste of research might be reduced by the development of live cumulative network meta-analyses.

Background

For many conditions, multiple competing treatments are available, many of which have been assessed in randomized trials [1]. Clinicians and patients who are making medical decisions need to know which treatment works best among all treatments for the condition of interest. They increasingly turn to systematic reviews and meta-analyses for current evidence-based assessments of the relative benefits and harms of treatments.

To decide the best treatment for a patient, clinicians and patients need a comprehensive, up-to-date synthesis of evidence for all treatments available for a given condition [2–4]. This synthesis could be provided by considering the whole set of conventional meta-analyses on all treatment comparisons or a network meta-analysis [5].

However, systematic reviews as currently performed may fail to meet clinicians’ and patients’ needs [6]. Systematic reviews and meta-analyses are insufficiently informative if they do not cover all alternative treatments or do not include all available current evidence. In fact, most meta-analyses have a narrow scope and focus on specific treatments [7]. Moreover, many meta-analyses become quickly out-of-date because clinically important evidence can accumulate rapidly, but updating a systematic review can be as costly and time-consuming as the original review [8, 9]. This failure to rigorously synthesize the totality of relevant evidence may have a detrimental effect on treatment decisions and future research planning.

The exponential growth in publications of randomized trials, especially in oncology, increases clinicians’ and patients’ need for broad meta-analyses encompassing all the evidence for all competing treatments [10]. Lung cancer, in particular, remains the fifth leading cause of disability-adjusted life years in developed countries and represents a key area of current therapeutic innovation [11]. With recent progresses in therapeutics, the number of patients with advanced non-small cell lung cancer (NSCLC) who receive second-line treatments is increasing, but which second-line treatment to recommend is unclear.

We used the example of NSCLC to quantify the waste of research related to systematic reviews failing to provide a complete and up-to-date synthesis of evidence over time.

Methods

We first used a comprehensive strategy to repeatedly identify all randomized trials, with published and unpublished results, and all systematic reviews of second-line treatments for advanced NSCLC available up to the end of each year from 2009 to 2015. Second, we sequentially assessed the amount of randomized evidence that was covered by systematic reviews collectively: for the years 2009 to 2015, we assessed the articles published up to December 31 of each of those years for proportion of treatments, treatment comparisons, trials, and patients covered by systematic reviews on this topic, with comparison to the total randomized evidence available at each time.

Identification of randomized trials

Eligibility criteria

We considered randomized trials of second-line treatments compared to each other or against a placebo or best supportive care in patients with advanced (stage IIIB–IV) NSCLC. We considered any cytotoxic monochemotherapy, any targeted treatment, any combination of a cytotoxic monochemotherapy and targeted treatment, and any combination of two targeted treatments (complete list in Additional file 1: Appendix 1). We excluded trials assessing doublet chemotherapy and comparing two different administration schemes. We excluded trials focusing exclusively on patients with epidermal growth factor receptor (EGFR)-activating mutation or anaplastic lymphoma kinase (ALK) rearrangement, because it represented a specific minority sub-group of all advanced NSCLC.

Search strategy

We searched for reports of randomized trials in the Cochrane Central Register of Controlled Trials, MEDLINE, and EMBASE (search equations in Additional file 1: Appendix 2) with no restriction on language, status, or year of publication, and searched other resources [12]: 1) previous systematic reviews (see below); 2) reference lists of all selected trials; 3) conference abstracts (from the American Society of Clinical Oncology Meeting, European Society of Medical Oncology Congress, and World Lung Cancer Conference); 4) non-industry trial registries and results databases (ClinicalTrials.gov and EudraCT); 5) industry trial registries and results databases; and 6) regulatory agency online databases (US Food and Drug Administration and European Medicines Agency); details in Additional file 1: Appendix 3. We contacted trialists to request complete results for all trials identified as “completed” on ClinicalTrials.gov but without published results and for all trials with conference abstracts but no full-text articles. The last search was conducted on March 2, 2015.

Identification of systematic reviews

Eligibility criteria

Systematic reviews of randomized trials of second-line treatments for advanced NSCLC were eligible. We selected reviews that addressed at least one comparison between the treatments considered previously, whether they included a meta-analysis or not. We excluded reviews that did not report clearly stated objectives, eligibility criteria for trials, or a systematic search strategy; reviews combining first- and second-line data were ineligible; reviews focusing exclusively on patients with EGFR mutation or ALK rearrangement or on doublet chemotherapy, or comparing two different administration schemes; and reviews that did not provide the list of included trials.

Search strategy

We searched the Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effects, MEDLINE, EMBASE (search equations in Additional file 1: Appendix 4), and other resources: conference abstracts from the American Society of Clinical Oncology Meeting, European Society of Medical Oncology Congress, and World Lung Cancer Conference; and the PROSPERO international prospective register of systematic reviews for completed or published systematic reviews (details in Additional file 1: Appendix 5). There was no restriction on language, status, or year of publication. The last search was conducted on March 2, 2015.

Selection of studies and extraction of data

Two authors independently and in duplicate examined titles, abstracts, and full-text articles to determine the eligibility of randomized trials and systematic reviews. We pilot-tested the eligibility criteria on a sample of 100 records (for the selection on titles and abstracts) and 10 reports (for the selection on full-text articles) to ensure that the selection criteria were applied consistently by the two authors. Disagreements were discussed with a third author. All data were independently extracted by two authors who used a standardized form.

For each trial, we extracted the dates of publication of the full-text article(s) (online publication, if any) and conference abstract(s), date of results posting on non-industry and industry trial registries, and date of publication of reports by regulatory agencies. We also extracted the treatments assessed, number of randomized patients in each arm, study phase (II or III), and reported outcomes (overall survival and progression-free survival).

For each systematic review, we extracted the list of relevant trials selected, the publication date (online publication, if any), date of last search of trials, number of trials included, and type of treatment compared. We assessed the funding source (industry, non-industry, no funding, or not reported), whether the review was an update of a previously published review, and whether a network meta-analysis was performed. Finally, we assessed the scope of each systematic review (i.e., interventions and comparators assessed). We identified whether the review focused on one specific treatment explicitly (alone or combined with other treatments), lumped different treatments of the same type together (e.g., monochemotherapy or targeted therapy), and lumped different types of treatments together (e.g., monochemotherapy and EGFR tyrosine kinase inhibitors considered the same intervention).

Two reviewers independently assessed the methodological quality of the systematic reviews, with a formal consensus process in case of disagreement. We used AMSTAR, a measurement tool created to assess the methodological quality of systematic reviews [13], which has been validated [14, 15]. We assessed the four items pertaining to duplicate study selection and data extraction, comprehensive literature search (at least two electronic sources and one supplementary strategy among reviews, experts, or reviewing the references), searching for reports regardless of their publication type, and providing a list of included and excluded trials. Searching for trials regardless of their publication type was judged inadequate when authors did not report searching the grey literature (conference abstracts, non-industry trial registries and results databases, industry trial registries and results databases, regulatory agency online databases) or excluding reports based on language. We focused on these four specific items because the methods used for the identification and selection of studies are directly related to a potential gap between the amount of randomized evidence covered by systematic reviews and the amount of randomized evidence available for inclusion, and other domains are unrelated.

Definition of randomized evidence available for inclusion in systematic reviews

We pre-specified the year 2009 as a starting point for our analyses in order to allow for a sufficient amount of evidence (in terms of both randomized trials and available systematic reviews) regarding the comparison between competing second-line treatments for advanced NSCLC to initiate a comparison between the available randomized evidence and that covered by systematic reviews. From 2009 to 2015, we identified the cumulative list of trials eligible for inclusion in systematic reviews; we checked that each trial identified would have been eligible for inclusion in at least one systematic review (i.e., corresponded to the selection criteria in terms of patients, interventions, and comparators). For each trial, we identified the earliest report of results and considered the corresponding publication date as when the trial became eligible for inclusion in systematic reviews. Considering the inevitable time lag between completion and publication, most recently published trials could not be selected by any systematic review, so we considered a 6-month lag period as recommended by the Cochrane Collaboration (i.e., we listed all trial results published up to July 1 each year, and up to August 31, 2014 for 2015) [16]. We also compiled the cumulative list of treatments and treatment comparisons assessed in eligible trials; finally, we calculated the cumulative number of patients included in trials as a measure of the available amount of randomized evidence.

Definition of randomized evidence covered by systematic reviews

We considered all systematic reviews published up to December 31 each year from 2009 to 2015 (up to March 2 for 2015). The reference date for a systematic review was the publication date of the full-text article or online publication date, if any. We compiled the cumulative list of all relevant trials selected by these systematic reviews and the cumulative list of treatments and treatment comparisons and cumulative number of included patients in the trials selected by the systematic reviews.

Assessment of randomized evidence not covered by systematic reviews

We evaluated the overall number and proportion of treatments, treatment comparisons, trials, and patients not covered by systematic reviews from 2009 to 2015.

We constructed cumulative networks of randomized evidence. Each node was a treatment and each edge was a treatment comparison (i.e., an edge connected two nodes when at least one randomized trial compared the two corresponding treatments). In multi-arm trials, doses of the same drug were lumped under a common node. The node size was proportional to the total number of patients randomly allocated to the corresponding treatment across all randomized trials available for inclusion; we represented the proportions of randomized patients not actually covered by systematic reviews by pie charts overlaid on nodes in the network. The edge width was proportional to the total number of randomized trials between the corresponding treatments available for inclusion; we represented the proportions of trials not selected by systematic reviews by a percentage bar chart overlaid on edges in the network. The evidence for a treatment comparison was considered partially covered when systematic reviews did not cover all the evidence available for this treatment comparison.

In sensitivity analyses, we discarded trials potentially ineligible for inclusion in any systematic review: trials of drugs that did not successfully pass phase II; trials that did not report treatment effects on overall survival or progression-free survival; and trials with results reported in conference abstracts only. In a last sensitivity analysis, the lag period to define randomized evidence available for inclusion in systematic reviews was defined by the last date of search for the last published systematic review.

Analyses involved use of R version 3.2.1 (R Development Core Team, Vienna, Austria).

Results

Randomized trials of second-line treatments for NSCLC

We identified 77 eligible trials with results available between May 2000 and November 2014 (Fig. 1). The results of 62 trials were published in 69 articles; among these 62 trials, 43 had results reported in other sources (16 in conference abstracts, 10 in non-industry trial registries and results databases or regulatory agency online databases, and 17 in both). The remaining 15 trials (20 %) had unpublished results (12 conference abstracts, 2 non-industry trial registries and results databases or regulatory agency online databases, and 1 both). In all, 61 trials (79 %) were registered at ClinicalTrials.gov. The 77 trials included 28,636 patients (median [Q1–Q3] 168 [100–559] patients); 72 included two arms, two compared three different treatments, and three compared one drug to two different doses of another drug, which were lumped together. Fig. 2 shows the complete network of evidence, with 45 different treatments and 54 treatment comparisons against each other or best supportive care or placebo. With 47 nodes in the network, there were 47 × 46/2 = 1,081 possible pairwise comparisons. The available direct evidence informs 54 comparisons (5 %).

Fig. 1

Flow diagram of selection of systematic reviews and randomized controlled trials of second-line treatments in advanced non-small cell lung cancer. ^*Additional full-text articles not identified by searching bibliographical databases; ^$63 full-text articles and 44 conference abstracts; ^¤69 full-text articles, 70 conference abstracts, 28 posted results, and 5 industry/FDA reports

Fig. 2

Network of 77 randomized controlled trials of second-line treatments in advanced non-small cell lung cancer. The thickness of connecting lines indicates the number of available comparisons. The size of each node is proportional to the number of patients allocated to the corresponding treatment. AFL: aflibercept; AMR: amrubicin; ARQ197: tivantinib; BEV: bevacizumab; BIBF1120: nintedanib; BSC: best supportive care; CET: cetuximab; DAC: dacomitinib; DOC: docetaxel; ERL: erlotinib; EVE: everolimus; FIGI: figitumumab; FULV: fulvestrant; GEF: gefitinib; ICO: icotinib; MAT: matuzumab; MK-0646: dalotuzumab; ONA: onartuzumab; PAZ: pazopanib; PBO: placebo; PDX: pralatrexate; PEM: pemetrexed; PTX: paclitaxel; RAM: ramucirumab; SEL: selumetinib; SOR: sorafenib; SUN: sunitinib; TOP: topotecan; TRA: trametinib; VAN: vandetanib; VFL: vinflunine; VIN: vinorelbine

Systematic reviews of second-line treatments for NSCLC

We identified 29 systematic reviews published from April 2001 to February 2015 (Table 1) [17–45]. The first published review was a Cochrane review and all subsequent reviews were published in journal venues. The number of reviews doubled in 2014, from 16 to 27 (Additional file 1: Appendix 6). The median time between the last review search and publication was 9 months [Q1–Q3 5–13 months]. The industry was involved in 10 % of the reviews and not involved in 41 %, and the funding source was unclear in 41 %. The reviews addressed 19 comparisons, with 26 reviews lumping different treatments or different types of treatments together, considering them as the same intervention or comparator. Six reviews focused on one specific treatment of main interest. Two reviews performed network meta-analyses but did not cover all available treatments. Only one of the 29 reviews was an update of a previously published review.

Table 1 Characteristics of 29 selected systematic reviews

Regarding the methodological quality of the 29 systematic reviews, 45 % of reviews lacked information on independent study selection and data extraction, 31 % a comprehensive literature search, and 45 % a search for reports regardless of their publication type. Of note, 17 % of reviews lacked information on a search for conference abstracts, 86 % a search for non-industry trial registries and results databases, 97 % a search for industry trial registries and results databases, and 97 % a search of regulatory agency online databases. In all, 79 % of systematic reviews did not report duplicate study selection and data extraction, comprehensive literature search, and searching for reports regardless of their publication type. Finally, 7 % of reviews provided a list of included and excluded trials.

Randomized evidence not covered or partially covered by systematic reviews

From 2009 to 2015, the amount of randomized evidence covered by existing systematic reviews was consistently incomplete: 40 % to 66 % of treatments; 38 % to 71 % of treatment comparisons; 45 % to 70 % of trials; and 30 % to 58 % of patients were missing (Fig. 3). In 2014, 27 reviews still did not cover 18 treatments (40 %), 20 treatment comparisons (38 %), 34 trials (46 %), and 8,486 patients (30 %).

Fig. 3

Amount of treatments, treatment comparisons, trials, and patients not covered by systematic reviews from 2009 to 2015. ^*The last search for randomized trials and systematic reviews was conducted on March 2, 2015

The cumulative networks of evidence show how the mismatch between available data and syntheses persisted from 2009 to 2015 (Fig. 4). Across all years, 10 % to 17 % of treatment comparisons were partially covered by systematic reviews and 55 % and 85 % were partially or not covered by systematic reviews. Moreover, the proportion of evidence covered by systematic reviews was unequally distributed across treatments.

Fig. 4

Cumulative networks of evidence showing the gap between the amount of randomized evidence covered by systematic reviews and the amount of randomized evidence available for inclusion. (a) 2009–2012 and (b) 2013–2015. ^*The last search for randomized trials and systematic reviews was conducted on March 2, 2015. From 2009 to 2015, we compared randomized controlled trials selected by systematic reviews published up to December 31 each year (up to March 2 for 2015) to all trials eligible for inclusion (i.e., all trial results published up to July 1 each year [up to August 31, 2014 for 2015]). Each node size is proportional to the total number of patients randomly allocated to the corresponding treatment across all randomized trials available for inclusion; we represented the proportion of randomized patients actually covered by systematic reviews by pie charts overlaid on nodes in the network. The thickness of each edge is proportional to the total number of randomized controlled trials between the corresponding treatments available for inclusion; we represented the proportion of trials actually selected by systematic reviews by a percentage bar chart overlaid on edges in the network

Sensitivity analyses gave consistent findings. When removing trials of drugs that did not pass phase II, 27 % to 64 % of trials, 27 % to 58 % of patients, 22 % to 53 % of treatments, and 21 % to 67 % of treatment comparisons were missing. When removing trials that did not report overall survival or progression-free survival, 41 % to 68 % of trials, 29 % to 58 % of patients, 36 % to 63 % of treatments, and 36 % to 69 % of treatment comparisons were missing. When discarding trials reported in conference abstracts only, 37 % to 68 % of trials, 24 % to 58 % of patients, 33 % to 63 % of treatments, and 33 % to 68 % of treatment comparisons were missing. When the lag period was the last date of search of the last systematic review, 42 % to 70 % of trials, 11 % to 53 % of patients, 36 % to 66 % of treatments, and 37 % to 72 % of treatment comparisons were missing.

Discussion

In this study, we assessed whether the whole set of conventional meta-analyses on pairwise treatment comparisons would allow for meeting clinicians’ and patients’ needs, to provide a comprehensive, up-to-date synthesis of evidence for all treatments. Our comparison of the amount of randomized evidence covered by systematic reviews and all randomized trials available for inclusion revealed a substantial waste related to the failure of systematic reviews to accumulate evidence scientifically: the evidence covered by existing systematic reviews on the topic was always substantially incomplete, with 40 % or more of treatments, treatment comparisons, and trials missing.

All meta-analyses on the same topic evaluating only a small fragment of the evidence has consequences for patient care and research planning. We need to identify the treatments with harmful effects and also treatments with side effects that have no advantage as compared with alternative treatments. As well, we need to be able to identify the treatment or group of treatments that works best. The relative beneficial effect of a treatment may be missed if some treatment comparisons of interest are not covered by systematic reviews. In our example, no systematic review encompassed all available treatments. Moreover, encompassing all the evidence for all treatments may have important implications for planning subsequent trials and helping prioritize future research to improve the evidence base [46]. In fact, based on the synthesis results and the geometry of the network of evidence, one could design a trial of treatments infrequently compared or a trial of the best potential treatment [47, 48].

Several reasons explain why the 29 systematic reviews did not cover all the randomized evidence. First, the trials may have addressed narrow and focused questions as compared to each review’s selection criteria (patients, interventions, and comparators). However, all trials we identified would have been eligible for inclusion in at least one systematic review; missing trials were not excluded from systematic reviews because they were not eligible.

Second, many meta-analyses become quickly out-of-date, sometimes by the time they are published. In fact, the systematic identification of trials is complex and time-consuming. In our case study, only one of 29 reviews was an update of a previous review and a large proportion of treatment comparisons covered by systematic reviews was out-of-date. This updating issue may be improved by the automation, as much as possible, of the whole trial search and selection process. These automated technologies would alleviate the burden on systematic reviewers. Third, another reason for the failure of systematic reviews to accumulate all available evidence is inadequate search methods for unpublished trials. In our case study, among the 15 trials (20 %) with unpublished results, only three (20 %) were included in systematic reviews. Among the 29 systematic reviews, 79 % could be considered at high risk of missing trials that would have met the inclusion criteria because they did not report duplicate study selection and data extraction, comprehensive literature search, or searching for reports regardless of publication type. Automated processes, such as meta-search engines, could systematically cover sources such as industry and non-industry trial registries and results databases to identify posted results and improve the evidence synthesis [49, 50]. Beyond the example at hand, the current way of conducting systematic reviews explains why they are inherently at risk of providing a fragmented, out-of-date panorama of the evidence for all treatments.

A broader scope on all evidence available for all treatments of a given condition is naturally provided by a systematic review with network meta-analysis, which allows for examining the totality of the randomized evidence using trial networks [51, 52]. Although the production of network meta-analyses follows an exponential growth, their number is still relatively limited [53]. With the increasing development of new treatments, examining networks of randomized trials is essential, and multiple treatment comparisons cannot be avoided [54]. Conventional meta-analysis focuses on comparisons of two treatments only. However, network meta-analysis allows for comparing all treatments to each other with direct and indirect comparisons, even if randomized trials are not available for some treatment comparisons. In our example, the available direct evidence informed 5 % of all possible comparisons between the 47 treatments. Ideally, the scoping exercise to define the criteria for considering trials in the systematic review should include all alternative treatments for the target condition [55]. It is still possible that a network meta-analysis will selectively choose treatments to include in the network, and excluding treatments may affect estimated treatment effects [56]. In our study, we identified two network meta-analyses; neither covered the whole evidence available at their time of analysis and considered a very restrictive network instead. Finally, network meta-analysis offers more flexibility to assess individual treatments. It offers the opportunity to overcome a common issue in standard meta-analysis whereby different treatments and sometimes different types of treatments are lumped together. In our example, erlotinib and gefitinib were frequently lumped together in EGFR tyrosine kinase inhibitors and docetaxel and pemetrexed were lumped together in monochemotherapy. This type of evidence lumping does not provide the relevant information for clinicians and patients who want to know which specific treatments work the best.

The paradox is that the set of systematic reviews fail to cover all the evidence, diminishing the value of research to clinicians by missing important comparisons. In fact, the production of meta-analyses has been substantially and rapidly increasing worldwide [57, 58]. In parallel, the evidence covered by systematic reviews features many gaps, and multiple overlapping meta-analyses on the same topic are common [59, 60].

As an alternative to classical systematic reviews performed at one point in time, a new theoretical framework is “living systematic reviews”, defined as high-quality online summaries of health research updated as new research becomes available [61]. We propose to push further the shift towards a new paradigm by switching: 1) from a series of standard meta-analyses focused on specific treatments (many treatments being not considered) to a single network meta-analysis covering all treatments; and 2) from meta-analyses performed at a given time and frequently out-of-date to a cumulative network meta-analysis systematically updated as soon as the results of a new trial become available, an approach to synthesis we call “live cumulative network meta-analysis”. In Fig. 5, we show the methodological steps we propose for live cumulative network meta-analysis.

Fig. 5

A new approach to synthesize evidence: live cumulative network meta-analysis. Starting from an initial NMA, a research community would regularly (e.g., every 3 months), search for, screen, and select trials with new results and, if any, extract data, assess the risk of bias, and update the NMA. NMA: network meta-analysis

We acknowledge that developing such methodology is challenging. In Table 2, we present some key challenges and potential solutions. The rigorous methodology of systematic reviews (exhaustive search of trials, minimizing subjectivity by independent duplicate assessments, assessing risk of bias within trials) is inherently demanding of resources and time, especially for a systematic review incorporating network meta-analysis. Moreover, keeping a systematic review up-to-date requires processes closer to those of rapid reviews (i.e., using accelerated and streamlined methods). Therefore, there is necessarily a trade-off between high-standard synthesis methods and real-time updating processes. Automated technologies may help define this trade-off by alleviating the burden of manual tasks for systematic reviewers. Several tools have been proposed to improve, hasten, and ease the search for trials, trial selection, extraction of data, and assessment of risk of bias [49, 50, 62, 63]. Live cumulative network meta-analysis may also raise issues regarding the current authoring and publishing system. Online posting may be more adequate to report periodically the findings of such “real-time” syntheses. Since Elliott et al. presented the theoretical framework of living systematic reviews, some examples have been published and have only partly addressed the aforementioned challenges [64–68]; for instance, by using accompanying open-access websites to disseminate the updates of the systematic review.

Table 2 Methodological steps of live cumulative network meta-analysis, key challenges, and potential solutions

Another challenge would be to consider all outcomes that are important or critical to patients for decision making in these live cumulative network meta-analyses [69, 70]. In our case study, we included trials regardless of reported outcomes; in a sensitivity analysis, we excluded trials that did not report treatment effects on overall survival or progression-free survival. However, there are other patient-important outcomes, in particular to measure the symptom burden of the disease and the quality of life of patients. More generally, it will be crucial to consider networks of trials according to the reporting of the different patient-important outcomes. In fact, the geometry of the network of trials could vary across outcomes because of differential reporting of outcomes (e.g., efficacy and safety outcomes) across drugs and trials.

Nonetheless, embracing the perspective of networks of trials of all alternative treatments for each condition, and in particular developing live cumulative network meta-analyses, could greatly benefit various stakeholders, including physicians, patients, and also guideline developers, funders, and decision-makers [71]. Networks of trials and their synthesis through network meta-analysis could increase the value of research when treatment recommendations are based on an exhaustive up-to-date network of randomized evidence [72]. Guideline developers and other decision-makers may further benefit from network meta-analyses if these implement recent developments to rate the quality of the body of evidence supporting treatment effect estimates for all patient-important outcomes (e.g. GRADE Summary of Findings tables) and rankings from network meta-analysis. For instance, in the field of medical therapies for open angle glaucoma, Li et al. recently showed that if a network meta-analysis had been conducted earlier, prostaglandins could have been shown as the most effective class in lowering intraocular pressure 7 years ahead of the guideline recommendation [73]. Moreover, biomedical research funding agencies could improve the prioritization of research proposals if they had access to a mapping of existing (and ongoing) trials evaluating all available treatments for a specific condition at the time a new trial is planned. Visualizing the network of trials and identifying which new trial maximizes the information can help stakeholders assess gaps in evidence and choose the next treatment comparison or trial that needs to be prioritized.

To our best knowledge, our study is the first to highlight the substantial waste associated with the failure of systematic reviews to accumulate evidence across all treatments (for the same disease). Moreover, our methodology based on a series of systematic overviews and networks of randomized evidence is novel and could be replicated in other fields. Our study has several limitations. First, we examined only one topic, second-line treatments for advanced NSCLC. However, our results should be generalizable, because the scope of meta-analyses is frequently narrow, with 81 % of standard meta-analyses that do not include all treatments and 43 % that cover only specific treatments [7]. Contrary to Haidich and colleagues, who assessed the evidence at the level of each systematic review, we assessed the cumulative evidence covered by all systematic reviews on a topic with a “bird’s-eye view”. Second, our gold standard included trials that meta-analysts may have considered ineligible for inclusion in their systematic reviews. However, we performed several sensitivity analyses and results were consistent. Third, we excluded trials in which the control group received different chemotherapy drugs at the discretion of the investigators. Such trials could also contribute to the synthesis in a network meta-analysis with a class effect model [74]; however, they do not allow for assessing the relative merits of the specific drugs. Similarly, we excluded randomized trials and systematic reviews comparing two different administration schemes because our focus was the comparison of alternative treatments against each other. Nonetheless, our analysis could be extended to such randomized trials and systematic reviews. In such cases, each relevant node would have several subnodes that relate to different administration schemes. Fourth, we may have missed some systematic reviews and randomized trials because of reporting bias, but we tried to minimize this possibility by an exhaustive search covering conference abstracts and registries. Fifth, we started our analysis for the year 2009; this pre-specified year was somehow arbitrary and we acknowledge that this starting point could have been earlier, considering that the first systematic review was published in April 2001; the second one was published in December 2005. However, there was no or little randomized evidence available between competing treatments at that time. Finally, we did not perform any outcome data synthesis at this stage. One may ask if covering all the randomized evidence available would have led to clinically important differences for the 10 % to 17 % of treatment comparisons partially covered by systematic reviews. In the framework we are promoting, a network meta-analysis would allow for estimating all treatment comparisons. Adding up to 40 % of missing evidence (about 8,000 patients) to the network would likely lead to clinically important differences, in particular for treatment comparisons partially covered by systematic reviews.

Conclusions

We illustrated that systematic reviews of a given condition provide a fragmented, out-of-date panorama of the evidence for all treatments. Embracing the perspective of networks of trials of all alternative treatments for each condition can have important consequences and should be adopted more generally. The waste of research associated with the failure to accumulate evidence could be reduced by the development of live cumulative network meta-analyses.