Skip to main content
Download PDF

Wasted research when systematic reviews fail to provide a complete and up-to-date evidence synthesis: the example of lung cancer

BMC Medicine volume 14, Article number: 8 (2016) Cite this article

A Commentary to this article was published on 29 March 2016

Abstract

Background

Multiple treatments are frequently available for a given condition, and clinicians and patients need a comprehensive, up-to-date synthesis of evidence for all competing treatments. We aimed to quantify the waste of research related to the failure of systematic reviews to provide a complete and up-to-date evidence synthesis over time.

Methods

We performed a series of systematic overviews and networks of randomized trials assessing the gap between evidence covered by systematic reviews and available trials of second-line treatments for advanced non-small cell lung cancer. We searched the Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, MEDLINE, EMBASE, and other resources sequentially by year from 2009 to March 2, 2015. We sequentially compared the amount of evidence missing from systematic reviews to the randomized evidence available for inclusion each year. We constructed cumulative networks of randomized evidence over time and evaluated the proportion of trials, patients, treatments, and treatment comparisons not covered by systematic reviews on December 31 each year from 2009 to 2015.

Results

We identified 77 trials (28,636 patients) assessing 47 treatments with 54 comparisons and 29 systematic reviews (13 published after 2013). From 2009 to 2015, the evidence covered by existing systematic reviews was consistently incomplete: 45 % to 70 % of trials; 30 % to 58 % of patients; 40 % to 66 % of treatments; and 38 % to 71 % of comparisons were missing. In the cumulative networks of randomized evidence, 10 % to 17 % of treatment comparisons were partially covered by systematic reviews and 55 % to 85 % were partially or not covered.

Conclusions

We illustrate how systematic reviews of a given condition provide a fragmented, out-of-date panorama of the evidence for all treatments. This waste of research might be reduced by the development of live cumulative network meta-analyses.

Peer Review reports

Background

For many conditions, multiple competing treatments are available, many of which have been assessed in randomized trials [1]. Clinicians and patients who are making medical decisions need to know which treatment works best among all treatments for the condition of interest. They increasingly turn to systematic reviews and meta-analyses for current evidence-based assessments of the relative benefits and harms of treatments.

To decide the best treatment for a patient, clinicians and patients need a comprehensive, up-to-date synthesis of evidence for all treatments available for a given condition [2–4]. This synthesis could be provided by considering the whole set of conventional meta-analyses on all treatment comparisons or a network meta-analysis [5].

However, systematic reviews as currently performed may fail to meet clinicians’ and patients’ needs [6]. Systematic reviews and meta-analyses are insufficiently informative if they do not cover all alternative treatments or do not include all available current evidence. In fact, most meta-analyses have a narrow scope and focus on specific treatments [7]. Moreover, many meta-analyses become quickly out-of-date because clinically important evidence can accumulate rapidly, but updating a systematic review can be as costly and time-consuming as the original review [8, 9]. This failure to rigorously synthesize the totality of relevant evidence may have a detrimental effect on treatment decisions and future research planning.

The exponential growth in publications of randomized trials, especially in oncology, increases clinicians’ and patients’ need for broad meta-analyses encompassing all the evidence for all competing treatments [10]. Lung cancer, in particular, remains the fifth leading cause of disability-adjusted life years in developed countries and represents a key area of current therapeutic innovation [11]. With recent progresses in therapeutics, the number of patients with advanced non-small cell lung cancer (NSCLC) who receive second-line treatments is increasing, but which second-line treatment to recommend is unclear.

We used the example of NSCLC to quantify the waste of research related to systematic reviews failing to provide a complete and up-to-date synthesis of evidence over time.

Methods

We first used a comprehensive strategy to repeatedly identify all randomized trials, with published and unpublished results, and all systematic reviews of second-line treatments for advanced NSCLC available up to the end of each year from 2009 to 2015. Second, we sequentially assessed the amount of randomized evidence that was covered by systematic reviews collectively: for the years 2009 to 2015, we assessed the articles published up to December 31 of each of those years for proportion of treatments, treatment comparisons, trials, and patients covered by systematic reviews on this topic, with comparison to the total randomized evidence available at each time.

Identification of randomized trials

Eligibility criteria

We considered randomized trials of second-line treatments compared to each other or against a placebo or best supportive care in patients with advanced (stage IIIB–IV) NSCLC. We considered any cytotoxic monochemotherapy, any targeted treatment, any combination of a cytotoxic monochemotherapy and targeted treatment, and any combination of two targeted treatments (complete list in Additional file 1: Appendix 1). We excluded trials assessing doublet chemotherapy and comparing two different administration schemes. We excluded trials focusing exclusively on patients with epidermal growth factor receptor (EGFR)-activating mutation or anaplastic lymphoma kinase (ALK) rearrangement, because it represented a specific minority sub-group of all advanced NSCLC.

Search strategy

We searched for reports of randomized trials in the Cochrane Central Register of Controlled Trials, MEDLINE, and EMBASE (search equations in Additional file 1: Appendix 2) with no restriction on language, status, or year of publication, and searched other resources [12]: 1) previous systematic reviews (see below); 2) reference lists of all selected trials; 3) conference abstracts (from the American Society of Clinical Oncology Meeting, European Society of Medical Oncology Congress, and World Lung Cancer Conference); 4) non-industry trial registries and results databases (ClinicalTrials.gov and EudraCT); 5) industry trial registries and results databases; and 6) regulatory agency online databases (US Food and Drug Administration and European Medicines Agency); details in Additional file 1: Appendix 3. We contacted trialists to request complete results for all trials identified as “completed” on ClinicalTrials.gov but without published results and for all trials with conference abstracts but no full-text articles. The last search was conducted on March 2, 2015.

Identification of systematic reviews

Eligibility criteria

Systematic reviews of randomized trials of second-line treatments for advanced NSCLC were eligible. We selected reviews that addressed at least one comparison between the treatments considered previously, whether they included a meta-analysis or not. We excluded reviews that did not report clearly stated objectives, eligibility criteria for trials, or a systematic search strategy; reviews combining first- and second-line data were ineligible; reviews focusing exclusively on patients with EGFR mutation or ALK rearrangement or on doublet chemotherapy, or comparing two different administration schemes; and reviews that did not provide the list of included trials.

Search strategy

We searched the Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effects, MEDLINE, EMBASE (search equations in Additional file 1: Appendix 4), and other resources: conference abstracts from the American Society of Clinical Oncology Meeting, European Society of Medical Oncology Congress, and World Lung Cancer Conference; and the PROSPERO international prospective register of systematic reviews for completed or published systematic reviews (details in Additional file 1: Appendix 5). There was no restriction on language, status, or year of publication. The last search was conducted on March 2, 2015.

Selection of studies and extraction of data

Two authors independently and in duplicate examined titles, abstracts, and full-text articles to determine the eligibility of randomized trials and systematic reviews. We pilot-tested the eligibility criteria on a sample of 100 records (for the selection on titles and abstracts) and 10 reports (for the selection on full-text articles) to ensure that the selection criteria were applied consistently by the two authors. Disagreements were discussed with a third author. All data were independently extracted by two authors who used a standardized form.

For each trial, we extracted the dates of publication of the full-text article(s) (online publication, if any) and conference abstract(s), date of results posting on non-industry and industry trial registries, and date of publication of reports by regulatory agencies. We also extracted the treatments assessed, number of randomized patients in each arm, study phase (II or III), and reported outcomes (overall survival and progression-free survival).

For each systematic review, we extracted the list of relevant trials selected, the publication date (online publication, if any), date of last search of trials, number of trials included, and type of treatment compared. We assessed the funding source (industry, non-industry, no funding, or not reported), whether the review was an update of a previously published review, and whether a network meta-analysis was performed. Finally, we assessed the scope of each systematic review (i.e., interventions and comparators assessed). We identified whether the review focused on one specific treatment explicitly (alone or combined with other treatments), lumped different treatments of the same type together (e.g., monochemotherapy or targeted therapy), and lumped different types of treatments together (e.g., monochemotherapy and EGFR tyrosine kinase inhibitors considered the same intervention).

Two reviewers independently assessed the methodological quality of the systematic reviews, with a formal consensus process in case of disagreement. We used AMSTAR, a measurement tool created to assess the methodological quality of systematic reviews [13], which has been validated [14, 15]. We assessed the four items pertaining to duplicate study selection and data extraction, comprehensive literature search (at least two electronic sources and one supplementary strategy among reviews, experts, or reviewing the references), searching for reports regardless of their publication type, and providing a list of included and excluded trials. Searching for trials regardless of their publication type was judged inadequate when authors did not report searching the grey literature (conference abstracts, non-industry trial registries and results databases, industry trial registries and results databases, regulatory agency online databases) or excluding reports based on language. We focused on these four specific items because the methods used for the identification and selection of studies are directly related to a potential gap between the amount of randomized evidence covered by systematic reviews and the amount of randomized evidence available for inclusion, and other domains are unrelated.

Definition of randomized evidence available for inclusion in systematic reviews

We pre-specified the year 2009 as a starting point for our analyses in order to allow for a sufficient amount of evidence (in terms of both randomized trials and available systematic reviews) regarding the comparison between competing second-line treatments for advanced NSCLC to initiate a comparison between the available randomized evidence and that covered by systematic reviews. From 2009 to 2015, we identified the cumulative list of trials eligible for inclusion in systematic reviews; we checked that each trial identified would have been eligible for inclusion in at least one systematic review (i.e., corresponded to the selection criteria in terms of patients, interventions, and comparators). For each trial, we identified the earliest report of results and considered the corresponding publication date as when the trial became eligible for inclusion in systematic reviews. Considering the inevitable time lag between completion and publication, most recently published trials could not be selected by any systematic review, so we considered a 6-month lag period as recommended by the Cochrane Collaboration (i.e., we listed all trial results published up to July 1 each year, and up to August 31, 2014 for 2015) [16]. We also compiled the cumulative list of treatments and treatment comparisons assessed in eligible trials; finally, we calculated the cumulative number of patients included in trials as a measure of the available amount of randomized evidence.

Definition of randomized evidence covered by systematic reviews

We considered all systematic reviews published up to December 31 each year from 2009 to 2015 (up to March 2 for 2015). The reference date for a systematic review was the publication date of the full-text article or online publication date, if any. We compiled the cumulative list of all relevant trials selected by these systematic reviews and the cumulative list of treatments and treatment comparisons and cumulative number of included patients in the trials selected by the systematic reviews.

Assessment of randomized evidence not covered by systematic reviews

We evaluated the overall number and proportion of treatments, treatment comparisons, trials, and patients not covered by systematic reviews from 2009 to 2015.

We constructed cumulative networks of randomized evidence. Each node was a treatment and each edge was a treatment comparison (i.e., an edge connected two nodes when at least one randomized trial compared the two corresponding treatments). In multi-arm trials, doses of the same drug were lumped under a common node. The node size was proportional to the total number of patients randomly allocated to the corresponding treatment across all randomized trials available for inclusion; we represented the proportions of randomized patients not actually covered by systematic reviews by pie charts overlaid on nodes in the network. The edge width was proportional to the total number of randomized trials between the corresponding treatments available for inclusion; we represented the proportions of trials not selected by systematic reviews by a percentage bar chart overlaid on edges in the network. The evidence for a treatment comparison was considered partially covered when systematic reviews did not cover all the evidence available for this treatment comparison.

In sensitivity analyses, we discarded trials potentially ineligible for inclusion in any systematic review: trials of drugs that did not successfully pass phase II; trials that did not report treatment effects on overall survival or progression-free survival; and trials with results reported in conference abstracts only. In a last sensitivity analysis, the lag period to define randomized evidence available for inclusion in systematic reviews was defined by the last date of search for the last published systematic review.

Analyses involved use of R version 3.2.1 (R Development Core Team, Vienna, Austria).

Results

Randomized trials of second-line treatments for NSCLC

We identified 77 eligible trials with results available between May 2000 and November 2014 (Fig. 1). The results of 62 trials were published in 69 articles; among these 62 trials, 43 had results reported in other sources (16 in conference abstracts, 10 in non-industry trial registries and results databases or regulatory agency online databases, and 17 in both). The remaining 15 trials (20 %) had unpublished results (12 conference abstracts, 2 non-industry trial registries and results databases or regulatory agency online databases, and 1 both). In all, 61 trials (79 %) were registered at ClinicalTrials.gov. The 77 trials included 28,636 patients (median [Q1–Q3] 168 [100–559] patients); 72 included two arms, two compared three different treatments, and three compared one drug to two different doses of another drug, which were lumped together. Fig. 2 shows the complete network of evidence, with 45 different treatments and 54 treatment comparisons against each other or best supportive care or placebo. With 47 nodes in the network, there were 47 × 46/2 = 1,081 possible pairwise comparisons. The available direct evidence informs 54 comparisons (5 %).

Fig. 1
figure 1

Flow diagram of selection of systematic reviews and randomized controlled trials of second-line treatments in advanced non-small cell lung cancer. *Additional full-text articles not identified by searching bibliographical databases; $63 full-text articles and 44 conference abstracts; ¤69 full-text articles, 70 conference abstracts, 28 posted results, and 5 industry/FDA reports

Full size image
Fig. 2
figure 2

Network of 77 randomized controlled trials of second-line treatments in advanced non-small cell lung cancer. The thickness of connecting lines indicates the number of available comparisons. The size of each node is proportional to the number of patients allocated to the corresponding treatment. AFL: aflibercept; AMR: amrubicin; ARQ197: tivantinib; BEV: bevacizumab; BIBF1120: nintedanib; BSC: best supportive care; CET: cetuximab; DAC: dacomitinib; DOC: docetaxel; ERL: erlotinib; EVE: everolimus; FIGI: figitumumab; FULV: fulvestrant; GEF: gefitinib; ICO: icotinib; MAT: matuzumab; MK-0646: dalotuzumab; ONA: onartuzumab; PAZ: pazopanib; PBO: placebo; PDX: pralatrexate; PEM: pemetrexed; PTX: paclitaxel; RAM: ramucirumab; SEL: selumetinib; SOR: sorafenib; SUN: sunitinib; TOP: topotecan; TRA: trametinib; VAN: vandetanib; VFL: vinflunine; VIN: vinorelbine

Full size image

Systematic reviews of second-line treatments for NSCLC

We identified 29 systematic reviews published from April 2001 to February 2015 (Table 1) [17–45]. The first published review was a Cochrane review and all subsequent reviews were published in journal venues. The number of reviews doubled in 2014, from 16 to 27 (Additional file 1: Appendix 6). The median time between the last review search and publication was 9 months [Q1–Q3 5–13 months]. The industry was involved in 10 % of the reviews and not involved in 41 %, and the funding source was unclear in 41 %. The reviews addressed 19 comparisons, with 26 reviews lumping different treatments or different types of treatments together, considering them as the same intervention or comparator. Six reviews focused on one specific treatment of main interest. Two reviews performed network meta-analyses but did not cover all available treatments. Only one of the 29 reviews was an update of a previously published review.

Table 1 Characteristics of 29 selected systematic reviews
Full size table

Regarding the methodological quality of the 29 systematic reviews, 45 % of reviews lacked information on independent study selection and data extraction, 31 % a comprehensive literature search, and 45 % a search for reports regardless of their publication type. Of note, 17 % of reviews lacked information on a search for conference abstracts, 86 % a search for non-industry trial registries and results databases, 97 % a search for industry trial registries and results databases, and 97 % a search of regulatory agency online databases. In all, 79 % of systematic reviews did not report duplicate study selection and data extraction, comprehensive literature search, and searching for reports regardless of their publication type. Finally, 7 % of reviews provided a list of included and excluded trials.

Randomized evidence not covered or partially covered by systematic reviews

From 2009 to 2015, the amount of randomized evidence covered by existing systematic reviews was consistently incomplete: 40 % to 66 % of treatments; 38 % to 71 % of treatment comparisons; 45 % to 70 % of trials; and 30 % to 58 % of patients were missing (Fig. 3). In 2014, 27 reviews still did not cover 18 treatments (40 %), 20 treatment comparisons (38 %), 34 trials (46 %), and 8,486 patients (30 %).

Fig. 3
figure 3

Amount of treatments, treatment comparisons, trials, and patients not covered by systematic reviews from 2009 to 2015. *The last search for randomized trials and systematic reviews was conducted on March 2, 2015

Full size image

The cumulative networks of evidence show how the mismatch between available data and syntheses persisted from 2009 to 2015 (Fig. 4). Across all years, 10 % to 17 % of treatment comparisons were partially covered by systematic reviews and 55 % and 85 % were partially or not covered by systematic reviews. Moreover, the proportion of evidence covered by systematic reviews was unequally distributed across treatments.

Fig. 4
figure 4

Cumulative networks of evidence showing the gap between the amount of randomized evidence covered by systematic reviews and the amount of randomized evidence available for inclusion. (a) 2009–2012 and (b) 2013–2015. *The last search for randomized trials and systematic reviews was conducted on March 2, 2015. From 2009 to 2015, we compared randomized controlled trials selected by systematic reviews published up to December 31 each year (up to March 2 for 2015) to all trials eligible for inclusion (i.e., all trial results published up to July 1 each year [up to August 31, 2014 for 2015]). Each node size is proportional to the total number of patients randomly allocated to the corresponding treatment across all randomized trials available for inclusion; we represented the proportion of randomized patients actually covered by systematic reviews by pie charts overlaid on nodes in the network. The thickness of each edge is proportional to the total number of randomized controlled trials between the corresponding treatments available for inclusion; we represented the proportion of trials actually selected by systematic reviews by a percentage bar chart overlaid on edges in the network

Full size image

Sensitivity analyses gave consistent findings. When removing trials of drugs that did not pass phase II, 27 % to 64 % of trials, 27 % to 58 % of patients, 22 % to 53 % of treatments, and 21 % to 67 % of treatment comparisons were missing. When removing trials that did not report overall survival or progression-free survival, 41 % to 68 % of trials, 29 % to 58 % of patients, 36 % to 63 % of treatments, and 36 % to 69 % of treatment comparisons were missing. When discarding trials reported in conference abstracts only, 37 % to 68 % of trials, 24 % to 58 % of patients, 33 % to 63 % of treatments, and 33 % to 68 % of treatment comparisons were missing. When the lag period was the last date of search of the last systematic review, 42 % to 70 % of trials, 11 % to 53 % of patients, 36 % to 66 % of treatments, and 37 % to 72 % of treatment comparisons were missing.

Discussion

In this study, we assessed whether the whole set of conventional meta-analyses on pairwise treatment comparisons would allow for meeting clinicians’ and patients’ needs, to provide a comprehensive, up-to-date synthesis of evidence for all treatments. Our comparison of the amount of randomized evidence covered by systematic reviews and all randomized trials available for inclusion revealed a substantial waste related to the failure of systematic reviews to accumulate evidence scientifically: the evidence covered by existing systematic reviews on the topic was always substantially incomplete, with 40 % or more of treatments, treatment comparisons, and trials missing.

All meta-analyses on the same topic evaluating only a small fragment of the evidence has consequences for patient care and research planning. We need to identify the treatments with harmful effects and also treatments with side effects that have no advantage as compared with alternative treatments. As well, we need to be able to identify the treatment or group of treatments that works best. The relative beneficial effect of a treatment may be missed if some treatment comparisons of interest are not covered by systematic reviews. In our example, no systematic review encompassed all available treatments. Moreover, encompassing all the evidence for all treatments may have important implications for planning subsequent trials and helping prioritize future research to improve the evidence base [46]. In fact, based on the synthesis results and the geometry of the network of evidence, one could design a trial of treatments infrequently compared or a trial of the best potential treatment [47, 48].

Several reasons explain why the 29 systematic reviews did not cover all the randomized evidence. First, the trials may have addressed narrow and focused questions as compared to each review’s selection criteria (patients, interventions, and comparators). However, all trials we identified would have been eligible for inclusion in at least one systematic review; missing trials were not excluded from systematic reviews because they were not eligible.

Second, many meta-analyses become quickly out-of-date, sometimes by the time they are published. In fact, the systematic identification of trials is complex and time-consuming. In our case study, only one of 29 reviews was an update of a previous review and a large proportion of treatment comparisons covered by systematic reviews was out-of-date. This updating issue may be improved by the automation, as much as possible, of the whole trial search and selection process. These automated technologies would alleviate the burden on systematic reviewers. Third, another reason for the failure of systematic reviews to accumulate all available evidence is inadequate search methods for unpublished trials. In our case study, among the 15 trials (20 %) with unpublished results, only three (20 %) were included in systematic reviews. Among the 29 systematic reviews, 79 % could be considered at high risk of missing trials that would have met the inclusion criteria because they did not report duplicate study selection and data extraction, comprehensive literature search, or searching for reports regardless of publication type. Automated processes, such as meta-search engines, could systematically cover sources such as industry and non-industry trial registries and results databases to identify posted results and improve the evidence synthesis [49, 50]. Beyond the example at hand, the current way of conducting systematic reviews explains why they are inherently at risk of providing a fragmented, out-of-date panorama of the evidence for all treatments.

A broader scope on all evidence available for all treatments of a given condition is naturally provided by a systematic review with network meta-analysis, which allows for examining the totality of the randomized evidence using trial networks [51, 52]. Although the production of network meta-analyses follows an exponential growth, their number is still relatively limited [53]. With the increasing development of new treatments, examining networks of randomized trials is essential, and multiple treatment comparisons cannot be avoided [54]. Conventional meta-analysis focuses on comparisons of two treatments only. However, network meta-analysis allows for comparing all treatments to each other with direct and indirect comparisons, even if randomized trials are not available for some treatment comparisons. In our example, the available direct evidence informed 5 % of all possible comparisons between the 47 treatments. Ideally, the scoping exercise to define the criteria for considering trials in the systematic review should include all alternative treatments for the target condition [55]. It is still possible that a network meta-analysis will selectively choose treatments to include in the network, and excluding treatments may affect estimated treatment effects [56]. In our study, we identified two network meta-analyses; neither covered the whole evidence available at their time of analysis and considered a very restrictive network instead. Finally, network meta-analysis offers more flexibility to assess individual treatments. It offers the opportunity to overcome a common issue in standard meta-analysis whereby different treatments and sometimes different types of treatments are lumped together. In our example, erlotinib and gefitinib were frequently lumped together in EGFR tyrosine kinase inhibitors and docetaxel and pemetrexed were lumped together in monochemotherapy. This type of evidence lumping does not provide the relevant information for clinicians and patients who want to know which specific treatments work the best.

The paradox is that the set of systematic reviews fail to cover all the evidence, diminishing the value of research to clinicians by missing important comparisons. In fact, the production of meta-analyses has been substantially and rapidly increasing worldwide [57, 58]. In parallel, the evidence covered by systematic reviews features many gaps, and multiple overlapping meta-analyses on the same topic are common [59, 60].

As an alternative to classical systematic reviews performed at one point in time, a new theoretical framework is “living systematic reviews”, defined as high-quality online summaries of health research updated as new research becomes available [61]. We propose to push further the shift towards a new paradigm by switching: 1) from a series of standard meta-analyses focused on specific treatments (many treatments being not considered) to a single network meta-analysis covering all treatments; and 2) from meta-analyses performed at a given time and frequently out-of-date to a cumulative network meta-analysis systematically updated as soon as the results of a new trial become available, an approach to synthesis we call “live cumulative network meta-analysis”. In Fig. 5, we show the methodological steps we propose for live cumulative network meta-analysis.

Fig. 5
figure 5

A new approach to synthesize evidence: live cumulative network meta-analysis. Starting from an initial NMA, a research community would regularly (e.g., every 3 months), search for, screen, and select trials with new results and, if any, extract data, assess the risk of bias, and update the NMA. NMA: network meta-analysis

Full size image

We acknowledge that developing such methodology is challenging. In Table 2, we present some key challenges and potential solutions. The rigorous methodology of systematic reviews (exhaustive search of trials, minimizing subjectivity by independent duplicate assessments, assessing risk of bias within trials) is inherently demanding of resources and time, especially for a systematic review incorporating network meta-analysis. Moreover, keeping a systematic review up-to-date requires processes closer to those of rapid reviews (i.e., using accelerated and streamlined methods). Therefore, there is necessarily a trade-off between high-standard synthesis methods and real-time updating processes. Automated technologies may help define this trade-off by alleviating the burden of manual tasks for systematic reviewers. Several tools have been proposed to improve, hasten, and ease the search for trials, trial selection, extraction of data, and assessment of risk of bias [49, 50, 62, 63]. Live cumulative network meta-analysis may also raise issues regarding the current authoring and publishing system. Online posting may be more adequate to report periodically the findings of such “real-time” syntheses. Since Elliott et al. presented the theoretical framework of living systematic reviews, some examples have been published and have only partly addressed the aforementioned challenges [64–68]; for instance, by using accompanying open-access websites to disseminate the updates of the systematic review.

Table 2 Methodological steps of live cumulative network meta-analysis, key challenges, and potential solutions
Full size table

Another challenge would be to consider all outcomes that are important or critical to patients for decision making in these live cumulative network meta-analyses [69, 70]. In our case study, we included trials regardless of reported outcomes; in a sensitivity analysis, we excluded trials that did not report treatment effects on overall survival or progression-free survival. However, there are other patient-important outcomes, in particular to measure the symptom burden of the disease and the quality of life of patients. More generally, it will be crucial to consider networks of trials according to the reporting of the different patient-important outcomes. In fact, the geometry of the network of trials could vary across outcomes because of differential reporting of outcomes (e.g., efficacy and safety outcomes) across drugs and trials.

Nonetheless, embracing the perspective of networks of trials of all alternative treatments for each condition, and in particular developing live cumulative network meta-analyses, could greatly benefit various stakeholders, including physicians, patients, and also guideline developers, funders, and decision-makers [71]. Networks of trials and their synthesis through network meta-analysis could increase the value of research when treatment recommendations are based on an exhaustive up-to-date network of randomized evidence [72]. Guideline developers and other decision-makers may further benefit from network meta-analyses if these implement recent developments to rate the quality of the body of evidence supporting treatment effect estimates for all patient-important outcomes (e.g. GRADE Summary of Findings tables) and rankings from network meta-analysis. For instance, in the field of medical therapies for open angle glaucoma, Li et al. recently showed that if a network meta-analysis had been conducted earlier, prostaglandins could have been shown as the most effective class in lowering intraocular pressure 7 years ahead of the guideline recommendation [73]. Moreover, biomedical research funding agencies could improve the prioritization of research proposals if they had access to a mapping of existing (and ongoing) trials evaluating all available treatments for a specific condition at the time a new trial is planned. Visualizing the network of trials and identifying which new trial maximizes the information can help stakeholders assess gaps in evidence and choose the next treatment comparison or trial that needs to be prioritized.

To our best knowledge, our study is the first to highlight the substantial waste associated with the failure of systematic reviews to accumulate evidence across all treatments (for the same disease). Moreover, our methodology based on a series of systematic overviews and networks of randomized evidence is novel and could be replicated in other fields. Our study has several limitations. First, we examined only one topic, second-line treatments for advanced NSCLC. However, our results should be generalizable, because the scope of meta-analyses is frequently narrow, with 81 % of standard meta-analyses that do not include all treatments and 43 % that cover only specific treatments [7]. Contrary to Haidich and colleagues, who assessed the evidence at the level of each systematic review, we assessed the cumulative evidence covered by all systematic reviews on a topic with a “bird’s-eye view”. Second, our gold standard included trials that meta-analysts may have considered ineligible for inclusion in their systematic reviews. However, we performed several sensitivity analyses and results were consistent. Third, we excluded trials in which the control group received different chemotherapy drugs at the discretion of the investigators. Such trials could also contribute to the synthesis in a network meta-analysis with a class effect model [74]; however, they do not allow for assessing the relative merits of the specific drugs. Similarly, we excluded randomized trials and systematic reviews comparing two different administration schemes because our focus was the comparison of alternative treatments against each other. Nonetheless, our analysis could be extended to such randomized trials and systematic reviews. In such cases, each relevant node would have several subnodes that relate to different administration schemes. Fourth, we may have missed some systematic reviews and randomized trials because of reporting bias, but we tried to minimize this possibility by an exhaustive search covering conference abstracts and registries. Fifth, we started our analysis for the year 2009; this pre-specified year was somehow arbitrary and we acknowledge that this starting point could have been earlier, considering that the first systematic review was published in April 2001; the second one was published in December 2005. However, there was no or little randomized evidence available between competing treatments at that time. Finally, we did not perform any outcome data synthesis at this stage. One may ask if covering all the randomized evidence available would have led to clinically important differences for the 10 % to 17 % of treatment comparisons partially covered by systematic reviews. In the framework we are promoting, a network meta-analysis would allow for estimating all treatment comparisons. Adding up to 40 % of missing evidence (about 8,000 patients) to the network would likely lead to clinically important differences, in particular for treatment comparisons partially covered by systematic reviews.

Conclusions

We illustrated that systematic reviews of a given condition provide a fragmented, out-of-date panorama of the evidence for all treatments. Embracing the perspective of networks of trials of all alternative treatments for each condition can have important consequences and should be adopted more generally. The waste of research associated with the failure to accumulate evidence could be reduced by the development of live cumulative network meta-analyses.

Abbreviations

ALK:

anaplastic lymphoma kinase

EGFR:

epidermal growth factor receptor

NSCLC:

non-small cell lung cancer

References

  1. Institute of Medicine. Initial national priorities for comparative effectiveness research. Washington, DC: The National Academies Press; 2009.

    Google Scholar 

  2. Gotzsche PC. Why we need a broad perspective on meta-analysis. It may be crucially important for patients. BMJ. 2000;321(7261):585–6.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Jefferson T, Doshi P, Thompson M, Heneghan C. Ensuring safe and effective drugs: who can do what it takes? BMJ. 2011;342:c7258.

    Article  PubMed  Google Scholar 

  4. Liberati A. Need to realign patient-oriented and commercial and academic research. Lancet. 2011;378(9805):1777–8.

    Article  PubMed  Google Scholar 

  5. Ioannidis JP. Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses. CMAJ. 2009;181(8):488–93.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Naci H, Ioannidis JP. How good is “evidence” from clinical studies of drug effects and why might such evidence fail in the prediction of the clinical utility of drugs? Annu Rev Pharmacol Toxicol. 2015;55:169–89.

    Article  CAS  PubMed  Google Scholar 

  7. Haidich AB, Pilalas D, Contopoulos-Ioannidis DG, Ioannidis JP. Most meta-analyses of drug interventions have narrow scopes and many focus on specific agents. J Clin Epidemiol. 2013;66(4):371–8.

    Article  PubMed  Google Scholar 

  8. Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, Sampson M, et al. When and how to update systematic reviews. Cochrane Database Syst Rev. 2008;1:MR000023.

    PubMed  Google Scholar 

  9. Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med. 2007;147(4):224–33.

    Article  PubMed  Google Scholar 

  10. Ioannidis JP, Karassa FB. The need to consider the wider agenda in systematic reviews and meta-analyses: breadth, timing, and depth of the evidence. BMJ. 2010;341:c4875.

    Article  PubMed  Google Scholar 

  11. Global Burden of Disease Cancer Collaboration, Fitzmaurice C, Dicker D, Pain A, Hamavid H, Moradi-Lakeh M, et al. The global burden of cancer 2013. JAMA Oncol. 2015;1(4):505–27.

    Article  PubMed Central  Google Scholar 

  12. Chan AW. Out of sight but not out of mind: how to search for unpublished clinical trial evidence. BMJ. 2012;344:d8013.

    Article  PubMed  Google Scholar 

  13. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z, et al. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS One. 2007;2(12):e1350.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62(10):1013–20.

    Article  PubMed  Google Scholar 

  16. Chandler J, Churchill R, Higgins J, Lasserson T, Tovey D. Methodological standards for the conduct of new Cochrane Intervention Reviews, Methodological expectations of Cochrane intervention reviews (MECIR). London: Cochrane Editorial Unit; 2013.

    Google Scholar 

  17. Al-Saleh K, Quinton C, Ellis PM. Role of pemetrexed in advanced non-small-cell lung cancer: meta-analysis of randomized controlled trials, with histology subgroup analysis. Curr Oncol. 2012;19(1):e9–e15.

    CAS  PubMed  PubMed Central  Google Scholar 

  18. Barlesi F, Jacot W, Astoul P, Pujol JL. Second-line treatment for advanced non-small cell lung cancer: a systematic review. Lung Cancer. 2006;51(2):159–72.

    Article  PubMed  Google Scholar 

  19. Bonfill X, Serra C, Sacristan M, Nogue M, Losa F, Montesinos J. Second-line chemotherapy for non-small cell lung cancer. Cochrane Database Syst Rev. 2002;2:CD002804.

    PubMed  Google Scholar 

  20. Cui J, Cai X, Zhu M, Liu T, Zhao N. The efficacy of bevacizumab compared with other targeted drugs for patients with advanced NSCLC: a meta-analysis from 30 randomized controlled clinical trials. PLoS One. 2013;8(4):e62038.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. Gao G, Ren S, Li A, He Y, Chen X, Li W, et al. A meta-analysis of comparing EGFR-TKI with chemotherapy as the second-line treatment of NSCLC patients with wild-type EGFR. J Clin Oncol. 2013;31(15):suppl 1.

  22. Hawkins N, Scott DA, Woods BS, Thatcher N. No study left behind: a network meta-analysis in non-small-cell lung cancer demonstrating the importance of considering all relevant data. Value Health. 2009;12(6):996–1003.

    Article  PubMed  Google Scholar 

  23. Jiang J, Huang L, Liang X, Zhou X, Huang R, Chu Z, et al. Gefitinib versus docetaxel in previously treated advanced non-small-cell lung cancer: a meta-analysis of randomized controlled trials. Acta Oncol. 2011;50(4):582–8.

    Article  CAS  PubMed  Google Scholar 

  24. Jin Y, Sun Y, Shi X, Zhao J, Shi L, Hong W, et al. Meta-analysis to assess the efficacy and toxicity of docetaxel-based doublet compared with docetaxel alone for patients with advanced NSCLC who failed first-line treatment. Clin Ther. 2014;36(12):1980–90.

    Article  CAS  PubMed  Google Scholar 

  25. Lee JK, Hahn S, Kim DW, Suh KJ, Keam B, Kim TM, et al. Epidermal growth factor receptor tyrosine kinase inhibitors vs conventional chemotherapy in non-small cell lung cancer harboring wild-type epidermal growth factor receptor: a meta-analysis. JAMA. 2014;311(14):1430–7.

    Article  PubMed  Google Scholar 

  26. Li N, Yang L, Ou W, Zhang L, Zhang SL, Wang SY. Meta-analysis of EGFR tyrosine kinase inhibitors compared with chemotherapy as second-line treatment in pretreated advanced non-small cell lung cancer. PLoS One. 2014;9(7):e102777.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Li X, Wang H, Lin W, Xu Q. Efficacy of combining targeted therapy with pemetrexed or docetaxel as second-line treatment in patients with advanced non-small-cell lung cancer: a meta-analysis of 14 randomized controlled trials. Curr Med Res Opin. 2014;30(11):2295–304.

    Article  CAS  PubMed  Google Scholar 

  28. Liang W, Wu X, Hong S, Zhang Y, Kang S, Fang W, et al. Multi-targeted antiangiogenic tyrosine kinase inhibitors in advanced non-small cell lung cancer: meta-analyses of 20 randomized controlled trials and subgroup analyses. PLoS One. 2014;9(10):e109757.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Perez-Moreno MA, Galvan-Banqueri M, Flores-Moreno S, Villalba-Moreno A, Cotrina-Luque J, Bautista-Paloma FJ. Systematic review of efficacy and safety of pemetrexed in non-small-cell-lung cancer. Int J Clin Pharm. 2014;36(3):476–87.

    Article  CAS  PubMed  Google Scholar 

  30. Popat S, Mellemgaard A, Fahrbach K, Martin A, Rizzo M, Kaiser R, et al. Nintedanib plus docetaxel as second-line therapy in patients with non-small-cell lung cancer: a network meta-analysis. Future Oncol. 2015;11(3):409–20.

    Article  CAS  PubMed  Google Scholar 

  31. Qi WX, Shen Z, Lin F, Sun YJ, Min DL, Tang LN, et al. Comparison of the efficacy and safety of EFGR tyrosine kinase inhibitor monotherapy with standard second-line chemotherapy in previously treated advanced non-small-cell lung cancer: a systematic review and meta-analysis. Asian Pac J Cancer Prev. 2012;13(10):5177–82.

    Article  PubMed  Google Scholar 

  32. Qi WX, Shen Z, Yao Y. Meta-analysis of docetaxel-based doublet versus docetaxel alone as second-line treatment for advanced non-small-cell lung cancer. Cancer Chemother Pharmacol. 2012;69(1):99–106.

    Article  CAS  PubMed  Google Scholar 

  33. Qi WX, Tang LN, He AN, Shen Z, Yao Y. The role of vandetanib in the second-line treatment for advanced non-small-cell-lung cancer: a meta-analysis of four randomized controlled trials. Lung. 2011;189(6):437–43.

    Article  CAS  PubMed  Google Scholar 

  34. Qi WX, Tang LN, He AN, Shen Z, Yao Y. Effectiveness and safety of pemetrexed-based doublet versus pemetrexed alone as second-line treatment for advanced non-small-cell lung cancer: a systematic review and meta-analysis. J Cancer Res Clin Oncol. 2012;138(5):745–51.

    Article  PubMed  Google Scholar 

  35. Qi WX, Wang Q, Jiang YL, Sun YJ, Tang LN, He AN, et al. Overall survival benefits for combining targeted therapy as second-line treatment for advanced non-small-cell-lung cancer: a meta-analysis of published data. PLoS One. 2013;8(2):e55637.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  36. Sun CT, Xu X, Sheng W, Wang XW, Wen SL, Han JQ. A meta-analysis of pemetrexed-based doublet compared with pemetrexed alone for the second-line treatment of advanced non-small-cell lung cancer. Bratisl Lek Listy. 2014;115(4):233–7.

    CAS  PubMed  Google Scholar 

  37. Sun L, Ma JT, Zhang SL, Zou HW, Han CB. Efficacy and safety of chemotherapy or tyrosine kinase inhibitors combined with bevacizumab versus chemotherapy or tyrosine kinase inhibitors alone in the treatment of non-small cell lung cancer: a systematic review and meta-analysis. Med Oncol. 2015;32(2):473.

    Article  PubMed  Google Scholar 

  38. Tao L, Zhuo W, Yang F, Zhu B. Vandetanib for advanced non-small cell lung cancer: a meta-analysis. Zhongguo Fei Ai Za Zhi. 2012;15(3):172–8.

    PubMed  Google Scholar 

  39. Tassinari D, Scarpi E, Sartori S, Drudi F, Castellani C, Carloni F, et al. Noninferiority trials in second-line treatments of nonsmall cell lung cancer: a systematic review of literature with meta-analysis of phase III randomized clinical trials. Am J Clin Oncol. 2012;35(6):593–9.

    Article  PubMed  Google Scholar 

  40. Tassinari D, Scarpi E, Sartori S, Tamburini E, Santelmo C, Tombesi P, et al. Second-line treatments in non-small cell lung cancer. A systematic review of literature and metaanalysis of randomized clinical trials. Chest. 2009;135(6):1596–609.

    Article  CAS  PubMed  Google Scholar 

  41. Vale CL, Burdett S, Fisher DJ, Navani N, Parmar MK, Copas AJ, et al. Should tyrosine kinase inhibitors be considered for advanced non-small-cell lung cancer patients with wild type EGFR? Two systematic reviews and meta-analyses of randomized trials. Clin Lung Cancer. 2015;16(3):173–82. e174.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  42. Wong KM, Victor C, Eng L, Verma S. The use of epidermal growth factor receptor tyrosine kinase inhibitors in treatment of advanced EGFR wild-type non-small cell lung cancer: a meta-analysis study. J Thorac Oncol. 2013;8(suppl 2):S1184.

  43. Xiao BK, Yang JY, Dong JX, Ji ZS, Si HY, Wang WL, et al. Meta-analysis of seven randomized control trials to assess the efficacy and toxicity of combining EGFR-TKI with chemotherapy for patients with advanced NSCLC who failed first-line treatment. Asian Pac J Cancer Prev. 2015;16(7):2915–21.

    Article  PubMed  Google Scholar 

  44. Yang X, Yang K, Kuang K. The efficacy and safety of EGFR inhibitor monotherapy in non-small cell lung cancer: a systematic review. Curr Oncol Rep. 2014;16(6):390.

    Article  PubMed  Google Scholar 

  45. Zhao N, Zhang XC, Yan HH, Yang JJ, Wu YL. Efficacy of epidermal growth factor receptor inhibitors versus chemotherapy as second-line treatment in advanced non-small-cell lung cancer with wild-type EGFR: a meta-analysis of randomized controlled clinical trials. Lung Cancer. 2014;85(1):66–73.

    Article  PubMed  Google Scholar 

  46. Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM, et al. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65.

    Article  PubMed  Google Scholar 

  47. DeSantis SM, Zhu H. A Bayesian mixed-treatment comparison meta-analysis of treatments for alcohol dependence and implications for planning future trials. Med Decis Making. 2014;34(7):899–910.

    Article  PubMed  PubMed Central  Google Scholar 

  48. Nikolakopoulou A, Mavridis D, Salanti G. Using conditional power of network meta-analysis (NMA) to inform the design of future clinical trials. Biom J. 2014;56(6):973–90.

    Article  PubMed  Google Scholar 

  49. Tsafnat G, Dunn A, Glasziou P, Coiera E. The automation of systematic reviews. BMJ. 2013;346:f139.

    Article  PubMed  Google Scholar 

  50. Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Syst Rev. 2014;3:74.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Salanti G, Kavvoura FK, Ioannidis JP. Exploring the geometry of treatment networks. Ann Intern Med. 2008;148(7):544–53.

    Article  PubMed  Google Scholar 

  52. Hutton B, Salanti G, Caldwell DM, Chaimani A, Schmid CH, Cameron C, et al. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Ann Intern Med. 2015;162(11):777–84.

    Article  PubMed  Google Scholar 

  53. Bafeta A, Trinquart L, Seror R, Ravaud P. Analysis of the systematic reviews process in reports of network meta-analyses: methodological systematic review. BMJ. 2013;347:f3675.

    Article  PubMed  PubMed Central  Google Scholar 

  54. Caldwell DM, Ades AE, Higgins JP. Simultaneous comparison of multiple treatments: combining direct and indirect evidence. BMJ. 2005;331(7521):897–900.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Dias S, Welton NJ, Sutton AJ, Ades AE. Evidence synthesis for decision making 1: introduction. Med Decis Making. 2013;33(5):597–606.

    Article  PubMed  PubMed Central  Google Scholar 

  56. Mills EJ, Kanters S, Thorlund K, Chaimani A, Veroniki AA, Ioannidis JP. The effects of excluding treatments from network meta-analyses: survey. BMJ. 2013;347:f5195.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Ioannidis JP, Chang CQ, Lam TK, Schully SD, Khoury MJ. The geometric increase in meta-analyses from China in the genomic era. PLoS One. 2013;8(6):e65602.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  58. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326.

    Article  PubMed  PubMed Central  Google Scholar 

  59. Helfer B, Prosser A, Samara MT, Geddes JR, Cipriani A, Davis JM, et al. Recent meta-analyses neglect previous systematic reviews and meta-analyses about the same topic: a systematic examination. BMC Med. 2015;13(1):82.

    Article  PubMed  PubMed Central  Google Scholar 

  60. Siontis KC, Hernandez-Boussard T, Ioannidis JP. Overlapping meta-analyses on the same topic: survey of published studies. BMJ. 2013;347:f4501.

    Article  PubMed  PubMed Central  Google Scholar 

  61. Elliott JH, Turner T, Clavisi O, Thomas J, Higgins JP, Mavergames C, et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 2014;11(2):e1001603.

    Article  PubMed  PubMed Central  Google Scholar 

  62. Jonnalagadda SR, Goyal P, Huffman MD. Automating data extraction in systematic reviews: a systematic review. Syst Rev. 2015;4:78.

    Article  PubMed  PubMed Central  Google Scholar 

  63. Marshall IJ, Kuiper J, Wallace BC. Automating risk of bias assessment for clinical trials. IEEE J Biomed Health Inform. 2015;19(4):1406–12.

    Article  PubMed  Google Scholar 

  64. Badgett RG, Vindhyal M, Stirnaman JT, Gibson CM, Halaby R. A living systematic review of nebulized hypertonic saline for acute bronchiolitis in infants. JAMA Pediatr. 2015;169(8):788–9.

    Article  PubMed  Google Scholar 

  65. Synnot A, Gruen RL, Menon D, Steyerberg EW, Buki A, Peul W, et al. A new approach to evidence synthesis in traumatic brain injury: living systematic reviews. J Neurotrauma. 2015. Epub ahead of print. doi:10.1089/neu.2015.4124.

  66. Brazinova A, Rehorcikova V, Taylor MS, Buckova V, Majdan M, Psota M, et al. Epidemiology of traumatic brain injury in Europe: a living systematic review. J Neurotrauma. 2015. Epub ahead of print. doi:10.1089/neu.2015.4126

  67. Cnossen MC, Scholten AC, Lingsma H, Synnot A, Tavender E, Gantner D, et al. Adherence to guidelines in adult patients with traumatic brain injury: A living systematic review. J Neurotrauma. 2015. Epub ahead of print. doi:10.1089/neu.2015.4121

  68. Ters P, Badgett RG. A living meta-analysis of colchicine for pericarditis. Ann Pharmacother. 2014;48(10):1398–9.

    Article  PubMed  Google Scholar 

  69. Handbook for grading the quality of evidence and the strength of recommendations using the GRADE approach. http://gdt.guidelinedevelopment.org/central_prod/_design/client/handbook/handbook.html. Accessed 2 Dec 2015.

  70. Gartlehner G, Flamm M. Is the Cochrane Collaboration prepared for the era of patient-centred outcomes research? Cochrane Database Syst Rev. 2013;3:ED000054.

    PubMed  Google Scholar 

  71. Naci H, O’Connor AB. Assessing comparative effectiveness of new drugs before approval using prospective network meta-analyses. J Clin Epidemiol. 2013;66(8):812–6.

    Article  PubMed  Google Scholar 

  72. Puhan MA, Schünemann HJ, Murad MH, Li T, Brignardello-Petersen R, Singh JA, et al. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ. 2014;349:g5630.

    Article  PubMed  Google Scholar 

  73. Li T, Rouse B, Shi Q, Dickersin K. Working from all angles. The Inaugural REWARD/EQUATOR Conference: increasing value and reducing waste in biomedical research conference, 28–30 September 2015, Edinburgh, UK.

  74. Haas DM, Caldwell DM, Kirkpatrick P, McIntosh JJ, Welton NJ. Tocolytic therapy for preterm delivery: systematic review and network meta-analysis. BMJ. 2012;345:e6226.

    Article  PubMed  PubMed Central  Google Scholar 

  75. Smalheiser NR, Lin C, Jia L, Jiang Y, Cohen AM, Yu C, et al. Design and implementation of Metta, a metasearch engine for biomedical literature retrieval intended for systematic reviewers. Health Inf Sci Syst. 2014;2:1.

    Article  PubMed  PubMed Central  Google Scholar 

  76. OpenTrials. All the Data, on All the Trials, Linked. http://opentrials.net/. Accessed 2 Dec 2015.

  77. Elliott J, Sim I, Thomas J, Owens N, Dooley G, Riis J, et al. #CochraneTech: technology and the future of systematic reviews. Cochrane Database Syst Rev. 2014;9:ED000091.

    PubMed  Google Scholar 

  78. Embase Screening. http://screening.metaxis.com/EMBASE/login.php. Accessed 2 Dec 2015.

  79. Wallace BC, Trikalinos TA, Lau J, Brodley C, Schmid CH. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics. 2010;11:55.

    Article  PubMed  PubMed Central  Google Scholar 

  80. O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4:5.

    Article  PubMed  PubMed Central  Google Scholar 

  81. Miwa M, Thomas J, O'Mara-Eves A, Ananiadou S. Reducing systematic review workload through certainty-based screening. J Biomed Inform. 2014;51:242–53.

    Article  PubMed  PubMed Central  Google Scholar 

  82. Rathbone J, Hoffmann T, Glasziou P. Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Syst Rev. 2015;4:80.

    Article  PubMed  PubMed Central  Google Scholar 

  83. Brown AW, Allison DB. Using crowdsourcing to evaluate published scientific literature: methods and example. PLoS One. 2014;9(7):e100647.

    Article  PubMed  PubMed Central  Google Scholar 

  84. Cepeda MS, Lobanov V, Berlin JA. From ClinicalTrials.gov trial registry to an analysis-ready database of clinical trial results. Clin Trials. 2013;10(2):347–8.

    Article  PubMed  Google Scholar 

  85. Tasneem A, Aberle L, Ananth H, Chakraborty S, Chiswell K, McCourt BJ, et al. The database for aggregate analysis of ClinicalTrials.gov (AACT) and subsequent regrouping by clinical specialty. PLoS One. 2012;7(3):e33677.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  86. Maruani A, Boutron I, Baron G, Ravaud P. Impact of sending email reminders of the legal requirement for posting results on ClinicalTrials.gov: cohort embedded pragmatic randomized controlled trial. BMJ. 2014;349:g5579.

    Article  PubMed  PubMed Central  Google Scholar 

  87. openMetaAnalysis. http://openmetaanalysis.github.io/. Accessed 2 Dec 2015.

  88. Rising J, Hsiang S. A tool for distributed meta-analysis. Social Science Research Network; 2014. http://ssrn.com/abstract=2458129. Accessed 2 Dec 2015

  89. Nikolakopoulou A, Mavridis D, Salanti G. Prospective use of network metaanalysis using formal statistical monitoring. 36th Annual Conference of the International Society for Clinical Biostatistics, 23–27 August 2015, Utrecht, The Netherlands.

Download references

Acknowledgments

This study was supported by a grant LEGS POIX 2015 (Chancellerie des Universités de Paris). The funding source had no role in the design of this study, its execution, analyses, interpretation of the data, and decision to submit results. The authors thank Laura Smales (BioMedEditing, Toronto, Canada) for language revision of the manuscript and Elise Diard for the conception of the Figures.

Author information

Author notes

    Authors and Affiliations

    1. Centre de Recherche Epidémiologie et Statistique Sorbonne Paris Cité, INSERM U1153, Paris, France

      Perrine Créquit, Ludovic Trinquart, Amélie Yavchitz & Philippe Ravaud

    2. Université Paris Descartes – Sorbonne Paris Cité, Paris, France

      Perrine Créquit, Ludovic Trinquart, Amélie Yavchitz & Philippe Ravaud

    3. Assistance Publique-Hôpitaux de Paris, Hôpital Hôtel-Dieu, Centre d’Epidémiologie Clinique, Paris, France

      Ludovic Trinquart, Amélie Yavchitz & Philippe Ravaud

    4. Cochrane France, Paris, France

      Ludovic Trinquart & Philippe Ravaud

    5. Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, USA

      Philippe Ravaud

    Authors
    1. Perrine Créquit
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Ludovic Trinquart
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Amélie Yavchitz
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Philippe Ravaud
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Ludovic Trinquart.

    Additional information

    Competing interests

    The authors declare that they have no competing interests.

    Authors’ contributions

    PC was involved in the study conception, selection of trials, data extraction, data analysis, interpretation of results, and drafting the manuscript. LT was involved in the study conception, selection of trials, data extraction, data analysis, interpretation of results, and drafting the manuscript. AY was involved in the data extraction, interpretation of results, and drafting the manuscript. PR was involved in the study conception, interpretation of results, and drafting the manuscript. All authors read and approved the final manuscript.

    Perrine Créquit and Ludovic Trinquart contributed equally to this work.

    A comment to this article is available at http://dx.doi.org/10.1186/s12916-016-0596-4.

    Additional file

    Additional file 1:

    Details regarding the methods and results. Appendix 1. List of eligible treatments. Appendix 2. Search equations for randomized controlled trials. Appendix 3. Other sources searched to identify randomized controlled trials. Appendix 4. Search equations for systematic reviews. Appendix 5. Other sources searched to identify systematic reviews. Appendix 6. Cumulative number of systematic reviews of second-line treatments in advsanced non-small cell lung cancer from 2009 to March 2015. (DOC 223 kb)

    Rights and permissions

    Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

    Reprints and Permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Créquit, P., Trinquart, L., Yavchitz, A. et al. Wasted research when systematic reviews fail to provide a complete and up-to-date evidence synthesis: the example of lung cancer. BMC Med 14, 8 (2016). https://doi.org/10.1186/s12916-016-0555-0

    Download citation

    • Received:

    • Accepted:

    • Published:

    • DOI: https://doi.org/10.1186/s12916-016-0555-0

    Keywords

    • Meta-analysis as topic
    • Systematic reviews
    • Randomized controlled trials
    • Network meta-analysis
    • Non-small cell lung cancer

    BMC Medicine

    ISSN: 1741-7015

    Contact us