Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

Wasted research when systematic reviews fail to provide a complete and up-to-date evidence synthesis: the example of lung cancer

Contributed equally
BMC Medicine201614:8

https://doi.org/10.1186/s12916-016-0555-0

Received: 3 September 2015

Accepted: 7 January 2016

Published: 20 January 2016

The Commentary to this article has been published in BMC Medicine 2016 14:59

Abstract

Background

Multiple treatments are frequently available for a given condition, and clinicians and patients need a comprehensive, up-to-date synthesis of evidence for all competing treatments. We aimed to quantify the waste of research related to the failure of systematic reviews to provide a complete and up-to-date evidence synthesis over time.

Methods

We performed a series of systematic overviews and networks of randomized trials assessing the gap between evidence covered by systematic reviews and available trials of second-line treatments for advanced non-small cell lung cancer. We searched the Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, MEDLINE, EMBASE, and other resources sequentially by year from 2009 to March 2, 2015. We sequentially compared the amount of evidence missing from systematic reviews to the randomized evidence available for inclusion each year. We constructed cumulative networks of randomized evidence over time and evaluated the proportion of trials, patients, treatments, and treatment comparisons not covered by systematic reviews on December 31 each year from 2009 to 2015.

Results

We identified 77 trials (28,636 patients) assessing 47 treatments with 54 comparisons and 29 systematic reviews (13 published after 2013). From 2009 to 2015, the evidence covered by existing systematic reviews was consistently incomplete: 45 % to 70 % of trials; 30 % to 58 % of patients; 40 % to 66 % of treatments; and 38 % to 71 % of comparisons were missing. In the cumulative networks of randomized evidence, 10 % to 17 % of treatment comparisons were partially covered by systematic reviews and 55 % to 85 % were partially or not covered.

Conclusions

We illustrate how systematic reviews of a given condition provide a fragmented, out-of-date panorama of the evidence for all treatments. This waste of research might be reduced by the development of live cumulative network meta-analyses.

Keywords

Meta-analysis as topic Systematic reviews Randomized controlled trials Network meta-analysis Non-small cell lung cancer

Background

For many conditions, multiple competing treatments are available, many of which have been assessed in randomized trials [1]. Clinicians and patients who are making medical decisions need to know which treatment works best among all treatments for the condition of interest. They increasingly turn to systematic reviews and meta-analyses for current evidence-based assessments of the relative benefits and harms of treatments.

To decide the best treatment for a patient, clinicians and patients need a comprehensive, up-to-date synthesis of evidence for all treatments available for a given condition [24]. This synthesis could be provided by considering the whole set of conventional meta-analyses on all treatment comparisons or a network meta-analysis [5].

However, systematic reviews as currently performed may fail to meet clinicians’ and patients’ needs [6]. Systematic reviews and meta-analyses are insufficiently informative if they do not cover all alternative treatments or do not include all available current evidence. In fact, most meta-analyses have a narrow scope and focus on specific treatments [7]. Moreover, many meta-analyses become quickly out-of-date because clinically important evidence can accumulate rapidly, but updating a systematic review can be as costly and time-consuming as the original review [8, 9]. This failure to rigorously synthesize the totality of relevant evidence may have a detrimental effect on treatment decisions and future research planning.

The exponential growth in publications of randomized trials, especially in oncology, increases clinicians’ and patients’ need for broad meta-analyses encompassing all the evidence for all competing treatments [10]. Lung cancer, in particular, remains the fifth leading cause of disability-adjusted life years in developed countries and represents a key area of current therapeutic innovation [11]. With recent progresses in therapeutics, the number of patients with advanced non-small cell lung cancer (NSCLC) who receive second-line treatments is increasing, but which second-line treatment to recommend is unclear.

We used the example of NSCLC to quantify the waste of research related to systematic reviews failing to provide a complete and up-to-date synthesis of evidence over time.

Methods

We first used a comprehensive strategy to repeatedly identify all randomized trials, with published and unpublished results, and all systematic reviews of second-line treatments for advanced NSCLC available up to the end of each year from 2009 to 2015. Second, we sequentially assessed the amount of randomized evidence that was covered by systematic reviews collectively: for the years 2009 to 2015, we assessed the articles published up to December 31 of each of those years for proportion of treatments, treatment comparisons, trials, and patients covered by systematic reviews on this topic, with comparison to the total randomized evidence available at each time.

Identification of randomized trials

Eligibility criteria

We considered randomized trials of second-line treatments compared to each other or against a placebo or best supportive care in patients with advanced (stage IIIB–IV) NSCLC. We considered any cytotoxic monochemotherapy, any targeted treatment, any combination of a cytotoxic monochemotherapy and targeted treatment, and any combination of two targeted treatments (complete list in Additional file 1: Appendix 1). We excluded trials assessing doublet chemotherapy and comparing two different administration schemes. We excluded trials focusing exclusively on patients with epidermal growth factor receptor (EGFR)-activating mutation or anaplastic lymphoma kinase (ALK) rearrangement, because it represented a specific minority sub-group of all advanced NSCLC.

Search strategy

We searched for reports of randomized trials in the Cochrane Central Register of Controlled Trials, MEDLINE, and EMBASE (search equations in Additional file 1: Appendix 2) with no restriction on language, status, or year of publication, and searched other resources [12]: 1) previous systematic reviews (see below); 2) reference lists of all selected trials; 3) conference abstracts (from the American Society of Clinical Oncology Meeting, European Society of Medical Oncology Congress, and World Lung Cancer Conference); 4) non-industry trial registries and results databases (ClinicalTrials.gov and EudraCT); 5) industry trial registries and results databases; and 6) regulatory agency online databases (US Food and Drug Administration and European Medicines Agency); details in Additional file 1: Appendix 3. We contacted trialists to request complete results for all trials identified as “completed” on ClinicalTrials.gov but without published results and for all trials with conference abstracts but no full-text articles. The last search was conducted on March 2, 2015.

Identification of systematic reviews

Eligibility criteria

Systematic reviews of randomized trials of second-line treatments for advanced NSCLC were eligible. We selected reviews that addressed at least one comparison between the treatments considered previously, whether they included a meta-analysis or not. We excluded reviews that did not report clearly stated objectives, eligibility criteria for trials, or a systematic search strategy; reviews combining first- and second-line data were ineligible; reviews focusing exclusively on patients with EGFR mutation or ALK rearrangement or on doublet chemotherapy, or comparing two different administration schemes; and reviews that did not provide the list of included trials.

Search strategy

We searched the Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effects, MEDLINE, EMBASE (search equations in Additional file 1: Appendix 4), and other resources: conference abstracts from the American Society of Clinical Oncology Meeting, European Society of Medical Oncology Congress, and World Lung Cancer Conference; and the PROSPERO international prospective register of systematic reviews for completed or published systematic reviews (details in Additional file 1: Appendix 5). There was no restriction on language, status, or year of publication. The last search was conducted on March 2, 2015.

Selection of studies and extraction of data

Two authors independently and in duplicate examined titles, abstracts, and full-text articles to determine the eligibility of randomized trials and systematic reviews. We pilot-tested the eligibility criteria on a sample of 100 records (for the selection on titles and abstracts) and 10 reports (for the selection on full-text articles) to ensure that the selection criteria were applied consistently by the two authors. Disagreements were discussed with a third author. All data were independently extracted by two authors who used a standardized form.

For each trial, we extracted the dates of publication of the full-text article(s) (online publication, if any) and conference abstract(s), date of results posting on non-industry and industry trial registries, and date of publication of reports by regulatory agencies. We also extracted the treatments assessed, number of randomized patients in each arm, study phase (II or III), and reported outcomes (overall survival and progression-free survival).

For each systematic review, we extracted the list of relevant trials selected, the publication date (online publication, if any), date of last search of trials, number of trials included, and type of treatment compared. We assessed the funding source (industry, non-industry, no funding, or not reported), whether the review was an update of a previously published review, and whether a network meta-analysis was performed. Finally, we assessed the scope of each systematic review (i.e., interventions and comparators assessed). We identified whether the review focused on one specific treatment explicitly (alone or combined with other treatments), lumped different treatments of the same type together (e.g., monochemotherapy or targeted therapy), and lumped different types of treatments together (e.g., monochemotherapy and EGFR tyrosine kinase inhibitors considered the same intervention).

Two reviewers independently assessed the methodological quality of the systematic reviews, with a formal consensus process in case of disagreement. We used AMSTAR, a measurement tool created to assess the methodological quality of systematic reviews [13], which has been validated [14, 15]. We assessed the four items pertaining to duplicate study selection and data extraction, comprehensive literature search (at least two electronic sources and one supplementary strategy among reviews, experts, or reviewing the references), searching for reports regardless of their publication type, and providing a list of included and excluded trials. Searching for trials regardless of their publication type was judged inadequate when authors did not report searching the grey literature (conference abstracts, non-industry trial registries and results databases, industry trial registries and results databases, regulatory agency online databases) or excluding reports based on language. We focused on these four specific items because the methods used for the identification and selection of studies are directly related to a potential gap between the amount of randomized evidence covered by systematic reviews and the amount of randomized evidence available for inclusion, and other domains are unrelated.

Definition of randomized evidence available for inclusion in systematic reviews

We pre-specified the year 2009 as a starting point for our analyses in order to allow for a sufficient amount of evidence (in terms of both randomized trials and available systematic reviews) regarding the comparison between competing second-line treatments for advanced NSCLC to initiate a comparison between the available randomized evidence and that covered by systematic reviews. From 2009 to 2015, we identified the cumulative list of trials eligible for inclusion in systematic reviews; we checked that each trial identified would have been eligible for inclusion in at least one systematic review (i.e., corresponded to the selection criteria in terms of patients, interventions, and comparators). For each trial, we identified the earliest report of results and considered the corresponding publication date as when the trial became eligible for inclusion in systematic reviews. Considering the inevitable time lag between completion and publication, most recently published trials could not be selected by any systematic review, so we considered a 6-month lag period as recommended by the Cochrane Collaboration (i.e., we listed all trial results published up to July 1 each year, and up to August 31, 2014 for 2015) [16]. We also compiled the cumulative list of treatments and treatment comparisons assessed in eligible trials; finally, we calculated the cumulative number of patients included in trials as a measure of the available amount of randomized evidence.

Definition of randomized evidence covered by systematic reviews

We considered all systematic reviews published up to December 31 each year from 2009 to 2015 (up to March 2 for 2015). The reference date for a systematic review was the publication date of the full-text article or online publication date, if any. We compiled the cumulative list of all relevant trials selected by these systematic reviews and the cumulative list of treatments and treatment comparisons and cumulative number of included patients in the trials selected by the systematic reviews.

Assessment of randomized evidence not covered by systematic reviews

We evaluated the overall number and proportion of treatments, treatment comparisons, trials, and patients not covered by systematic reviews from 2009 to 2015.

We constructed cumulative networks of randomized evidence. Each node was a treatment and each edge was a treatment comparison (i.e., an edge connected two nodes when at least one randomized trial compared the two corresponding treatments). In multi-arm trials, doses of the same drug were lumped under a common node. The node size was proportional to the total number of patients randomly allocated to the corresponding treatment across all randomized trials available for inclusion; we represented the proportions of randomized patients not actually covered by systematic reviews by pie charts overlaid on nodes in the network. The edge width was proportional to the total number of randomized trials between the corresponding treatments available for inclusion; we represented the proportions of trials not selected by systematic reviews by a percentage bar chart overlaid on edges in the network. The evidence for a treatment comparison was considered partially covered when systematic reviews did not cover all the evidence available for this treatment comparison.

In sensitivity analyses, we discarded trials potentially ineligible for inclusion in any systematic review: trials of drugs that did not successfully pass phase II; trials that did not report treatment effects on overall survival or progression-free survival; and trials with results reported in conference abstracts only. In a last sensitivity analysis, the lag period to define randomized evidence available for inclusion in systematic reviews was defined by the last date of search for the last published systematic review.

Analyses involved use of R version 3.2.1 (R Development Core Team, Vienna, Austria).

Results

Randomized trials of second-line treatments for NSCLC

We identified 77 eligible trials with results available between May 2000 and November 2014 (Fig. 1). The results of 62 trials were published in 69 articles; among these 62 trials, 43 had results reported in other sources (16 in conference abstracts, 10 in non-industry trial registries and results databases or regulatory agency online databases, and 17 in both). The remaining 15 trials (20 %) had unpublished results (12 conference abstracts, 2 non-industry trial registries and results databases or regulatory agency online databases, and 1 both). In all, 61 trials (79 %) were registered at ClinicalTrials.gov. The 77 trials included 28,636 patients (median [Q1–Q3] 168 [100–559] patients); 72 included two arms, two compared three different treatments, and three compared one drug to two different doses of another drug, which were lumped together. Fig. 2 shows the complete network of evidence, with 45 different treatments and 54 treatment comparisons against each other or best supportive care or placebo. With 47 nodes in the network, there were 47 × 46/2 = 1,081 possible pairwise comparisons. The available direct evidence informs 54 comparisons (5 %).
Fig. 1

Flow diagram of selection of systematic reviews and randomized controlled trials of second-line treatments in advanced non-small cell lung cancer. *Additional full-text articles not identified by searching bibliographical databases; $63 full-text articles and 44 conference abstracts; ¤69 full-text articles, 70 conference abstracts, 28 posted results, and 5 industry/FDA reports

Fig. 2

Network of 77 randomized controlled trials of second-line treatments in advanced non-small cell lung cancer. The thickness of connecting lines indicates the number of available comparisons. The size of each node is proportional to the number of patients allocated to the corresponding treatment. AFL: aflibercept; AMR: amrubicin; ARQ197: tivantinib; BEV: bevacizumab; BIBF1120: nintedanib; BSC: best supportive care; CET: cetuximab; DAC: dacomitinib; DOC: docetaxel; ERL: erlotinib; EVE: everolimus; FIGI: figitumumab; FULV: fulvestrant; GEF: gefitinib; ICO: icotinib; MAT: matuzumab; MK-0646: dalotuzumab; ONA: onartuzumab; PAZ: pazopanib; PBO: placebo; PDX: pralatrexate; PEM: pemetrexed; PTX: paclitaxel; RAM: ramucirumab; SEL: selumetinib; SOR: sorafenib; SUN: sunitinib; TOP: topotecan; TRA: trametinib; VAN: vandetanib; VFL: vinflunine; VIN: vinorelbine

Systematic reviews of second-line treatments for NSCLC

We identified 29 systematic reviews published from April 2001 to February 2015 (Table 1) [1745]. The first published review was a Cochrane review and all subsequent reviews were published in journal venues. The number of reviews doubled in 2014, from 16 to 27 (Additional file 1: Appendix 6). The median time between the last review search and publication was 9 months [Q1–Q3 5–13 months]. The industry was involved in 10 % of the reviews and not involved in 41 %, and the funding source was unclear in 41 %. The reviews addressed 19 comparisons, with 26 reviews lumping different treatments or different types of treatments together, considering them as the same intervention or comparator. Six reviews focused on one specific treatment of main interest. Two reviews performed network meta-analyses but did not cover all available treatments. Only one of the 29 reviews was an update of a previously published review.
Table 1

Characteristics of 29 selected systematic reviews

Systematic review

Last search

Publication date

Number of trials

Funding source

Intervention

Comparator

Specific treatment

Different treatments lumped together

Different types of treatments lumped together

Bonfill 2002

Jul 2001

Apr 2001

1

Non-industry

CTx

PBO or BSC

No

Yes

No

Tassinari 2009

Jul 2008

Feb 2009

3

NR

CTx or EGFRTKI

BSC

No

Yes

Yes

Yang 2014

Dec 2013

May 2014

2

NR

EGFRTKI

PBO

No

Yes

No

Wong 2013

SMay 2012

Oct 2013

4

NR

EGFRTKI

CTx or PBO

No

Yes

Yes

Barlesi 2006

Feb 2005

Dec 2005

4

NR

DOC

CTx or BSC

No

Yes

Yes

Al-Saleh 2012

Jan 2010

Feb 2012

1

Industry

PEM

CTx

Yes

Yes

No

Perez-Moreno 2014

Apr 2012

Mar 2014

1

Non-industry

PEM

CTx

Yes

Yes

No

Jiang 2011

Feb 2010

Dec 2010

4

Non-industry

GEF

DOC

Yes

No

No

Qi 2012c

Mar 2012

Oct 2012

8

Non-industry

EGFRTKI

CTx

No

Yes

No

Gao 2013

NR

Jun 2013

3

NR

EGFRTKI

CTx

No

Yes

No

Lee 2014

Dec 2013

Apr 2014

7

Non-industry

EGFRTKI

CTx

No

Yes

No

Zhao 2014

Jul 2013

Apr 2014

6

Non-industry

EGFRTKI

CTx

No

Yes

No

Li 2014b

Jul 2013

Jul 2014

10

None

EGFRTKI

CTx

No

Yes

No

Vale 2014

Jan 2014

Nov 2014

14

Non-industry

EGFRTKI

CTx

No

Yes

No

Qi 2012a

Mar 2011

May 2011

8

NR

DOC + (CTx or TT)

DOC

No

Yes

Yes

Jin 2014*

Dec 2013

Sep 2014

12

Non-industry

DOC + (CTx or TT)

DOC

No

Yes

Yes

Qi 2012b

May 2011

Jan 2012

5

NR

PEM + (CTx or TT)

PEM

No

Yes

Yes

Sun 2014

Feb 2012

Apr 2014

4

NR

PEM + (CTx or TT)

PEM

No

Yes

Yes

Qi 2011

Jul 2011

Oct 2011

4

NR

CTx + VAN or VAN

CTx or EGFRTKI

Yes

Yes

Yes

Tao 2012

Sep 2011

Mar 2012

5

NR

CTx + VAN or VAN

CTx or EGFRTKI

Yes

Yes

Yes

Tassinari 2012

Jun 2010

Dec 2012

4

NR

DOC

CTx or EGFRTKI

No

Yes

Yes

Qi 2013

May 2012

Feb 2013

8

Non-industry

ERL + TT

ERL

No

Yes

No

Cui 2013

Dec 2011

Apr 2013

8

Non-industry

BEV + (CTx or EGFRTKI)

CTx or EGFRTKI

Yes

Yes

Yes

     

EGFRTKI

CTx or PBO

No

Yes

Yes

Li 2014a

Dec 2013

Apr 2014

14

None

CTx + TT

CTx

No

Yes

No

Liang 2014

Jan 2014

Oct 2014

10

Non-industry

MATKI + (CTx or EGFRTKI) or MATKI

CTx or EGFRTKI or PBO

No

Yes

Yes

Sun 2015

Oct 2014

Jan 2015

2

NR

BEV + EGFRTKI

EGFRTKI

No

Yes

No

Xiao 2015

Sep 2014

Feb 2015

5

Non-industry

CTx + EGFRTKI

CTx or EGFRTKI

No

Yes

Yes

Hawkins 2009

Oct 2007

Apr 2009

6

Industry

DOC vs PEM vs ERL vs GEF

No

No

No

Popat 2015

Mar 2014

Dec 2014

9

Industry

CTx vs TT vs CTx + TT vs (PBO or BSC)

No

No

No

*Update of Qi 2012a; network meta-analysis; conference abstracts. BEV: bevacizumab; BSC: best supportive care; CTx: monochemotherapy; DOC: docetaxel; EGFRTKI: EGFR tyrosine kinase inhibitors; ERL: erlotinib; GEF: gefitinib; MATKI: multi-targeted antiangiogenic tyrosine kinase inhibitors; NR: not reported; PBO: placebo; PEM: pemetrexed; TT: targeted therapy; VAN: vandetanib

Regarding the methodological quality of the 29 systematic reviews, 45 % of reviews lacked information on independent study selection and data extraction, 31 % a comprehensive literature search, and 45 % a search for reports regardless of their publication type. Of note, 17 % of reviews lacked information on a search for conference abstracts, 86 % a search for non-industry trial registries and results databases, 97 % a search for industry trial registries and results databases, and 97 % a search of regulatory agency online databases. In all, 79 % of systematic reviews did not report duplicate study selection and data extraction, comprehensive literature search, and searching for reports regardless of their publication type. Finally, 7 % of reviews provided a list of included and excluded trials.

Randomized evidence not covered or partially covered by systematic reviews

From 2009 to 2015, the amount of randomized evidence covered by existing systematic reviews was consistently incomplete: 40 % to 66 % of treatments; 38 % to 71 % of treatment comparisons; 45 % to 70 % of trials; and 30 % to 58 % of patients were missing (Fig. 3). In 2014, 27 reviews still did not cover 18 treatments (40 %), 20 treatment comparisons (38 %), 34 trials (46 %), and 8,486 patients (30 %).
Fig. 3

Amount of treatments, treatment comparisons, trials, and patients not covered by systematic reviews from 2009 to 2015. *The last search for randomized trials and systematic reviews was conducted on March 2, 2015

The cumulative networks of evidence show how the mismatch between available data and syntheses persisted from 2009 to 2015 (Fig. 4). Across all years, 10 % to 17 % of treatment comparisons were partially covered by systematic reviews and 55 % and 85 % were partially or not covered by systematic reviews. Moreover, the proportion of evidence covered by systematic reviews was unequally distributed across treatments.
Fig. 4

Cumulative networks of evidence showing the gap between the amount of randomized evidence covered by systematic reviews and the amount of randomized evidence available for inclusion. (a) 2009–2012 and (b) 2013–2015. *The last search for randomized trials and systematic reviews was conducted on March 2, 2015. From 2009 to 2015, we compared randomized controlled trials selected by systematic reviews published up to December 31 each year (up to March 2 for 2015) to all trials eligible for inclusion (i.e., all trial results published up to July 1 each year [up to August 31, 2014 for 2015]). Each node size is proportional to the total number of patients randomly allocated to the corresponding treatment across all randomized trials available for inclusion; we represented the proportion of randomized patients actually covered by systematic reviews by pie charts overlaid on nodes in the network. The thickness of each edge is proportional to the total number of randomized controlled trials between the corresponding treatments available for inclusion; we represented the proportion of trials actually selected by systematic reviews by a percentage bar chart overlaid on edges in the network

Sensitivity analyses gave consistent findings. When removing trials of drugs that did not pass phase II, 27 % to 64 % of trials, 27 % to 58 % of patients, 22 % to 53 % of treatments, and 21 % to 67 % of treatment comparisons were missing. When removing trials that did not report overall survival or progression-free survival, 41 % to 68 % of trials, 29 % to 58 % of patients, 36 % to 63 % of treatments, and 36 % to 69 % of treatment comparisons were missing. When discarding trials reported in conference abstracts only, 37 % to 68 % of trials, 24 % to 58 % of patients, 33 % to 63 % of treatments, and 33 % to 68 % of treatment comparisons were missing. When the lag period was the last date of search of the last systematic review, 42 % to 70 % of trials, 11 % to 53 % of patients, 36 % to 66 % of treatments, and 37 % to 72 % of treatment comparisons were missing.

Discussion

In this study, we assessed whether the whole set of conventional meta-analyses on pairwise treatment comparisons would allow for meeting clinicians’ and patients’ needs, to provide a comprehensive, up-to-date synthesis of evidence for all treatments. Our comparison of the amount of randomized evidence covered by systematic reviews and all randomized trials available for inclusion revealed a substantial waste related to the failure of systematic reviews to accumulate evidence scientifically: the evidence covered by existing systematic reviews on the topic was always substantially incomplete, with 40 % or more of treatments, treatment comparisons, and trials missing.

All meta-analyses on the same topic evaluating only a small fragment of the evidence has consequences for patient care and research planning. We need to identify the treatments with harmful effects and also treatments with side effects that have no advantage as compared with alternative treatments. As well, we need to be able to identify the treatment or group of treatments that works best. The relative beneficial effect of a treatment may be missed if some treatment comparisons of interest are not covered by systematic reviews. In our example, no systematic review encompassed all available treatments. Moreover, encompassing all the evidence for all treatments may have important implications for planning subsequent trials and helping prioritize future research to improve the evidence base [46]. In fact, based on the synthesis results and the geometry of the network of evidence, one could design a trial of treatments infrequently compared or a trial of the best potential treatment [47, 48].

Several reasons explain why the 29 systematic reviews did not cover all the randomized evidence. First, the trials may have addressed narrow and focused questions as compared to each review’s selection criteria (patients, interventions, and comparators). However, all trials we identified would have been eligible for inclusion in at least one systematic review; missing trials were not excluded from systematic reviews because they were not eligible.

Second, many meta-analyses become quickly out-of-date, sometimes by the time they are published. In fact, the systematic identification of trials is complex and time-consuming. In our case study, only one of 29 reviews was an update of a previous review and a large proportion of treatment comparisons covered by systematic reviews was out-of-date. This updating issue may be improved by the automation, as much as possible, of the whole trial search and selection process. These automated technologies would alleviate the burden on systematic reviewers. Third, another reason for the failure of systematic reviews to accumulate all available evidence is inadequate search methods for unpublished trials. In our case study, among the 15 trials (20 %) with unpublished results, only three (20 %) were included in systematic reviews. Among the 29 systematic reviews, 79 % could be considered at high risk of missing trials that would have met the inclusion criteria because they did not report duplicate study selection and data extraction, comprehensive literature search, or searching for reports regardless of publication type. Automated processes, such as meta-search engines, could systematically cover sources such as industry and non-industry trial registries and results databases to identify posted results and improve the evidence synthesis [49, 50]. Beyond the example at hand, the current way of conducting systematic reviews explains why they are inherently at risk of providing a fragmented, out-of-date panorama of the evidence for all treatments.

A broader scope on all evidence available for all treatments of a given condition is naturally provided by a systematic review with network meta-analysis, which allows for examining the totality of the randomized evidence using trial networks [51, 52]. Although the production of network meta-analyses follows an exponential growth, their number is still relatively limited [53]. With the increasing development of new treatments, examining networks of randomized trials is essential, and multiple treatment comparisons cannot be avoided [54]. Conventional meta-analysis focuses on comparisons of two treatments only. However, network meta-analysis allows for comparing all treatments to each other with direct and indirect comparisons, even if randomized trials are not available for some treatment comparisons. In our example, the available direct evidence informed 5 % of all possible comparisons between the 47 treatments. Ideally, the scoping exercise to define the criteria for considering trials in the systematic review should include all alternative treatments for the target condition [55]. It is still possible that a network meta-analysis will selectively choose treatments to include in the network, and excluding treatments may affect estimated treatment effects [56]. In our study, we identified two network meta-analyses; neither covered the whole evidence available at their time of analysis and considered a very restrictive network instead. Finally, network meta-analysis offers more flexibility to assess individual treatments. It offers the opportunity to overcome a common issue in standard meta-analysis whereby different treatments and sometimes different types of treatments are lumped together. In our example, erlotinib and gefitinib were frequently lumped together in EGFR tyrosine kinase inhibitors and docetaxel and pemetrexed were lumped together in monochemotherapy. This type of evidence lumping does not provide the relevant information for clinicians and patients who want to know which specific treatments work the best.

The paradox is that the set of systematic reviews fail to cover all the evidence, diminishing the value of research to clinicians by missing important comparisons. In fact, the production of meta-analyses has been substantially and rapidly increasing worldwide [57, 58]. In parallel, the evidence covered by systematic reviews features many gaps, and multiple overlapping meta-analyses on the same topic are common [59, 60].

As an alternative to classical systematic reviews performed at one point in time, a new theoretical framework is “living systematic reviews”, defined as high-quality online summaries of health research updated as new research becomes available [61]. We propose to push further the shift towards a new paradigm by switching: 1) from a series of standard meta-analyses focused on specific treatments (many treatments being not considered) to a single network meta-analysis covering all treatments; and 2) from meta-analyses performed at a given time and frequently out-of-date to a cumulative network meta-analysis systematically updated as soon as the results of a new trial become available, an approach to synthesis we call “live cumulative network meta-analysis”. In Fig. 5, we show the methodological steps we propose for live cumulative network meta-analysis.
Fig. 5

A new approach to synthesize evidence: live cumulative network meta-analysis. Starting from an initial NMA, a research community would regularly (e.g., every 3 months), search for, screen, and select trials with new results and, if any, extract data, assess the risk of bias, and update the NMA. NMA: network meta-analysis

We acknowledge that developing such methodology is challenging. In Table 2, we present some key challenges and potential solutions. The rigorous methodology of systematic reviews (exhaustive search of trials, minimizing subjectivity by independent duplicate assessments, assessing risk of bias within trials) is inherently demanding of resources and time, especially for a systematic review incorporating network meta-analysis. Moreover, keeping a systematic review up-to-date requires processes closer to those of rapid reviews (i.e., using accelerated and streamlined methods). Therefore, there is necessarily a trade-off between high-standard synthesis methods and real-time updating processes. Automated technologies may help define this trade-off by alleviating the burden of manual tasks for systematic reviewers. Several tools have been proposed to improve, hasten, and ease the search for trials, trial selection, extraction of data, and assessment of risk of bias [49, 50, 62, 63]. Live cumulative network meta-analysis may also raise issues regarding the current authoring and publishing system. Online posting may be more adequate to report periodically the findings of such “real-time” syntheses. Since Elliott et al. presented the theoretical framework of living systematic reviews, some examples have been published and have only partly addressed the aforementioned challenges [6468]; for instance, by using accompanying open-access websites to disseminate the updates of the systematic review.
Table 2

Methodological steps of live cumulative network meta-analysis, key challenges, and potential solutions

Methodological steps

Key challenges

Potential solutions

0. Initial network meta-analysis

Resource intensive but commonly one-shot investment

Setting-up of a research community (preferentially international) in charge of designing a high-quality and clinically relevant network meta-analysis and keeping it up-to-date for a given mandate (e.g., a 5- or 10-year period)

Redundant meta-analyses frequently commissioned by different groups

Need to consider all patient-important outcomes

Perform iterations at regular intervals (e.g., every 3 months) through steps 1–5

1. Search for trials

Need to identify trials of novel drugs. For instance, six to nine new second-line therapies per year in advanced NSCLC

Community expert monitoring would identify pipeline therapies assessed in clinical trials and allow adapting the search equations

 

Querying repeatedly a wide range of sources to identify trials with published and unpublished results is time consuming and labor intensive

Metasearch engine script designed for the question at hand would allow querying automatically and simultaneously the multiple sources [75]

Need to identify multiple reports of the same trial. For instance, there were on average two reports per trial of second-line treatments in advanced NSCLC

The OpenTrials database would contain all openly available data and documents on all clinical trials threaded together by trial ID [76]

Need to update the list of treatments, of trials, and multiple reports for the same trial

2. Screening of reports and selection of trials

Screening repeatedly may be resource intensive depending on the clinical question. In second-line therapies of advanced NSCLC we estimated that the workload would be manageable (about 50 new records to screen each month for CENTRAL, MEDLINE, EMBASE, and around 600 conference abstracts per year)

Using crowdsourcing for screening would allow distributing microtasks to community experts and dealing with increasing amounts of evidence [77, 78]

Future automated technologies would help community experts in the screening process; for instance, natural language processing methods using the semantic features of the reports and could help identify potentially relevant trial reports [49, 50, 7982]

If required only (at least one trial with new results), continue with steps 3–5

3. Data extraction

Extracting data and assessing the risk of bias repeatedly may be resource intensive depending on the number of trials with new results. In second-line therapies of advanced NSCLC we estimated that the workload would be manageable (about 10 to 15 new trials per year)

Using crowdsourcing for data extraction would allow distributing microtasks to experts and dealing with increasing amounts of evidence [77, 83]

4. Assessment of risk of bias

Need to check for consistency in extracted data between multiple reports for the same trial; in cases of inconsistency, need to justify the choice of a specific source

Automatic data extraction is possible depending on the source. For instance, it is possible to abstract automatically posted results from ClinicalTrials.gov [8486]

Future automated technologies could help experts to extract data or to assess the risk of bias within trials [49, 50, 62, 63]

5. Updating of network meta-analysis

Need to develop online software for updating the network meta-analysis*

Online solutions in development for conventional meta-analysis could be extended to network meta-analysis [87, 88]

6. Dissemination

Need to make the results publicly available after each iteration

A freely accessible website would allow reporting the live cumulative network meta-analysis, including all details regarding methods and processes, graphs, and data

Need for transparent reporting of the whole process

Need for peer-review

Alternative forms of peer-review (e.g., post-publication peer-review) could be implemented

*Eventually incorporating adjustment for multiple testing in live cumulative network meta-analysis to account for the inflated type I error, depending on ongoing discussion [89]. NSCLC: non-small cell lung cancer

Another challenge would be to consider all outcomes that are important or critical to patients for decision making in these live cumulative network meta-analyses [69, 70]. In our case study, we included trials regardless of reported outcomes; in a sensitivity analysis, we excluded trials that did not report treatment effects on overall survival or progression-free survival. However, there are other patient-important outcomes, in particular to measure the symptom burden of the disease and the quality of life of patients. More generally, it will be crucial to consider networks of trials according to the reporting of the different patient-important outcomes. In fact, the geometry of the network of trials could vary across outcomes because of differential reporting of outcomes (e.g., efficacy and safety outcomes) across drugs and trials.

Nonetheless, embracing the perspective of networks of trials of all alternative treatments for each condition, and in particular developing live cumulative network meta-analyses, could greatly benefit various stakeholders, including physicians, patients, and also guideline developers, funders, and decision-makers [71]. Networks of trials and their synthesis through network meta-analysis could increase the value of research when treatment recommendations are based on an exhaustive up-to-date network of randomized evidence [72]. Guideline developers and other decision-makers may further benefit from network meta-analyses if these implement recent developments to rate the quality of the body of evidence supporting treatment effect estimates for all patient-important outcomes (e.g. GRADE Summary of Findings tables) and rankings from network meta-analysis. For instance, in the field of medical therapies for open angle glaucoma, Li et al. recently showed that if a network meta-analysis had been conducted earlier, prostaglandins could have been shown as the most effective class in lowering intraocular pressure 7 years ahead of the guideline recommendation [73]. Moreover, biomedical research funding agencies could improve the prioritization of research proposals if they had access to a mapping of existing (and ongoing) trials evaluating all available treatments for a specific condition at the time a new trial is planned. Visualizing the network of trials and identifying which new trial maximizes the information can help stakeholders assess gaps in evidence and choose the next treatment comparison or trial that needs to be prioritized.

To our best knowledge, our study is the first to highlight the substantial waste associated with the failure of systematic reviews to accumulate evidence across all treatments (for the same disease). Moreover, our methodology based on a series of systematic overviews and networks of randomized evidence is novel and could be replicated in other fields. Our study has several limitations. First, we examined only one topic, second-line treatments for advanced NSCLC. However, our results should be generalizable, because the scope of meta-analyses is frequently narrow, with 81 % of standard meta-analyses that do not include all treatments and 43 % that cover only specific treatments [7]. Contrary to Haidich and colleagues, who assessed the evidence at the level of each systematic review, we assessed the cumulative evidence covered by all systematic reviews on a topic with a “bird’s-eye view”. Second, our gold standard included trials that meta-analysts may have considered ineligible for inclusion in their systematic reviews. However, we performed several sensitivity analyses and results were consistent. Third, we excluded trials in which the control group received different chemotherapy drugs at the discretion of the investigators. Such trials could also contribute to the synthesis in a network meta-analysis with a class effect model [74]; however, they do not allow for assessing the relative merits of the specific drugs. Similarly, we excluded randomized trials and systematic reviews comparing two different administration schemes because our focus was the comparison of alternative treatments against each other. Nonetheless, our analysis could be extended to such randomized trials and systematic reviews. In such cases, each relevant node would have several subnodes that relate to different administration schemes. Fourth, we may have missed some systematic reviews and randomized trials because of reporting bias, but we tried to minimize this possibility by an exhaustive search covering conference abstracts and registries. Fifth, we started our analysis for the year 2009; this pre-specified year was somehow arbitrary and we acknowledge that this starting point could have been earlier, considering that the first systematic review was published in April 2001; the second one was published in December 2005. However, there was no or little randomized evidence available between competing treatments at that time. Finally, we did not perform any outcome data synthesis at this stage. One may ask if covering all the randomized evidence available would have led to clinically important differences for the 10 % to 17 % of treatment comparisons partially covered by systematic reviews. In the framework we are promoting, a network meta-analysis would allow for estimating all treatment comparisons. Adding up to 40 % of missing evidence (about 8,000 patients) to the network would likely lead to clinically important differences, in particular for treatment comparisons partially covered by systematic reviews.

Conclusions

We illustrated that systematic reviews of a given condition provide a fragmented, out-of-date panorama of the evidence for all treatments. Embracing the perspective of networks of trials of all alternative treatments for each condition can have important consequences and should be adopted more generally. The waste of research associated with the failure to accumulate evidence could be reduced by the development of live cumulative network meta-analyses.

Notes

Abbreviations

ALK: 

anaplastic lymphoma kinase

EGFR: 

epidermal growth factor receptor

NSCLC: 

non-small cell lung cancer

Declarations

Acknowledgments

This study was supported by a grant LEGS POIX 2015 (Chancellerie des Universités de Paris). The funding source had no role in the design of this study, its execution, analyses, interpretation of the data, and decision to submit results. The authors thank Laura Smales (BioMedEditing, Toronto, Canada) for language revision of the manuscript and Elise Diard for the conception of the Figures.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Centre de Recherche Epidémiologie et Statistique Sorbonne Paris Cité, INSERM U1153
(2)
Université Paris Descartes – Sorbonne Paris Cité
(3)
Assistance Publique-Hôpitaux de Paris, Hôpital Hôtel-Dieu, Centre d’Epidémiologie Clinique
(4)
Cochrane France
(5)
Department of Epidemiology, Mailman School of Public Health, Columbia University

References

  1. Institute of Medicine. Initial national priorities for comparative effectiveness research. Washington, DC: The National Academies Press; 2009.Google Scholar
  2. Gotzsche PC. Why we need a broad perspective on meta-analysis. It may be crucially important for patients. BMJ. 2000;321(7261):585–6.View ArticlePubMedPubMed CentralGoogle Scholar
  3. Jefferson T, Doshi P, Thompson M, Heneghan C. Ensuring safe and effective drugs: who can do what it takes? BMJ. 2011;342:c7258.View ArticlePubMedGoogle Scholar
  4. Liberati A. Need to realign patient-oriented and commercial and academic research. Lancet. 2011;378(9805):1777–8.View ArticlePubMedGoogle Scholar
  5. Ioannidis JP. Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses. CMAJ. 2009;181(8):488–93.View ArticlePubMedPubMed CentralGoogle Scholar
  6. Naci H, Ioannidis JP. How good is “evidence” from clinical studies of drug effects and why might such evidence fail in the prediction of the clinical utility of drugs? Annu Rev Pharmacol Toxicol. 2015;55:169–89.View ArticlePubMedGoogle Scholar
  7. Haidich AB, Pilalas D, Contopoulos-Ioannidis DG, Ioannidis JP. Most meta-analyses of drug interventions have narrow scopes and many focus on specific agents. J Clin Epidemiol. 2013;66(4):371–8.View ArticlePubMedGoogle Scholar
  8. Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, Sampson M, et al. When and how to update systematic reviews. Cochrane Database Syst Rev. 2008;1:MR000023.PubMedGoogle Scholar
  9. Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med. 2007;147(4):224–33.View ArticlePubMedGoogle Scholar
  10. Ioannidis JP, Karassa FB. The need to consider the wider agenda in systematic reviews and meta-analyses: breadth, timing, and depth of the evidence. BMJ. 2010;341:c4875.View ArticlePubMedGoogle Scholar
  11. Global Burden of Disease Cancer Collaboration, Fitzmaurice C, Dicker D, Pain A, Hamavid H, Moradi-Lakeh M, et al. The global burden of cancer 2013. JAMA Oncol. 2015;1(4):505–27.View ArticlePubMed CentralGoogle Scholar
  12. Chan AW. Out of sight but not out of mind: how to search for unpublished clinical trial evidence. BMJ. 2012;344:d8013.View ArticlePubMedGoogle Scholar
  13. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10.View ArticlePubMedPubMed CentralGoogle Scholar
  14. Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z, et al. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS One. 2007;2(12):e1350.View ArticlePubMedPubMed CentralGoogle Scholar
  15. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62(10):1013–20.View ArticlePubMedGoogle Scholar
  16. Chandler J, Churchill R, Higgins J, Lasserson T, Tovey D. Methodological standards for the conduct of new Cochrane Intervention Reviews, Methodological expectations of Cochrane intervention reviews (MECIR). London: Cochrane Editorial Unit; 2013.Google Scholar
  17. Al-Saleh K, Quinton C, Ellis PM. Role of pemetrexed in advanced non-small-cell lung cancer: meta-analysis of randomized controlled trials, with histology subgroup analysis. Curr Oncol. 2012;19(1):e9–e15.PubMedPubMed CentralGoogle Scholar
  18. Barlesi F, Jacot W, Astoul P, Pujol JL. Second-line treatment for advanced non-small cell lung cancer: a systematic review. Lung Cancer. 2006;51(2):159–72.View ArticlePubMedGoogle Scholar
  19. Bonfill X, Serra C, Sacristan M, Nogue M, Losa F, Montesinos J. Second-line chemotherapy for non-small cell lung cancer. Cochrane Database Syst Rev. 2002;2:CD002804.PubMedGoogle Scholar
  20. Cui J, Cai X, Zhu M, Liu T, Zhao N. The efficacy of bevacizumab compared with other targeted drugs for patients with advanced NSCLC: a meta-analysis from 30 randomized controlled clinical trials. PLoS One. 2013;8(4):e62038.View ArticlePubMedPubMed CentralGoogle Scholar
  21. Gao G, Ren S, Li A, He Y, Chen X, Li W, et al. A meta-analysis of comparing EGFR-TKI with chemotherapy as the second-line treatment of NSCLC patients with wild-type EGFR. J Clin Oncol. 2013;31(15):suppl 1.Google Scholar
  22. Hawkins N, Scott DA, Woods BS, Thatcher N. No study left behind: a network meta-analysis in non-small-cell lung cancer demonstrating the importance of considering all relevant data. Value Health. 2009;12(6):996–1003.View ArticlePubMedGoogle Scholar
  23. Jiang J, Huang L, Liang X, Zhou X, Huang R, Chu Z, et al. Gefitinib versus docetaxel in previously treated advanced non-small-cell lung cancer: a meta-analysis of randomized controlled trials. Acta Oncol. 2011;50(4):582–8.View ArticlePubMedGoogle Scholar
  24. Jin Y, Sun Y, Shi X, Zhao J, Shi L, Hong W, et al. Meta-analysis to assess the efficacy and toxicity of docetaxel-based doublet compared with docetaxel alone for patients with advanced NSCLC who failed first-line treatment. Clin Ther. 2014;36(12):1980–90.View ArticlePubMedGoogle Scholar
  25. Lee JK, Hahn S, Kim DW, Suh KJ, Keam B, Kim TM, et al. Epidermal growth factor receptor tyrosine kinase inhibitors vs conventional chemotherapy in non-small cell lung cancer harboring wild-type epidermal growth factor receptor: a meta-analysis. JAMA. 2014;311(14):1430–7.View ArticlePubMedGoogle Scholar
  26. Li N, Yang L, Ou W, Zhang L, Zhang SL, Wang SY. Meta-analysis of EGFR tyrosine kinase inhibitors compared with chemotherapy as second-line treatment in pretreated advanced non-small cell lung cancer. PLoS One. 2014;9(7):e102777.View ArticlePubMedPubMed CentralGoogle Scholar
  27. Li X, Wang H, Lin W, Xu Q. Efficacy of combining targeted therapy with pemetrexed or docetaxel as second-line treatment in patients with advanced non-small-cell lung cancer: a meta-analysis of 14 randomized controlled trials. Curr Med Res Opin. 2014;30(11):2295–304.View ArticlePubMedGoogle Scholar
  28. Liang W, Wu X, Hong S, Zhang Y, Kang S, Fang W, et al. Multi-targeted antiangiogenic tyrosine kinase inhibitors in advanced non-small cell lung cancer: meta-analyses of 20 randomized controlled trials and subgroup analyses. PLoS One. 2014;9(10):e109757.View ArticlePubMedPubMed CentralGoogle Scholar
  29. Perez-Moreno MA, Galvan-Banqueri M, Flores-Moreno S, Villalba-Moreno A, Cotrina-Luque J, Bautista-Paloma FJ. Systematic review of efficacy and safety of pemetrexed in non-small-cell-lung cancer. Int J Clin Pharm. 2014;36(3):476–87.View ArticlePubMedGoogle Scholar
  30. Popat S, Mellemgaard A, Fahrbach K, Martin A, Rizzo M, Kaiser R, et al. Nintedanib plus docetaxel as second-line therapy in patients with non-small-cell lung cancer: a network meta-analysis. Future Oncol. 2015;11(3):409–20.View ArticlePubMedGoogle Scholar
  31. Qi WX, Shen Z, Lin F, Sun YJ, Min DL, Tang LN, et al. Comparison of the efficacy and safety of EFGR tyrosine kinase inhibitor monotherapy with standard second-line chemotherapy in previously treated advanced non-small-cell lung cancer: a systematic review and meta-analysis. Asian Pac J Cancer Prev. 2012;13(10):5177–82.View ArticlePubMedGoogle Scholar
  32. Qi WX, Shen Z, Yao Y. Meta-analysis of docetaxel-based doublet versus docetaxel alone as second-line treatment for advanced non-small-cell lung cancer. Cancer Chemother Pharmacol. 2012;69(1):99–106.View ArticlePubMedGoogle Scholar
  33. Qi WX, Tang LN, He AN, Shen Z, Yao Y. The role of vandetanib in the second-line treatment for advanced non-small-cell-lung cancer: a meta-analysis of four randomized controlled trials. Lung. 2011;189(6):437–43.View ArticlePubMedGoogle Scholar
  34. Qi WX, Tang LN, He AN, Shen Z, Yao Y. Effectiveness and safety of pemetrexed-based doublet versus pemetrexed alone as second-line treatment for advanced non-small-cell lung cancer: a systematic review and meta-analysis. J Cancer Res Clin Oncol. 2012;138(5):745–51.View ArticlePubMedGoogle Scholar
  35. Qi WX, Wang Q, Jiang YL, Sun YJ, Tang LN, He AN, et al. Overall survival benefits for combining targeted therapy as second-line treatment for advanced non-small-cell-lung cancer: a meta-analysis of published data. PLoS One. 2013;8(2):e55637.View ArticlePubMedPubMed CentralGoogle Scholar
  36. Sun CT, Xu X, Sheng W, Wang XW, Wen SL, Han JQ. A meta-analysis of pemetrexed-based doublet compared with pemetrexed alone for the second-line treatment of advanced non-small-cell lung cancer. Bratisl Lek Listy. 2014;115(4):233–7.PubMedGoogle Scholar
  37. Sun L, Ma JT, Zhang SL, Zou HW, Han CB. Efficacy and safety of chemotherapy or tyrosine kinase inhibitors combined with bevacizumab versus chemotherapy or tyrosine kinase inhibitors alone in the treatment of non-small cell lung cancer: a systematic review and meta-analysis. Med Oncol. 2015;32(2):473.View ArticlePubMedGoogle Scholar
  38. Tao L, Zhuo W, Yang F, Zhu B. Vandetanib for advanced non-small cell lung cancer: a meta-analysis. Zhongguo Fei Ai Za Zhi. 2012;15(3):172–8.PubMedGoogle Scholar
  39. Tassinari D, Scarpi E, Sartori S, Drudi F, Castellani C, Carloni F, et al. Noninferiority trials in second-line treatments of nonsmall cell lung cancer: a systematic review of literature with meta-analysis of phase III randomized clinical trials. Am J Clin Oncol. 2012;35(6):593–9.View ArticlePubMedGoogle Scholar
  40. Tassinari D, Scarpi E, Sartori S, Tamburini E, Santelmo C, Tombesi P, et al. Second-line treatments in non-small cell lung cancer. A systematic review of literature and metaanalysis of randomized clinical trials. Chest. 2009;135(6):1596–609.View ArticlePubMedGoogle Scholar
  41. Vale CL, Burdett S, Fisher DJ, Navani N, Parmar MK, Copas AJ, et al. Should tyrosine kinase inhibitors be considered for advanced non-small-cell lung cancer patients with wild type EGFR? Two systematic reviews and meta-analyses of randomized trials. Clin Lung Cancer. 2015;16(3):173–82. e174.View ArticlePubMedPubMed CentralGoogle Scholar
  42. Wong KM, Victor C, Eng L, Verma S. The use of epidermal growth factor receptor tyrosine kinase inhibitors in treatment of advanced EGFR wild-type non-small cell lung cancer: a meta-analysis study. J Thorac Oncol. 2013;8(suppl 2):S1184.Google Scholar
  43. Xiao BK, Yang JY, Dong JX, Ji ZS, Si HY, Wang WL, et al. Meta-analysis of seven randomized control trials to assess the efficacy and toxicity of combining EGFR-TKI with chemotherapy for patients with advanced NSCLC who failed first-line treatment. Asian Pac J Cancer Prev. 2015;16(7):2915–21.View ArticlePubMedGoogle Scholar
  44. Yang X, Yang K, Kuang K. The efficacy and safety of EGFR inhibitor monotherapy in non-small cell lung cancer: a systematic review. Curr Oncol Rep. 2014;16(6):390.View ArticlePubMedGoogle Scholar
  45. Zhao N, Zhang XC, Yan HH, Yang JJ, Wu YL. Efficacy of epidermal growth factor receptor inhibitors versus chemotherapy as second-line treatment in advanced non-small-cell lung cancer with wild-type EGFR: a meta-analysis of randomized controlled clinical trials. Lung Cancer. 2014;85(1):66–73.View ArticlePubMedGoogle Scholar
  46. Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM, et al. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65.View ArticlePubMedGoogle Scholar
  47. DeSantis SM, Zhu H. A Bayesian mixed-treatment comparison meta-analysis of treatments for alcohol dependence and implications for planning future trials. Med Decis Making. 2014;34(7):899–910.View ArticlePubMedPubMed CentralGoogle Scholar
  48. Nikolakopoulou A, Mavridis D, Salanti G. Using conditional power of network meta-analysis (NMA) to inform the design of future clinical trials. Biom J. 2014;56(6):973–90.View ArticlePubMedGoogle Scholar
  49. Tsafnat G, Dunn A, Glasziou P, Coiera E. The automation of systematic reviews. BMJ. 2013;346:f139.View ArticlePubMedGoogle Scholar
  50. Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Syst Rev. 2014;3:74.View ArticlePubMedPubMed CentralGoogle Scholar
  51. Salanti G, Kavvoura FK, Ioannidis JP. Exploring the geometry of treatment networks. Ann Intern Med. 2008;148(7):544–53.View ArticlePubMedGoogle Scholar
  52. Hutton B, Salanti G, Caldwell DM, Chaimani A, Schmid CH, Cameron C, et al. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Ann Intern Med. 2015;162(11):777–84.View ArticlePubMedGoogle Scholar
  53. Bafeta A, Trinquart L, Seror R, Ravaud P. Analysis of the systematic reviews process in reports of network meta-analyses: methodological systematic review. BMJ. 2013;347:f3675.View ArticlePubMedPubMed CentralGoogle Scholar
  54. Caldwell DM, Ades AE, Higgins JP. Simultaneous comparison of multiple treatments: combining direct and indirect evidence. BMJ. 2005;331(7521):897–900.View ArticlePubMedPubMed CentralGoogle Scholar
  55. Dias S, Welton NJ, Sutton AJ, Ades AE. Evidence synthesis for decision making 1: introduction. Med Decis Making. 2013;33(5):597–606.View ArticlePubMedPubMed CentralGoogle Scholar
  56. Mills EJ, Kanters S, Thorlund K, Chaimani A, Veroniki AA, Ioannidis JP. The effects of excluding treatments from network meta-analyses: survey. BMJ. 2013;347:f5195.View ArticlePubMedPubMed CentralGoogle Scholar
  57. Ioannidis JP, Chang CQ, Lam TK, Schully SD, Khoury MJ. The geometric increase in meta-analyses from China in the genomic era. PLoS One. 2013;8(6):e65602.View ArticlePubMedPubMed CentralGoogle Scholar
  58. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326.View ArticlePubMedPubMed CentralGoogle Scholar
  59. Helfer B, Prosser A, Samara MT, Geddes JR, Cipriani A, Davis JM, et al. Recent meta-analyses neglect previous systematic reviews and meta-analyses about the same topic: a systematic examination. BMC Med. 2015;13(1):82.View ArticlePubMedPubMed CentralGoogle Scholar
  60. Siontis KC, Hernandez-Boussard T, Ioannidis JP. Overlapping meta-analyses on the same topic: survey of published studies. BMJ. 2013;347:f4501.View ArticlePubMedPubMed CentralGoogle Scholar
  61. Elliott JH, Turner T, Clavisi O, Thomas J, Higgins JP, Mavergames C, et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 2014;11(2):e1001603.View ArticlePubMedPubMed CentralGoogle Scholar
  62. Jonnalagadda SR, Goyal P, Huffman MD. Automating data extraction in systematic reviews: a systematic review. Syst Rev. 2015;4:78.View ArticlePubMedPubMed CentralGoogle Scholar
  63. Marshall IJ, Kuiper J, Wallace BC. Automating risk of bias assessment for clinical trials. IEEE J Biomed Health Inform. 2015;19(4):1406–12.View ArticlePubMedGoogle Scholar
  64. Badgett RG, Vindhyal M, Stirnaman JT, Gibson CM, Halaby R. A living systematic review of nebulized hypertonic saline for acute bronchiolitis in infants. JAMA Pediatr. 2015;169(8):788–9.View ArticlePubMedGoogle Scholar
  65. Synnot A, Gruen RL, Menon D, Steyerberg EW, Buki A, Peul W, et al. A new approach to evidence synthesis in traumatic brain injury: living systematic reviews. J Neurotrauma. 2015. Epub ahead of print. doi:https://doi.org/10.1089/neu.2015.4124.
  66. Brazinova A, Rehorcikova V, Taylor MS, Buckova V, Majdan M, Psota M, et al. Epidemiology of traumatic brain injury in Europe: a living systematic review. J Neurotrauma. 2015. Epub ahead of print. doi:https://doi.org/10.1089/neu.2015.4126
  67. Cnossen MC, Scholten AC, Lingsma H, Synnot A, Tavender E, Gantner D, et al. Adherence to guidelines in adult patients with traumatic brain injury: A living systematic review. J Neurotrauma. 2015. Epub ahead of print. doi:https://doi.org/10.1089/neu.2015.4121
  68. Ters P, Badgett RG. A living meta-analysis of colchicine for pericarditis. Ann Pharmacother. 2014;48(10):1398–9.View ArticlePubMedGoogle Scholar
  69. Handbook for grading the quality of evidence and the strength of recommendations using the GRADE approach. http://gdt.guidelinedevelopment.org/central_prod/_design/client/handbook/handbook.html. Accessed 2 Dec 2015.
  70. Gartlehner G, Flamm M. Is the Cochrane Collaboration prepared for the era of patient-centred outcomes research? Cochrane Database Syst Rev. 2013;3:ED000054.PubMedGoogle Scholar
  71. Naci H, O’Connor AB. Assessing comparative effectiveness of new drugs before approval using prospective network meta-analyses. J Clin Epidemiol. 2013;66(8):812–6.View ArticlePubMedGoogle Scholar
  72. Puhan MA, Schünemann HJ, Murad MH, Li T, Brignardello-Petersen R, Singh JA, et al. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ. 2014;349:g5630.View ArticlePubMedGoogle Scholar
  73. Li T, Rouse B, Shi Q, Dickersin K. Working from all angles. The Inaugural REWARD/EQUATOR Conference: increasing value and reducing waste in biomedical research conference, 28–30 September 2015, Edinburgh, UK.Google Scholar
  74. Haas DM, Caldwell DM, Kirkpatrick P, McIntosh JJ, Welton NJ. Tocolytic therapy for preterm delivery: systematic review and network meta-analysis. BMJ. 2012;345:e6226.View ArticlePubMedPubMed CentralGoogle Scholar
  75. Smalheiser NR, Lin C, Jia L, Jiang Y, Cohen AM, Yu C, et al. Design and implementation of Metta, a metasearch engine for biomedical literature retrieval intended for systematic reviewers. Health Inf Sci Syst. 2014;2:1.View ArticlePubMedPubMed CentralGoogle Scholar
  76. OpenTrials. All the Data, on All the Trials, Linked. http://opentrials.net/. Accessed 2 Dec 2015.
  77. Elliott J, Sim I, Thomas J, Owens N, Dooley G, Riis J, et al. #CochraneTech: technology and the future of systematic reviews. Cochrane Database Syst Rev. 2014;9:ED000091.PubMedGoogle Scholar
  78. Embase Screening. http://screening.metaxis.com/EMBASE/login.php. Accessed 2 Dec 2015.
  79. Wallace BC, Trikalinos TA, Lau J, Brodley C, Schmid CH. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics. 2010;11:55.View ArticlePubMedPubMed CentralGoogle Scholar
  80. O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4:5.View ArticlePubMedPubMed CentralGoogle Scholar
  81. Miwa M, Thomas J, O'Mara-Eves A, Ananiadou S. Reducing systematic review workload through certainty-based screening. J Biomed Inform. 2014;51:242–53.View ArticlePubMedPubMed CentralGoogle Scholar
  82. Rathbone J, Hoffmann T, Glasziou P. Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Syst Rev. 2015;4:80.View ArticlePubMedPubMed CentralGoogle Scholar
  83. Brown AW, Allison DB. Using crowdsourcing to evaluate published scientific literature: methods and example. PLoS One. 2014;9(7):e100647.View ArticlePubMedPubMed CentralGoogle Scholar
  84. Cepeda MS, Lobanov V, Berlin JA. From ClinicalTrials.gov trial registry to an analysis-ready database of clinical trial results. Clin Trials. 2013;10(2):347–8.View ArticlePubMedGoogle Scholar
  85. Tasneem A, Aberle L, Ananth H, Chakraborty S, Chiswell K, McCourt BJ, et al. The database for aggregate analysis of ClinicalTrials.gov (AACT) and subsequent regrouping by clinical specialty. PLoS One. 2012;7(3):e33677.View ArticlePubMedPubMed CentralGoogle Scholar
  86. Maruani A, Boutron I, Baron G, Ravaud P. Impact of sending email reminders of the legal requirement for posting results on ClinicalTrials.gov: cohort embedded pragmatic randomized controlled trial. BMJ. 2014;349:g5579.View ArticlePubMedPubMed CentralGoogle Scholar
  87. openMetaAnalysis. http://openmetaanalysis.github.io/. Accessed 2 Dec 2015.
  88. Rising J, Hsiang S. A tool for distributed meta-analysis. Social Science Research Network; 2014. http://ssrn.com/abstract=2458129. Accessed 2 Dec 2015
  89. Nikolakopoulou A, Mavridis D, Salanti G. Prospective use of network metaanalysis using formal statistical monitoring. 36th Annual Conference of the International Society for Clinical Biostatistics, 23–27 August 2015, Utrecht, The Netherlands.Google Scholar

Copyright

© Créquit et al. 2016

Advertisement