- Open Access
Living cumulative network meta-analysis to reduce waste in research: A paradigmatic shift for systematic reviews?
BMC Medicinevolume 14, Article number: 59 (2016)
The original article was published in BMC Medicine 2016 14:8
In a recent research article in BMC Medicine, Créquit and colleagues demonstrate how published systematic reviews in lung cancer provide a fragmented, out-of-date picture of the evidence for all treatments. The results and conclusions drawn from this study, based on cumulative network meta-analyses (NMA) of evidence from randomized clinical trials over time, are quite compelling. The inherent waste of research resulting from incomplete evidence synthesis has wide-reaching implications for a range of target groups including developers of systematic reviews and guidelines and their end-users, health care professionals and patients at the point of care. Building on emerging concepts for living systematic reviews and NMA, the authors propose "living cumulative NMA" as a potential solution and paradigmatic shift. Here we describe how recent innovations within authoring, dissemination, and updating of systematic reviews and trustworthy guidelines may greatly facilitate the production of living NMA. Some additional challenges need to be solved for NMA in general, and for living cumulative NMA in particular, before a paradigmatic shift for systematic reviews can become reality.
Please see related research article: https://bmcmedicine.biomedcentral.com/articles/10.1186/s12916-016-0555-0
Health care professionals and other decision makers responsible for providing patients with safe and high quality care need access to the best current research evidence about diagnosis, treatment and prognosis . High quality systematic reviews synthesize the best available evidence regarding an intervention’s benefits and harms to answer a clinical question. Clinical practice guidelines combine this evidence with other relevant considerations such as patients’ values and preferences . Together, reviews and guidelines constitute key tools to provide the best current evidence and optimal recommendations for choosing among alternative diagnostic and treatment strategies .
Over the past decades the world has seen large advances in standards, methods, and systems for conducting and reporting systematic reviews and trustworthy guidelines [4–7]. These advances include systematic reviews that use network meta-analyses (NMA), the statistical technique that allows comparing multiple treatments at a time [8, 9], the methods of which are rapidly evolving .
In a study recently published in BMC Medicine, Créquit and colleagues illustrate how published systematic reviews in lung cancer provide a fragmented, out-of-date picture of the evidence for all treatments . The evidence covered by existing systematic reviews for second-line treatments for advanced non-small cell lung cancer was incomplete, with 40 % or more of treatments, treatment comparisons, and trials missing. The incomplete evidence was detected through the construction of cumulative networks of evidence from randomized clinical trials over time and an evaluation of the proportion of trials, patients, treatments, and treatment comparisons not covered by systematic reviews on a yearly basis from 2009 onwards.
The study has limitations, including lack of consideration of the scope of the systematic reviews (limited scope may have been a legitimate reason to not include all available trials), and failure to address the possibility that it would have been inappropriate to pool together the results of all the randomized trials detected. Although these limitations may have resulted in an exaggeration of the magnitude of the problem, the results and conclusions are nevertheless compelling. As stated by the authors "available pairwise treatment comparisons do not allow for meeting clinicians’ and patients’ needs in decision making. Nor do they meet the needs of other target audiences such as developers of clinical practice guidelines, decision aids, funders and decision-makers in health care systems" .
Living cumulative NMA
Building on emerging concepts for living systematic reviews and network meta-analysis, the authors conclude that this waste of research might be reduced by "living cumulative NMA" (living NMA hereafter). With this approach, the focus would be on developing NMAs that include all available treatments (as opposed to a series of head-to-head comparisons), which are updated as soon as new trials become available. The authors propose methodological steps for living NMA and present a series of challenges and potential solutions for each step from creation to dissemination and updating. Their proposed solution represents an innovative input to existing challenges for systematic reviews and also relates to international calls for increased value and reduced waste in research, here within the context of evidence synthesis of available trials .
Although producing living NMAs is challenging, recent developments could greatly facilitate their production. Improved methods, strategies, and tools for more efficient authoring, dissemination, and updating of systematic reviews and trustworthy guidelines are emerging [13–15]. Tools, such as the Epistemonikos database, allow efficient and intuitive searching for systematic reviews and individual studies . The use of GRADE combined with innovative health information technology (e.g., MAGICapp) facilitates dynamically updated digitally structured evidence summaries, guidelines, and decision aids that clinicians and patients can access on all devices in user-friendly formats [3, 14, 15, 17, 18]. Presentations on these devices include desirable and undesirable consequences of treatment alternatives customized for both shared decision-making and integration in electronic medical records linked to patient specific data [14, 19].
These innovations, however, have not fully addressed all the challenges related to NMA in general, and to living cumulative NMA in particular. We will now comment on some additional challenges and suggest potential solutions from the perspective of developers of systematic reviews and guidelines and their end-users, health care professionals and patients at the point of care.
Rating quality of evidence in NMA
Systematic reviews need to perform an appropriate assessment of the quality of the body of evidence (synonyms: certainty or confidence in the evidence) for patient-important outcomes [5, 7]. State of the art reviews now provide evidence summaries using GRADE Summary of Findings (SoF) tables that provide quality of evidence ratings to inform clinicians, patients, guideline developers, and policy decision-makers .
Créquit and co-authors point out that guideline developers and other decision-makers may further benefit from NMAs if they rate the quality of the body of evidence supporting treatment effect estimates for all patient-important outcomes. Rating quality of evidence in NMA presents additional challenges and represents a methodological frontier. Authors have already presented two systems based on GRADE principles for making quality of evidence ratings in NMAs [21, 22]. Much work remains in the testing, application, and refinement of these methods, currently being undertaken by (among others) the GRADE NMA project group .
Patient-important outcomes in NMA
Systematic reviews and trustworthy guidelines need to provide evidence for all outcomes to allow an assessment of the balance between the desirable and undesirable consequences of a course of action [4, 24]. The Créquit study focused on the availability of treatment comparisons for lung cancer rather than on the extent to which authors synthesized evidence across all patient-important outcomes. As acknowledged by the authors, the identification and reporting of all outcomes that are important to patients for decision-making in living NMA represents a particular challenge, including differential reporting across outcomes resulting in variation in the geometry of the network of trials. Further methodological work in NMA will be required to address these issues.
Dissemination of results from NMA to the point of care
Health care professionals and patients will, as outlined above, increasingly be able to access the best current evidence from systematic reviews and recommendations in guidelines at the point of care in new and user-friendly formats [3, 14, 19]. Interactive Summary of Findings (iSoF) tables represent one such innovation with new and improved presentation formats of evidence summaries linked to systematic reviews and guidelines. iSoF tables allow end-users to balance benefits and harms in absolute numbers, also taking into account the quality of the evidence .
Current SoF tables are structured only for paired comparisons, and so do not address the multiple comparisons of NMA. How to best present multiple comparisons for all patient-important outcomes to end-users remains uninvestigated. Members of the GRADE NMA project group are exploring both how to optimally present the results from NMA in SoF tables and how to translate relative estimates of effects (the output obtained from NMA) into absolute estimates of effect that facilitate the decision-making process .
The traditional methods of conducting systematic reviews are associated with a risk of presenting an incomplete picture of the relevant evidence addressing a clinical question. Living NMA represents a promising approach to providing a broad, complete, and updated presentation of the evidence regarding all the available management options. This might well represent a paradigmatic shift for systematic reviews. However, as is the case for any new technology, living NMA will require overcoming a number of challenges. Investigators are currently addressing these challenges which suggest that living NMA as a new tool for addressing clinical questions may, in the near future, become available.
Guyatt G, Rennie D, Meade M, Cook D. Users’ Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice. 3rd ed. New York, NY: McGraw-Hill Education, American Medical Association; 2015.
Andrews JC, Schunemann HJ, Oxman AD, et al. GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation's direction and strength. J Clin Epidemiol. 2013;66(7):726–35.
Treweek S, Oxman AD, Alderson P, et al. Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results. Implement Sci. 2013;8:6.
Laine C, Taichman DB, Mulrow C. Trustworthy clinical guidelines. Ann Intern Med. 2011;154(11):774–5.
Guyatt GH, Oxman AD, Kunz R, et al. What is "quality of evidence" and why is it important to clinicians? BMJ. 2008;336(7651):995–8.
Guyatt GH, Oxman AD, Kunz R, et al. Going from evidence to recommendations. BMJ. 2008;336(7652):1049–51.
Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. 2011. http://www.cochrane.org/handbook. Accessed 5 Mar 2016.
Salanti G. Indirect and mixed-treatment comparison, network, or multiple-treatments meta-analysis: many names, many benefits, many concerns for the next generation evidence synthesis tool. Res Syn Meth. 2012;3:80–97.
Lu G, Ades AE. Combination of direct and indirect evidence in mixed treatment comparisons. Stat Med. 2004;23(20):3105–24.
Efthimiou O, Debray TP, van Valkenhoef G, et al. GetReal in network meta-analysis: a review of the methodology. Res Syn Meth. 2016. doi:10.1002/jrsm.
Crequit P, Trinquart L, Yavchitz A, Ravaud P. Wasted research when systematic reviews fail to provide a complete and up-to-date evidence synthesis: the example of lung cancer. BMC Med. 2016;14(1):8.
Macleod MR, Michie S, Roberts I, et al. Biomedical research: increasing value, reducing waste. Lancet. 2014;383(9912):101–4.
Elliott JH, Turner T, Clavisi O, et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 2014;11(2), e1001603.
Vandvik PO, Brandt L, Alonso-Coello P, et al. Creating clinical practice guidelines we can trust, use, and share: a new era is imminent. Chest. 2013;144(2):381–9.
MAGIC authoring and publication platform (MAGICapp) for improving patient care through guidelines, evidence summaries and decision aids that we can all trust, use and share. http://www.magicapp.org. Accessed 5 Mar 2016.
Epistemonikos. A Free, Relational, Collaborative, Multilingual Database of Health Evidence. http://www.epistemonikos.org Accessed 5 Mar 2016.
Agoritsas T, Heen AF, Brandt L, et al. Decision aids that really promote shared decision making: the pace quickens. BMJ. 2015;350:g7624.
Kristiansen A, Brandt L, Alonso-Coello P, et al. Development of a novel, multilayered presentation format for clinical practice guidelines. Chest. 2015;147(3):754–63.
Elwyn G, Quinlan C, Mulley A, Agoritsas T, Vandvik PO, Guyatt G. Trustworthy guidelines - excellent; customized care tools - even better. BMC Med. 2015;13(1):199.
Guyatt G, Oxman AD, Akl EA, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94.
Puhan MA, Schunemann HJ, Murad MH, et al. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ. 2014;349:g5630.
Salanti G, Del Giovane C, Chaimani A, Caldwell DM, Higgins JP. Evaluating the quality of evidence from a network meta-analysis. PLoS One. 2014;9(7), e99682.
GRADE working group. http://www.gradeworkinggroup.org. Accessed 5 Mar 2016.
Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–6.
iSoF. The key information you need to understand the benefits and harms of treatments. http://isof.epistemonikos.org. Accessed 5 Mar 2016.
This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
The authors report no financial conflicts of interests but have potential non-financial conflicts of interest through their engagement in the GRADE working group , the MAGIC research and innovation program , and methodological research of relevance to the content of this commentary.
POV conceptualized and drafted the manuscript. RBP and GG subsequently edited and critically revised the manuscript for scientific and factual content. All authors read and approved the final manuscript.