Skip to content

Advertisement

  • Commentary
  • Open Access

Informing real-world practice with real-world evidence: the value of PRECIS-2

BMC Medicine201816:76

https://doi.org/10.1186/s12916-018-1071-1

Received: 3 May 2018

Accepted: 10 May 2018

Published: 21 May 2018

Abstract

Real-world evidence is needed to inform real-world practice. Pragmatic controlled trials are intended to provide such evidence by assessing the effectiveness of medicines and other interventions in real-world settings, as opposed to explanatory trials that assess efficacy in highly controlled settings. Dal-Ré and colleagues (BMC Med 16:49, 2018) recently performed a literature review of studies published between 2014 and 2017 to assess the degree to which studies that self-identified as pragmatic were truly so. The authors found that over one-third of randomized controlled trials of drugs and biologics that were self-labeled as pragmatic used placebo controls (as opposed to usual care), tested medicines before licensing, or were conducted in a single site. Further, they proposed that, in order to improve the reliability of the ‘pragmatic’ label, investigators should assess their trials using the PRECIS-2 tool upon submission to funders, ethics boards, or journals. We appreciate the value of PRECIS-2 as an indicator to assess the pragmatic versus explanatory features in a trial, and we herein highlight the potential challenges and opportunities that may arise with its systematic and widespread use.

Keywords

Pragmatic trialsExplanatory trialsReal-world evidenceEffectivenessUsual clinical practicePRECIS-2

Background

The term ‘pragmatic controlled trial’ (PCT) was first coined in 1967 by Schwartz and Lellouch [1], and is broadly defined as a randomized controlled trial (RCT) whose purpose is to inform decisions about clinical practice. PCTs assess the effectiveness of an intervention “under practical conditions,” maximizing external validity by studying interventions in the context of routine clinical practice conditions [2]. In contrast, explanatory trials measure efficacy and thus prioritize internal validity. While critical for demonstrating the efficacy and safety of medicines, explanatory trials can be poor predictors of real-world effectiveness. The main advantage of PCTs is the fact that they are primarily designed to answer decision-making questions about real-world applicability and generalizability, for example, by comparing the effectiveness of two treatment approaches among heterogeneous patients within the context of real-world practice, whilst using outcomes that matter to end users, including physicians, patients, and administrators [35].

The distinction between an explanatory trial and a PCT is not dichotomous, but can be viewed on a continuum and along a variety of dimensions [5]. With this in mind, a CONSORT working group on PCTs developed a tool, called the Pragmatic-Explanatory Continuum Indicator Summary (PRECIS), to help trialists make study design decisions suited to their intended purpose [6]. After initial use, the PRECIS tool was further refined and validated (PRECIS-2) [5], mapping nine domains of a trial onto a wheel, where each spoke represents a feature of the trial that can be characterized on an explanatory–pragmatic spectrum. The nine domains of the tool address patient eligibility and recruitment; study setting; features of how the intervention is delivered, including organizational resources requirements, flexibility in delivery protocol and adherence assessments, and the closeness of follow-up; patient-centeredness of the primary outcome; and the extent to which all data are included in the primary analyses [7]. The PRECIS-2 tool has been used for both trial design and assessment [5, 8, 9].

Dal-Ré et al. [10] recently reviewed published literature between 2014 and 2017 to assess the degree to which studies that self-identified as pragmatic are truly so. They found that over one-third of RCTs of drugs and biologics that were self-labeled as pragmatic used design features viewed as incongruent with the pragmatic approach such as placebo controls (as opposed to usual care), investigational medicines, or single-site settings. The authors proposed that, in order to improve the reliability of the ‘pragmatic’ label, investigators should assess their trials using PRECIS-2 upon submission to funders, ethics boards, or journals, and should include ratings with explanations and supporting documentation. We agree that there is a need for systematic consideration of external validity, and we appreciate the value of PRECIS-2 as one such indicator. Herein, we draw attention to the potential challenges and opportunities that may arise with the systematic and widespread use of PRECIS-2.

Challenges and opportunities

As Dal-Ré et al. indicate [10], there is increasing interest in real-world evidence, especially that which is generated from high-quality PCTs. This interest is due, in part, to a ‘voltage drop’ in effectiveness as interventions tested in explanatory trials move into real-world situations [11]. Further, given the emphasis of PCTs on generating information that is salient to decision-making, this type of study can provide insights about outcomes, such as cost or quality of life in usual care, to fill the gap remaining after market approval and to fully guide decisions by patients, physicians, and policymakers in selecting the optimal treatment [12, 13].

Ensuring fit-for-purpose is critical to effective interpretation of evidence. Dal-Ré et al.’s [10] suggestion to provide PRECIS-2 ratings and supporting documents could help decision-makers assess the degree to which study proposals and research findings are appropriately characterized. Further, detailed information on how and why studies are pragmatic in different domains helps to understand the applicability of evidence in particular settings and patient populations, for example, such as when a clinic is deciding to implement a new finding in their practice [14].

Nevertheless, caution should be exercised regarding the proposal. The PRECIS-2 tool is designed to be used by a group and to foster discussion about study features. In a previous study [9], we used the PRECIS-2 tool to assess five PCTs. Raters often struggled to use the tool, and large differences in inter-rater reliability were observed despite having access to detailed study information and common training. In follow-up conversations, different perspectives on various aspects, such as what constitutes ‘usual care’, emerged. Similarly, Bratton et al. [15] described how, throughout the development of the BLISTER trial, team members debated whether the trial was pragmatic or explanatory; despite having used the PRECIS wheel, the team needed extra guidance and scores ranged widely on several domains. The authors also highlighted the value of discussion, not just the mean score, to reach a greater consensus on the degree of pragmatism.

Absolute ratings are thus challenging when using PRECIS-2 and, even when accompanied by an explanation and supporting documentation, may be difficult for use by reviewers evaluating journal, protocol, or grant submissions. This is not a criticism of the usability of PRECIS-2, but rather a comment on the challenges of evaluating external validity. Even when the trial team self-rates their study, which we agree is more appropriate than an independent external evaluation, the use of rating information is complicated. For grant review, reviewers would be required to understand how to use and interpret the ratings and consider if PRECIS-2 is in fact the best tool. For peer-review publishing, an additional challenge emerges in that there is no clear or consensus threshold for what should be labeled as pragmatic.

One example of the complexity of the PRECIS-2 tool is the issue of a single-center trial, which, as Dal-Ré et al. indicate [10], would be likely to receive a more explanatory rating. However, if the trial is designed to address a question such as, for example, patient satisfaction with medication infusion timing, which typically occurs in such a setting, it could be pragmatic. This sort of nuance might be lost among the volume of details assessed by a busy grant reviewer. The investigator could justify giving themselves a rating of 3 instead of 1 on the setting domain by, for example, explaining that their intervention is typically performed in a specialized setting represented by their single center. The reviewer, in turn, would need to understand this potential complexity to decide whether they agreed with the pragmatic assessment.

Another example of the complexity of applying the PRECIS-2 tool is the value of capturing trial changes over time. Given that a trial design may change from initial concept to execution, the degree of pragmatism may change. Assessing the degree of pragmatism at one time point could miss valuable insights about how the intervention had to be adapted to better fit a specific setting. For example, constraints in a healthcare system where an intervention is being tested may result in changes to flexibility in delivery or adherence. The circumstances that led to these changes could be important to capture for end users. The PRECIS-2 criteria can be used to assess pragmatic elements at multiple time points and across different settings, but cannot in isolation capture critical information on adaptations and how an intervention could work in broader settings given these adaptations [16].

Conclusions

The PRECIS-2 tool can be a valuable instrument to characterize the applicability of a trial and enhance the reliability of the ‘pragmatic’ label. However, we caution that the tool may be limited in its interpretation and its widespread application would require ongoing training in using the tool for both trialists and reviewers evaluating journal, protocol, or grant submissions. Nevertheless, PRECIS-2 could enhance transparency and accountability in the reporting of RCTs and ensure that real-world evidence appropriately incorporates real-world treatment patterns and decision-making needs.

Declarations

Acknowledgements

The authors wish to acknowledge Eric Larson and David Chambers for their input on this commentary.

Authors’ contributions

GN and KEJ conducted the literature research, and drafted, revised, and approved the final manuscript.

Authors’ information

GN is an epidemiologist and program officer in Implementation Science at the National Cancer Institute’s Division of Cancer Control and Population Sciences. KEJ is a senior research associate at IMPAQ International in the Pharmaceutical Health Services Research practice area. Their previous research focused on the application of PRECIS-2 to rate PCT domains.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
, Rockville, USA
(2)
, Seattle, USA

References

  1. Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, Oxman AD, Moher D, CONSORT group, Pragmatic Trials in Healthcare (PRACTIHC) group. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 2008;337:a2390.View ArticlePubMedPubMed CentralGoogle Scholar
  2. Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis. 1967;20:637–48.View ArticlePubMedGoogle Scholar
  3. Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?”. Lancet. 2005;365(9453):256–65.View ArticlePubMedGoogle Scholar
  4. Treweek S, Zwarenstein M. Making trials matter: pragmatic and explanatory trials and the problem of applicability. Trials. 2009;10:37.View ArticlePubMedPubMed CentralGoogle Scholar
  5. Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147.View ArticlePubMedGoogle Scholar
  6. Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, Tunis S, Bergel E, Harvey I, Magid DJ, Chalkidou K. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62(5):464–75.View ArticlePubMedGoogle Scholar
  7. PRECIS-2. https://www.precis-2.org/. Accessed 23 Apr 2018.
  8. Luoma KA, Leavitt IM, Marrs JC, Nederveld AL, Regensteiner JG, Dunn AL, et al. How can clinical practices pragmatically increase physical activity for patients with type 2 diabetes? A systematic review. Transl Behav Med. 2017;7:751–72.View ArticlePubMedPubMed CentralGoogle Scholar
  9. Johnson KE, Neta G, Dember LM, Coronado GD, Suls J, Chambers DA, et al. Use of PRECIS ratings in the National Institutes of Health (NIH) health care systems research Collaboratory. Trials. 2016;17:32.View ArticlePubMedPubMed CentralGoogle Scholar
  10. Dal-Ré R, Janiaud P, Ioannidis JPA. Real-world evidence: how pragmatic are randomized controlled trials labeled as pragmatic? BMC Med. 2018;16:49.View ArticlePubMedPubMed CentralGoogle Scholar
  11. Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8:117.View ArticlePubMedPubMed CentralGoogle Scholar
  12. Zuidgeest MGP, Goetz I, Groenwold RHH, Irving E, van Thiel GJMW, Grobbee DE. Pragmatic trials and real-world evidence – an introduction to the series. J Clin Epidemiol. 2017;88:7–13.View ArticlePubMedGoogle Scholar
  13. Ford I, Norrie J. Pragmatic trials. N Engl J Med. 2016;375(5):454–63.View ArticlePubMedGoogle Scholar
  14. Johnson K, Tuzzio L, Renz A, Baldwin L-M, Parchman M. Decision-to-implement worksheet for evidence-based interventions: from the WWAMI region practice and research network. J Am Board Fam Med. 2016;29(5):553–62.View ArticlePubMedPubMed CentralGoogle Scholar
  15. Bratton DJ, Nunn AJ, Wojnarowska F, Kirtschig G, Sandell A, Williams HC. The value of the pragmatic-explanatory continuum indicator summary wheel in an ongoing study: the bullous pemphigoid steroids and tetracyclines study. Trials. 2012;13:50.View ArticlePubMedPubMed CentralGoogle Scholar
  16. Chambers DA, Norton WE. The Adaptome: advancing the science of intervention adaptation. Am J Prev Med. 2016;51(4 Suppl 2):S124–31.View ArticlePubMedPubMed CentralGoogle Scholar

Copyright

© The Author(s). 2018

Advertisement