We’re sorry, something doesn't seem to be working properly.
Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.
Towards understanding the de-adoption of low-value clinical practices: a scoping review
BMC Medicinevolume 13, Article number: 255 (2015)
Low-value clinical practices are common in healthcare, yet the optimal approach to de-adopting these practices is unknown. The objective of this study was to systematically review the literature on de-adoption, document current terminology and frameworks, map the literature to a proposed framework, identify gaps in our understanding of de-adoption, and identify opportunities for additional research.
MEDLINE, EMBASE, the Cochrane Central Register of Controlled Trials, the Cochrane Database of Systematic Reviews, the Cochrane Database of Abstracts and Reviews of Effects, and CINAHL Plus were searched from 1 January 1990 to 5 March 2014. Additional citations were identified from bibliographies of included citations, relevant websites, the PubMed ‘related articles’ function, and contacting experts in implementation science. English-language citations that referred to de-adoption of clinical practices in adults with medical, surgical, or psychiatric illnesses were included. Citation selection and data extraction were performed independently and in duplicate.
From 26,608 citations, 109 were included in the final review. Most citations (65 %) were original research with the majority (59 %) published since 2010. There were 43 unique terms referring to the process of de-adoption—the most frequently cited was “disinvest” (39 % of citations). The focus of most citations was evaluating the outcomes of de-adoption (50 %), followed by identifying low-value practices (47 %), and/or facilitating de-adoption (40 %). The prevalence of low-value practices ranged from 16 % to 46 %, with two studies each identifying more than 100 low-value practices. Most articles cited randomized clinical trials (41 %) that demonstrate harm (73 %) and/or lack of efficacy (63 %) as the reason to de-adopt an existing clinical practice. Eleven citations described 13 frameworks to guide the de-adoption process, from which we developed a model for facilitating de-adoption. Active change interventions were associated with the greatest likelihood of de-adoption.
This review identified a large body of literature that describes current approaches and challenges to de-adoption of low-value clinical practices. Additional research is needed to determine an ideal strategy for identifying low-value practices, and facilitating and sustaining de-adoption. In the meantime, this study proposes a model that providers and decision-makers can use to guide efforts to de-adopt ineffective and harmful practices.
Clinical practice evolves in response to scientific evidence through a process of discovery (novel practice introduced into clinical practice, e.g., systemic thrombolysis for acute ST-elevation myocardial infarction (STEMI) ), replacement (newer, more effective practice supplants current practice, e.g. tenecteplase superior to alteplase among patients with STEMI ), or reversal (current practice shown to be ineffective or harmful, e.g., suppression of ventricular ectopy after a myocardial infarction using encainide, flecainide, or moricizine ) . Discovery and replacement introduce novel, beneficial therapies into clinical practice, while reversal implies that patients receive no benefit and may be at risk of harm . The adoption of clinical practices that are later de-adopted imposes substantial inefficiencies on the healthcare system wherein resources that could have been dedicated to other purposes are instead devoted to a practice that was ineffective or harmful (e.g., self-monitoring of blood glucose in patients with type 2 diabetes mellitus managed without insulin) .
Practice reversal is common [5, 7, 8]. A recent review of articles published in a major general medical journal between 2001 and 2010 found that 27 % of original articles re-examined the efficacy of an established practice, among which 40 % found evidence for practice reversal . In another review, commissioned by the Australian government’s Comprehensive Management Framework for managing their Medical Benefits Schedule, Elshaug and colleagues triangulated data from searches of the peer-reviewed literature, targeted health technology databases, and opportunistic sampling of stakeholder groups to identify 156 potentially unsafe and/or ineffective practices .
Medical reversal may be an unavoidable consequence of evidence-based medicine and/or early technology adoption; however, it is important that its incidence remain low given the threat that it poses to providing high-quality healthcare. It is equally important that any intervention with evidence for medical reversal be rapidly de-adopted. We were unable to identify any knowledge synthesis that systematically examined the de-adoption of established clinical practices. We conducted this scoping review to describe the literature on de-adoption, document current terminology and frameworks, map the literature to a proposed conceptual framework (Table 1), identify gaps in the understanding of this important concept, and identify opportunities for more detailed evidence syntheses and/or empirical research.
We developed a conceptual framework for this work that employed the key features of Everett Rogers’ Innovation-Decision model to conceptualize de-adoption (Table 1) . De-adoption was defined as the discontinuation of a clinical practice after it was previously adopted . We followed established scoping review methodology [10, 11], and used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines to report the methods and results .
We included English-language citations that referred to the de-adoption of any clinical practice in adults (mean age ≥ 18 years) with medical, surgical, or psychiatric illnesses. All original and non-original quantitative and qualitative research citations were eligible; however, we excluded citations that exclusively described the adoption of practices or appropriateness of resource use (e.g., selected use of antimicrobials, appropriate use of surgical procedures, appropriate use of lumbar spine radiography among patients with lower back pain). Although de-adoption is a component within the larger issue of resource optimization, the “appropriateness” of a clinical practice embodies more than simply discontinuing its use. Therefore, we excluded citations primarily focused on clinical practice appropriateness.
Search strategy and data sources
With the help of a medical librarian, we searched the following electronic databases from 1 January 1990 to 5 March 2014: Ovid MEDLINE, Ovid EMBASE, the Cochrane Central Register of Controlled Trials (CENTRAL), the Cochrane Database of Systematic Reviews, the Cochrane Database of Abstracts and Reviews of Effects, and CINAHL Plus. Pilot searches in MEDLINE suggested that none of the currently available Medical Subject Heading (MeSH) terms were specific to articles reporting de-adoption. Therefore, the MEDLINE search was confined to use of text words that included combinations and synonyms of de-adoption and healthcare technologies (Additional file 1: Appendix). Search terms were combined using the appropriate Boolean logic, and included wildcards to account for plural words and variations in spelling. The search strategy included similar combinations of terms within the other databases. To ensure reproducibility, the MEDLINE search strategy was peer reviewed by a second medical librarian using the Peer Review of Electronic Search Strategies (PRESS) checklist .
To increase the sensitivity of the search strategy, we also searched the gray literature according to recommendations from the Canadian Agency for Drugs and Technologies in Health . Relevant websites included The Canadian Agency for Drugs and Technologies in Health, Program for Assessment of Technology in Health, Australian Government Medical Services Advisory Committee, Austrian Institute of Technology Assessment, National Institute for Health and Care Excellence, Agency for Healthcare Research and Quality, Blue Cross & Blue Shield Association, Choosingwisely.org and Choosingwiselycanada.org, and the Trip Database. Additional citations were identified by (1) contacting experts in implementation science; (2) using the PubMed “related articles” function; and (3) hand-searching bibliographies from important implementation science/adoption of innovations textbooks [9, 15, 16], and reference lists of included citations. Reference management was performed in EndNote (version X7, Thomson Reuters).
Prior to the screening of titles and abstracts, the citation screening form was calibrated by three team members (DJN, KJM, JKH) independently with a random sample of 50 citations. Once consistent citation selection was achieved (kappa ≥ 0.8) , all citations were screened for inclusion independently and in duplicate by three reviewers through a two-stage process. During level-one screening, titles and abstracts were reviewed to determine citations that met the inclusion/exclusion criteria. The full text of any citation classified as “include” or “unclear” was reviewed to determine whether it met study inclusion criteria (level-two screening). Eligibility disagreements were resolved by consensus, or arbitration by a third reviewer. Agreement between reviewers at all stages of citation selection was quantified using the kappa statistic .
Data extraction and synthesis
Three reviewers independently extracted data from all included citations using a pre-designed electronic form that was pilot tested using a random sample of 10 citations. Once data were consistently abstracted (kappa ≥ 0.8) , reviewers proceeded with full data extraction. Extracted data pertained to (1) the citation (e.g., original research, non-original research, website); (2) the term(s) used to refer to de-adoption (e.g., discontinuance, medical reversal, rejection); (3) characteristics of the target condition(s) or clinical practice(s) (e.g., use of nesiritide in acute decompensated heart failure ); (4) characteristics of evidence suggesting de-adoption (e.g., original research versus non-original research); (5) whether barriers and facilitators to de-adoption were reported; and (6) whether conceptual frameworks to promote low-value practice de-adoption were used/cited.
Independently, and in duplicate, reviewers mapped the abstracted data onto the proposed conceptual framework. Articles were summarized using counts, proportions, mean (standard deviation), or median (inter-quartile range, IQR) where appropriate. Data were managed and analyzed using Stata version 13.1 (Stata Corp, College Station, TX, USA).
The electronic database and gray literature searches identified 26,557 unique citations (Fig. 1) that were screened for inclusion, from which 110 full text citations were retrieved for further assessment. An additional 51 articles were identified through review of bibliographies, and consultation with knowledge translation experts. From these 161 full text citations, 109 were included in the final review. The most common reason citations were excluded after full text review was owing to an explicit focus on the adoption and/or appropriateness of clinical practices (n = 25).
Description of the included citations
A description of the included citations is provided in Table 2. Most citations were original research studies (65 %), with the majority being either quasi-experimental (28 %) or cohort studies (14 %). Among the non-original citations, most were editorials or letters to the editor (19 %), or narrative reviews (15 %). Most articles originated in North America (60 %) with the USA representing the most common country (47 % of all articles). The majority of articles were published from 2010 onwards (59 %), with very few published prior to 2000 (3 %). Most articles described the de-adoption of therapeutic interventions (62 %), with comparatively fewer describing the de-adoption of diagnostic interventions (30 %). The randomized clinical trial was most frequently cited (41 %) as the level of evidence that should trigger de-adoption, and most articles cited risk of harm (73 %), and/or lack of efficacy (63 %) as the reason practices should be de-adopted. Among the articles that reported the original reason for clinical practice adoption (n = 16, 15 %), most (n = 10, 63 %) cited observational research (case series and cohort studies) as the evidence that shaped adoption. A detailed, referenced bibliography of the citations is provided in Additional file 1: Table S1.
We identified 43 unique terms representative of the process of de-adoption (Table 3). The majority of citations (65 %) referred to de-adoption using more than one term, and among these the median (IQR) number of terms per citation was 3 (2–3). Disinvest* was the most frequently cited term (39 % of included citations). Other commonly cited terms included decrease use (24 %), discontinu* (16 %), abandon* (16 %), reassess* (14 %), obsole* (12 %), medical reversal (11 %), and contradict* (10 %). Terms such as de-implement* and de-adopt* were infrequently cited (4 % and 3 %, respectively). A term representative of the process of de-adoption was found in the title or abstract of 86 % of citations and most frequently included disinvest* (31 %), decrease use (12 %), reassess* (7 %), withdraw* (7 %), medical reversal (6 %), discontinu* (6 %), and obsole* (6 %). Each of the 43 unique terms was mapped onto our conceptual framework. The majority of terms (n = 22, 51 %) referred to facilitating the de-adoption process. Seventeen terms (40 %) mapped to more than one category within the conceptual framework, with the most common cross-classification being facilitate de-adoption and sustain de-adoption (13/17, 76 %).
Barriers and facilitators to de-adoption
Barriers and facilitators to de-adoption were cited within 51 and 48 of the included citations respectively. The bulk of articles citing barriers to or facilitators of de-adoption were original research (Fig. 2).
Mapping citations to the de-adoption conceptual framework
Articles frequently mapped to more than one category within our conceptual framework (Fig. 3). The primary focus among included citations was evaluating de-adoption outcomes (50 %), identifying low-value practices (47 %), and facilitating the de-adoption process (40 %). Two articles (2 %) discussed sustaining de-adoption. Most articles whose focus was on evaluating de-adoption outcomes were original research (80 %), whereas the majority of articles that discussed identifying low-value practices were non-original research (63 %).
Frameworks for the de-adoption of low-value clinical practices were provided in 11 citations (Table 4), of which half were derived from original research (n = 5, 45 %). Two citations documented clinical application of their framework [19, 20]. Seven citations described frameworks for identifying and prioritizing candidate low-value practices, and nine citations described frameworks for facilitating the de-adoption process. Among citations that described a framework for identifying low-value practices, common mechanisms included consultation with clinical stakeholders, monitoring for new scientific evidence, examining for practices with large between-provider variation, and/or embedding the notion of health technology reassessment within the life cycle of any given practice. Commonly proposed criteria for prioritizing the de-adoption of low-value practices included the availability of evidence that a candidate practice is ineffective or harmful, the safety of the low-value practice (i.e., harmful practices prioritized ahead of those that are simply ineffective), potential health and cost impact of de-adoption, and availability of alternative practices. Among citations that described frameworks for facilitating the de-adoption process, common mechanisms included restructuring of funding associated with the given practice, changes to local and/or regional policies, and more consistent integration of health technology reassessment within existing health technology assessment programs.
Lists of low-value practices were provided by eight citations (Table 5). Searches of the published literature were the most frequently employed means of identifying low-value practices (n = 7 citations, 88 %); however, the sources searched and the approach to defining a low-value practice varied by citation. Evidence was combined with stakeholder engagement to identify low-value practices in three citations [8, 21, 22], and one citation identified low-value practices as those shown to have high variability in rates of use between providers . Among the seven citations that used the published literature to identify low-value practices, the prevalence of low-value practices ranged from 16 %  to 46 % , with two studies each identifying more than 100 low-value practices [7, 8].
The impact of de-adoption efforts was evaluated and reported in 39 original research citations (Table 6). Most studies used interrupted time series methodology (n = 21, 54 %) and obtained data from large administrative databases or clinical registries (n = 30, 76 %). The most common target conditions were cardiovascular disease (n = 11, 28 %), arthritides (n = 8, 21 %), and menopause (n = 7, 18 %). All but one of the practices (pulmonary artery catheter) examined were therapeutic interventions. The most frequently examined therapies included cyclo-oxygenase-2 (COX-2) inhibitors and other non-steroidal anti-inflammatory drugs (NSAIDs) (n = 8, 21 %), hormone replacement therapy (n = 7, 18 %), and percutaneous coronary intervention (n = 3, 8 %). Thirteen studies reported on de-adoption efforts that followed an active change intervention, all of which demonstrated reductions in the target low-value practice [25–37]. The most common intervention was withdrawal of a low-value drug from the market (n = 9, 23 %). Other active change interventions commonly included an education component targeted at patients and/or providers. Of the 26 studies that did not report on the effects of an active change intervention, 23 (88 %) demonstrated reductions in the target practice. Of the 27 and 11 studies that examined de-adoption efforts for harmful or ineffective practices, respectively, 25 (92 %) and 9 (81 %) demonstrated reductions in the target practice.
De-adoption of low-value clinical practices is essential to improve healthcare quality and create a sustainable healthcare system. To our knowledge this is the first knowledge synthesis to comprehensively examine the de-adoption of low-value clinical practices. We identified 109 citations, most of which were published within the last five years, and concentrated on evaluating changes in practice that occurred following the publication of evidence for medical reversal. We identified 43 terms used to refer to the process of de-adoption, with disinvest being the most frequently cited term. We also identified 13 frameworks that conceptualize individual components of the de-adoption process, and from these frameworks propose a model for de-adoption (Fig. 4). These results provide foundations for guiding the de-adoption of ineffective and harmful clinical practices from patient care as well as directing future research.
The first major finding from our study pertains to the diverse list of terms used to refer to de-adoption with no clearly established taxonomy. The implication of this is that communication is impaired, which may impact “branding” of de-adoption and efficient searching for relevant literature. Furthermore, it is unclear how different concepts and initiatives such as “less is more” , reducing research waste , and Choosing Wisely  are related. Conversely, knowledge translation and implementation science are increasingly recognized terms in healthcare research, facilitating understanding and communication of the related concepts. Terms such as de-adoption and de-implementation that have a more general connotation, and are natural antonyms of adoption and implementation, ought to be used as terms that brand the process of reducing or removing low-value clinical practices. Other terms, such as disinvest, describe specific elements of the de-adoption process and are not ideal candidates to brand this process. Interestingly, de-adoption and de-implementation were infrequently cited within the included citations, whereas disinvest was the most commonly cited term. Given this lack of clarity with regard to de-adoption terminology, there is an urgent need to develop a taxonomy of terms.
Using the proposed conceptual framework (Table 1), themes common to the frameworks identified in the scoping review (Table 4), and the Knowledge-to-Action framework , we derived the second major result from this study, a synthesis framework for facilitating de-adoption (Fig. 4). At the heart of this framework is the identification and prioritization of low-value practices. The identification process involves determining the low-value practice(s) and selection of the knowledge unit that defines a practice as low-value (i.e., randomized clinical trial, systematic review, and/or clinical practice guideline). With regard to prioritization when there is more than one low-value practice identified, current literature suggests prioritizing based on strength of evidence supporting lack of efficacy, safety of the low-value practice (i.e., harmful practices eliminated first), potential health and cost impact of de-adoption, and availability of alternative practices. To permit more of an integrated de-adoption process, and thus improve the probability of success, we suggest stakeholder engagement take place concomitant with practice identification and prioritization. The de-adoption process is then envisioned to follow a similar action cycle as in the original Knowledge-to-Action cycle . However, given the anticipated challenges associated with discontinuing established clinical practices , the analysis of barriers and facilitators will require a greater in-depth exploration of both scientific (e.g., presence and quality of evidence supporting de-adoption) and non-scientific (e.g., historical, political, social, and economic factors) barriers to de-adoption . In addition, the intervention that guides de-adoption will likely need to be more closely integrated into clinical care pathways compared to that for adoption, with policy changes and/or changes to funding models predicted to have the greatest likelihood of facilitating de-adoption. Implementation of the intervention will need to be evaluated, and outcomes such as low-value practice use, costs, and potential harms assessed. Finally, any de-adoption intervention should include a sustainability plan; else it is highly likely that healthcare providers will (knowingly or unknowingly) revert to using the practice to which they have become habituated .
The third important result from this review is the identification of key questions that require additional research to advance the science of de-adoption. For example, there are multiple factors that likely determine when a practice should be de-adopted (e.g., nature of the intervention, lack of effectiveness or degree of harm, nature of the evidence) but the role of each factor and the interplay among them that ultimately determines when to de-adopt is not clear. In addition, what do we do with clinical practices that are ineffective for a broad population, but may be effective in a small subgroup that is difficult to study? To answer these and other questions we need additional knowledge syntheses that establish a taxonomy of de-adoption terminology, summarize barriers and facilitators to de-adoption, and quantify the impact of past examples of de-adoption. We also need empirical research to examine optimal strategies for identifying candidate low-value practices, and to determine which de-adoption strategies are likely to have the greatest impact. Furthermore, given existing fiscal climates with limited resources, we also need to balance the need to refine and prioritize the science of de-adoption with the need to do the same for adopting new practices.
While we await this additional research, what can healthcare decision-makers practically do with the existing knowledge base? First, this review highlights that de-adoption requires a multi-dimensional construct that is far more complex than simply ceasing to provide a given practice. Second, several studies have demonstrated that de-adoption does occur in response to publication of new evidence (Table 6), with the most consistent de-adoption occurring in response to an active change intervention. The intervention with the greatest likelihood of de-adoption is market withdrawal of a harmful drug. However, the real challenge lies in how to actively facilitate de-adoption when market withdrawal is not possible (e.g., insulin ), or not clearly indicated (e.g., practices that are simply ineffective). Interventions cited as having the greatest likelihood of effecting de-adoption include changes to policies, and/or restructuring of funding associated with the low-value practice, the latter through strategies of disinvestment, reinvestment, or defunding. However, this scoping review did not identify any studies that applied a strategy of disinvestment in response to evidence for medical reversal. At this point, pending further research, we suggest use of our proposed synthesis model (Fig. 4) as a starting point for anyone interested in promoting the de-adoption of low-value practices.
There are limitations to this review. First, our search may have missed relevant articles due to the lack of indexing terminology specific to de-adoption that for practical reasons forced us to restrict the search to English language articles published from 1990 onwards. However, the majority of included citations were published after 1999, and originated in high-income countries, therefore it is unlikely that we missed any broad concepts related to de-adoption. Second, grouping articles and de-adoption terminology according to the main categories in the conceptual framework, even though completed in duplicate by independent reviewers, is partly subjective. Finally, we elected to conduct a scoping review in order to provide an inclusive and broad description of what is known about de-adoption and therefore are limited in our ability to present granular details. Our work identifies opportunity for future systematic reviews.
De-adoption of low-value clinical practices is essential to improve healthcare quality and create a sustainable healthcare system. We identified a large body of literature that describes current approaches, and challenges to the de-adoption of low-value clinical practices. Our results should promote future research in at least two areas. First, knowledge syntheses are required to explore areas wherein there is an abundance of literature, such as establishing a taxonomy of de-adoption terminology, summarizing barriers and facilitators to de-adoption, and quantifying the impact of past examples of de-adoption. Second, empirical research is required to examine optimal strategies for identifying candidate low-value practices, and to determine which de-adoption strategies are likely to have the greatest impact. In the meantime, we have developed a conceptual model that providers and decision-makers can use to guide efforts to de-adopt ineffective and harmful practices and describe examples of successful de-adoption that can be used to inform efforts.
Canadian Agency for Drugs and Technologies in Health
Cochrane Central Register of Controlled Trials
Guideline for Not Funding Health Technology
Heart and Estrogen/progestin Replacement Study
hormone replacement therapy
health technology reassessment
intermittent positive pressure breathing
Medical Subject Heading
National Institute for Health and Care Excellence
non-steroidal anti-inflammatory drugs
pulmonary artery catheter
program budgeting and marginal analysis
percutaneous coronary intervention
Primary Care Trusts
proton pump inhibitor
Peer Review of Electronic Search Strategies
Preferred Reporting Items for Systematic Reviews and Meta-analyses
randomized controlled trial
retropubic radical prostatectomy
ST-elevation myocardial infarction
Women’s Health Initiative
Second International Study of Infarct Survival (ISIS-2) Collaborative Group. Randomised trial of intravenous streptokinase, oral aspirin, both, or neither among 17,187 cases of suspected acute myocardial infarction: ISIS-2. Lancet. 1988;332:349–60.
Assessment of the Safety and Efficacy of a New Thrombolytic (ASSENT-2) Investigators, Van De Werf F, Adgey J, Ardissino D, Armstrong PW, Aylward P, et al. Single-bolus tenecteplase compared with front-loaded alteplase in acute myocardial infarction: the ASSENT-2 double-blind randomised trial. Lancet. 1999;354:716–22.
Echt DS, Liebson PR, Mitchell LB, Peters RW, Obias-Manno D, Barker AH, et al. Mortality and morbidity in patients receiving encainide, flecainide, or placebo. The Cardiac Arrhythmia Suppression Trial. N Engl J Med. 1991;324:781–8.
Prasad V, Cifu A. Medical reversal: why we must raise the bar before adopting new technologies. Yale J Biol Med. 2011;84:471–8.
Prasad V, Gall V, Cifu A. The frequency of medical reversal. Arch Intern Med. 2011;171:1675–6.
Cameron C, Coyle D, Ur E, Klarenbach S. Cost-effectiveness of self-monitoring of blood glucose in patients with type 2 diabetes mellitus managed without insulin. CMAJ. 2010;182:28–34.
Prasad V, Vandross A, Toomey C, Cheung M, Rho J, Quinn S, et al. A decade of reversal: an analysis of 146 contradicted medical practices. Mayo Clinic Proc. 2013;88:790–8.
Elshaug AG, Watt AM, Mundy L, Willis CD. Over 150 potentially low-value health care practices: an Australian study. Med J Austr. 2012;197:556–60.
Rogers EM. The innovation-decision process. In: Diffusion of Innovations. 5th ed. New York, New York: Free Press; 2003. p. 168–218.
Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8:19–32.
Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5:69.
Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151:264–9. W264.
Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009;62:944–52.
Grey Matters: a practical search tool for evidence-based medicine. Canadian Agency for Drugs and Technologies in Health. 2013. http://www.cadth.ca/en/resources/grey-matters. Accessed 1 Mar 2014.
Straus SE, Tetroe J, Graham ID. Knowledge translation in health care: moving from evidence to practice. 2nd ed. Oxford, UK: John Wiley & Sons, Ltd.; 2013.
Greenhalgh T, Robert G, Bate P, Macfarlane F, Kyriakidou O. Diffusion of innovations in health service organisations: a systematic literature review. Malden, MA: Blackwell Publishing Ltd.; 2005.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74.
Hauptman PJ, Schnitzler MA, Swindle J, Burroughs TE. Use of nesiritide before and after publications suggesting drug-related risks in patients with acute decompensated heart failure. JAMA. 2006;296:1877–84.
Polisena J, Clifford T, Elshaug AG, Mitton C, Russell E, Skidmore B. Case studies that illustrate disinvestment and resource allocation decision-making processes in health care: a systematic review. Int J Technol Assess Health Care. 2013;29:174–84.
Levin L. Disinvestment strategies based on evidence guided adoption and obsolescence of technologies: the Ontario experience. Canadian Foundation for Healthcare Improvement. 2011. http://www.cfhi-fcass.ca/Libraries/CEO_Forum_files/LevinENG.sflb.ashx. Accessed 4 Mar 2014.
Choosing Wisely: Five Things Physicians and Patients Should Question. American Board of Internal Medicine Foundation. 2012. http://www.choosingwisely.org/clinician-lists/. Accessed 4 Mar 2014.
Choosing Wisely Canada. Canadian Medical Association. 2014. http://www.choosingwiselycanada.org. Accessed 1 May 2014.
Hollingworth W, Busby J, Jones H, Sterne J. Can variation in hospital procedure rates identify candidates for health technology reassessment and disinvestment? Value Health. 2013;16:A470.
Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA. 2005;294:218–28.
Roumie CL, Grogan EL, Falbe W, Awad J, Speroff T, Dittus RS, et al. A three-part intervention to change the use of hormone replacement therapy in response to new evidence. Ann Intern Med. 2004;141:118–25.
Sindby JE, Brocki BC, Rasmussen BS, Gorst-Rasmussen A, Andreasen JJ. Efforts to change transfusion practice behaviour and reduce transfusion rates are effective in coronary artery bypass surgery. J Cardiothorac Vasc Anesth. 2011;1:S46.
Krol N, Wensing M, Haaijer-Ruskamp F, Muris JWM, Numans ME, Schattenberg G, et al. Patient-directed strategy to reduce prescribing for patients with dyspepsia in general practice: a randomized trial. Aliment Pharmacol Ther. 2004;19:917–22.
Kulawik D, Sands JJ, Mayo K, Fenderson M, Hutchinson J, Woodward C, et al. Focused vascular access education to reduce the use of chronic tunneled hemodialysis catheters: results of a network quality improvement initiative. Semin Dial. 2009;22:692–7.
Ross-Degnan D, Soumerai SB, Fortess EE, Gurwitz JH. Examining product risk in context. Market withdrawal of zomepirac as a case study. JAMA. 1993;270:1937–42.
Williams D, Singh M, Hind C. The effect of the withdrawal of rofecoxib on prescribing patterns of COX-2 inhibitors in Scotland. Br J Clin Pharmacol. 2006;62:366–8.
Thiebaud P, Patel BV, Nichol MB. Impact of rofecoxib withdrawal on cyclooxygenase-2 utilization among patients with and without cardiovascular risk. Value Health. 2006;9:361–8.
Barozzi N, Tett SE. What happened to the prescribing of other COX-2 inhibitors, paracetamol and non-steroidal anti-inflammatory drugs when rofecoxib was withdrawn in Australia? Pharmacoepidemiol Drug Saf. 2007;16:1184–91.
Sun SX, Lee KY, Bertram CT, Goldstein JL. Withdrawal of COX-2 selective inhibitors rofecoxib and valdecoxib: impact on NSAID and gastroprotective drug prescribing and utilization. Curr Med Res Opin. 2007;23:1859–66.
Setakis E, Leufkens HG, van Staa TP. Changes in the characteristics of patients prescribed selective cyclooxygenase 2 inhibitors after the 2004 withdrawal of rofecoxib. Arthritis Rheum. 2008;59:1105–11.
Sukel MP, van der Linden MW, Chen C, Erkens JA, Herings RM. Large-scale stopping and switching treatment with COX-2 inhibitors after the rofecoxib withdrawal. Pharmacoepidemiol Drug Saf. 2008;17:9–19.
Hsiao FY, Tsai YW, Huang WF. Changes in physicians’ practice of prescribing cyclooxygenase-2 inhibitor after market withdrawal of rofecoxib: a retrospective study of physician-patient pairs in Taiwan. Clin Ther. 2009;31:2618–27.
Stafford RS, Radley DC. National trends in antiobesity medication use. Arch Intern Med. 2003;163:1046–50.
Grady D, Redberg RF. Less is more: how less health care can result in better health. Arch Intern Med. 2010;170:749–50.
Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JP, et al. Biomedical research: increasing value, reducing waste. Lancet. 2014;383:101–4.
Cassel CK, Guest JA. Choosing wisely: helping physicians and patients make smart decisions about their care. JAMA. 2012;307:1801–2.
Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? J Contin Educ Health Prof. 2006;26:13–24.
Haas M, Hall J, Viney R, Gallego G. Breaking up is hard to do: why disinvestment in medical technology is harder than investment. Aust Health Rev. 2012;36:148–52.
Montini T, Graham ID. “Entrenched practices and other biases”: unpacking the historical, economic, professional, and social resistance to de-implementation. Implement Sci. 2015;10:24.
Duhigg C. The power of habit: why we do what we do in life and business. New York: Random House Trade Paperbacks; 2012.
Niven DJ, Rubenfeld GD, Kramer AA, Stelfox HT. Effect of published scientific evidence on glycemic control in adult intensive care units. JAMA Intern Med. 2015;175:801–9.
Watt AM, Hiller JE, Braunack-Mayer AJ, Moss JR, Buchan H, Wale J, et al. The ASTUTE Health study protocol: deliberative stakeholder engagements to inform implementation approaches to healthcare disinvestment. Implement Sci. 2012;7:101.
Watt AM, Willis CD, Hodgetts K, Elshaug AG, Hiller JE. Engaging clinicians in evidence-based disinvestment: role and perceptions of evidence. Int J Technol Assess Health Care. 2012;28:211–9.
Elshaug AG, Moss JR, Littlejohns P, Karnon J, Merlin TL, Hiller JE. Identifying existing health care services that do not provide value for money. Med J Austr. 2009;190:269–73.
Elshaug AG, Hiller JE, Moss JR. Exploring policy-makers’ perspectives on disinvestment from ineffective healthcare practices. Int J Technol Assess Health Care. 2008;24:1–9.
Nieuwlaat R, Schwalm JD, Khatib R, Yusuf S. Why are we failing to implement effective therapies in cardiovascular disease? Eur Heart J. 2013;34:1262–9.
Garner S, Littlejohns P. Disinvestment from low value clinical interventions: NICEly done? BMJ. 2011;343:d4519.
Henshall C, Schuller T, Mardhani-Bayne L. Using health technology assessment to support optimal use of technologies in current practice: the challenge of “disinvestment”. Int J Technol Assess Health Care. 2012;28:203–10.
Moynihan RN. A healthy dose of disinvestment. Med J Austr. 2012;196:158–8.
Ibargoyen-Roteta N, Gutierrez-Ibarluzea I, Asua J, Benguria-Arrate G, Galnares-Cordero L. Scanning the horizon of obsolete technologies: possible sources for their identification. Int J Technol Assess Health Care. 2009;25:249–54.
Ibargoyen-Roteta N, Gutierrez-Ibarluzea I, Asua J. Guiding the process of health technology disinvestment. Health Policy. 2010;98:218–26.
Chamberlain CA, Martin RM, Busby J, Gilbert R, Cahill DJ, Hollingworth W. Trends in procedures for infertility and caesarean sections: was NICE disinvestment guidance implemented? NICE recommendation reminders. BMC Public Health. 2013;13:112.
Gerdvilaite J, Nachtnebel A. Disinvestment: overview of disinvestment experiences and challenges in selected countries. Ludwig Boltzman Institut fur Health Technology Assessment. 2011. http://eprints.hta.lbg.ac.at/926/##. Accessed 4 Mar 2014.
Leggett L, Noseworthy TW, Zarrabi M, Lorenzetti D, Sutherland LR, Clement FM. Health technology reassessment of non-drug technologies: current practices. Int J Technol Assess Health Care. 2012;28:220–7.
Leggett LE, Mackean G, Noseworthy TW, Sutherland L, Clement F. Current status of health technology reassessment of non-drug technologies: survey and key informant interviews. Health Res Policy Syst. 2012;10:38.
Cooper C, Starkey K. Disinvestment in health care. BMJ. 2010;340:c1413–3.
Karnon J, Carlton J, Czoski-Murray C, Smith K. Informing disinvestment through cost-effectiveness modelling: is lack of data a surmountable barrier? Appl Health Econ Health Policy. 2009;7:1–9.
Hodgetts K, Elshaug AG, Hiller JE. What counts and how to count it: physicians constructions of evidence in a disinvestment context. Soc Sci Med. 2012;75:2191–9.
Hughes DA, Ferner RE. New drugs for old: disinvestment and NICE. BMJ. 2010;340:c572.
Street JM, Hennessy SE, Watt AM, Hiller JE, Elshaug AG. News and social media: windows into community perspectives on disinvestment. Int J Technol Assess Health Care. 2011;27:376–83.
Prasad V, Ioannidis JP. Evidence-based de-implementation for contradicted, unproven, and aspiring healthcare practices. Implement Sci. 2014;9:1.
Elshaug AG, Hiller JE, Tunis SR, Moss JR. Challenges in Australian policy processes for disinvestment from existing, ineffective health care practices. Aust New Zealand Health Policy. 2007;4:23.
Williams IP, Bryan S. Cost-effectiveness analysis and formulary decision making in England: findings from research. Soc Sci Med. 2007;65:2116–29.
Mortimer D. Reorienting programme budgeting and marginal analysis (PBMA) towards disinvestment. BMC Health Serv Res. 2010;10:288.
Nuti S, Vainieri M, Bonini A. Disinvestment for re-allocation: a process to identify priorities in healthcare. Health Policy. 2010;95:137–43.
Garner S, Docherty M, Somner J, Sharma T, Choudhury M, Clarke M, et al. Reducing ineffective practice: challenges in identifying low-value health care using Cochrane systematic reviews. J Health Serv Res Policy. 2013;18:6–12.
Noseworthy T, Clement F. Health technology reassessment: scope, methodology, & language. Int J Technol Assess Health Care. 2012;28:201–2.
Donaldson C, Bate A, Mitton C, Dionne F, Ruta D. Rational disinvestment. QJM. 2010;103:801–7.
Haines T, O'Brien L, McDermott F, Markham D, Mitchell D, Watterson D, et al. A novel research design can aid disinvestment from existing health technologies with uncertain effectiveness, cost-effectiveness, and/or safety. J Clin Epidemiol. 2014;67:144–51.
Paulden M. Investment and disinvestment of health technologies: The need for two cost-effectiveness thresholds. Value Health. 2012;15:A33.
Pearson S, Littlejohns P. Reallocating resources: how should the National Institute for Health and Clinical Excellence guide disinvestment efforts in the National Health Service? J Health Serv Res Policy. 2007;12:160–5.
Hislop JM. Societal preferences for health technology disinvestment policy: Views of Scottish taxpayers - a qualitative study. Value Health. 2011;14:A356–7.
Kelly M. Public health programmes and interventions and disinvestment. National Institute for Health and Care Excellence. 2006. https://www.nice.org.uk/proxy/?sourceUrl=http%3A%2F%2Fwww.nice.org.uk%2FniceMedia%2Fpdf%2Fsmt%2F040406item5.pdf . Accessed 4 Mar 2014.
Garner S. Disinvestment: the UK experience. International Society for Pharmacoeconomics and Outcomes Research, 15th Annual European Congress. 2012. http://www.ispor.org/congresses/berlin1112/presentations/W10_Garner.pdf. Accessed 4 Mar 2014.
Harris E, Mundy L, Hewson K, Jacobsen N. Disinvestment in Australia and New Zealand. HealthPACT. 2013. http://www.health.qld.gov.au/healthpact. Accessed 4 Mar 2014.
Joshi NP, Stahnisch FW, Noseworthy TW. Reassessment of health technologies: obsolescence and waste. Canadian Agency for Drugs & Technologies in Health. 2009. https://www.cadth.ca/reassessment-health-technologies-obsolescence-and-waste. Accessed 22 October 2013.
Leng G. Introduction of new disinvestment programmes. National Institute for Health & Care Excellence. 2006. https://www.nice.org.uk/proxy/?sourceUrl=http%3A%2F%2Fwww.nice.org.uk%2FniceMedia%2Fpdf%2Fsmt%2F210206item3.pdf. Accessed 4 Mar 2014.
Kowalczyk KJ, Levy JM, Caplan CF, Lipsitz SR, Yu H-y. Gu X, et al. Temporal national trends of minimally invasive and retropubic radical prostatectomy outcomes from 2003 to 2007: results from the 100 % Medicare sample. Eur Urol. 2012;61:803–9.
Ioannidis JP. In reply II-reversal of medical practices. Mayo Clinic Proc. 2013;88:1184.
Bonakdar tehrani A, Howard D. Potential cost savings from comparative effectiveness research: lessons from COURAGE study. Value Health. 2011;14:A2.
Prasad V, Cifu A. In reply I-reversal of medical practices. Mayo Clinic Proc. 2013;88:1183–4.
Tatsioni A, Siontis GCM, Ioannidis JPA. Partisan perspectives in the medical literature: a study of high frequency editorialists favoring hormone replacement therapy. J Gen Intern Med. 2010;25:914–9.
Koo KK, Sun JC, Zhou Q, Guyatt G, Cook DJ, Walter SD, et al. Pulmonary artery catheters: evolving rates and reasons for use. Crit Care Med. 2011;39:1613–8.
Luetmer MT, Kallmes DF. Have referral patterns for vertebroplasty changed since publication of the placebo-controlled trials? Am J Neuroradiol. 2011;32:647–8.
Deyell MW, Buller CE, Miller LH, Wang TY, Dai D, Lamas GA, et al. Impact of National Clinical Guideline recommendations for revascularization of persistently occluded infarct-related arteries on clinical practice in the United States. Arch Int Med. 2011;171:1636–43.
Atwater BD, Oujiri J, Wolff MR. The immediate impact of the Clinical Outcomes Utilizing Revascularization and Aggressive Drug Evaluation (COURAGE) trial on the management of stable angina. Clin Cardiol. 2009;32:E1–3.
Wiener RS, Welch HG. Trends in the use of the pulmonary artery catheter in the United States, 1993–2004. JAMA. 2007;298:423–9.
Murphy DJ, Needham DM, Netzer G, Zeger SL, Colantuoni E, Ness P, et al. RBC transfusion practices among critically ill patients. Crit Care Med. 2013;41:2344–53.
Ahmed B, Dauerman HL, Piper WD, Robb JF, Verlee MP, Ryan Jr TJ, et al. Recent changes in practice of elective percutaneous coronary intervention for stable angina. Circ Cardiovasc Qual Outcomes. 2011;4:300–5.
Ehrenstein V, Hernandez RK, Ulrichsen SP, Rungby J, Lash TL, Riis AH, et al. Rosiglitazone use and post-discontinuation glycaemic control in two European countries, 2000–2010. BMJ Open. 2013;3, e003424.
Xie F, Petitti DB, Chen W. Prescribing patterns for antihypertensive drugs after the Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial: report of experience in a health maintenance organization. Am J Hypertens. 2005;18:464–9.
Stafford RS, Furberg CD, Finkelstein SN, Cockburn IM, Alehegn T, Ma J. Impact of clinical trial results on national trends in alpha-blocker prescribing, 1996–2002. JAMA. 2004;291:54–62.
Majumdar SR, Inui TS, Gurwitz JH, Gillman MW, McLaughlin TJ, Soumerai SB. Influence of physician specialty on adoption and relinquishment of calcium channel blockers and other treatments for myocardial infarction. J Gen Intern Med. 2001;16:351–9.
Majumdar SR, Almasi EA, Stafford RS. Promotion and prescribing of hormone therapy after report of harm by the Women’s Health Initiative. JAMA. 2004;292:1983–8.
Austin PC, Mamdani MM, Tu K, Jaakkimainen L. Prescriptions for estrogen replacement therapy in Ontario before and after publication of the Women’s Health Initiative Study. JAMA. 2003;289:3241–2.
Massatti RR, Sweeney HA, Panzano PC, Roth D. The de-adoption of innovative mental health practices (IMHP): why organizations choose not to sustain an IMHP. Adm Policy Mental Health. 2008;35:50–65.
Howard DH, Kenline C, Lazarus HM, Lemaistre CF, Maziarz RT, McCarthy Jr PL, et al. Abandonment of high-dose chemotherapy/hematopoietic cell transplants for breast cancer following negative trial results. Health Serv Res. 2011;46:1762–77.
Azermai M, Vander Stichele RR, Van Bortel LM, Elseviers MM. Barriers to antipsychotic discontinuation in nursing homes: an exploratory study. Aging Ment Health. 2014;18:346–53.
Hersh AL, Stefanick ML, Stafford RS. National use of postmenopausal hormone therapy: annual trends and response to recent evidence. JAMA. 2004;291:47–53.
Tatsioni A, Bonitsis NG, Ioannidis JPA. Persistence of contradicted claims in the literature. JAMA. 2007;298:2517–26.
Duffy SQ, Farley DE. The protracted demise of medical technology. The case of intermittent positive pressure breathing. Med Care. 1992;30:718–36.
Howard DH, Shen YC. Comparative effectiveness research, technological abandonment, and health care spending. Adv Health Econ Health Serv Res. 2012;23:103–21.
Prasad V, Cifu A, Ioannidis JPA. Reversals of established medical practices: evidence to abandon ship. JAMA. 2012;307:37–8.
Banta HD, Thacker SB. The case for reassessment of health care technology. Once is not enough. JAMA. 1990;264:235–40.
Elshaug AG, Watt AM, Moss JR, Hiller JE. Policy perspectives on the obsolescence of health technologies in Canada. Canadian Agency for Drugs and Technologies in Health. 2009. https://www.cadth.ca/media/pdf/Obsolescence%20of%20Health%20Technologies%20in%20Canada_Policy_Forum_e.pdf. Accessed 21 Oct 2013.
MacKean G, Noseworthy T, Elshaug AG, Leggett L, Littlejohns P, Berezanski J, et al. Health technology reassessment: the art of the possible. Int J Technol Assess Health Care. 2013;29:418–23.
Towards better patient care: drugs to avoid. Prescrire Int. 2013, 22:108–111.
Kiechle FL, Arcenas RC, Rogers LC. Establishing benchmarks and metrics for disruptive technologies, inappropriate and obsolete tests in the clinical laboratory. Clin Chim Acta. 2014;427:131–6.
Prasad V, Vandross A. Cardiovascular primary prevention: how high should we set the bar? Arch Intern Med. 2012;172:656–9. discussion 659.
Warner JL, Yang P, Alterovitz G. Reversal of medical practices. Mayo Clinic Proc. 2013;88:1182–3.
Medical Reversal. Life in the FastLane. 2014. http://lifeinthefastlane.com/education/ccc/medical-reversal. Accessed 3 Jun 2014.
Moscucci M. Medical reversal, clinical trials, and the “late” open artery hypothesis in acute myocardial infarction. Arch Intern Med. 2011;171:1643–4.
Fatovich DM. Medical reversal: what are you doing wrong for your patient today? Emerg Med Australas. 2013;25:1–3.
Haas JS, Kaplan CP, Gerstenberger EP, Kerlikowske K. Changes in the use of postmenopausal hormone therapy after the publication of clinical trial results. Ann Intern Med. 2004;140:184–8.
Siontis GCM, Tatsioni A, Katritsis DG, Ioannidis JPA. Persistent reservations against contradicted percutaneous coronary intervention indications: citation content analysis. Am Heart J. 2009;157:695–701.
Brunt ME, Murray MD, Hui SL, Kesterson J, Perkins AJ, Tierney WM. Mass media release of medical research results: an analysis of antihypertensive drug prescribing in the aftermath of the calcium channel blocker scare of March 1995. J Gen Intern Med. 2003;18:84–94.
Smalley W, Shatin D, Wysowski DK, Gurwitz J, Andrade SE, Goodman M, et al. Contraindicated use of cisapride: impact of food and drug administration regulatory action. JAMA. 2000;284:3036–9.
Huang WF, Tsai YW, Hsiao FY, Liu WC. Changes of the prescription of hormone therapy in menopausal women: an observational study in Taiwan. BMC Public Health. 2007;7:56.
Naylor CD. The complex world of prescribing behavior. JAMA. 2004;291:104–6.
Lawton B, Rose S, McLeod D, Dowell A. Changes in use of hormone replacement therapy after the report from the Women’s Health Initiative: cross sectional survey of users. BMJ. 2003;327:845–6.
Voorn VM, de Mheen PJ. M-v, So-Osman C, Vlieland TP, Koopman-van Gemert AW, Nelissen RG, et al. Designing a strategy to implement cost-effective blood transfusion management in elective hip and knee arthroplasties: a study protocol. Implement Sci. 2012;7:58.
Atkins D. Connecting research and patient care: lessons from the VA’s Quality Enhancement Research Initiative. J General Intern Med. 2009;25:1–2.
Elshaug AG, McWilliams JM, Landon BE. The value of low-value lists. JAMA. 2013;309:775–6.
Gershengorn HB, Wunsch H. Understanding changes in established practice: pulmonary artery catheter use in critically ill patients. Crit Care Med. 2013;41:2667–76.
Colmenares P. Proposal for a state health technology assessment program. WMJ. 2012;111:176–82.
‘Do not do’ recommendations. National Institute for Health and Care Excellence. 2014 http://www.nice.org.uk/proxy/?sourceUrl=http%3a%2f%2fwww.nice.org.uk%2fusingguidance%2fdonotdorecommendations%2fsearch.jsp. Accessed 3 Jun 2014.
We would like to acknowledge Diane Lorenzetti, MLS (University of Calgary) for assistance with the literature search strategy, and Laure Perrier, MSc (University of Toronto) for peer review of the literature search strategy. We also acknowledge Andrea Patey, MSc (School of Health Sciences, City University London), and Dr Sumit Majumdar, MD MSc FRCPC (University of Alberta) for identifying additional relevant citations.
DJN is funded through a Clinician Fellowship Award from Alberta Innovates – Health Solutions, and a Knowledge Translation Canada Student Fellowship and Training Program grant. HTS is supported by a Population Health Investigator Award from Alberta Innovates – Health Solutions. SES is funded by a Tier 1 Canada Research Chair. BRH is supported by the Roy and Vi Baay Chair in Kidney Research. The funding agencies did not contribute to design and conduct of the study; collection, management, analysis, or interpretation of the data; or preparation, review, or approval of the final manuscript.
The authors declare they have no competing interests.
DJN designed the study, screened citations for inclusion, extracted data from included citations, performed and interpreted all analyses, and drafted the final manuscript. KJM and JKH screened citations for inclusion, extracted data from included citations, interpreted analyses, and revised the manuscript for important intellectual content. BRH and LPJ assisted with study design, interpreted analyses, and revised the manuscript for important intellectual content. SES and HTS supervised the conduct of the study, and as such were involved with study design, interpretation of analyses, and contributed key revisions to the intellectual content of the manuscript. All authors agree to be accountable for all aspects of the work and approved the final manuscript for submission.
Table S1. Description of articles included in the scoping review. Appendix. MEDLINE search. (DOCX 201 kb)