- Research article
- Open Access
- Open Peer Review
Is the endocrine research pipeline broken? A systematic evaluation of the Endocrine Society clinical practice guidelines and trial registration
BMC Medicine volume 13, Article number: 187 (2015)
Very low quality (VLQ) evidence translates into very low confidence in the balance of risk and benefits based on the estimates drawn from the body of evidence. Consequently, this assessment highlights gaps in the research evidence, i.e. knowledge gaps, for important clinical questions. In this way, expert guideline panels identify priority knowledge gaps that, arguably, should inform the research agenda and prioritize scarce research economical resources. The extent to which the research agenda reflects the knowledge gaps identified in clinical practice guidelines is unknown.
A systematic evaluation of the Endocrine Society (ES) clinical practice guidelines portfolio from 2008 to 2014 was conducted with the objectives to identify (1) recommendations in the ES clinical practice guidelines based on VLQ evidence reflecting knowledge gaps in endocrinology, and (2) active research designed to address these gaps by searching the clinical trial registry, clinicaltrials.gov, using terms describing patients (diseases), interventions, comparison, and outcomes.
In 25 ES guidelines, we found 660 recommendations, of which 131 (20 %) were supported by VLQ evidence. Clinical trialists are attempting to answer 28 (21 %) of these knowledge gaps by performing 69 clinical trials.
The research enterprise is addressing one in five knowledge gaps identified in clinical practice recommendations in endocrinology. These findings suggest an inefficiency in the allocation of very scarce research economical resources. Linking the research agenda to evidence gaps in clinical practice guidelines may improve both the efficiency of the research enterprise and the translation of evidence into more confident clinical practice.
The Endocrine Society (ES) has created clinical practice guidelines to aid clinicians in the care of patients with endocrine disorders. In 2005, the ES adopted the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) Group system [1–4]. The GRADE approach rates the panel’s confidence in the risk estimates of favorable and unfavorable outcomes. This confidence in the estimates is captured by classifying the quality of the body of evidence supporting a recommendation into one of four categories: high, moderate, low, and very low quality (VLQ) [1–4]. VLQ evidence results when the body of evidence is comprised of studies at high risk of bias, yielding imprecise results or results of indirect relevance to the recommendation, and/or results that are inconsistent across studies or are not fully reported. VLQ evidence translates into very low confidence in the balance of risk and benefits based on the estimates drawn from the body of evidence [3, 4]. Consequently, this assessment highlights gaps in the research evidence, i.e. knowledge gaps, for important clinical questions. Since these knowledge gaps affect the assessments about the balance of benefits and harms of alternative courses of action they reduce our confidence that patients will be better off were they to receive care consistent with recommendations based on VLQ evidence.
A recent study showed that low to very low quality evidence, largely derived from small observational studies, supported most endocrinology guideline recommendations [5, 6]. To this extent, guidelines that explicitly account for confidence in estimates from the body of evidence, such as ES guidelines, can be used to identify important knowledge gaps and guide the research agenda. This is particularly important in the face of scarce, research dollars. Moreover, it has been previously estimated that 85 % of research is of low impact or wasted mainly due to being unnecessary, poorly designed, biased, unusable, incompletely published, or simply addressing the wrong research question. A better connection between knowledge gaps identified in practice guidelines and the research enterprise could reduce research waste [7–12]. To explore the integrity of the pipeline connecting knowledge gaps with ongoing research, we conducted a study using ES guidelines.
We conducted a systematic evaluation of the available ES clinical practice guidelines to identify clinical recommendations that are based on VLQ evidence and that potentially reflect knowledge gaps in endocrinology. Using the ES guideline web site, we identified and retrieved all ES clinical practice guidelines issued from 2008 to 2014 [13–38]. For each guideline, two reviewers working independently searched and extracted the number of graded recommendations in each guideline and those rated as based on VLQ evidence. Guideline panels following the GRADE approach, as is the case with ES guidelines, rate evidence as VLQ when the body of evidence produces estimates about which we have very low confidence. Studies produce low-confidence estimates when they are at high risk of bias, produce results that are of indirect relevance to the recommendation, imprecise or inconsistent, or when there is evidence of incomplete and biased reporting [3, 4, 39].
Because of problems with classification, some recommendations based on VLQ evidence should not have been graded as they represent best practice statements in which there is no sensible alternative. In this sense, best practice recommendations are thought to do substantially more good than harm (or vice versa) and therefore no one would consider doing a study to definitively establish the answer to the implicit question . Examples of best practice recommendations include: (1) “We suggest that female-to-male transsexual persons evaluate the risks and benefits of including total hysterectomy and oophorectomy as part of sex reassignment surgery”, and (2) “In patients presenting with heart failure, initial assessment should be made of the patient’s ability to perform routine/desired activities of daily living” [20, 38, 41]. After excluding these, the crude inter-observer agreement for the identification of recommendations based of VLQ evidence was 96 %.
We then identified the research questions relevant to these knowledge gaps in terms of patients, interventions, comparisons, and outcomes (PICO). For each VLQ evidence item, we drafted a research question that a clinical trial could answer using the PICO format (e.g. Patients – patients with large adrenal pheochromocytomas; Intervention – minimally invasive adrenalectomy; Comparison – open resection adrenalectomy; Outcome – complete tumor resection, tumor rupture avoidance, and local recurrence rates) [1, 35, 39]. The objective of this step was to make the underlying questions explicit thus ensuring reproducibility of our methods. To calibrate this process, two researchers independently produced these questions for 11 recommendations with an initial agreement of 80 %. This process was repeated until the agreement was 100 %, achieved after 20 recommendations.
We then searched the clinicaltrials.gov database for active studies (randomized and observational) addressing the questions identified from the guidelines. Since 2000, this registry has emerged as the most complete including trials from 188 countries. A clinical trial was deemed eligible if it addressed the PICO question with at least one of the necessary outcomes. For each eligible clinical trial, we extracted the study design, source of funding, year of entry, and the sample size. Although we did not look for publication of results, we excluded trials completed 5 or more years prior to the date of guideline publication. Reviewers working independently searched the clinical trial registry until 100 % agreement, a point reached after searching for 20 questions. A single reviewer completed the search for the rest of the questions.
We performed a descriptive analysis and summarized continuous variables as mean (SD) and presented percentages in cases of categorical variables.
We identified 25 Clinical Practice Guidelines from the ES with 660 recommendations, of which 209 (32 %) were supported by VLQ evidence [8–32]. After excluding 78 (12 %) best practice statements, the total was 131 (20 %) recommendations based on VLQ evidence (Fig. 1). The majority of the guidelines supported the care of patients with pituitary, gonadal, and adrenal disorders , and most recommendations supported by VLQ evidence came from these guidelines (24 %; Fig. 2).
Active research was identified for 28 (21 %) of these 131 recommendations represented by 69 clinical studies (mean of 2.5 studies per each recommendation – when taking into consideration those with active research; Fig. 1). Of these, 35 (51 %) were randomized trials and 34 (49 %) were observational. Thirteen reported industry funding and six were multicenter studies. Of the 69 active studies, 42 (60 %) were in the guidelines of thyroid dysfunction during pregnancy, testosterone therapy in adult men, and diagnosis of Cushing syndrome which had a total of 32 recommendations supported by VLQ evidence, of which 15 had at least one active research study.
Most of the identified clinical trials addressed knowledge gaps affecting recommendations for patients with thyroid disorders (70 %); the least dealt with were gaps in the evidence for care of patients with diabetes, obesity, and cardiovascular disease (16 %; Fig. 2).
Important knowledge gaps are evident in 10 of every 50 ES clinical practice guideline recommendations, of which the research enterprise is actively addressing, at best, two. Moreover, of the active trials, 60 % are trying to improve the quality of evidence in only three of the 25 guidelines. In some cases, several studies are actively addressing the same question. The research enterprise, thus characterized, poorly reflects the knowledge gaps at the frontline of endocrine practice, covering these gaps incompletely and sometimes redundantly. Multiple explanations exist for these observations. The research enterprise may not be aware of these gaps because funding agencies and researchers do not use clinical practice guidelines to identify knowledge gaps, or they may be responding to these gaps with basic studies that are not in the registry. Alternatively, ES expert ratings of confidence in estimates evidence have not been explicitly communicated to researchers and funders to facilitate the development of a practice-relevant research agenda.
This cross-sectional study contributes to the understanding of a seemingly broken research pipeline. However, there are dynamic aspects that only a longitudinal study could answer. For example, in its 2014 version, the ES guideline on Androgen Therapy in Women had seven recommendations based on VLQ evidence, all new since its 2006 version . Conversely, the evidence supporting two of their 2006 recommendations was upgraded in 2014 from warranting very low to warranting low confidence in the estimates [38, 42]. How these changes result from the manner in which the research enterprise responds to knowledge gaps merits further study.
To our knowledge, there is no comprehensive repository of knowledge gaps in endocrinology. This study partially reflects the coverage of the current ES guideline portfolio based on our selection of recommendations supported by VLQ evidence, instead of the much larger set of recommendations based on low quality evidence. A key advantage of the portfolio that enabled this study is the ES’s early adoption of GRADE, which enables the separation of the rating of evidence (used here) from the grading of the strength of recommendation . Further, we used the clinicaltrials.gov registry which, despite being the largest, may have missed research (e.g. well-designed observational studies, not registered studies), particularly outside the United States. Additionally, studies published after the publication of the clinical guidelines that could have assessed these gaps were not part of our analysis. Nevertheless, we identified knowledge gaps in the ES clinical practice guidelines in duplicate and performed a search of clinical trials to address them after high reproducibility was achieved between each of the independent reviewers.
We are also aware that, even though panels may be the best way to identify research gaps, it could be easily argued that this does not automatically mean that trials should be conducted in those areas, e.g. the needed studies might be prohibitively expensive, research gaps might be difficult to study, or the science field is not developed enough to support the conduct of clinical trials. On the other hand, panels choose areas of clinical relevance to formulate recommendations. Often, these represent ongoing practice, which is unlikely to be supported solely by mechanistic hypotheses or basic science data. While we do not know the extent to which our findings apply to other areas of medicine, they may very well represent a trigger to examine the research pipeline and improve the quality and relevance of the evidence. A functional pipeline connecting evidence to recommendations and back can ultimately better support decision making by patients, clinicians, policy makers, and funding agencies.
Researchers are addressing only one in five knowledge gaps identified in clinical practice recommendations in endocrinology. Linking the research agenda to evidence gaps in guidelines may improve both the efficiency of the research enterprise and the translation of evidence into practice by increasing the value and reducing the waste in research.
Grading of Recommendations, Assessment, Development and Evaluation System
Patient, Intervention/Exposure, Comparison and Outcome
Very low quality
Guyatt GH, Oxman AD, Schünemann HJ, Tugwell P, Knottnerus A. GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. J Clin Epidemiol. 2011;64:380–2.
GRADE online learning modules. McMaster University. http://cebgrade.mcmaster.ca/. Accessed April 16, 2015.
Balshem H, Helfand M, Schünemann HJ, Oxman AD, Kunz R, Brozek J, et al. GRADE guidelines: Rating the quality of evidence. J Clin Epidemiol. 2011;64:401–6.
Swiglo BA, Murad MH, Schünemann HJ, Kunz R, Vigersky RA, Guyatt GH, et al. A case for clarity, consistency, and helpfulness: state-of-the-art clinical practice guidelines in endocrinology using the grading of recommendations, assessment, development, and evaluation system. J Clin Endocrinol Metab. 2008;93:666–73.
Brito JP, Domecq JP, Murad MH, Guyatt GH, Montori VM. The Endocrine Society guidelines: when the confidence cart goes before the evidence horse. J Clin Endocrinol Metab. 2013;98:3246–52. doi:10.1210/jc.2013-1814.
Hazlehurst JM, Armstrong MJ, Sherlock M, Rowe IA, O’Reilly MW, Franklyn JA, et al. A comparative quality assessment of evidence-based clinical guidelines in endocrinology. Clin Endocrinol (Oxf). 2013;78:183–90. doi:10.1111/j.1365-2265.2012.04441.x.
Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374:86–9.
Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gülmezoglu AM, et al. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383:156–65.
Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383:166–75.
Al-Shahi Salman R, Beller E, Kagan J, Hemminki E, Phillips RS, Savulescu J, et al. Increasing value and reducing waste in biomedical research regulation and management. Lancet. 2014;383:176–85.
Chan AW, Song F, Vickers A, Jefferson T, Dickersin K, Gøtzsche PC, et al. Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014;383:257–66.
Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383:267–76.
The Endocrine Society. The Endocrine Society published guidelines. 2014. https://www.endocrine.org/education-and-practice-management/clinical-practice-guidelines.
Martin KA, Chang RJ, Ehrmann DA, Ibanez L, Lobo RA, Rosenfield RL, et al. Evaluation and treatment of hirsutism in premenopausal women: an endocrine society clinical practice guideline. J Clin Endocrinol Metab. 2008;93:1105–20.
Nieman LK, Biller BM, Findling JW, Newell-Price J, Savage MO, Stewart PM, et al. The diagnosis of Cushing’s syndrome: an Endocrine Society Clinical Practice Guideline. J Clin Endocrinol Metab. 2008;93:1526–40.
Funder JW, Carey RM, Fardella C, Gomez-Sanchez CE, Mantero F, Stowasser M, et al. Endocrine Society. Case detection, diagnosis, and treatment of patients with primary aldosteronism: an endocrine society clinical practice guideline. J Clin Endocrinol Metab. 2008;93:3266–81.
Rosenzweig JL, Ferrannini E, Grundy SM, Haffner SM, Heine RJ, Horton ES, et al. Primary prevention of cardiovascular disease and type 2 diabetes in patients at metabolic risk: an endocrine society clinical practice guideline. J Clin Endocrinol Metab. 2008;93:3671–89.
August GP, Caprio S, Fennoy I, Freemark M, Kaufman FR, Lustig RH, et al. Endocrine Society. Prevention and treatment of pediatric obesity: an endocrine society clinical practice guideline based on expert opinion. J Clin Endocrinol Metab. 2008;93:4576–99.
Cryer PE, Axelrod L, Grossman AB, Heller SR, Montori VM, Seaquist ER, et al. Endocrine Society. Evaluation and management of adult hypoglycemic disorders: an Endocrine Society Clinical Practice Guideline. J Clin Endocrinol Metab. 2009;94:709–28.
Hembree WC, Cohen-Kettenis P, de Delemarre-van de Waal HA, Gooren LJ, Meyer 3rd WJ, Spack NP, et al. Endocrine Society. Endocrine treatment of transsexual persons: an Endocrine Society clinical practice guideline. J Clin Endocrinol Metab. 2009;94:3132–54.
Bhasin S, Cunningham GR, Hayes FJ, Matsumoto AM, Snyder PJ, Swerdloff RS, et al. Task Force, Endocrine Society. Testosterone therapy in men with androgen deficiency syndromes: an Endocrine Society clinical practice guideline. J Clin Endocrinol Metab. 2010;95:2536–59. doi:10.1210/jc.2009-2354.
Speiser PW, Azziz R, Baskin LS, Ghizzoni L, Hensle TW, Merke DP, et al. Endocrine Society. Congenital adrenal hyperplasia due to steroid 21-hydroxylase deficiency: an Endocrine Society clinical practice guideline. J Clin Endocrinol Metab. 2010;95:4133–60.
Melmed S, Casanueva FF, Hoffman AR, Kleinberg DL, Montori VM, Schlechte JA, et al. Endocrine Society. Diagnosis and treatment of hyperprolactinemia: an Endocrine Society clinical practice guideline. J Clin Endocrinol Metab. 2011;96:273–88. doi:10.1210/jc.2010-1692.
Heber D, Greenway FL, Kaplan LM, Livingston E, Salvador J, Still C. Endocrine Society. Endocrine and nutritional management of the post-bariatric surgery patient: an Endocrine Society Clinical Practice Guideline. J Clin Endocrinol Metab. 2010;95:4823–43.
Freda PU, Beckers AM, Katznelson L, Molitch ME, Montori VM, Post KD, et al. Endocrine Society. Pituitary incidentaloma: an endocrine society clinical practice guideline. J Clin Endocrinol Metab. 2011;96:894–904. doi:10.1210/jc.2010-1048.
Molitch ME, Clemmons DR, Malozowski S, Merriam GR, Vance ML, Endocrine Society. Evaluation and treatment of adult growth hormone deficiency: an Endocrine Society clinical practice guideline. J Clin Endocrinol Metab. 2011;96:1587–609. doi:10.1210/jc.2011-0179.
Klonoff DC, Buckingham B, Christiansen JS, Montori VM, Tamborlane WV, Vigersky RA, et al. Endocrine Society. Continuous glucose monitoring: an Endocrine Society Clinical Practice Guideline. J Clin Endocrinol Metab. 2011;96:2968–79.
Holick MF, Binkley NC, Bischoff-Ferrari HA, Gordon CM, Hanley DA, Heaney RP, et al. Endocrine Society. Evaluation, treatment, and prevention of vitamin D deficiency: an Endocrine Society clinical practice guideline. J Clin Endocrinol Metab. 2011;96:1911–30.
Umpierrez GE, Hellman R, Korytkowski MT, Kosiborod M, Maynard GA, Montori VM, et al. Endocrine Society. Management of hyperglycemia in hospitalized patients in non-critical care setting: an endocrine society clinical practice guideline. J Clin Endocrinol Metab. 2012;97:16–38.
Watts NB, Adler RA, Bilezikian JP, Drake MT, Eastell R, Orwoll ES, et al. Endocrine Society. Osteoporosis in men: an Endocrine Society clinical practice guideline. J Clin Endocrinol Metab. 2012;97:1802–22.
De Groot L, Abalovich M, Alexander EK, Amino N, Barbour L, Cobin RH, et al. Management of thyroid dysfunction during pregnancy and postpartum: an Endocrine Society clinical practice guideline. J Clin Endocrinol Metab. 2012;97:2543–65.
Berglund L, Brunzell JD, Goldberg AC, Goldberg IJ, Sacks F, Murad MH, et al. Endocrine society. Evaluation and treatment of hypertriglyceridemia: an Endocrine Society clinical practice guideline. J Clin Endocrinol Metab. 2012;97:2969–89.
Blumer I, Hadar E, Hadden DR, Jovanovič L, Mestman JH, Murad MH, et al. Diabetes and pregnancy: an endocrine society clinical practice guideline. J Clin Endocrinol Metab. 2013;98:4227–49. doi:10.1210/jc.2013-2465.
Legro RS, Arslanian SA, Ehrmann DA, Hoeger KM, Murad MH, Pasquali R, et al. Endocrine Society. Diagnosis and treatment of polycystic ovary syndrome: an Endocrine Society clinical practice guideline. J Clin Endocrinol Metab. 2013;98:4565–92.
Lenders JW, Duh QY, Eisenhofer G, Gimenez-Roqueplo AP, Grebe SK, Murad MH, et al. Endocrine Society. Pheochromocytoma and paraganglioma: an endocrine society clinical practice guideline. J Clin Endocrinol Metab. 2014;99:1915–42.
Katznelson L, Laws Jr ER, Melmed S, Molitch ME, Murad MH, Utz A, et al. Endocrine Society. Acromegaly: an endocrine society clinical practice guideline. J Clin Endocrinol Metab. 2014;99:3933–51.
Singer FR, Bone 3rd HG, Hosking DJ, Lyles KW, Murad MH, Reid IR, et al. Paget’s disease of bone: an endocrine society clinical practice guideline. J Clin Endocrinol Metab. 2014;99:4408–22.
Wierman ME, Arlt W, Basson R, Davis SR, Miller KK, Murad MH, et al. Androgen therapy in women: a reappraisal: an Endocrine Society clinical practice guideline. Clin Endocrinol Metab. 2014;99:3489–510.
Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64:383–94.
Guyatt GH, Schünemann HJ, Djulbegovic B, Akl EA. Guideline panels should not GRADE good practice statements. J Clin Epidemiol. 2015;68:597–600.
Hunt SA, Abraham WT, Chin MH, Feldman AM, Francis GS, Ganiats TG, et al. 2009 focused update incorporated into the ACC/AHA 2005 Guidelines for the Diagnosis and Management of Heart Failure in Adults: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines: developed in collaboration with the International Society for Heart and Lung Transplantation. Circulation. 2009;119:e391–479.
Wierman ME, Basson R, Davis SR, Khosla S, Miller KK, Rosner W, et al. Androgen therapy in women: an Endocrine Society Clinical Practice guideline. J Clin Endocrinol Metab. 2006;91:3697–710.
The authors declare that they have no competing (financial and non-financial) interests.
NSO, RRG, and VMM designed the study, wrote the protocol, served as overall principal investigators, and wrote and reviewed the manuscript. JPB and WFY helped design the study, wrote and made a critical review of the manuscript, and assisted with protocol adaptations and submissions. NSO, RRG, and VMM are guarantors. All authors read and approved the final manuscript.
Naykky Singh Ospina and Rene Rodriguez-Gutierrez contributed equally to this work.