- Commentary
- Open Access
- Published:
The National Institutes of Health and guidance for reporting preclinical research
BMC Medicine volume 13, Article number: 34 (2015)
- The Erratum to this article has been published in BMC Medicine 2015 13:80
Abstract
The quality of reporting clinical and preclinical research is not optimal. Reporting guidelines can help make reports of research more complete and transparent, thus increasing their value and making them more useful to all readers. Getting reporting guidelines into practice is complex and expensive, and involves several stakeholders, including prospective authors, peer reviewers, journal editors, guideline developers, and implementation scientists. Working together will help ensure their maximum uptake and penetration. We are all responsible for helping to ensure that all research is reported so completely that it is of value to everybody.
Please see related article: http://dx.doi.org/10.1186/s12916-015-0266-y
Background
There is a large body of evidence indicating the very inadequate reporting of research, both clinical and preclinical. For example, in a survey of highly cited (>500 citations) animal studies from seven leading journals by impact factor (Science, Nature, Cell, Nature Medicine, Nature Genetics, Nature Immunology, and Nature Biotechnology) less than 20% of them described the randomization processes of the animals or blinding [1]. These findings have been consistent across preclinical research content areas (e.g., [2,3]).
To help increase the number of published papers that are completely and transparently reported, and thus reduce and/or eliminate the number of unusable publications, reporting guidelines have been developed since the early 1990s. There has been a proliferation in their numbers, particularly in the last decade. Reporting guidelines are popular; the EQUATOR Network’s library of reporting guidelines indicates more than 200 in existence, covering a wide variety of designs, data, populations, as well as clinical, preclinical, and other characteristics, with at least another score in development [4]. The Network is working to harmonize these activities and has also published advice on how to develop reporting guidelines [4], thus helping to ensure their rigorous development and usefulness. One reason for their popularity is that, when used appropriately by prospective authors, and endorsed and implemented optimally by journals, they have their intended effect, namely, of improving the completeness of research reports [5-7], thus increasing the value of research findings to clinicians, patients, and other interested readers.
Recently published reporting guidelines
Another reporting guideline – CoBRA, indicating how to optimally cite bioresources – has been published in BMC Medicine [8]. At least two other reporting guidelines have been published this year already [9,10]. In 2013, the National Institutes of Health entered the fray, supported by the Nature Publishing Group and Science magazine, and proposed the Principles and Guidelines for Reporting Preclinical Research (NIHPG) [11] with a meeting involving 30 journal editors [12]. This triumvirate is an important and influential group. The NIHPG include the better performance of statistical analysis, transparency in reporting, data and material sharing, consideration of refutations, and consideration of establishing best practice guidelines, with a focus on authors providing complete information about the methods used in their studies to allow interested readers to replicate them. Within a short period of time the NIHPG have been endorsed by an impressive, and growing, number of journals, such as Blood, and groups, including the American Association for Cancer Research and the World Association of Medical Editors [13].
Reporting preclinical research: how does NIHPG compare with existing guidelines?
In addition to the NIHPG, there are several other options [14-17] for prospective authors of preclinical research to report what they did and found, including the Animal Research: Reporting of In Vivo Experiments (ARRIVE) [18]. ARRIVE is a reporting guideline developed in partnership with the National Centre for the Replacement, Refinement and Reduction of Animals in Research [19]. The guidance was rigorously developed and followed a review of published animal studies that provided a rationale for moving forward. The development process also included a consensus meeting with a broad spectrum of stakeholders interested in animal research. At its core, the ARRIVE guidance is a 20-item checklist covering what authors should report in the Introduction, Methods, Results, and Discussion sections of their papers. For example, item 8 asks authors to report on the experimental animals used: “Provide details of the animals used, including species, strain, sex, developmental stage (e.g. mean or median age plus age range) and weight”. There is evidence that authors are not providing this information [20], despite it being essential if others are interested in replication [21].
The ARRIVE guidelines and NIHPG have different scope but similar objectives. The NIHPG is primarily focused on journal policy; for instance, the guidance recommends that journals have either no or generous limits on the length of their methods sections. In contrast, the ARRIVE guidelines are focused on reporting within the manuscript itself with its 20-item checklist, compared to the six ‘core’ reporting items found in the NIHPG. There is little difference between the six proposed items and their corresponding items (sub-items) in the ARRIVE checklist. Thus, the NIHPG items may be considered a minimum set of reporting items that are focused on the internal validity and statistics of a paper, whereas the ARRIVE guidelines are comprehensive and cover the entire paper from title to discussion. Indeed, the NIHPG even recommends following community standards such as the ARRIVE guidelines, suggesting that it is intended to co-exist with these other standards.
While the NIHPG and ARRIVE efforts to improve the reporting of preclinical research, are admirable, it is possible they might be seen as confusing, particularly to prospective authors. For example, in journals that endorse both ARRIVE (also endorsed by many journals – [22]) and NIHPG, authors might not know whether to use the NIHPG and/or ARRIVE and may elect to use neither, thus negating the efforts of both groups (in areas of overlap, such as statistics, should authors choose one guidance over another one and, if so, which one?). Indeed, a similar situation occurred during the developing of the CONSORT Statement for reporting randomized trials. The CONSORT guidance was produced about 6 months before the Asilomar guidance. Dr. Drummond Rennie, then deputy editor of JAMA, recommended the two groups work together to produce a single guidance thus providing more clarity and direction for prospective authors. Similarly, the CONSORT group wanted to incorporate the TIDieR guidance for describing interventions [23] into the CONSORT checklist. Members of both groups discussed how best to do this and provided some guidance on this for prospective authors [23].
While the NIHPG guidance appears to have face validity and support, a more pressing question is whether it will have its intended effect, namely, improving the completeness and transparency when reporting preclinical research, as well as facilitating the likelihood of reproducible studies being done. There is likely little merit in editors asking authors to use the guidance – the intervention – if it has only a minimal effect on relevant outcomes such as better reporting and enabling others to adequately replicate methods. Unfortunately, an assessment of the benefits (and harms) of reporting guidelines has not been a top priority of guideline developers [24], to date. We hope the NIHPG developers will actively plan to conduct an evaluation of their guidance and encourage others to do likewise. Attention to relevant outcomes will be important to consider. Evaluations assessing adherence to reporting guidelines (i.e., in addition to endorsement) may provide a more meaningful insight into its impact on completeness of reporting when used at different stages of the editorial process. One such recently performed evaluation [25] incorporated these concepts by comparing the use and non-use of reporting guidelines during peer review on author-revised manuscript quality.
Having produced the NIHPG – an intervention to help improve the transparency and better reporting of preclinical research – it is important to consider its endorsement and implementation. More generally, are journals using similar explicit language regarding the endorsement of either guidance? Our experiences are that there is a wide variability in the language of endorsement and that this is confusing to prospective authors, thus potentially reducing the intended effectiveness of the guidance. Guideline developers are starting to provide greater guidance to help journal editors achieve a more standard and successful endorsement strategy [9].
Implementing reporting guidelines
While endorsement is an important step, how journals implement reporting guidelines is a critical factor. There is wide variability in how journals implement reporting guidelines [26] possibly because guideline developers have not focused on how to do this until recently [27,28]. If editorial procedures are not consistently available online it will be difficult to assess journal implementation of NIHPG endorsement (that is, verification by the journals’ editorial team of author adherence to it). Editors may want to develop explicit statements about their journals’ endorsement of NIHPG, and other reporting guidelines, in their ‘Instructions to Authors’, and optimally to recommend submission of relevant checklists at the time of manuscript submission. To further implementation practices, editors may also want to recommend peer reviewers use reporting guidelines during their manuscript review assessment [29]. Such active implementation policies by journals will lead to more complete, clear, transparent, and reproducible publications, ultimately increasing the usability and value of preclinical research reports.
Well-developed reporting guidelines can improve the reporting and usefulness of all research. If studies are reported with enough detail for the findings to be implemented in practice, they are more helpful to health care providers, policy makers, and patients for making decisions. Transparent reporting also allows decision makers to judge the internal validity and applicability of the research, and enables others to reproduce the findings. Currently we do not know which reporting guidelines are most helpful in achieving this goal because of the variability in their development; this was a discussion theme at a recent EQUATOR Network meeting [30]. The development of a tool to assess the robustness (and other features) of reporting guidelines will help all stakeholders, particularly research funders and journal editors, to make decisions on which reporting guidelines to focus their endorsement and implementation efforts. Similarly, robust guidelines are likely the ones for prospective authors to spend their time using.
Getting reporting guidelines into practice is complex and expensive, and involves many players, including prospective authors, peer reviewers, journal editors, guideline developers, and implementation scientists. Working together will help ensure their maximum uptake and penetration. Technology is also likely to facilitate implementation of reporting guidelines, such as software to help prospective authors ensure they remember to report on every aspect of the research they are reporting, and journal editors to assess the degree of compliance to guidelines they endorse. Unfortunately, to date, there has been very limited funding available to develop and help maximize the potential of reporting guideline uptake. Perhaps having the NIH involved in reporting guideline efforts will encourage them and others to commit more resources to these efforts. We are all responsible for helping to ensure that all research is reported so completely that it is of value to everybody.
Abbreviations
- ARRIVE:
-
Animal Research: Reporting of In Vivo Experiments
- NIHPG:
-
National Institutes of Health Principles and Guidelines for Reporting Preclinical Research
References
- 1.
Hackam DG, Redelmeier DA. Translation of research evidence from animals to humans. JAMA. 2006;296:1731–2.
- 2.
Kilkenny C, Parsons N, Kadyszewski E, Festing MFW, Cuthill IC, Fry D, et al. Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PLoS ONE. 2009;4:e7824.
- 3.
Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383:166–75.
- 4.
http://www.equator-network.org/library/.
- 5.
Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7:e1000217.
- 6.
Turner L, Shamseer L, Altman DG, Weeks L, Peters J, Kober T, et al. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomized controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev. 2012;11, MR000030.
- 7.
Smidt N, Rutjes AW, van der Windt DA, Ostelo RW, Bossuyt PM, Reitsma JB, et al. The quality of diagnostic accuracy studies since the STARD statement: has it improved? Neurology. 2006;67:792–7.
- 8.
Tunis AS, McInnes MD, Hanna R, Esmail K. Association of study quality with completeness of reporting: have completeness of reporting and quality of systematic reviews and meta-analyses in major radiology journals changed since publication of the PRISMA statement? Radiology. 2013;269:413–26.
- 9.
Bravo E, Calzolari A, De Castro P, Mabile L, Napolitani F, Rossi AM, et al. Developing a guideline to standardize the citation of bioresources in journal article (CoBRA). BMC Med. 2015. [Ahead of print.]
- 10.
Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4:1.
- 11.
http://www.nih.gov/about/reporting-preclinical-research.htm.
- 12.
Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement. Ann Intern Med. 2015;162:55–63.
- 13.
Nature editorial. Announcement: Reducing our irreproducibility. Nature. 2013;496:398.
- 14.
Hooijmans CR, Leenaars M, Ritskes-Hoitinga M. A gold standard publication checklist to improve the quality of animal studies, to fully integrate the Three Rs, and to make systematic reviews more feasible. Altern Lab Anim. 2010;38:167–82.
- 15.
Institute for Laboratory Animal Research. Guidance for the Description of Animal Research in Scientific Publications. Washington, DC: The National Academies Press; 2011. p. 1–31.
- 16.
Landis SC, Amara SG, Asadullah K, Austin CP, Blumenstein R, Bradley EW, et al. A call for transparent reporting to optimize the predictive value of preclinical research. Nature. 2012;490:187–91.
- 17.
Nature. Reporting Checklist for Life Sciences Articles. 2013. http://www.nature.com/authors/policies/checklist.pdf. Accessed on: 01-15-2015.
- 18.
Kilkenny C, Browne WJ, Cuthill IC, Emerson M, Altman DG. The ARRIVE guidelines animal research: reporting in vivo experiments. PLoS Biol. 2010;8:e1000412.
- 19.
Baker D, Lidster K, Sottomayor A, Amor S. Two years later: journals are not yet enforcing the ARRIVE guidelines on reporting standards for pre-clinical animal studies. PLoS Biol. 2013;11:e1001756.
- 20.
Collins FS, Tabak LA. NIH plans to enhance reproducibility. Nature. 2014;505:612–3.
- 21.
Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687.
- 22.
Stevens A, Shamseer L, Weinstein E, Yazdi F, Turner L, Thielman J, et al. Relation of completeness of reporting of health research to journals’ endorsement of reporting guidelines: systematic review. BMJ. 2014;348:g3804.
- 23.
Cobo E, Cortés J, Ribera JM, Cardellach F, Selva-O'Callaghan A, Kostov B, et al. Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trial. BMJ. 2011;343:d6783.
- 24.
Hirst A, Altman DG. Are peer reviewers encouraged to use reporting guidelines? A survey of 116 health research journals. PLoS ONE. 2012;7:e35621.
- 25.
Michie S, Johnston M. Theories and techniques of behaviour change: developing a cumulative science of behaviour change. Health Psychol Rev. 2012;6:1–6.
- 26.
French SD, Green SE, O’Connor DA, McKenzie JE, Francis JJ, Michie S, et al. Developing theory-informed behaviour change interventions to implement evidence into practice: a systematic approach using the Theoretical Domains Framework. Implement Sci. 2012;7:38.
- 27.
- 28.
- 29.
https://www.nc3rs.org.uk/arrive-animal-research-reporting-vivo-experiments.
- 30.
Author information
Affiliations
Corresponding author
Additional information
Competing interests
Drs. Avey and Antes declare no competing interests. Dr. Moher and Professor Altman (chair) are members of the EQUATOR Network’s steering committee.
Authors’ contributions
DM drafted the commentary and the other authors provide critical input on all subsequent versions. All authors approve the final version of this manuscript.
An erratum to this article is available at http://dx.doi.org/10.1186/s12916-015-0321-8.
Rights and permissions
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
About this article
Cite this article
Moher, D., Avey, M., Antes, G. et al. The National Institutes of Health and guidance for reporting preclinical research. BMC Med 13, 34 (2015). https://doi.org/10.1186/s12916-015-0284-9
Received:
Accepted:
Published:
Keywords
- Implementation
- Preclinical research
- Quality of reporting
- Reporting guidelines
