The problem with peer review in medicine
Recent innovations in peer review seem to be driven by biologists with medical research ‘tagging along’. However, systems which might help biological research to thrive, might not necessarily be appropriate for research that directly influences patient care. There is no agreement on who a ‘peer’ or what ‘peer review’ actually is [[11]]. It is not clear what peer review aims to achieve [[28]] and no evidence that peer review works [[29]]. Journal instructions for peer reviewers [[30]] and the criteria for eligibility to peer review are variable (Table 1). There has been little evaluation of any of the more recent innovations in peer review for any outcomes. Furthermore, the whole system is based on honesty and trust and, as a consequence, is not designed to detect fraud.
Despite this, peer review is still seen by researchers as important and necessary for scientific communication [[31]] and publication in a peer reviewed medical journal is still the only valid or legitimate route to disseminating clinical research. In 2006, Richard Smith of the BMJ commented that it was, ‘odd that science should be rooted in belief’ [[11]]. In the world of evidence based medicine, it is astonishing that the evidence on which medical treatment is based is itself based on such precarious foundations with so many untested assumptions. Today, a junior doctor still relies on faith in the peer review system when judging a clinical trial and a patient searching, ‘Should I have my wisdom teeth removed if they don’t hurt?’ would get more than a million results on Google (search date 12 May 2014) with no guidance on the relevance or trustworthiness of any of them, leaving them as much in the dark as I was when I first asked that question. The difference between now and then is that then, information was just not available or accessible, and now, there is so much information available of varying quality that it is impossible to make sense of it all without some specialist knowledge. For example, if the lay person knows what to search for (prophylactic extraction of third molar) and which sources they can trust (the Cochrane library), the relevant information can be found easily. According to a Cochrane review I found [[32]], there is no evidence either way of the benefit of having wisdom teeth removed if they are asymptomatic. I feel reassured I made the right decision all those years ago. However, not all clinical questions can be answered so easily or can afford the luxury of waiting for a Cochrane systematic review to be done. When there is no ready-made Cochrane review, a system that provides some sort of quality check for individual studies might serve as an important consideration for patients (and doctors) who need to weigh up, using the available evidence, the risks and benefits of a course of action and make definitive, time dependent, decisions that could be life changing.
A UK Parliamentary enquiry on peer review in 2011 [[33]] concluded that different types of peer review are suitable for different disciplines and encouraged increased recognition that peer-review quality is independent of journal business model. With this in mind, is there a need to redesign peer review specifically for clinical research and ensure that this is driven by the clinical community?
Training and specialization in peer review
With peer review as a vague and undefined process it is not surprising that in a survey of peer review conducted by Sense about Science, 56% of reviewers in a survey said there was a lack of guidance on how to review and 68% thought formal training would help [[31]]. Training and mentoring schemes for peer review have shown little impact [[34]-[37]] and even a decline in peer reviewer performance with time [[38]]. It may be that by the time a researcher has reached the stage in their career when they start to peer review, it is too late to teach peer review.
Although reporting guidelines have been available for two decades, many researchers and reviewers still do not understand what they are or the need for them. This is further compounded by inconsistent guidance from journals for authors on how to use reporting guidelines [[30]] and a lack of awareness of how they can improve the reporting of RCTs [[39]] and, thereby, aid peer review. There are misunderstandings about trial registration and even what constitutes an RCT. There is evidence that reviewers fail to detect deliberately introduced errors [[34],[37]] and do not detect deficiencies in reporting methods, sometimes even suggesting inappropriate revisions [[40]]. Manuscripts reporting poorly conducted clinical research get published in peer reviewed journals and their findings inform systematic reviews, which in turn could also be poorly conducted and reported. These systematic reviews have the potential to inform clinical judgments.
The need for a concerted effort across disciplines to investigate the effects of peer review has been recognized [[28]], but before the effects can be investigated, the aims of peer review need to be defined. This is a daunting challenge if one aim, or a small number of aims, is intended to fulfill all peer review needs for all fields, specialties and study designs. A more manageable way may be to introduce specialization into peer review, so that specific fields can define the purpose and aims of peer review to suit their own needs and design training to meets those aims.
Since the methodology for conducting and reporting of RCTs has been defined by the CONSORT statement [[41]] which improves the reporting of RCTs [[39]] and, thereby, aids the peer review process, peer review of RCTs lends itself to such specialization. CONSORT could form the framework for the content of a training program and help to define the knowledge and skills that are needed by a given individual to appraise an RCT critically. Peer reviewers could be taught to spot fundamental flaws and be periodically evaluated to make sure they do, in the same way that any other knowledge or skill that affects patient care is.
Peer review of RCTs should be recognized as a professional skill in this way. Every RCT, and its peer review reports if made public, whether published online, on paper, open access or subscription only, with open or closed peer review, or peer reviewed before or after publication could then have a searchable ‘quality assurance’ symbol (like the ‘kite-mark’ used by the British Standards Institute [[42]]) or a word, so that readers know whether a study was reviewed by at least one appropriately trained and accredited expert. Such a system could accommodate all peer review models (Figure 1).
To achieve this, major organizations including medical schools, medical regulatory and accreditation organizations (such as the General Medical Council and Royal Colleges in the UK), funding bodies, publishers and journal editors and lay people need to come to a consensus on the definition, purpose, standards and training requirements of peer review of RCTs. Training should begin in medical schools and be ongoing.
By recognizing peer review as a professional skill with measurable standards which are separate from the journal, publisher or peer review model, peer review is separated from commercial considerations, peer reviewers get recognition for their work, and researchers, clinicians and patients get some indication of quality on which to base their judgments. Publishers and journals are then free to innovate while still maintaining consistency of peer review for RCTS, editors have clear criteria on which to base their choice of peer reviewer for a given manuscript and a baseline is set that allows for future research into the effectiveness of peer review per se and comparative studies on the effectiveness and quality of emerging innovations.
