Table 1 General characteristics of the trials included
Author, Year, Journal publishing the study | Intervention/Comparator | Journal(s) involved in the study | Peer reviewers (n) | Manuscripts (n)/Peer review reports (n) | Unit of randomization | Outcomes (Scale) | Sample size (randomized/analyzed) | Risk of bias |
|---|---|---|---|---|---|---|---|---|
Training/Mentoring/Feedback | ||||||||
Callaham, JAMA 2002 Study 1a [36] | Feedback by editors/usual process | Annals of Emergency Medicine | Low volume low qualitya peer reviewers (n = 51) | Manuscripts submitted to the journal (n = NR)/182 peer review reports | Peer reviewers | The quality of peer review report (using editor routine quality scale) | 51 randomized/35 analyzed | - Rs: Low - Al: Low - D: Low - At: High |
Callaham, JAMA 2002 Study 1b [36] | Feedback by editors/usual process | Annals of Emergency Medicine | Low volume, average qualityb peer reviewers (n = 127) | Manuscripts submitted to the journal (n = NR)/324 peer review reports | Peer reviewers | The quality of peer review report (using editor routine quality scale) | 127 randomized/95 analyzed | - Rs: Low - Al: Low - D: Low - At: High |
Callaham, 2002 Ann Emerg Med Study 2 [37] | Feedback by editors/usual process | Annals of Emergency Medicine | Average qualityb peer reviewers (n = 150) | Manuscripts submitted to the journal (n = NR)/79 peer review reports | Peer reviewers | The quality of peer review report (using editor routine quality scale) | 150 randomized/22 analyzed | - Rs: Low - Al: Low - D: Low - At: High |
Houry, 2012 BMC Med Educ [38] | Training (workshop)/usual process | Annals of Emergency Medicine | New peer reviewers (n = 50) | Manuscripts submitted to the journal (n = 490)/490 peer review reports | Peer reviewers | The quality of peer review report (using editor routine quality scale) | 50 randomized/46 analyzed | - Rs: Low - Al: Unclear - D: Unclear - At: Low |
Schroter, 2004 BMJ [35] | Training (face-to-face or self-training)/usual process (The two intervention groups, face-to-face and self-training, were pooled in the meta-analysis) | British Medical Journal | Consenting peer reviewers (n = 609) | One fabricated manuscript with errors (n = 1)c/418 peer review reports | Peer reviewers | 1) The quality of peer review report (using the RQI Version 3.2) 2) The rejection rate 3) The time spent on the review | 609 randomized/418 analyzed | - Rs: Low - Al: Low - D: Low - At: High |
Statistical peer review | ||||||||
Arnau, 2003 Med Clin (Barc) [39] | Adding a statistical peer reviewer/usual process | Medicina Clinica | Statistical peer reviewers (n = NR) | Manuscripts submitted to the journal (n = 82) | Manuscripts | The final manuscript quality (using the MQAI) | 82 randomized/43 analyzed | - Rs: Low - Al: Low - D: Low - At: High |
Cobo, 2007 PLOS One [40] | 2 × 2 factorial design comparing/adding a statistical peer reviewer/use of a reporting guidelines checklist/both/usual process (we selected only the two groups adding a statistical peer reviewer/usual process in the analysis)d | Medicina Clinica | Statistical peer reviewers (n = 39) | Consecutive manuscripts submitted to the journal (n = 68) | Manuscripts | The final manuscript quality (using the MQAI) | 68 randomized/62 analyzed | - Rs: Low - Al: Low - D: Low - At: Low |
Checklist | ||||||||
Cobo, 2007 PLOS One [40] | 2 × 2 factorial design comparing adding a statistical peer reviewer/use of a reporting guidelines checklist/both/usual process (we selected only the two groups use of a reporting guidelines checklist/usual process in the analysis)d | Medicina Clinica | Statistical peer reviewers (n = 39) | Consecutive manuscripts submitted to the journal (n = 69) | Manuscripts | The final manuscript quality (using the MQAI) | 69 randomized/60 analyzed | - Rs: Low - Al: Low - D: Low - At: Low |
Cobo, 2011 BMJ [41] | Use of a reporting guidelines checklist/usual process | Medicina Clinica | A senior statistician (n = 1) | Consecutive manuscripts submitted to the journal (n = 92) | Manuscripts | The final manuscript quality (using the MQAI) | 92 randomized/92 analyzed | - Rs: Low - Al: Low - D: Low - At: Low |
Open peer review (i.e., identity of peer reviewers revealed to the authors, other peer reviewers, and/or general public) | ||||||||
Das Sinha, 1999 Natl Med J India [42] | Pairs of reviewers were identified to assess each manuscript and the two reviewers were randomized; one to be informed they would have their identity revealed to the other peer reviewer and one to remain anonymous | The National Medical Journal of India | Peer reviewers of the journal (n = 156, 78 pairs) | Manuscripts submitted to the journal (n = 100)/156 peer review reports | Pairs of peer reviewers | 1) The quality of peer review report (using editor routine quality scale) 2) The rejection rate | 100 randomized/78 analyzed | - Rs: Low - Al: Low - D: Low - At: Low |
Godlee, 1998 JAMA [47] | Five groups: 1) ask to sign their report + blinded to authors name and affiliation; 2) ask to remain anonymous + blinded to authors name and affiliation; 3) ask to sign their report + unblinded to authors name and affiliation; 4) ask to remain anonymous + unblinded to authors name and affiliation/usual process We pooled groups 1 and 3 vs. 2 and 4 A fifth group where peer reviewers were unaware of the study for which the manuscript was sent according to the usual process was not taken into account in the analysis | British Medical Journal | Peer reviewers of the journal (n = 420) | One fabricated manuscript with errors (n = 1)/184 peer review reports | Peer reviewers | The rejection rate | 360 randomized/184 analyzed | - Rs: Low - Al: Low - D: Low - At: High |
Van Rooyen, 1998 JAMA [48] | Open to peer reviewers (combination of blinded and unblinded to authors identity) vs. anonymous (combination of blinded and unblinded to authors identity) | British Medical Journal | Peer reviewers of the journal (n = NR) | Consecutive manuscripts submitted to the journal (n = 527) | Manuscripts & peer reviewers | 1) The quality of peer review report (using the RQI Version 3.2) 2) The time spent on the review | 527 manuscript randomized/598 reviews analyzed | - Rs: Low - Al: Unclear - D: Low - At: Unclear |
Van Rooyen, 1999 BMJ [43] | Identity revealed to authors/peer reviewers remained anonymous to authors For each manuscript, a pair of reviewers were identified and each reviewer was randomized to have their identity revealed to authors or remain anonymous | British Medical Journal | Peer reviewers of the journal (n = 250) | Consecutive manuscripts submitted to the journal (n = 125)/113 manuscripts assessed/226 peer review reports | Peer reviewers | 1) The quality of peer review report (using the RQI Version 4) 2) The rejection rate 3) The time spent on the review | 250 randomized/226 analyzed | - Rs: Low - Al: Unclear - D: Low - At: High |
Van Rooyen, 2010 BMJ [44] | Identity revealed to general public/peer reviewers signed their review for authors and other peer reviewers | British Medical Journal | Peer reviewers of the journal (n = 471) | Consecutive manuscripts submitted to the journal (n = 558) | Manuscripts | 1) The quality of peer review report (using the RQI Version 4) 2) The rejection rate 3) The time spent on the review | 558 manuscript randomized/471 analyzed | - Rs: Unclear - Al: Unclear - D: Low - At: Low |
Vinther, 2012 Dan Med [45] | Pairs of reviewers were identified to assess each manuscript; for each manuscript, peer reviewers were randomized to have their identity revealed to authors/remained anonymous to authors | The Journal of Danish Medical Association | Peer reviewers of the journal (n = 380) | Manuscripts submitted to the journal (n = 190)/364 peer review reports | Peer reviewers | 1) The quality of peer review report (using the RQI Version 4) 2) The rejection rate | 380 randomized/364 analyzed | - Rs: Unclear - Al: Unclear - D: Low - At: Low |
Walsh, 2000 Br J Psychiatry [46] | Identity revealed to authors/peer reviewers remained anonymous to authors | British Journal of Psychiatry | Peer reviewers of the journal (n = 245) | Manuscripts submitted to the journal (n = 408)/354 peer review reports | Manuscript | 1) The quality of peer review report (using the RQI Version 3.2) 2) The rejection rate 3) The time spent on the review | 408 manuscripts randomized/354 analyzed | - Rs: Low - Al: Unclear - D: Low - At: High |
Blinded peer review (i.e., peer reviewers are blinded of the authors name and affiliation) | ||||||||
Alam, 2011 Br J Dermatol [49] | Randomization of four peer reviewers for each manuscript, two randomized to assess a blinded version of the manuscript, two to assess an unblinded version of the manuscript | Dermatologic Surgery | Volunteer peer reviewers of the journal (n = 20) | Consecutive manuscripts submitted to the journal (n = 40)/160 peer review reports | Peer reviewers & manuscript | The rejection rate | 20 peer reviewers/40 manuscripts 160 peer review reports analyzed 6 peer reviewers failed to do their review and were replaced (3 blinded/3 unblinded) | - Rs: Unclear - Al: Unclear - D: Low - At: Low |
Fisher, 1994 JAMA [50] | Identification of four peer reviewers for each manuscript, two randomized to assess a blinded version of the manuscript, two to assess an unblinded version of the manuscript | Journal of Developmental and Behavioral Pediatrics | Peer reviewers of the journal (n = 228) | Consecutive manuscripts submitted to the journal (n = 57)/228 peer review reports | Peer reviewers | The rejection rate | 228 randomized/220 analyzed | - Rs: Low - Al: Unclear - D: Low - At: Low |
Godlee, 1998 JAMA [47] | Five groups: 1) ask to sign their report + blinded to authors name and affiliation; 2) ask to remain anonymous + blinded to authors name and affiliation; 3) ask to sign their report + unblinded to authors name and affiliation; 4) ask to remain anonymous + unblinded to authors name and affiliation/usual process We pooled groups 1 and 2 vs. 3 and 4 A fifth group where peer reviewers were unaware of the study for which the manuscript was sent according to the usual process was not taken into account in the analysis | British Medical Journal | Peer reviewers of the journal (n = 360) | One fabricated manuscript with errors (n = 1)/130 peer review reports | Peer reviewers | The rejection rate | 240 randomized/130 analyzed | - Rs: Low - Al: Low - D: Low - At: High |
Justice, 1998 JAMA [51] | Identification of two peer reviewers for each manuscript, one randomized to assess a blinded version of the manuscript, one to assess an unblinded version of the manuscript | Annals of Emergency Medicine, Annals of Internal Medicine, JAMA, Obstetrics & Gynecology and Ophthalmology | Peer reviewers of journals (n = NR) | Manuscripts submitted to journals (n = 92)/77 manuscripts with two peer review reports | Peer reviewers | The quality of peer review report (using editor routine quality scale) | 92 manuscript with two reviewers/77 manuscript with two reviewers reports analyzed | - Rs: Low - Al: Unclear - D: Low - At: Unclear |
McNutt, 1990 JAMA [52] | Identification of two peer reviewers for each manuscript, one randomized to assess a blinded version of the manuscript, one to assess an unblinded version of the manuscript | Journal of General Internal Medicine | Peer reviewers of the journal (n = NR) | Manuscripts submitted to the journal (n = 127)/252 peer review reports | Peer reviewers | 1) The quality of peer review report (using editor routine quality scale) 2) The time spent on the review | 127 manuscript with two reviewers/127 and 125 reviewers reports analyzed | - Rs: Low - Al: Unclear - D: Low - At: Low |
Van Rooyen, 1998 JAMA [48] | Identification of two peer reviewers for each manuscript, one randomized to assess a blinded version of the manuscript, one to assess an unblinded version of the manuscript This study also assessed masked versus unmasked review (See above in open peer review intervention) | British Medical Journal | Peer reviewers of the journal (n = NR) | Consecutive manuscripts submitted to the journal included in this study (n = 309) | Peer reviewers & manuscripts | 1) The quality of peer review report (using the RQI Version 3.2) 2) The time spent on the review | 309 manuscripts with two reviewers randomized/618 peer reviews reports analyzed | - Rs: Low - Al: Unclear - D: Low - At: Unclear |
Accelerate the peer review process | ||||||||
Johnston, 2007 Ann Neurol [54] | Early screening by editors/formal external review | Annals of Neurology | Peer reviewers of the journal (n = 386) | Consecutive manuscripts submitted to the journal (n = 351) | Manuscripts | The time to a manuscript decision | 351 manuscripts randomized/351 reviews analyzed | - Rs: Unclear - Al: Low - D: low - At: Low |
Neuhauser, 1989 Medical Care [55] | Calling first peer reviewers/sending out manuscript without a prior phone call | Medical Care | Peer reviewers of the journal (n = NR) | Manuscripts submitted to the journal (n = 95) | Peer reviewers | The overall time for the peer review process | 95 manuscripts with two peer reviewers randomized | - Rs: Unclear - Al: Unclear - D: Low - At: Low |
Pitkin, 2002 JAMA [53] | Asking first: referees received information about manuscript by fax and indicated their willingness to review/editors mailed the manuscript and asked to return the review | Obstetrics & Gynecology | Peer reviewers of the journal (n = NR) | Consecutive manuscripts submitted to the journal (n = 283) Identification of two peer reviewers for each manuscript | Peer reviewers | 1) The overall time for the peer review process 2) The quality of peer review report (using editor routine quality scale) | 283 manuscripts with two reviewers randomized and analyzed | - Rs: Low - Al: Low - D: Low - At: Low |