Skip to main content

Advertisement

Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Table 1 General characteristics of the trials included

From: Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis

Author, Year, Journal publishing the study Intervention/Comparator Journal(s) involved in the study Peer reviewers (n) Manuscripts (n)/Peer review reports (n) Unit of randomization Outcomes (Scale) Sample size (randomized/analyzed) Risk of bias
Training/Mentoring/Feedback
Callaham, JAMA 2002 Study 1a [36] Feedback by editors/usual process Annals of Emergency Medicine Low volume low qualitya peer reviewers (n = 51) Manuscripts submitted to the journal (n = NR)/182 peer review reports Peer reviewers The quality of peer review report (using editor routine quality scale) 51 randomized/35 analyzed - Rs: Low
- Al: Low
- D: Low
- At: High
Callaham, JAMA 2002 Study 1b [36] Feedback by editors/usual process Annals of Emergency Medicine Low volume, average qualityb peer reviewers (n = 127) Manuscripts submitted to the journal (n = NR)/324 peer review reports Peer reviewers The quality of peer review report (using editor routine quality scale) 127 randomized/95 analyzed - Rs: Low
- Al: Low
- D: Low
- At: High
Callaham, 2002 Ann Emerg Med Study 2 [37] Feedback by editors/usual process Annals of Emergency Medicine Average qualityb peer reviewers (n = 150) Manuscripts submitted to the journal (n = NR)/79 peer review reports Peer reviewers The quality of peer review report (using editor routine quality scale) 150 randomized/22 analyzed - Rs: Low
- Al: Low
- D: Low
- At: High
Houry, 2012 BMC Med Educ [38] Training (workshop)/usual process Annals of Emergency Medicine New peer reviewers (n = 50) Manuscripts submitted to the journal (n = 490)/490 peer review reports Peer reviewers The quality of peer review report (using editor routine quality scale) 50 randomized/46 analyzed - Rs: Low
- Al: Unclear
- D: Unclear
- At: Low
Schroter, 2004 BMJ [35] Training (face-to-face or self-training)/usual process (The two intervention groups, face-to-face and self-training, were pooled in the meta-analysis) British Medical Journal Consenting peer reviewers (n = 609) One fabricated manuscript with errors (n = 1)c/418 peer review reports Peer reviewers 1) The quality of peer review report (using the RQI Version 3.2)
2) The rejection rate
3) The time spent on the review
609 randomized/418 analyzed - Rs: Low
- Al: Low
- D: Low
- At: High
Statistical peer review
Arnau, 2003 Med Clin (Barc) [39] Adding a statistical peer reviewer/usual process Medicina Clinica Statistical peer reviewers (n = NR) Manuscripts submitted to the journal (n = 82) Manuscripts The final manuscript quality (using the MQAI) 82 randomized/43 analyzed - Rs: Low
- Al: Low
- D: Low
- At: High
Cobo, 2007 PLOS One [40] 2 × 2 factorial design comparing/adding a statistical peer reviewer/use of a reporting guidelines checklist/both/usual process (we selected only the two groups adding a statistical peer reviewer/usual process in the analysis)d Medicina Clinica Statistical peer reviewers (n = 39) Consecutive manuscripts submitted to the journal (n = 68) Manuscripts The final manuscript quality (using the MQAI) 68 randomized/62 analyzed - Rs: Low
- Al: Low
- D: Low
- At: Low
Checklist
Cobo, 2007 PLOS One [40] 2 × 2 factorial design comparing adding a statistical peer reviewer/use of a reporting guidelines checklist/both/usual process (we selected only the two groups use of a reporting guidelines checklist/usual process in the analysis)d Medicina Clinica Statistical peer reviewers (n = 39) Consecutive manuscripts submitted to the journal (n = 69) Manuscripts The final manuscript quality (using the MQAI) 69 randomized/60 analyzed - Rs: Low
- Al: Low
- D: Low
- At: Low
Cobo, 2011 BMJ [41] Use of a reporting guidelines checklist/usual process Medicina Clinica A senior statistician (n = 1) Consecutive manuscripts submitted to the journal (n = 92) Manuscripts The final manuscript quality (using the MQAI) 92 randomized/92 analyzed - Rs: Low
- Al: Low
- D: Low
- At: Low
Open peer review (i.e., identity of peer reviewers revealed to the authors, other peer reviewers, and/or general public)
Das Sinha, 1999 Natl Med J India [42] Pairs of reviewers were identified to assess each manuscript and the two reviewers were randomized; one to be informed they would have their identity revealed to the other peer reviewer and one to remain anonymous The National Medical Journal of India Peer reviewers of the journal (n = 156, 78 pairs) Manuscripts submitted to the journal (n = 100)/156 peer review reports Pairs of peer reviewers 1) The quality of peer review report (using editor routine quality scale)
2) The rejection rate
100 randomized/78 analyzed - Rs: Low
- Al: Low
- D: Low
- At: Low
Godlee, 1998 JAMA [47] Five groups: 1) ask to sign their report + blinded to authors name and affiliation; 2) ask to remain anonymous + blinded to authors name and affiliation; 3) ask to sign their report + unblinded to authors name and affiliation; 4) ask to remain anonymous + unblinded to authors name and affiliation/usual process
We pooled groups 1 and 3 vs. 2 and 4
A fifth group where peer reviewers were unaware of the study for which the manuscript was sent according to the usual process was not taken into account in the analysis
British Medical Journal Peer reviewers of the journal (n = 420) One fabricated manuscript with errors (n = 1)/184 peer review reports Peer reviewers The rejection rate 360 randomized/184 analyzed - Rs: Low
- Al: Low
- D: Low
- At: High
Van Rooyen, 1998 JAMA [48] Open to peer reviewers (combination of blinded and unblinded to authors identity) vs. anonymous (combination of blinded and unblinded to authors identity) British Medical Journal Peer reviewers of the journal (n = NR) Consecutive manuscripts submitted to the journal (n = 527) Manuscripts & peer reviewers 1) The quality of peer review report (using the RQI Version 3.2)
2) The time spent on the review
527 manuscript randomized/598 reviews analyzed - Rs: Low
- Al: Unclear
- D: Low
- At: Unclear
Van Rooyen, 1999 BMJ [43] Identity revealed to authors/peer reviewers remained anonymous to authors
For each manuscript, a pair of reviewers were identified and each reviewer was randomized to have their identity revealed to authors or remain anonymous
British Medical Journal Peer reviewers of the journal (n = 250) Consecutive manuscripts submitted to the journal (n = 125)/113 manuscripts assessed/226 peer review reports Peer reviewers 1) The quality of peer review report (using the RQI Version 4)
2) The rejection rate
3) The time spent on the review
250 randomized/226 analyzed - Rs: Low
- Al: Unclear
- D: Low
- At: High
Van Rooyen, 2010 BMJ [44] Identity revealed to general public/peer reviewers signed their review for authors and other peer reviewers British Medical Journal Peer reviewers of the journal (n = 471) Consecutive manuscripts submitted to the journal (n = 558) Manuscripts 1) The quality of peer review report (using the RQI Version 4)
2) The rejection rate
3) The time spent on the review
558 manuscript randomized/471 analyzed - Rs: Unclear
- Al: Unclear
- D: Low
- At: Low
Vinther, 2012 Dan Med [45] Pairs of reviewers were identified to assess each manuscript; for each manuscript, peer reviewers were randomized to have their identity revealed to authors/remained anonymous to authors The Journal of Danish Medical Association Peer reviewers of the journal (n = 380) Manuscripts submitted to the journal (n = 190)/364 peer review reports Peer reviewers 1) The quality of peer review report (using the RQI Version 4)
2) The rejection rate
380 randomized/364 analyzed - Rs: Unclear
- Al: Unclear
- D: Low
- At: Low
Walsh, 2000 Br J Psychiatry [46] Identity revealed to authors/peer reviewers remained anonymous to authors British Journal of Psychiatry Peer reviewers of the journal (n = 245) Manuscripts submitted to the journal (n = 408)/354 peer review reports Manuscript 1) The quality of peer review report (using the RQI Version 3.2)
2) The rejection rate
3) The time spent on the review
408 manuscripts randomized/354 analyzed - Rs: Low
- Al: Unclear
- D: Low
- At: High
Blinded peer review (i.e., peer reviewers are blinded of the authors name and affiliation)
Alam, 2011 Br J Dermatol [49] Randomization of four peer reviewers for each manuscript, two randomized to assess a blinded version of the manuscript, two to assess an unblinded version of the manuscript Dermatologic Surgery Volunteer peer reviewers of the journal (n = 20) Consecutive manuscripts submitted to the journal (n = 40)/160 peer review reports Peer reviewers & manuscript The rejection rate 20 peer reviewers/40 manuscripts
160 peer review reports analyzed
6 peer reviewers failed to do their review and were replaced (3 blinded/3 unblinded)
- Rs: Unclear
- Al: Unclear
- D: Low
- At: Low
Fisher, 1994 JAMA [50] Identification of four peer reviewers for each manuscript, two randomized to assess a blinded version of the manuscript, two to assess an unblinded version of the manuscript Journal of Developmental and Behavioral Pediatrics Peer reviewers of the journal (n = 228) Consecutive manuscripts submitted to the journal (n = 57)/228 peer review reports Peer reviewers The rejection rate 228 randomized/220 analyzed - Rs: Low
- Al: Unclear
- D: Low
- At: Low
Godlee, 1998 JAMA [47] Five groups: 1) ask to sign their report + blinded to authors name and affiliation; 2) ask to remain anonymous + blinded to authors name and affiliation; 3) ask to sign their report + unblinded to authors name and affiliation; 4) ask to remain anonymous + unblinded to authors name and affiliation/usual process
We pooled groups 1 and 2 vs. 3 and 4
A fifth group where peer reviewers were unaware of the study for which the manuscript was sent according to the usual process was not taken into account in the analysis
British Medical Journal Peer reviewers of the journal (n = 360) One fabricated manuscript with errors (n = 1)/130 peer review reports Peer reviewers The rejection rate 240 randomized/130 analyzed - Rs: Low
- Al: Low
- D: Low
- At: High
Justice, 1998 JAMA [51] Identification of two peer reviewers for each manuscript, one randomized to assess a blinded version of the manuscript, one to assess an unblinded version of the manuscript Annals of Emergency Medicine, Annals of Internal Medicine, JAMA, Obstetrics & Gynecology and Ophthalmology Peer reviewers of journals (n = NR) Manuscripts submitted to journals (n = 92)/77 manuscripts with two peer review reports Peer reviewers The quality of peer review report (using editor routine quality scale) 92 manuscript with two reviewers/77 manuscript with two reviewers reports analyzed - Rs: Low
- Al: Unclear
- D: Low
- At: Unclear
McNutt, 1990 JAMA [52] Identification of two peer reviewers for each manuscript, one randomized to assess a blinded version of the manuscript, one to assess an unblinded version of the manuscript Journal of General Internal Medicine Peer reviewers of the journal (n = NR) Manuscripts submitted to the journal (n = 127)/252 peer review reports Peer reviewers 1) The quality of peer review report (using editor routine quality scale)
2) The time spent on the review
127 manuscript with two reviewers/127 and 125 reviewers reports analyzed - Rs: Low
- Al: Unclear
- D: Low
- At: Low
Van Rooyen, 1998 JAMA [48] Identification of two peer reviewers for each manuscript, one randomized to assess a blinded version of the manuscript, one to assess an unblinded version of the manuscript
This study also assessed masked versus unmasked review (See above in open peer review intervention)
British Medical Journal Peer reviewers of the journal (n = NR) Consecutive manuscripts submitted to the journal included in this study (n = 309) Peer reviewers & manuscripts 1) The quality of peer review report (using the RQI Version 3.2)
2) The time spent on the review
309 manuscripts with two reviewers randomized/618 peer reviews reports analyzed - Rs: Low
- Al: Unclear
- D: Low
- At: Unclear
Accelerate the peer review process
Johnston, 2007 Ann Neurol [54] Early screening by editors/formal external review Annals of Neurology Peer reviewers of the journal (n = 386) Consecutive manuscripts submitted to the journal (n = 351) Manuscripts The time to a manuscript decision 351 manuscripts randomized/351 reviews analyzed - Rs: Unclear
- Al: Low
- D: low
- At: Low
Neuhauser, 1989 Medical Care [55] Calling first peer reviewers/sending out manuscript without a prior phone call Medical Care Peer reviewers of the journal (n = NR) Manuscripts submitted to the journal (n = 95) Peer reviewers The overall time for the peer review process 95 manuscripts with two peer reviewers randomized - Rs: Unclear
- Al: Unclear
- D: Low
- At: Low
Pitkin, 2002 JAMA [53] Asking first: referees received information about manuscript by fax and indicated their willingness to review/editors mailed the manuscript and asked to return the review Obstetrics & Gynecology Peer reviewers of the journal (n = NR) Consecutive manuscripts submitted to the journal (n = 283)
Identification of two peer reviewers for each manuscript
Peer reviewers 1) The overall time for the peer review process
2) The quality of peer review report (using editor routine quality scale)
283 manuscripts with two reviewers randomized and analyzed - Rs: Low
- Al: Low
- D: Low
- At: Low
  1. Rs, Random sequence generation; Al, Allocation concealment; D, Detection bias (blinding of outcome assessment); At, Attrition bias (incomplete outcome data); NR, not reported. The Manuscript Quality Assessment Instrument (MQAI) is a 34-item scale, each item scored from 1 to 5, aimed to evaluate the quality of the research report. The Review Quality Instrument (RQI) is an 8-item scale, each item scored from 1 to 5, aimed to evaluate the quality of the peer review report
  2. aReviewers with a median score of 3 or lower on a quality scale of 1 to 5 routinely used by editors for the reviews they performed in the previous 2 years
  3. bReviewers with a median score of 4 or lower on a quality scale of 1 to 5 routinely used by editors for the reviews they performed in the previous 2 years
  4. cPeer reviewers provided three peer review reports for three different manuscripts; we selected the assessment of the last manuscript
  5. dWe did not consider the group: “clinical reviewer + statistician + checklist reviewer”, because we were interested in the effect of the use of a checklist or adding a statistician reviewer, not the effect of both