Kenneth Schulz, FHI and School of Medicine, UNC Chapel Hill
25 June 2010
Bacchetti wrote that “Reports of completed studies should not include power calculations, and guidelines requiring them (he cites CONSORT 2001) should be changed to instead discourage them.” Unfortunately, Dr. Bacchetti is incorrect and misinterprets the intent of CONSORT. CONSORT does not require “power calculations” and, furthermore, does not require any specific trial conduct, other than clear and transparent reporting.
The CONSORT 2001 checklist (1), cited by Bacchetti, focuses only on reporting and simply asks “How sample size was determined . . .” It does not require “power calculations” or any other specific approach. Moreover, on Mar 24, 2010 we published an updated version, CONSORT 2010, in the BMJ (2), with simultaneous publication in 8 other journals. The sample size item queries “How sample size was determined.” For example, Dr. Bacchetti’s value of information methods for choosing sample size could aptly be reported under CONSORT 2010. With that clear and transparent reporting, readers could decide the appropriateness of his approach or, for that matter, other approaches proposed by anyone else.
CONSORT does not have any official view of Bacchetti’s position on sample size methods. Individuals in the CONSORT Group have a broad range of views on his position. Indeed, that range includes views extremely similar to Bacchetti. One of us published a similar challenge to current sample size dogma that predated Bacchetti’s challenge (3). In that article are statements such as “We question the branding of trials as unethical based solely on an inherently subjective, imprecise sample size calculation process.” But again, regardless of the range of views, all of the individuals in the CONSORT Group agree that authors should report how the sample size was determined. Sample size is a key element of the design of a randomised trial so a full and transparent account of how the trial was planned should include how the sample size was determined.
Moreover, that central theme of focusing on reporting extends to all the items in the checklist. Bacchetti and other readers should recognize that CONSORT is solely about reporting. CONSORT does not advise on trial design or conduct, and will not do so in the future.
Kenneth F. Schulz, PhD David Moher, PhD Douglas G. Altman, DSc
Reference List
1. Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001; 357:1191-4. 2. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 2010; 340:c332 3. Schulz KF, Grimes DA. Sample size calculations in randomised trials: mandatory and mystical. Lancet 2005; 365:1348-53.
Competing interests
None declared
Good intentions versus CONSORT’s actual effect
Peter Bacchetti, University of California, San Francisco
5 July 2010
I thank Drs. Schulz, Moher, and Altman for their comments, and I apologize for my careless wording. I should have written, “Reports of completed studies should not include power calculations, and guidelines that have the effect of requiring that they be reported should be changed to instead discourage their reporting.” I was addressing reports of completed studies rather than their conduct, and my concern is with the effect of the CONSORT guideline rather than its intentions. While I understand the principled perspective of Schulz et al., a more practical perspective is also relevant and supports my call for change.
Some laudably flexible phrasing in the 2010 CONSORT explanation document [1] was also present in the 2001 version [2], but this has not changed several crucial facts: 1) researchers frequently determine the planned sample size based on cost or feasibility; 2) they generally believe that they should provide power calculations instead of discussing how sample size was actually determined (this is what warps the guideline into effectively being a requirement to report power); 3) many readers and researchers believe that calculations showing high power allow P>0.05 to be interpreted as supporting negative conclusions, without considering the estimate and its confidence interval. I believe that these facts cause the CONSORT sample size guideline to reduce transparency and promote misinterpretation, exactly contrary to its intended purpose.
Unfortunately, I see little in the 2010 update [1] that seems likely to mitigate this perversion of its intent. Both of its examples use power calculations, and there is considerable discussion of power calculation but no mention of other ways to determine sample size. The text even states that describing how sample size was determined “reveals the power of the trial to readers”. This presumes that a power calculation will always be reported, which seems very similar to the misstatement of mine that prompted Schulz et al. to comment. The text fails to emphasize that power calculations should not be reported when they were only used to justify the sample size, not to determine it. The problem of unwarranted claims of no difference is discussed as if it were a sample size issue instead of in the interpretation section (item 22), where it could be more effectively addressed.
I propose that changing the guideline to simply require stating the original planned sample size would eliminate the harmful effects without sacrificing anything of scientific importance, because the motivation for the originally planned sample size (including the claimed power) is irrelevant for interpreting a study’s results. Transparency does not require disclosure of irrelevant information. Less drastic changes (e.g., to the accompanying explanatory text) might also do some good, and I urge the CONSORT group to consider how to more effectively combat the problems I have described.
The CONSORT statement is certainly not to blame for the facts I note above, but that does not mean it should not adapt to them. Regardless of the guideline’s intention, what happens in actual practice also matters.
1. Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, Elbourne D, Egger M, Altman DG: CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials. British Medical Journal 2010, 340:28.
2. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gotzsche PC, Lang T: The revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Annals of Internal Medicine 2001, 134:663-694.
A Fundamental Misinterpretation of CONSORT
25 June 2010
Bacchetti wrote that “Reports of completed studies should not include power calculations, and guidelines requiring them (he cites CONSORT 2001) should be changed to instead discourage them.” Unfortunately, Dr. Bacchetti is incorrect and misinterprets the intent of CONSORT. CONSORT does not require “power calculations” and, furthermore, does not require any specific trial conduct, other than clear and transparent reporting.
The CONSORT 2001 checklist (1), cited by Bacchetti, focuses only on reporting and simply asks “How sample size was determined . . .” It does not require “power calculations” or any other specific approach. Moreover, on Mar 24, 2010 we published an updated version, CONSORT 2010, in the BMJ (2), with simultaneous publication in 8 other journals. The sample size item queries “How sample size was determined.” For example, Dr. Bacchetti’s value of information methods for choosing sample size could aptly be reported under CONSORT 2010. With that clear and transparent reporting, readers could decide the appropriateness of his approach or, for that matter, other approaches proposed by anyone else.
CONSORT does not have any official view of Bacchetti’s position on sample size methods. Individuals in the CONSORT Group have a broad range of views on his position. Indeed, that range includes views extremely similar to Bacchetti. One of us published a similar challenge to current sample size dogma that predated Bacchetti’s challenge (3). In that article are statements such as “We question the branding of trials as unethical based solely on an inherently subjective, imprecise sample size calculation process.” But again, regardless of the range of views, all of the individuals in the CONSORT Group agree that authors should report how the sample size was determined. Sample size is a key element of the design of a randomised trial so a full and transparent account of how the trial was planned should include how the sample size was determined.
Moreover, that central theme of focusing on reporting extends to all the items in the checklist. Bacchetti and other readers should recognize that CONSORT is solely about reporting. CONSORT does not advise on trial design or conduct, and will not do so in the future.
Kenneth F. Schulz, PhD
David Moher, PhD
Douglas G. Altman, DSc
Reference List
1. Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001; 357:1191-4.
2. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 2010; 340:c332
3. Schulz KF, Grimes DA. Sample size calculations in randomised trials: mandatory and mystical. Lancet 2005; 365:1348-53.
Competing interests
None declared
Good intentions versus CONSORT’s actual effect
5 July 2010
I thank Drs. Schulz, Moher, and Altman for their comments, and I apologize for my careless wording. I should have written, “Reports of completed studies should not include power calculations, and guidelines that have the effect of requiring that they be reported should be changed to instead discourage their reporting.” I was addressing reports of completed studies rather than their conduct, and my concern is with the effect of the CONSORT guideline rather than its intentions. While I understand the principled perspective of Schulz et al., a more practical perspective is also relevant and supports my call for change.
Some laudably flexible phrasing in the 2010 CONSORT explanation document [1] was also present in the 2001 version [2], but this has not changed several crucial facts: 1) researchers frequently determine the planned sample size based on cost or feasibility; 2) they generally believe that they should provide power calculations instead of discussing how sample size was actually determined (this is what warps the guideline into effectively being a requirement to report power); 3) many readers and researchers believe that calculations showing high power allow P>0.05 to be interpreted as supporting negative conclusions, without considering the estimate and its confidence interval. I believe that these facts cause the CONSORT sample size guideline to reduce transparency and promote misinterpretation, exactly contrary to its intended purpose.
Unfortunately, I see little in the 2010 update [1] that seems likely to mitigate this perversion of its intent. Both of its examples use power calculations, and there is considerable discussion of power calculation but no mention of other ways to determine sample size. The text even states that describing how sample size was determined “reveals the power of the trial to readers”. This presumes that a power calculation will always be reported, which seems very similar to the misstatement of mine that prompted Schulz et al. to comment. The text fails to emphasize that power calculations should not be reported when they were only used to justify the sample size, not to determine it. The problem of unwarranted claims of no difference is discussed as if it were a sample size issue instead of in the interpretation section (item 22), where it could be more effectively addressed.
I propose that changing the guideline to simply require stating the original planned sample size would eliminate the harmful effects without sacrificing anything of scientific importance, because the motivation for the originally planned sample size (including the claimed power) is irrelevant for interpreting a study’s results. Transparency does not require disclosure of irrelevant information. Less drastic changes (e.g., to the accompanying explanatory text) might also do some good, and I urge the CONSORT group to consider how to more effectively combat the problems I have described.
The CONSORT statement is certainly not to blame for the facts I note above, but that does not mean it should not adapt to them. Regardless of the guideline’s intention, what happens in actual practice also matters.
1. Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, Elbourne D, Egger M, Altman DG: CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials. British Medical Journal 2010, 340:28.
2. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gotzsche PC, Lang T: The revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Annals of Internal Medicine 2001, 134:663-694.
Competing interests
None