Table 18 Methods for multiple testing corrections: Bonferroni correction, Holm’s procedure, Westfall-Young permutation procedure

Bonferroni correction

 The Bonferroni correction specifies that when m statistical tests are conducted, each one should use a critical level of α/m where α is the desired type I error for the full collection of tests. For example, a Bonferroni correction applied in the setting of 10,000 hypothesis tests would require that an individual test reaches statistical significance at a critical level = 0.05/10,000 = 0.000005. Achieving this level of significance would require an extremely large sample size or effect size (e.g., magnitude of association) in order for an individual test to have reasonable power

Holm’s procedure

 Order the p-values from smallest to largest as p₍₁₎, p₍₂₎,..., p_(m), where m is the number of tests. Beginning with p₍₁₎, proceed in order, comparing each p_(i) to the critical value α/(m-i + 1). Stop the first time that p_(i) exceeds the critical value α/(m-i + 1). Call this index j. Declare all p-values p₍₁₎, p₍₂₎,..., p_(j-1) to be statistically significant

 This procedure controls the FWER to be no more than α. It is clear from comparison of the sequential Holm critical values to the fixed Bonferroni critical value that the Holm procedure has the potential to reject more tests and therefore offers greater power, although when the number of tests m is very large, as often in HDD, the actual difference in critical values can be extremely small

Westfall-Young permutation procedure

 The Westfall-Young permutation procedure [104] is a multivariate permutation procedure to control the FWER that is more efficient (powerful) than Bonferroni-like procedures (as Bonferroni and Holm’s procedure) in finding true discoveries. It exploits the correlations among variables, which are preserved in the permutation process, since all variables are permuted at the same time. The method is a step-down procedure similar to the Holm method. After p-values are calculated for all variables and ranked, multiple times class labels are permuted and corresponding p-values are calculated. Then the successive minima of these new p-values are retained and compared to the original p-values. For each variable, the proportion of number of permutations where the minimum new p-value is less than the original p-value is the adjusted p-value