Skip to content

Partek has Dunn-Sidak Multiple Test Correction. Is this the same/similar to any of R's p.adjust.methods?

3 messages · Earl F. Glynn, Peter Dalgaard, Martin Maechler

Original
Earl F. Glynn
# 
The Partek package (www.partek.com) allows only two selections for Multiple
Test Correction:  Bonferroni and Dunn-Sidak.  Can anyone suggest why Partek
implemented Dunn-Sidak and not the other methods that R has?  Is there any
particular advantage to the Dunn-Sidak method?
R knows about these methods (in R 2.1.1):
[1] "holm" "hochberg" "hommel" "bonferroni" "BH" "BY" "fdr"
[8] "none"

BH is Benjamini & Hochberg (1995) and is also called "fdr" (I wish R's
documentation said this clearly).  BY is Benjamini & Yekutieli (2001).

I found a few hits from Google on Dunn-Sidak, but I'm curious if anyone can
tell me on a "conservative-liberal" scale, where the Dunn-Sidak method
falls? My guess is it's less conservative than Bonferroni (but aren't all
the other methods?), but how does it compare to the other methods?

A limited numerical experiment suggested this order to me:  bonferroni (most
conservative), hochberg and holm about the same, BY, BH (also called fdr),
and then none.

Thanks for any of  thoughts on this.

efg
Peter Dalgaard
# 
"Earl F. Glynn" <efg at stowers-institute.org> writes:
As far as I gather, D-S is exact for independent tests, conservative
for comparisons of group means, and liberal for mutually exclusive
tests (in which case Bonferroni is exact). It is always less
conservative than Bonferroni, but the difference is small for typical
significance levels: when the Bonferroni level is p, the D-S level is

   1 - (1-p/N)^N

and if you put p=0.05 and vary N you'll find that it varies from 0.05
at N=1 down to 0.04877 at N=100000. (Exercise for the students: what
is the limit as N goes to infinity?)

The three H-methods play a somewhat different game, basically by only
requiring multiple-testing adjustment for non-significant tests.  The
FDR methods play yet differently by allowing the per test level to
increase with the number of significant tests.
I'd expect the differences to be fairly small in scenarios where the
global null hypothesis is true (excluding "none"). The main difference
comes in when some of the nulls are actually false. Also, it depends
on your definitions: With the exception of "BY" and "none" the
p.adjust methods agree on the smallest adjusted p value, so have the
same familywise error rate under the global null. If you count the
total number of rejected tests, then you get a difference due to
"cascading" in the non-bonferroni cases.
Martin Maechler
# 
Earl> The Partek package (www.partek.com) allows only two selections for Multiple
    Earl> Test Correction:  Bonferroni and Dunn-Sidak.  Can anyone suggest why Partek
    Earl> implemented Dunn-Sidak and not the other methods that R has?  Is there any
    Earl> particular advantage to the Dunn-Sidak method?
    Earl> R knows about these methods (in R 2.1.1):

    >> p.adjust.methods
    Earl> [1] "holm" "hochberg" "hommel" "bonferroni" "BH" "BY" "fdr"
    Earl> [8] "none"

    Earl> BH is Benjamini & Hochberg (1995) and is also called "fdr" (I wish R's
    Earl> documentation said this clearly).  BY is Benjamini & Yekutieli (2001).

The current R docu has

  >>   The '"BH"' and '"BY"' method of Benjamini, Hochberg, and Yekutieli
  >>   control the false discovery rate, the expected proportion of false
  >>   discoveries amongst the rejected hypotheses.  The false discovery
  >>   rate is a less stringent condition than the family wise error
  >>   rate, so these methods are more powerful than the others.

so both "BH" and "BY"   ``are FDR versions''. 
"fdr" was used - unfortunately - in some older versions of R,
so we kept it working as an *alias* for the time being.  
You should rather not know about it :-)
and use "BH" or "BY" (and maybe other methods in the future) instead.

Regards,

Martin