cor.test(method = spearman, exact = TRUE) not exact (PR#14095)

Sun, Nov 29, 2009 7:00 PM #

Full_Name: David Simcha
Version: 2.10
OS: Windows XP Home
Submission from: (NULL) (173.3.208.5)

Spearman's rank correlation rho

data:  a and b 
S = 0, p-value < 2.2e-16
alternative hypothesis: true rho is greater than 0 
sample estimates:
rho 
  1

[1] 2.755732e-07

Since we have perfect rank correlation and only one permutation out of 10! could
give this for N = 10, the p-value should be 1/10!.  Reading the code in prho.c,
it appears that the "exact" calculation uses the Edgeworth approximation for N >
9.  This makes sense because, for similar examples with N <= 9, the results are
as expected (1 / N!).

The "exact" p-value calculation is good enough for most practical purposes, but
is clearly not exact.  Some informal testing I've done indicates that it can
even be less accurate than the "approximate" p-value calculation in some cases. 
I think it's absurd to call these p-values "exact" when they are clearly based
on an asymptotic approximation that can be off by orders of magnitude in some
cases.

Petr Savicky

Mon, Nov 30, 2009 3:58 AM #

On Mon, Nov 30, 2009 at 04:00:12AM +0100, dsimcha at gmail.com wrote:

The Edgeworth approximation is not exact, although the error is quite small.
Computing the exact values has time complexity roughly 2^n, if Ryser's formula
for permanent is used. In cases, where one needs the exact values for small n,
it is possible to use the package pspearman at CRAN, which includes precomuted
values up to n=22. See also the paper
  M.A. van de Wiel and A. Di Bucchianico,
  Fast computation of the exact null distribution of Spearman's rho and Page's L statistic
  for samples with and without ties, J. Stat. Plann. Inf. 92 (2001), pp. 133-145.
which deals with this question.

  library(pspearman)
  a <- c(1:10)
  b <- c(1:10)
  out <- spearman.test(a, b, alternative = "greater", approximation="exact")
  out
  #         Spearman's rank correlation rho
  # data:  a and b 
  # S = 0, p-value = 2.756e-07
  # alternative hypothesis: true rho is greater than 0 
  # sample estimates:
  # rho 
  #   1 
  out$p.value - 1/factorial(10) # [1] 0

PS.