correlation significance testing with multiple factor levels

If I understand your problem correctly, then one option is to use a permutation test.  This tests the null hypothesis that set A and set B are identical and the only reason you are seeing a difference is due to random chance.  The procedure is to randomly shuffle the observations between the sets and recomputed your test statistic (difference of mean of correlations).

Here is one example of how you could do this using some simulated data (you should probably try different rho matrices to see how this performs under different situations).

library(MASS)

rho1 <- matrix(.6, 3, 3)
diag(rho1) <- 1
rho2 <- matrix(.1, 3, 3)
diag(rho2) <- 1

x1 <- mvrnorm(25, rep(0,3), rho1)
x2 <- mvrnorm(25, rep(0,3), rho2)

x <- rbind(x1,x2)
g <- rep(1:2, each=25)

tsfunc <- function(g){
	c1 <- cor(x[g==1,])
	c2 <- cor(x[g==2,])
	m1 <- mean( c1[ lower.tri(c1) ] )
	m2 <- mean( c2[ lower.tri(c2) ] )
	m1-m2
}

out <- replicate(999, tsfunc( sample(g) ) )
out <- c(tsfunc(g), out)

hist(out)
abline(v=out[1])

mean( out >= out[1] )  # one-sided p-value

hope this helps,

correlation significance testing with multiple factor levels

Thread (2 messages)