Remove highly correlated variables from a data frame or matrix

keeprows<-apply(calc.rho,1,function(x) return(sum(x>0.8)<3))
ro246.lt.8<-calc.rho[keeprows,keeprows]
ro246.lt.8[ro246.lt.8 == 1] <- NA
(mmax <- max(abs(ro246.lt.8), na.rm=TRUE))

tree = hclust(1-calc.rho, method = "average")

I suspect that you want to identify which variables are highly
correlated, and then keep only "representative" variables, i.e.,
remove redundant ones. This is a bit of a risky procedure but I have
done such things before as well sometimes to simplify large sets of
highly related variables. If your threshold of 0.8 is approximate, you
could simply use average linkage hierarchical clustering with
dissimilarity = 1-correlation, cut the tree at the appropriate height
(1-0.8=0.2), and from each cluster keep a single representative (e.g.,
the one with the highest mean correlation with other members of the
cluster). Something along these lines (untested)

tree = hclust(1-calc.rho, method = "average")
clusts = cutree(tree, h = 0.2)
clustLevels = sort(unique(clusts))
representatives = unlist(lapply(clustLevels, function(cl)
{
  inClust = which(clusts==cl);
  rho1 = calc.rho[inClust, inClust, drop = FALSE];
  repr = inClust[ which.max(colSums(rho1)) ]
  repr
}))

the variable representatives now contains indices of the variables you
want to retain, so you could subset the calc.rho matrix as
rho.retained = calc.rho[representatives, representatives]

I haven't tested the code and it may contain bugs, but something along
these lines should get you where you want to be.

Oh, and depending on how strict you want to be with the remaining
correlations, you could use complete linkage clustering (will retain
more variables, some correlations will be above 0.8).

Peter

On Thu, Nov 14, 2019 at 10:50 AM Ana Marija <sokovic.anamarija at gmail.com> wrote:

Hello,

I have a data frame like this (a matrix):
head(calc.rho)
            rs9900318 rs8069906 rs9908521 rs9908336 rs9908870 rs9895995
rs56192520      0.903     0.268     0.327     0.327     0.327     0.582
rs3764410       0.928     0.276     0.336     0.336     0.336     0.598
rs145984817     0.975     0.309     0.371     0.371     0.371     0.638
rs1807401       0.975     0.309     0.371     0.371     0.371     0.638
rs1807402       0.975     0.309     0.371     0.371     0.371     0.638
rs35350506      0.975     0.309     0.371     0.371     0.371     0.638

dim(calc.rho)

[1] 246 246

I would like to remove from this data all highly correlated variables,
with correlation more than 0.8

I tried this:

data<- calc.rho[,!apply(calc.rho,2,function(x) any(abs(x) > 0.80))]
dim(data)

[1] 246   0

Can you please advise,

Thanks
Ana

But this removes everything.