Skip to content

q about memory usage progression

2 messages · Juliet Hannah

Juliet Hannah
# 
I monitored the usage of memory on a script that I ran. It ran 30K
regressions and it stores p-values for one of the
coefficients. It read in a file that has 3000 rows and about 30K
columns. The size of the file is about 170 MB.

My understanding is that memory usage started out at 2.2G and went up to 23G:


cpu=00:03:08, mem=172.75822 GBs, io=0.00000, vmem=2.224G, maxvmem=2.224G
cpu=00:42:35, mem=29517.64894 GBs, io=0.00000, vmem=23.612G, maxvmem=23.612G

I know very little about how memory works, but I thought the hardest
part would be reading the file in. Could
someone explain why there is such a substantial increase over the
course of the script.

Thanks,

Juliet
Juliet Hannah
# 
One more note. In case it is helpful, I am including the code for my loop:

# data is read in

numSNPs <- ncol(myData);
pvalues <- rep(-1,numSNPs);
names(pvalues) <- colnames(myData);


for (SNPnum in 1:numSNPs)
{
   is.na(pvalues[SNPnum]) <- TRUE;
   try({
      fit.yags <- yags(log(myPhenos$PHENOTYPE) ~
myPhenos$AGE+myPhenos$SEX*myData[,SNPnum], id=myPhenos$id,
family=gaussian,corstr="exchangeable",alphainit=0.05)
      z.gee <- fit.yags at coefficients[5]/sqrt(fit.yags at robust.parmvar[5,5]);
      pval <- 2 * pnorm(abs(z.gee), lower.tail = FALSE);
      pvalues[SNPnum] <- pval;
})
}

pvalues <- format(pvalues,digits=3);
On Mon, Dec 29, 2008 at 11:59 AM, Juliet Hannah <juliet.hannah at gmail.com> wrote: