Hello,
i am trying to subset a dataframe multiple times:
something like:
stats <- by(df, list(items), ttestData)
ttestData <- function(df){
t.test( df[,c(2,3,4), df[,c(5,6,7)]
}
While this works for small data, it is to slow for my
actual data: 500000 rows dataframe with
about 135000 different indices, subsetting the
dataframe into chunks of 5 on average.
Do you have any suggestions how I could speed this up?
I tried changing to call by reference with
the package ref, but by does not like that.
Thank you very much in advance
Ido Tamir
extracting from data frame multiple times
2 messages · Ido M. Tamir, Thomas Lumley
On Fri, 14 Oct 2005, Ido M. Tamir wrote:
Hello,
i am trying to subset a dataframe multiple times:
something like:
stats <- by(df, list(items), ttestData)
ttestData <- function(df){
t.test( df[,c(2,3,4), df[,c(5,6,7)]
}
While this works for small data, it is to slow for my
actual data: 500000 rows dataframe with
about 135000 different indices, subsetting the
dataframe into chunks of 5 on average.
Do you have any suggestions how I could speed this up?
The first step is to find out what is too slow, using Rprof(). It may be the t.test or it may be the by(). If it is the by() you could put the numeric data into two matrices x1<-df[,2:4] x2<-df[,5:7] order them so that the same "item" entries were adjacent, compute the start and end indices for each group, and do something like lapply(1:howevermany, function(i) t.test(x1[start[i]:end[i],],x2[start[i]:end[i])) Even just turning df into a matrix might help If it is the repeated t.test() calls that are too slow you need to speed them up. You can probably rowsum() to compute means and variances for all the t-tests at once. -thomas