Replacing for loop with tapply!?

Fri, Jun 10, 2005 11:05 AM

Dear all,

Dimitris and Andy, thanks for your great help. I have progressed to the 
following code which runs very fast and effective:

mat <- matrix(sample(-15:50, 15 * 10, TRUE), 15, 10)
mat[mat>45] <- NA
mat<-NA
mat
temps <- c(35, 37, 39)
ind <- rbind(
     t(sapply(temps, function(temp)
       rowSums(mat > temp, na.rm=TRUE) )),
     rowSums(!is.na(mat), na.rm=FALSE),
     apply(mat, 1, max, na.rm=TRUE))
ind <- t(ind)
ind

However, some weather stations have missing values for the whole year. 
Unfortunately, the code breaks down (when uncommenting mat<-NA).

I have tried 'ifelse' statements in the functions, but it becomes even 
more of a mess. I could subset the matrix before hand, but this would 
mean merging with a complete matrix afterwards to make it compatible 
with other years. That would slow things down.

How can I make the code robust for rows containing all missing values?

Thanks for your help,

Sander.

Dimitris Rizopoulos wrote:

for the maximum you could use something like:

ind[, 1] <- apply(mat, 2, max)

I hope it helps.

Best,
Dimitris

----
Dimitris Rizopoulos
Ph.D. Student
Biostatistical Centre
School of Public Health
Catholic University of Leuven

Address: Kapucijnenvoer 35, Leuven, Belgium
Tel: +32/16/336899
Fax: +32/16/337015
Web: http://www.med.kuleuven.ac.be/biostat/
     http://www.student.kuleuven.ac.be/~m0390867/dimitris.htm



----- Original Message ----- 
From: "Sander Oom" <slist at oomvanlieshout.net>
To: "Dimitris Rizopoulos" <dimitris.rizopoulos at med.kuleuven.be>
Cc: <r-help at stat.math.ethz.ch>
Sent: Friday, June 10, 2005 12:10 PM
Subject: Re: [R] Replacing for loop with tapply!?

Thanks Dimitris,

Very impressive! Much faster than before.

Thanks to new found R.basic, I can simply rotate the result with
rotate270{R.basic}:

mat <- matrix(sample(-15:50, 365 * 15000, TRUE), 365, 15000)
temps <- c(37, 39, 41)
#################
#ind <- matrix(0, length(temps), ncol(mat))
ind <- matrix(0, 4, ncol(mat))
(startDate <- date())

[1] "Fri Jun 10 12:08:01 2005"

for(i in seq(along = temps)) ind[i, ] <- colSums(mat > temps[i])
ind[4, ] <- colMeans(max(mat))

Error in colMeans(max(mat)) : 'x' must be an array of at least two
dimensions

(endDate <- date())

[1] "Fri Jun 10 12:08:02 2005"

ind <- rotate270(ind)
ind[1:10,]

  V4 V3 V2 V1
1   0 56 75 80
2   0 46 53 60
3   0 50 58 67
4   0 60 72 80
5   0 59 68 76
6   0 55 67 74
7   0 62 77 93
8   0 45 57 67
9   0 57 68 75
10  0 61 66 76

However, I have not managed to get the row maximum using your 
method? It
should be 50 for most rows, but my first guess code gives an error!

Any suggestions?

Sander



Dimitris Rizopoulos wrote:

maybe you are looking for something along these lines:

mat <- matrix(sample(-15:50, 365 * 15000, TRUE), 365, 15000)
temps <- c(37, 39, 41)
#################
ind <- matrix(0, length(temps), ncol(mat))
for(i in seq(along = temps)) ind[i, ] <- colSums(mat > temps[i])
ind


I hope it helps.

Best,
Dimitris

----
Dimitris Rizopoulos
Ph.D. Student
Biostatistical Centre
School of Public Health
Catholic University of Leuven

Address: Kapucijnenvoer 35, Leuven, Belgium
Tel: +32/16/336899
Fax: +32/16/337015
Web: http://www.med.kuleuven.ac.be/biostat/
    http://www.student.kuleuven.ac.be/~m0390867/dimitris.htm


----- Original Message ----- 
From: "Sander Oom" <slist at oomvanlieshout.net>
To: <r-help at stat.math.ethz.ch>
Sent: Friday, June 10, 2005 10:50 AM
Subject: [R] Replacing for loop with tapply!?

Dear all,

We have a large data set with temperature data for weather stations
across the globe (15000 stations).

For each station, we need to calculate the number of days a certain
temperature is exceeded.

So far we used the following S code, where mat88 is a matrix
containing
rows of 365 daily temperatures for each of 15000 weather stations:

m <- 37
n <- 2
outmat88 <- matrix(0, ncol = 4, nrow = nrow(mat88))
for(i in 1:nrow(mat88)) {
# i <- 3
row1 <- as.data.frame(df88[i,  ])
temprow37 <- select.rows(row1, row1 > m)
temprow39 <- select.rows(row1, row1 > m + n)
temprow41 <- select.rows(row1, row1 > m + 2 * n)
outmat88[i, 1] <- max(row1, na.rm = T)
outmat88[i, 2] <- count.rows(temprow37)
outmat88[i, 3] <- count.rows(temprow39)
outmat88[i, 4] <- count.rows(temprow41)
}
outmat88

We have transferred the data to a more potent Linux box running R,
but
still hope to speed up the code.

I know a for loop should be avoided when looking for speed. I also
know
the answer is in something like tapply, but my understanding of
these
commands is still to limited to see the solution. Could someone 
show
me
the way!?

Thanks in advance,

Sander.
--

Replacing for loop with tapply!?

Thread (9 messages)