Skip to content

Large 3d array manipulation

4 messages · Vemuri, Aparna, Duncan Murdoch, David Winsemius

Vemuri, Aparna
# 
I have a large 3 dimensional array of size (243,246,768)
 The first dimension is Rows, second is columns and the third is Time. 

So for each row and column, I want to calculate the mean of time steps
1:8, 2:9, 3:10 and so on and assign the values to a new array. For this
I am using the following script.

for(i in 1:243)
{
for(j in 1:246)
{
for(k in 1:768)
{
newVar[i,j,k] <- mean( myVar[i,j,k:k+7])
}
}
}

This works, but needless to mention it take a very long time to loop
over all the rows, columns and time periods. I was wondering if there is
a simpler way to do this.

Thanks
Aparna
Duncan Murdoch
#
On 27/02/2009 6:15 PM, Vemuri, Aparna wrote:
Yes, vectorize it.  I think this would work:

newVar <- array(NA, c(243, 246, 768))

for (k in 1:768)
   newVar[,,k] <- apply(myVar, 1:2, function(x) mean(x[k:(k+7)]))

but of course I haven't tried it.

Duncan Murdoch
David Winsemius
#
On Feb 27, 2009, at 7:49 PM, Duncan Murdoch wrote:

            

      
      
      
    

        
That's rather interesting. I had not realized that one could use that  
construction with apply. I had earlier tried to substitute rollmean  
for the inner loop. For one thing I thought that trying to index 768+7  
was going to create some problems with out-of-range indexing. I got  
the same error with my effort to insert rollmean in the OP's  
construction as I do in this construction:

myVar <- array(1:(243*246*768), dim=c(243,246,768))
 > myVar[1,1,1:8]
[1]      1  59779 119557 179335 239113 298891 358669 418447

newVar <-array(,dim=c(243,246,768))
library(zoo)
for (k in 1:768)  newVar[,,k] <- apply(myVar, 1:2, function(x)  
mean(x[k:(k+7)]))
Error in newVar[, , k] <- apply(myVar, 1:2, function(x) rollmean(x,  
8)) :
   number of items to replace is not a multiple of replacement length

I am guessing that at some point the assignment function cannot  
resolve which index in newVar to use. So I tried redimensioning newVar  
to only have 761 as it third dimension and using:

  myVar[1,1,1:8]

for (k in 1:761)  {newVar[ , , ] <- apply(myVar, 1:2, function(x)  
mean(x[k:(k+7)])) ; print(k)}

This may be working. This executes in less than 20 seconds:

 > newVar <-array(,dim=c(243,246,761))
 > str(newVar); Sys.time()
  logi [1:243, 1:246, 1:761] NA NA NA NA NA NA ...
[1] "2009-02-27 22:35:48 EST"
 >  newVar[,,] <- apply(myVar, 1:2, function(x) rollmean(x, 8));  
Sys.time()
[1] "2009-02-27 22:35:57 EST"
 > str(newVar)
  num [1:243, 1:246, 1:761] 209224 269002 328780 388558 448336 ...
 >

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      David Winsemius
    

    

      
      #
    

  


  

      
On Feb 27, 2009, at 10:37 PM, David Winsemius wrote:

        

            

      
      
      
    

        
I forgot to mention that at this point I decided that the for loop  
might be entirely superfluous if one were returning a 761 length  
vector from rollmean. So I tested my theory: