Faster way to get raster average?
Thank you all for the responses. All of them helped me to learn new approaches. Since I wanted to stick to R (even though I call CDO from R), I ended up adopting Michael's suggestion to reshape data as a matrix before taking the averages. While relatively hard to read and understand, the time saved in the operation will probably make a huge difference in the final script looping through thousands of files.?Greetings, -- Thiago V. dos Santos PhD student Land and Atmospheric Science University of Minnesota http://www.laas.umn.edu/CurrentStudents/MeettheStudents/ThiagodosSantos/index.htm Phone: (612) 323 9898
On Wednesday, May 6, 2015 7:25 AM, Byman HIkanyona <bymanh at gmail.com> wrote:
Hi, My approach, in brief, is similar to what Micheal? Summer suggested, i.e using the netcdf tools (CDO, NCO, etc.). Whether in Linux, Mac,or windows. these tools run more efficiently - memory use and speed. I sometimes use R to automate the run since most are run on (DOS prompt like environment, batch). You can also 'pipe' the operation for example in CDO to avoid saving extra files. For example (CDO - windows) 'R CMD BATCH cdo?ensmean?ifile[1-100]?ofile'? computes the ensemble mean of 100 files listed and saves the result in output file, which you can read using Raster packages. I don't know if this helps. Cheers, Byman
On Wed, May 6, 2015 at 10:02 AM, Rainer M Krug <Rainer at krugs.de> wrote:
"Thiago V. dos Santos" <thi_veloso at yahoo.com.br> writes:
Oh, I forgot to mention that I work on a quad-core Mac with 12GB.
So, if multi-core can potentially accelerate cellStats, or whatever function is faster to get a raster average, I would be glad to see some examples on how to implement it.?Greetings,
The key question is: how often do you have to do the calculations and if it would be feasible to just let them run for a few days and be done with it - optimization takes time! Cheers, Rainer
-- Thiago V. dos Santos PhD student Land and Atmospheric Science University of Minnesota http://www.laas.umn.edu/CurrentStudents/MeettheStudents/ThiagodosSantos/index.htm Phone: (612) 323 9898 ? ? ? On Tuesday, May 5, 2015 8:48 PM, Thiago V. dos Santos <thi_veloso at yahoo.com.br> wrote: ? Hi all, I am working with some terabytes of CMIP5 climate files.?Each file is a netcdf with multiple layers (timesteps) representing monthly data.? For each file, I need to extract the average value of the raster and put all values in a data frame. This is my current approach:--------------- library(raster) # make up some datacmip <- brick(nc=150, nr=114, nl=1872)cmip <- setValues(cmip, matrix(rep(1:17100, 1872), nc=1872)) # get mean values (area average) as data framescmip.mean <- as.data.frame(cellStats(cmip, mean, na.rm=T))--------------- which works pretty fast in this example:
system.time(as.data.frame(cellStats(cmip, mean, na.rm=T)))
? user? system elapsed ? 0.069? 0.012? 0.081 However, the calculation with my actual data is substantially slower:
system.time(as.data.frame(cellStats(cmip, mean, na.rm=T)))
? user? system elapsed ? 4.600? 1.105? 5.704 Since I will have to deal with thousands of files, here comes my question: is there a faster way to get a the average value of a?raster??? Many thanks, -- Thiago V. dos Santos PhD student Land and Atmospheric Science University of Minnesota http://www.laas.umn.edu/CurrentStudents/MeettheStudents/ThiagodosSantos/index.htm Phone: (612) 323 9898 ??? [[alternative HTML version deleted]]
_______________________________________________ R-sig-Geo mailing list R-sig-Geo at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo ? ? ? ?[[alternative HTML version deleted]] _______________________________________________ R-sig-Geo mailing list R-sig-Geo at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo
-- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Stellenbosch University South Africa Tel :? ? ? ?+33 - (0)9 53 10 27 44 Cell:? ? ? ?+33 - (0)6 85 62 59 98 Fax :? ? ? ?+33 - (0)9 58 10 27 44 Fax (D):? ? +49 - (0)3 21 21 25 22 44 email:? ? ? Rainer at krugs.de Skype:? ? ? RMkrug PGP: 0x0F52F982 _______________________________________________ R-sig-Geo mailing list R-sig-Geo at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo
"The highest reward for a person's toil is not what they get for it, but what they become by it." --John Ruskin, British art critic Byman Hamududu [[alternative HTML version deleted]]