Faster way to get raster average?
"Thiago V. dos Santos" <thi_veloso at yahoo.com.br> writes:
Hi all, I am working with some terabytes of CMIP5 climate files.?Each file is a netcdf with multiple layers (timesteps) representing monthly data.? For each file, I need to extract the average value of the raster and put all values in a data frame. This is my current approach:--------------- library(raster) # make up some datacmip <- brick(nc=150, nr=114, nl=1872)cmip <- setValues(cmip, matrix(rep(1:17100, 1872), nc=1872)) # get mean values (area average) as data framescmip.mean <- as.data.frame(cellStats(cmip, mean, na.rm=T))--------------- which works pretty fast in this example:
system.time(as.data.frame(cellStats(cmip, mean, na.rm=T)))
user system elapsed 0.069 0.012 0.081 However, the calculation with my actual data is substantially slower:
system.time(as.data.frame(cellStats(cmip, mean, na.rm=T)))
user system elapsed 4.600 1.105 5.704 Since I will have to deal with thousands of files, here comes my question: is there a faster way to get a the average value of a?raster???
I am not an expert on netcdf files, but I would probably look outside R to do these calculations. GRASS, and possibly even gdal, come to mind. Depending on how often you have to do the calculations, putting them into a spatial database and do the calculations there might be an option? Concerning paralelization: if, as Robert points out, the disk access is the bottleneck (memory allocation in R will also take some time, depending on the size of the individual map), paralelization will not speed up (possibly even slow it down). In this case, your option is an SSD, or a raid, and, as you are using a Mac, an internal or Thunderbolt raid (all faster then even USB 3) - and definitely not an USB 2.0 HDD. Otherwise, just using mcapply() over all maps would help. Cheers, Rainer
Many thanks, -- Thiago V. dos Santos PhD student Land and Atmospheric Science University of Minnesota http://www.laas.umn.edu/CurrentStudents/MeettheStudents/ThiagodosSantos/index.htm Phone: (612) 323 9898 [[alternative HTML version deleted]]
_______________________________________________ R-sig-Geo mailing list R-sig-Geo at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Stellenbosch University South Africa Tel : +33 - (0)9 53 10 27 44 Cell: +33 - (0)6 85 62 59 98 Fax : +33 - (0)9 58 10 27 44 Fax (D): +49 - (0)3 21 21 25 22 44 email: Rainer at krugs.de Skype: RMkrug PGP: 0x0F52F982 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 494 bytes Desc: not available URL: <https://stat.ethz.ch/pipermail/r-sig-geo/attachments/20150506/2f4d3235/attachment.bin>