Memory limit problems in R / import of maps
On Tue, 22 Apr 2008, Tomislav Hengl wrote:
Dear list, I know that much has already been said about the memory limit problems. If there is any progress about this problem, we would be interested to hear. In our project, we are importing 24 maps/bands, each consists of 1,450,000 pixels. We further would like to glue all maps into a single data frame (e.g. 'ksc' class in adehabitat package; or 'SpatialGridDataFrame' in sp package), but this seems to be impossible. We tried to run this under windows (after following http://cran.r-project.org/bin/windows/base/rw-FAQ.html#There-seems-to-be-a-limit-on-the-memory-it-us es_0021 and setting the --max-mem-size) and under Linux Ubuntu, but still get the same error message (seems that there is no difference in use of memory under the two OS): "Error: Cannot allocate vector of size 11.1 Mb" The R workspace with 24 loaded grids is also really small (18 MB), but any further gluing and calculation is blocked due the vector size error message.
No, about 11MB per grid times 24 is not 18MB, you are probably looking at the wrong row in gc(). A lot depends on how you are reading them, are you using the rgdal package, or reading ASCII grids? The question is mostly how to organise the workflow to reduce copying to a minimum.
For a comparison, in a GIS such as ArcGIS or SAGA/ILWIS (open source) we have no problems of loading and processing 3-4 times more grids.
Recall that these are different kinds of systems, and very likely not holding and copying all the data in memory.
Should we simply give up on running spatial analysis using large grids (>10 million grids) in R?
Do you mean grids with over 10M grid cells, or 10K grids each of 1M grid cells? If the analyses can be done in tiles, rgdal provides access to the full ability of GDAL to select subscenes. Hope this helps, Roger
Any suggestion or comment is more than welcome, Tom Hengl http://spatial-analyst.net
_______________________________________________ R-sig-Geo mailing list R-sig-Geo at stat.math.ethz.ch https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Roger Bivand Economic Geography Section, Department of Economics, Norwegian School of Economics and Business Administration, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43 e-mail: Roger.Bivand at nhh.no