memory issue on R with Linux 64

Hi List,

I get an error using readGDAL{rgdal}: cannot allocate vector of size 3.1 Gb

I am using Linux 64bit (opensuse 11) with 4 gig swap and 4 gig Ram and R 2.8.0. 

The load monitor shows that most of Ram is used up and then when Swap use starts increasing, R returns the error.

Is there anything I should do within R to circumvent this?


Any help appreciated
Thanks
Herry

Hi List,

I get an error using readGDAL{rgdal}: cannot allocate vector of size 3.1 
Gb

I am using Linux 64bit (opensuse 11) with 4 gig swap and 4 gig Ram and R 
2.8.0.

The load monitor shows that most of Ram is used up and then when Swap 
use starts increasing, R returns the error.

Is there anything I should do within R to circumvent this?


Any help appreciated
Thanks
Herry

On Thu, 29 Jan 2009, Alexander.Herr at csiro.au wrote:

Hi List,

I get an error using readGDAL{rgdal}: cannot allocate vector of size 3.1
Gb

This is a tile of your 73K by 80K raster, right? One possibility is to use
smaller tiles, another to get more memory (as Edzer wrote), a third to use
lower level functions in rgdal to avoid duplication (and repeated gc()) - in
readGDAL the data read by getRasterData() are copied, so at least doubling
memory usage.

Do you need to read the raster? If this is the overlay problem, you should
be able to use the output of GDALinfo for your raster to build a
GridTopology and SpatialGrid, and overlay (tiles of) that on the
SpatialPolygons (tiles of that because overlay() will generate cell centre
coordinates and do point in polygon, so you're still stuck with too many
coordinates). The next issue would be to copy out the polygon IDs, or the
extracted values, as a raster - here the forthcoming raster package on
R-Forge may be what you need.

Roger

I am using Linux 64bit (opensuse 11) with 4 gig swap and 4 gig Ram and R
2.8.0.

The load monitor shows that most of Ram is used up and then when Swap use
starts increasing, R returns the error.

Is there anything I should do within R to circumvent this?


Any help appreciated
Thanks
Herry

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no

Herry,

This is how you can do it in package{raster}. (revision 209;  a build
should be available within 24 hours).

Following Edzer's example:

require(raster)
library(maptools)
# read a SpatialPolygonsDataFrame
nc <- readShapePoly(system.file("shapes/sids.shp",
     package="maptools")[1], proj4string=CRS("+proj=longlat +datum=NAD27"))
# create a new RasterLayer object from the polygon bounding box and
set row / cols.
rs <- rasterFromBbox(nc, nrows=54, ncols=176)
# transfer polygons to raster, use values of column 13 in the polygon dataframe
rs <- polygonsToRaster(nc, rs, field=13)
# plot, either directly
plot(rs)
plot(nc, add=T, border="blue")
# or via sp for a real map
x11()
spplot(asSpGrid(rs), col.regions=bpy.colors())

also see the example in ?polygonsToRaster

The polygonsToRaster function works for very large rasters (row by row
processing if you provide an output file name). I have only tested if
for a very limited number of very simple cases, so user beware...
The algorithm needs optimization for speed, so that might be a problem
for your very large grids. (particularly of your polygons are also
complex). It also needs some tweaking (and options) for deciding when
a polygon is in a grid cell. As is, the intention is that a polygon
has to overlap with the center of a cell to be considered inside.

Robert




On Thu, Jan 29, 2009 at 3:09 PM, Roger Bivand <Roger.Bivand at nhh.no> wrote:

On Thu, 29 Jan 2009, Alexander.Herr at csiro.au wrote:

Hi List,

I get an error using readGDAL{rgdal}: cannot allocate vector of size 3.1
Gb

This is a tile of your 73K by 80K raster, right? One possibility is to use
smaller tiles, another to get more memory (as Edzer wrote), a third to use
lower level functions in rgdal to avoid duplication (and repeated gc()) - in
readGDAL the data read by getRasterData() are copied, so at least doubling
memory usage.

Do you need to read the raster? If this is the overlay problem, you should
be able to use the output of GDALinfo for your raster to build a
GridTopology and SpatialGrid, and overlay (tiles of) that on the
SpatialPolygons (tiles of that because overlay() will generate cell centre
coordinates and do point in polygon, so you're still stuck with too many
coordinates). The next issue would be to copy out the polygon IDs, or the
extracted values, as a raster - here the forthcoming raster package on
R-Forge may be what you need.

Roger

I am using Linux 64bit (opensuse 11) with 4 gig swap and 4 gig Ram and R
2.8.0.

The load monitor shows that most of Ram is used up and then when Swap use
starts increasing, R returns the error.

Is there anything I should do within R to circumvent this?


Any help appreciated
Thanks
Herry

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no

Hi List,

I get an error using readGDAL{rgdal}: cannot allocate vector of size 
3.1 Gb

I am using Linux 64bit (opensuse 11) with 4 gig swap and 4 gig Ram and R 2.8.0. 

The load monitor shows that most of Ram is used up and then when Swap use starts increasing, R returns the error.

Is there anything I should do within R to circumvent this?


Any help appreciated
Thanks
Herry

On Thu, 29 Jan 2009, Alexander.Herr at csiro.au wrote:

Hi List,

I get an error using readGDAL{rgdal}: cannot allocate vector of size 
3.1 Gb

This is a tile of your 73K by 80K raster, right? One possibility is to 
use smaller tiles, another to get more memory (as Edzer wrote), a 
third to use lower level functions in rgdal to avoid duplication (and 
repeated gc()) - in readGDAL the data read by getRasterData() are 
copied, so at least doubling memory usage.

Do you need to read the raster? If this is the overlay problem, you 
should be able to use the output of GDALinfo for your raster to build 
a GridTopology and SpatialGrid, and overlay (tiles of) that on the 
SpatialPolygons (tiles of that because overlay() will generate cell 
centre coordinates and do point in polygon, so you're still stuck with 
too many coordinates). The next issue would be to copy out the polygon 
IDs, or the extracted values, as a raster - here the forthcoming 
raster package on R-Forge may be what you need.

Roger

I am using Linux 64bit (opensuse 11) with 4 gig swap and 4 gig Ram 
and R 2.8.0.

The load monitor shows that most of Ram is used up and then when Swap 
use starts increasing, R returns the error.

Is there anything I should do within R to circumvent this?


Any help appreciated
Thanks
Herry

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School 
of Economics and Business Administration, Helleveien 30, N-5045 
Bergen, Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no