Skip to content

getValues: error extracting values of large raster

2 messages · Alexander.Herr at csiro.au, Jonathan Greenberg

Alexander.Herr at csiro.au
# 
Hello,
I am getting the following:


While extracting all values with:
Error in getRasterData(con, offset = offs, region.dim = reg, band = object at data@band): long vectors not supported yet: memory.c:3308
class       : RasterLayer 
dimensions  : 76740, 80200, 6154548000  (nrow, ncol, ncell)
resolution  : 50, 50  (x, y)
extent      : -1888000, 2122000, -4847000, -1010000  (xmin, xmax, ymin, ymax)
coord. ref. : +proj=aea +lat_1=-18 +lat_2=-36 +lat_0=0 +lon_0=132 +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs 
data source : /data/her134/BA/GDA94albers/clum50m0314 
names       : clum50m0314 
values      : 100, 663  (min, max)
attributes  :
        ID  COUNT LU_CODEV7 LU_CODEV7N                                  TERTIARY_V7
 from: 100 540931     1.0.0        100  1.0.0 Conservation and natural environments
 to  : 663   1755     6.6.3        663 6.6.3 Estuary/coastal waters - intensive use
                              SECONDARY_V7                              PRIMARY_V7 CLASSES_18
 1.0 Conservation and natural environments 1 Conservation and natural environments          1
                6.6 Estuary/coastal waters                                 6 Water         17
           C18_DESCRIPTION
 Nature conservation (1.1)
               Water (6.0)



Using:
platform       x86_64-unknown-linux-gnu    
arch           x86_64                      
os             linux-gnu                   
system         x86_64, linux-gnu           
status                                     
major          3                           
minor          1.0                         
year           2014                        
month          04                          
day            10                          
svn rev        65387                       
language       R                           
version.string R version 3.1.0 (2014-04-10)
nickname       Spring Dance        

raster version 2.2.31

Linux:
LSB Version:	core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch
Distributor ID:	SUSE LINUX
Description:	SUSE Linux Enterprise Server 11 (x86_64)
Release:	11
Codename:	n/a


I used to be able to handle this sized raster in earlier versions. Any ideas of a workaround?

Cheers and thanks
Herry
1 day later
Jonathan Greenberg
# 
Herry:

This is going to be an issue with your R install, not something
specific to raster -- you are attempting to load an entire raster into
an in-memory vector/matrix, and some versions/installs of R only allow
vectors of length 2^31-1 = 2,147,483,647 values.  Your raster has
6,154,548,000 values, approximately 3x this size.  If your data is
being read in as a 16 bit integer, this means you need 6,154,548,000
times 16 bits equals approximately 12 gb of RAM.  If it is being read
in as a double precision floating point, this means the vector needs
to have at least 48gb of RAM.

You might want to search for that error "long vectors not supported
yet" and see what solutions people have.

As a general rule, you should try to avoid loading an entire raster
into memory if you can avoid it -- why are you trying to do a full
read-into-memory?  Can you do chunk-based processing?  Subsampling?

--j
On Tue, Jul 15, 2014 at 1:07 AM, <Alexander.Herr at csiro.au> wrote: