Error in vector("double", length) : vector size specified is too large....VLDs

"Tom Colson" <tom_colson at ncsu.edu> writes:

rm(data1)
variog.1.b <- variog(raw.data)

variog: computing omnidirectional variogram
Error in vector("double", length) : vector size specified is too large

Turns out I was wrong re: # of rows...it's 304,000


Same problem. Version is 2.1.1, hardware is Dual Xeon 3.6 4 GB RAM, XP Pro
64 Bit. Can reproduce the problem with 64Bit R 2.1.1 running on Fedora 4,
same hardware. 

Variograms involve the differences between all pairs of points which
can become a rather large number of values. 304000*303999/2 in your
case, about 344GB by my reckoning. And the distances between them
makes for a similar quantity.

Now, some algorithms may be smarter than to keep all values in memory,
but you haven't even told us where you got the variog() from. It
doesn't seem to be in the standard packages, although we do have
variogram() and Variogram() in spatial and nlme.

O__  ---- Peter Dalgaard             ??ster Farimagsgade 5, Entr.B
  c/ /'_ --- Dept. of Biostatistics     PO Box 2099, 1014 Cph. K
 (*) \(*) -- University of Copenhagen   Denmark          Ph:  (+45) 35327918
~~~~~~~~~~ - (p.dalgaard at biostat.ku.dk)                  FAX: (+45) 35327907

"Tom Colson" <tom_colson at ncsu.edu> writes:

rm(data1)
variog.1.b <- variog(raw.data)

variog: computing omnidirectional variogram
Error in vector("double", length) : vector size specified is too large

Turns out I was wrong re: # of rows...it's 304,000


Same problem. Version is 2.1.1, hardware is Dual Xeon 3.6 4 GB RAM, XP Pro
64 Bit. Can reproduce the problem with 64Bit R 2.1.1 running on Fedora 4,
same hardware. 

Variograms involve the differences between all pairs of points which
can become a rather large number of values. 304000*303999/2 in your
case, about 344GB by my reckoning. And the distances between them
makes for a similar quantity.

Now, some algorithms may be smarter than to keep all values in memory,
but you haven't even told us where you got the variog() from. It
doesn't seem to be in the standard packages, although we do have
variogram() and Variogram() in spatial and nlme.

Right, this is from geoR, which uses full matrices. I think both fields 
and gstat can work with larger data sets. Whether model-based 
geostatistics is what you need for interpolating a digital elevation model 
is another question.

Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no

Yes, using geoR. 

I can interpolate the DEM quite easily in Grass (v.surf.rst, kriging) and
block kriging in ArcInfo. What we need, though, is to be able to "estimate"
or even nail down the variogram for these data sets. Where am I going with
this? I'm guessing that variables such as slope, ruggedness, etc.. are going
to produce different sill, range, and nugget values, which I can then use to
fine tune the interpolation process, rather than using the same spline or
kriging parameters on say, a whole state boundary worth of Lidar data.  And
yes, I can estimate the variogram in ArcInfo (limited to 10000 points) and
can also import the DEM from grass into R using spgrass....but the point is
to analyze the point data BEFORE I make the DEM. 

So I'm guessing the geoR isn't ever going to handle this size data, and I
need to be using gstat? (As I write this, gstat(variogram) is plugging away
for last 10 minute with no errors.....)

Thanks for quick replies


Thomas Colson
North Carolina State University
Department of Forestry and Environmental Resources
(919) 673 8023
tom_colson at ncsu.edu

Calendar:
www4.ncsu.edu/~tpcolson
 

-----Original Message-----
From: Roger Bivand [mailto:Roger.Bivand at nhh.no] 
Sent: Thursday, September 15, 2005 3:28 PM
To: Peter Dalgaard
Cc: Tom Colson; r-help at stat.math.ethz.ch
Subject: Re: [R] Error in vector("double", length) : vector size specified
is too large....VLDs

"Tom Colson" <tom_colson at ncsu.edu> writes:

rm(data1)
variog.1.b <- variog(raw.data)

variog: computing omnidirectional variogram Error in 
vector("double", length) : vector size specified is too large

Turns out I was wrong re: # of rows...it's 304,000


Same problem. Version is 2.1.1, hardware is Dual Xeon 3.6 4 GB RAM, 
XP Pro
64 Bit. Can reproduce the problem with 64Bit R 2.1.1 running on 
Fedora 4, same hardware.

Variograms involve the differences between all pairs of points which 
can become a rather large number of values. 304000*303999/2 in your 
case, about 344GB by my reckoning. And the distances between them 
makes for a similar quantity.

Now, some algorithms may be smarter than to keep all values in memory, 
but you haven't even told us where you got the variog() from. It 
doesn't seem to be in the standard packages, although we do have
variogram() and Variogram() in spatial and nlme.

Right, this is from geoR, which uses full matrices. I think both fields and
gstat can work with larger data sets. Whether model-based geostatistics is
what you need for interpolating a digital elevation model is another
question.

Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no