Creating very large spatial weight matrix
Sorry, I also realize that your vector is just too long - R is limited to a 32-bit index, so it just cannot be done without special tricks in separate vectors. The ff package documentation mentions that 64-bit indexing may be included in a future release, and there have been discussions on this for base R. (perhaps for R 3.0) Cheers, Mike
On Fri, Nov 19, 2010 at 10:40 AM, Michael Sumner <mdsumner at gmail.com> wrote:
And, please report your OS and version of R (64-bit presumably?). On Fri, Nov 19, 2010 at 10:39 AM, Michael Sumner <mdsumner at gmail.com> wrote:
In general you need at least twice the required memory, and it has to be contiguous. Try with a fresh instance of R and try to create a single vector of that size, that might show that you *could* do it. Otherwise, check out the ff package, and see other options in the High Performance Computing Task View on CRAN. There may be other techniques you can use to solve the problem, but those two things are my direct answers to your questions. Cheers, Mike. On Fri, Nov 19, 2010 at 10:28 AM, Aleksandr Andreev <aleksandr.andreev at gmail.com> wrote:
Hello list,
I have 120,000 geocoded observations, for which I'm trying to create a
distance-based spatial weighting matrix so that I can perform a Moran
test.
Each observation has Lat and Lon.
Unfortunately, when I run
dists <- as.matrix(dist(cbind(Lon, Lat)))
I get the message:
Error in vector("double", length) : vector size specified is too large
Now I realize that 120,000^2 / 2 is on the order of 6 GB. However, I
seem to be running into software limitations on the vector size before
I hit RAM limitations. Also, in principle, it should be possible
(though slow) to use hard disk space to store this matrix. Does anyone
have any ideas on how to do this in R?
Thanks,
------------------------
Aleksandr Andreev
Graduate Student - Department of Economics
University of North Carolina at Chapel Hill
_______________________________________________ R-sig-Geo mailing list R-sig-Geo at stat.math.ethz.ch https://stat.ethz.ch/mailman/listinfo/r-sig-geo
-- Michael Sumner Institute for Marine and Antarctic Studies, University of Tasmania Hobart, Australia e-mail: mdsumner at gmail.com
-- Michael Sumner Institute for Marine and Antarctic Studies, University of Tasmania Hobart, Australia e-mail: mdsumner at gmail.com
Michael Sumner Institute for Marine and Antarctic Studies, University of Tasmania Hobart, Australia e-mail: mdsumner at gmail.com