errorsarlm out of memory (method Chebyshev undetected in help)

Thanks for reminding.

method "Sparse" and "Chebyshev"" unavailable here

memry.size ()  TRUE=1535.12  / FALSE=1147.03
memory.limit() size=NA  1535

1. spdep  ver 2.10.1

2. the code is as below.

rm(list=ls())
datam <-read.csv("c:/migration/M_R_20100618_t2.csv",header=T, row.names=1)
dim(datam)
datam[1,]

#SAR err
library(ncf)
require (spdep)

# Define coordinates, neighbours, and spatial weights
coords<-cbind(datam$lat,datam$lon)
coords<-as.matrix(coords)

#Define neighbourhood (here distance 9)
nb10<-dnearneigh(coords,0,9)    # 9

#######################
#Spatial weights, illustrated with coding style "W" (row standardized)
nb10.w<-nb2listw(nb10, glist=NULL, style="W", zero.policy=FALSE)

#style="C"(sums over all links to n), or"B"(binary) for method "Matrix"or
"spam" in errorsarlm
#"C" and "B" = symmetric but "W"(sums over all links to n), "S" = similar to
symmertic
#"U"(sums over all links to unity)  , "S"( variance-stabilizing coding )


glm1<- glm (datam$bird_w~datam$topo_var)
summary(glm1)
res.glm1 <- residuals(glm1)


#######################
#1. Spatial SAR error model
sem.nb10.w <- errorsarlm(glm1,

listw=nb10.w,na.action=na.fail,tol.solve=1.0e-12)
summary(sem.nb1.5.w) #Gives a summary of the SAR error model
res.sem.nb1.5.w <- residuals(sem.nb1.5.w)
#######################




3. error message
method=default
warnings
cannot allocate vector of size 181.2 Mb

warnings()

Warning messages:
1: In errorsarlm(glm1, listw = nb10.w, na.action = na.fail,  ... :
 Reached total allocation of 1535Mb: see help(memory.size)
2: In errorsarlm(glm1, listw = nb10.w, na.action = na.fail,  ... :
 Reached total allocation of 1535Mb: see help(memory.size)
3: In crossprod(WA) : Reached total allocation of 1535Mb: see
help(memory.size)
4: In crossprod(WA) : Reached total allocation of 1535Mb: see
help(memory.size)
5: In array(0, c(n, p)) :
 Reached total allocation of 1535Mb: see help(memory.size)
6: In array(0, c(n, p)) :
 Reached total allocation of 1535Mb: see help(memory.size)
7: In t.default(W) : Reached total allocation of 1535Mb: see
help(memory.size)
8: In t.default(W) : Reached total allocation of 1535Mb: see
help(memory.size)
9: Reached total allocation of 1535Mb: see help(memory.size)
10: Reached total allocation of 1535Mb: see help(memory.size)
11: In array(0, c(n, p)) :
 Reached total allocation of 1535Mb: see help(memory.size)
12: In array(0, c(n, p)) :
 Reached total allocation of 1535Mb: see help(memory.size)
13: In y[1L + 0L:(m - 1L) * (n + 1L)] <- x :
 Reached total allocation of 1535Mb: see help(memory.size)
14: In y[1L + 0L:(m - 1L) * (n + 1L)] <- x :
 Reached total allocation of 1535Mb: see help(memory.size)
15: In colnames(b) <- rownames(a) :
 Reached total allocation of 1535Mb: see help(memory.size)
16: In colnames(b) <- rownames(a) :
 Reached total allocation of 1535Mb: see help(memory.size)
17: In storage.mode(a) <- "double" :
 Reached total allocation of 1535Mb: see help(memory.size)
18: In storage.mode(a) <- "double" :
 Reached total allocation of 1535Mb: see help(memory.size)
19: In rownames(b) <- colnames(a) :
 Reached total allocation of 1535Mb: see help(memory.size)
20: In rownames(b) <- colnames(a) :
 Reached total allocation of 1535Mb: see help(memory.size)
21: In rownames(b) <- colnames(a) :
 Reached total allocation of 1535Mb: see help(memory.size)
22: In rownames(b) <- colnames(a) :
 Reached total allocation of 1535Mb: see help(memory.size)

method ="Matrix" (style ="B", the remaining part of code is identical as
above)

?????????if (mean(as(last, "matrix")) < tol) { :
 something lacking in TRUE/FALSE value
??????: ?????? 40 ????????? (??? warnings() ?????????)
Warning messages:
1: In determinant(x, TRUE) : This version of the Matrix package returns
|determinant(L)| instead of determinant(A), i.e., a
*DIFFERENT* value.
If still necessary, do change your code, following
http://matrix.r-forge.r-project.org

2: In optimize(sar.error.f, interval = interval, maximum = TRUE,  ... :
 NA/Inf is converted to the maximum positive value
.........
23: In .local(object, ...) :
 Reached total allocation of 1535Mb: see help(memory.size)
24: In .local(object, ...) :
 Reached total allocation of 1535Mb: see help(memory.size)

25: In optimize(sar.error.f, interval = interval, maximum = TRUE,  ... :
 NA/Inf is converted to the maximum positive value
40: the same


method"spam"
Loading required package: spam
Package 'spam' is loaded. Spam version 0.22-0 (2010-06-08).
Type demo( spam) for some demos, help( Spam) for an overview
of this package.
Help for individual functions is optained by adding the
suffix '.spam' to the function name, e.g. 'help(chol.spam)'.

Attaching package: 'spam'


       The following object(s) are masked from package:Matrix :

        norm


       The following object(s) are masked from package:base :

        backsolve,
        forwardsolve

warnings()

Warning messages:
1: package 'spam' was built under R version 2.10.1
2: In optimize(sar.error.f, interval = interval, maximum = TRUE,  ... :
 NA/Inf is converted to the maximum positive value
.....
16: the same
17: In t.spam(x) : Reached total allocation of 1535Mb: see help(memory.size)
18: In t.spam(x) : Reached total allocation of 1535Mb: see help(memory.size)
19: In optimize(sar.error.f, interval = interval, maximum = TRUE,  ... :
 NA/Inf is converted to the maximum positive value
...
40:  the same





On Fri, Jun 25, 2010 at 6:10 PM, Roger Bivand <Roger.Bivand at nhh.no> wrote:

On Fri, 25 Jun 2010, elaine kuo wrote:

 Dear List,

I am using errorsarlm (spdep) to measure SAR autocorrelation but failed as
detailed below.

system: windows XP and Vista
RAM : 2G (XP) and 4G(Vista)

sample size : 4500
variable: 4
nb10<-dnearneigh(coords,0,9)

the error issue: out of memory of 180 Mb

The similar messages in the archive were checked and tried but failed to
solve the issues.
1. enlarging the R space to 2047 Mb
2. updating the neighboring unit from 0.5 to 9

Please kindly help how to solve the problem, for I found SAR and CAR seem
to
be more reliable for autocorrelation than autocovariate.

No, you have not included the verbatim code leading to the memory problem,
nor the output of traceback() after the error, nor the output of
sessionInfo() to tell us which versions of packages you are using. This is
almost certainly nothing to do with dnearneigh(), unless the (0, 9)
thresholds include all n observations as neighbours for each neighbour, and
even then the neighbour object would not be very large.

Did you read the help page for errorsarlm() or spautolm()? No? Then do so
and look carefully at the method= argument. You are almost certainly using
the default method, which uses dense matrices. Use your choice of sparse or
approximation alternatives, for example "Matrix" for updating sparse
Cholesky Jacobians, or "Chebyshev" for a fast approximate Jacobians.

Using sparse matrix methods to overcome memory constraints has been
mentioned on this and other lists many times. With n=25000 ther is no
problem on a 1GB machine, so your problem doesn't even count as large.

The list isn't really an alternative for reading help pages, and crucially
accessing the references given on the help pages. I'll add a link to
?do_ldet on the function fitting pages, and add LeSage J and RK Pace (2009)
Introduction to Spatial Econometrics. CRC Press, Boca Raton too, as this is
the key reference that is required reading in this area.

Hope this helps,

Roger

Thanks

Elaine

       [[alternative HTML version deleted]]

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no

	[[alternative HTML version deleted]]