Skip to content

error in errorsarlm

9 messages · elaine kuo, Danlin Yu, Roger Bivand

Original
Danlin Yu
# 
Elaine:

Based on the error message (cannot allocate vector of size 121.8 Mb), it 
is very possibly you have way too many cases in your data set. A 
potential solution would be randomly sample your dataset to get a 
representative sample set and run the spatial regression, or you can 
always try a computer with more physic memory, or try memory.limit() to 
manually allocate more memories. See ?memory.limit for more details.

Hope this helps.

Cheers,
Dr. Danlin Yu

elaine kuo ??:

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Roger Bivand
    

    

      
      #
    

  


  

      
On Fri, 16 Jul 2010, elaine kuo wrote:

        

            

      
      
      
    

        
Beyond what Danlin said, I am very unsure where to start. Please provide 
adequate information about your input data, specifically the number of 
observations, and the number of links in the neighbour list

length(nb10)
sum(card(nb10))

In addition, the implicit discription of nb10 looks confusing (is the 
distance threshold 1.5, 9, or 10?). The ordering of lat and lon looks 
wrong. You are not using Great Circle distances (and probably should?). So 
memory issues in errorsarlm are the least of your problems. My guess is 
that with dnearneigh() and a threshold of 9 degrees, you are including 
most of a large data set in the neighbour list, and if n is large, this 
will give a very large almost dense set of spatial weights, which was not 
what you intended.

Get the weights correct first, and the rest will likely follow.

Roger

            

      
      
      
    

        

      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        
      

    

    

      
      

        


  

        

      Roger Bivand
    

    

      
      #
    

  


  

      
On Sat, 17 Jul 2010, elaine kuo wrote:

        

            

      
      
      
    

        
But 9 what, metres, degrees? Should your code be: 
cbind(datam$lat,datam$lon), as the normal ordering is eastings then 
northings, not the opposite?

            

      
      
      
    

        
This is not a good reason, as graph neighbours can be sparse and link over 
longer distances.

            

      
      
      
    

        
This gives an average of 181.7 neighbours per observation, and is the root 
of the problem. The Chebyshev method uses a set of low powers of the 
weights, which fill in quickly for large average numbers of neighbours. 
Did you try the Matrix method (your error message suggests that you may 
have done, but the code does not)?

            

      
      
      
    

        
Windows (your OS?) memory handling is very poor for larger objects, and 
the 121Mb is extra memory needed. However, the main problem is your design 
of weights, which are far too dense to make sense. Choose an alternative 
weights definition and try again.

Roger

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    
2 days later

    

      
      

        
      

    

    

      
      

        


  

        

      Roger Bivand
    

    

      
      #
    

  


  

      
On Tue, 20 Jul 2010, elaine kuo wrote:

        

            

      
      
      
    

        
Just do not use distance as a criterion when the observations are very 
irregularly spaced! Use perhaps a symmetricised k-nearest neighbour 
definition. And make sure that you do use Great Circle distances - you are 
not doing so now. With 200 neighbours, you have no viable spatial process, 
really.

You can accept no-neighbour observations when there is no good reason to 
suppose that they could interact, but then have to attend to your 
zero.policy settings. It all depends on what processes are thought to be 
causing the observations to influence each other.

            

      
      
      
    

        
Warning not a problem, but the error suggests scaling problems in the 
covariance matrix of the coefficients. The X variables are most likely 
measured in inappropriate metrics (some coefficients of size 1000, others 
of size 0.0001, for example).

Please find local help to resolve the underlying problems, related to why 
you want these weights, and why you are trying to fit this kind of model 
(that is: is your research question sensible and what authority 
(literature) do you have for your choices?). Solve all of the problems 
first on a small subset of your data, and scale up afterwards.

Roger

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    
2 days later

    

      
      

        
      

    

    
7 days later

    

      
      

        


  

        

      Roger Bivand
    

    

      
      #
    

  


  

      
On Fri, 23 Jul 2010, elaine kuo wrote:

        

            

      
      
      
    

        
The warning is irrelevant. The error is caused by your weights.

            

      
      
      
    

        
The threshold is in km when longlat=TRUE, but the coordinates are in 
decimal degrees. Please state the representation of your coordinates - are 
they in decimal degrees or not?

            

      
      
      
    

        
No, simply that you are including too many neighbours leading to too much 
smoothing for many observations.

Please try everything on small examples until you understand what you are 
doing. Best, use publically available small examples, and include complete 
verbatim code (or post code on a website), max. 10 lines.