Skip to content

RPostgreSQL and memory usage error

5 messages · Alan Eng, Paul Gilbert, Tim Keitt +2 more

Original
Alan Eng
# 
Hi there,

My name is Alan and I work in analytics for a startup in San Francisco. My
company has been using the RPostgreSQL package as an interface to our
Postgres DB for the past year but recently an error surfaced that I could
not find the solution to on Google.

With small or large queries, the error "RS-DBI driver: (could not Retrieve
the result : ERROR: out of memory DETAIL: Failed on request of size
3145728" comes up. The "3145728" seems random because other times I get
size of 36 or 24.

Thanks in advance for any tips/advice on resolving this issue. Here's some
info on package versions:

R version 3.0.1 (2013-05-16)
RPostgreSQL_0.4
DBI_0.2-7
sqldf_0.4-7.1
1 day later
Paul Gilbert
# 
One possibility is that you are mistakenly doing a query that returns a 
much larger number of records than you are expecting. Try count(*) to 
check how many records you are trying to return.

If the client and server are the same machine, there is a possibility 
that the server is grabbing a lot of memory, maybe for a join or 
something like that. I'm really not very familiar with the memory 
demands of the server. I guess I am not even sure if this message comes 
from the R side of the DBI or is passed back from the server and DBI is 
just the messenger. Others on this list probably know the answer to that.

Paul
On 05/05/2014 01:15 PM, Alan Eng wrote:
Tim Keitt
#
On Tue, May 6, 2014 at 4:47 PM, Paul Gilbert <pgilbert902 at gmail.com> wrote:

            

      
      
      
    

        
Googling that error message leads to posts about postgresql so its coming
from the backend. Almost certainly an error in the SQL query.

THK

            

      
      
      
    

        

      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Neil Tiffin
    

    

      
      #
    

  


  

      
Alan,

Adding to what Paul said, we would need some more info in order to help you.  The best would be sample tables, data, and queries that reproduces the problem.  If the problem is based on a large data set, we can generate data if we know how to make the problem happen.

Regards,

Neil

        
On May 5, 2014, at 12:15 PM, Alan Eng <aeng at stitchfix.com> wrote:

        

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      NISHIYAMA Tomoaki
    

    

      
      #
    

  


  

      
Hi, 

The version 0.4 of RPostgreSQL is known to have memory leak issues,
which at least in part was fixed in the latest svn branch.
https://code.google.com/p/rpostgresql/issues/detail?id=57
If you have run for a long time on a low memory computer, they might
hurt you. 

For the release of the fixed version, we need to change the tests structure 
so that the output on a machine with and without the DB setup will not be
too much different and clearly understandable to persuade CRAN maintainers.

The part "ERROR: out of memory DETAIL: Failed on request of size 3145728",
came from the libpq function call.