memory error

Tue, Apr 6, 2010 8:27 AM

I don't have an absolute estimate, but try data.table::merge, as it needs 
less
working memory than base::merge.

20 million rows of 5 columns isn't beyond 32bit :
   (1*4 + 4*8)*19758564/1024^3 = 0.662GB

Also try sqldf to do the join.

Matthew


"Sharpie" <chuck at sharpsteen.net> wrote in message
news:1270102758449-1747733.post at n4.nabble.com...


Janet Choate-2 wrote:

Thanx for clarification on stating my problem, Charlie.

I am attempting to merge to files, i.e.:
hi39 = merge(comb[,c("hillID","geo")], hi.h39, by=c("hillID"))

if this is relevant or helps to explain:
the file 'comb' is 3 columns and 1127 rows
the file 'hi.h39' is 5 columns and 19758564 rows

i started a new clean R session in which i was able to read those 2 files
in, but get the following error when i try to merge them:

R(2175) malloc: *** mmap(size=79036416) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
R(2175) malloc: *** mmap(size=79036416) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
R(2175) malloc: *** mmap(size=158068736) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
R(2175) malloc: *** mmap(size=158068736) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
R(2175) malloc: *** mmap(size=158068736) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Error: cannot allocate vector of size 150.7 Mb

so the final error is "Cannot allocate vector of size 150.7 Mb", as
suggested when R runs out of memory.

i am running R version 2.9.2, on mac os X 10.5 - leopard.

any suggestion on how to increase R's memory on a mac?
thanx for any much needed help!
Janet

Ah, so it is indeed a shortage of memory problem.  With R 2.9.2, you are
likely running a 32 bit version of R which will be limited to accessing at
most 4 GB of RAM. You may want to try the newest version of R, 2.10.1, as
it includes a 64 bit version that will allow you to access significantly
more memory- provided you have the RAM installed on your system.

I'm not too hot on memory usage calculation, but someone else on this list
may be able to give you a ballpark estimate of how much RAM this merge
would
require.  If it turns out to be a ridiculous amount, you will need to
consider breaking the merge up into chunks or finding an out-of-core (i.e.
not dependent on RAM for storage) merge tool.

Hope this helps!

-Charlie

-----
Charlie Sharpsteen
Undergraduate-- Environmental Resources Engineering
Humboldt State University
-- 
View this message in context:
http://n4.nabble.com/memory-error-tp1747357p1747733.html
Sent from the R help mailing list archive at Nabble.com.