R scripts slowing down after repeated called to compiled code
I work with images with a lot of processing done in C code. Quite often I allocate memory there up to several gigs in chunks of 10-15 Mb each plus hundreds of protected dims, names etc. I had a similar problem only once when due to some erroneous use of an external library, internally created objects were not freed correctly. Otherwise, after correcting this, I never have seen any slow down on large number of objects created and manipulated. And then, it was so difficult to track the memory leak that I would really suggest to double and triple check all the memory allocations. Your code does not use any temp files? This could be a real pain. Oleg
Dirk Eddelbuettel wrote:
On 25 May 2007 at 19:12, Michael Braun wrote: | So I'm stuck. Can anyone help? It sounds like a memory issue. Your memory may just get fragmented. One tool that may help you find leaks is valgrind -- see the 'R Extensions' manual. I can also recommend the visualisers like kcachegrind (part of KDE). But it may not be a leak. I found that R just doesn't cope well with many large memory allocations and releases -- I often loop over data request that I subset and process. This drives my 'peak' memory use to 1.5 or 1.7gb on 32bit/multicore machine with 4gb, 6gb or 8gb (but 32bit leading to the hard 3gb per process limit) . And I just can't loop over many such task. So I now use the littler frontend to script this, dump the processed chunks as Rdata files and later re-read the pieces. That works reliably. So one think you could try is to dump your data in 'gsl ready' format from R, quit R, leave it out of the equation and then see if what happens if you do the iterations in only GSL and your code. Hth, Dirk
Dr Oleg Sklyar | EBI-EMBL, Cambridge CB10 1SD, UK | +44-1223-494466