Applying user function over a large matrix
Hi Jim, Thanks for your prompt response, I am using a fairly powerful Mac with Leopard OS and 17GB RAM and 2x3 GhZ intel zeon processor so I do not think the system is paging. I also using the Rmpi and snow utilities to parallelize it but even then it takes 3.5-4 hours to just complete one chunk of matrices. You mentioned about storing the data and applying on 1 column at a time. Any hint on how I should I go about doing that? I cam across the filehash package but am not sure how to use apply over an environment variable. So any help in this direction will be most welcome. thanks ---- Original message ----
Date: Tue, 29 Apr 2008 16:05:41 -0400 From: "jim holtman" <jholtman at gmail.com> Subject: Re: [R] Applying user function over a large matrix To: "Sudipta Sarkar" <ssarkar at lanworth.com> What size machine do you have. A single copy of your object will require 1.5GB of memory. How slow is slow? Is the operating
system
paging because it does not have enough physical memory? can
you store
the data and only operate on 1 column at a time -- this
reduces the
size of the object to 72MB. On Tue, Apr 29, 2008 at 3:16 PM, Sudipta Sarkar
<ssarkar at lanworth.com> wrote:
Respected R experts, I am trying to apply a user function that basically calls and applies the R loess function from stat package over each time series. I have a large matrix of size 21 X 9000000 and I need to apply the loess for each column and hence I have implemented this separate user function that applies loess over each column and I am calling this function foo as follows: xc<-apply(t,2,foo) where t is my 21 X 9000000 matrix and loess. This is turning out to be a very slow process and I need to repeat this step for 25-30 such large matrix chunks. Is there any trick I can use to make this work faster? Any help will be deeply appreciated. Regards Sudipta Sarkar PhD Senior Analyst/Scientist Lanworth Inc. (Formerly Forest One Inc.) 300 Park Blvd., Ste 425 Itasca, IL Ph: 630-250-0468
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained,
reproducible code.
-- Jim Holtman Cincinnati, OH +1 513 646 9390 What is the problem you are trying to solve?
Sudipta Sarkar PhD Senior Analyst/Scientist Lanworth Inc. (Formerly Forest One Inc.) 300 Park Blvd., Ste 425 Itasca, IL Ph: 630-250-0468