Memory(RAM) issues
Hi Lui, Anupam and my other R-user friends, Thanks for your previous suggestions. As for my issue, it is clearly RAM problem as my code is running perfectly as long as my input data size is small and code has been refined a number of times to increase efficiency [ by using matrix more in the context of manipulations and reducing order of time complexity wherever possible]. Yet it has gotta run 2000 iterations and in each of the iterations it repeatedly modifies few [0.5 Mn * 15] tables After few basic investigation it seems that the problem MAY be resolved if 64 bit version of R is installed in 64 bit OS as it claims to be able use potentially a 8TB RAM ( for my case I think 4 GB RAM may serve the purpose but 2 GB is not enough) if available.I don't have any domain knowledge in IT since I hail from a statistics background. I work in a company as a statistical analyst. The IT folks of my company are ready to deploy any free solution but I need to tell them what software needs to be installed and what all OS configurations would be required. They normally run all of the applications on unix servors. So I need to know if any free 64 bit of R version can be installed in unix servor or if not unix, may be on other servor. So hereby I request my R-user friends to please let me know if any free 64 bit version of R is available that can be installed on any unix servor. If that is not available is there any other FREE solution and if available, how to get that and what all configuration is required. Awaiting your replies, Regards, Abhisek P.S. Please forward this mail to any other R-mailing list if you deem it fit for any of them.
On Sat, Jun 11, 2011 at 4:38 PM, Lui ## <lui.r.project at googlemail.com> wrote:
Hello Abhisek, maybe you wanna try it on just a bigger machine (I guess you are working at a university, so I hope you do have access to them). In case getting computing time or the like is a major issue, you may wanna try Amazon AWS: For a few rupees (about 50-100 per hour) you can "rent" pretty fast computers (20 Ghz, 8BG of RAM). You may want to try out the Windows version (little bit more expensive) which is easily accessible via remote desktop. Installing Revolution (which is free for academic purposes) (64 Bit Version) might give you a good start. Maybe its not a viable option in a long term view (pricing), but it might help to get a clue whether the problem can be solved on a bigger machine and just trying it out without a bigger hastle... Good luck! Lui On Sat, Jun 11, 2011 at 7:47 AM, Abhisek Saha <tading at gmail.com> wrote:
Thanks Anupam for your inputs. I believe there are two ways to circumvent the issue...1> making the code more efficient 1> Increasing the memory in some way.The reasons why I did not paste the code are 1> It is long and using quite a number of functions that ?I created 2> Secondly my intention is not to make the code more efficient if that's possible. Here landing into a memory problem with 2 GB RAM is natural as my analysis entails 1500 simulations from huge multivariate distributions that change after every simulation and tomorrow I may have to do similar analysis with 10 million observations * 20 columns. In view of above I shall be needing more memory sometime later and my IT friends are ready to support me for that(probably with a sandbox) but I am not sure whether I can install probably a servor version of R that can be capable of working with 8GB or so RAM. So it is more of technical help I need and I have no clue regarding the plausibility of the solution mentioned( i.e. a servor version of R that is capable of more memory). Regards, Abhisek On Sat, Jun 11, 2011 at 10:10 AM, Anupam <anupamtg at gmail.com> wrote:
It will be helpful on this forum to use metric measures: 12 Lakh is 1.2 million, thus your data is 1.2 million observations x 15 variables. I do not know the intricacies of R. You may have to wait for someone with that knowledge to respond. Including some relevant portions of error messages and code in your query can also help someone to respond to your message. Anupam. -----Original Message----- From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On Behalf Of Abhisek Saha Sent: Saturday, June 11, 2011 6:25 AM To: r-help at r-project.org Subject: [R] Memory(RAM) issues Dear All, I have been working with R(desktop version) in VISTA. I have the latest version R 2.13.0. I have been working with few data-sets of size 12Lakh * 15 and my code is quite computing intensive ( applying MCMC gibbs sampler on a posterior of 144 variables) that often runs into a memory issue such as memory can not allocate the vector ,full size(shows to have reached something like 1.5 GB) reached or something to this effect. I have a RAM of 2GB. ?I checked with the option like memory.size and it says a 64 bit R if sat on 64 bit windows then max memory capability is 8TB. Now I don't have ?background to understand the definitions and differences between 32 and 64 bit machines and other technical requirements like servor etc but it would be of great help if anyone can let me have a feel of it. Could any of you tell me whether some servor version of R would resolve my issue or not (I am not sure now what kind of servor my company would allow R to be installed at this point ,may be linux type) and if that's the case could any of you guide me about how to go about installing that onto a sevor. Thank you, Abhisek ? ? ? ?[[alternative HTML version deleted]]
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.