bam (mgcv) not using the specified number of cores
Hi Simon, Thanks for the reply. I've tried to reproduce the error, but I don't know how to show output from `top` any other way than to attach screenshots, so please excuse that. I'm attaching screenshots of what happens when I run with two and three cores. In the former, it seems to be working on one core, and in the latter, it appears to be working on three. When reproducing the error, I'm getting odd behavior that isn't entirely consistent -- sometimes it "behaves" and operates on the asked-for number of cores, and other times not. I'm also attaching a screenshot showing terminal output from a remote cluster when I run my full model (N=67K) rather than a subset (N=7K) -- I get that error "Error in qr.qty(qrx, f) : right-hand side should have 60650 not 118451 rows ..." I suppose this is a memory overload problem? Any suggestions on how to get bam to not call for more memory than the node has available would be welcome, though I suspect that is a supercomputing problem rather than a R problem. Thanks, Andrew sessionInfo() for local machine: 1> sessionInfo() R version 3.0.2 (2013-09-25) Platform: x86_64-pc-linux-gnu (64-bit) locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 [7] LC_PAPER=en_US.UTF-8 LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] parallel stats graphics grDevices utils datasets methods [8] base other attached packages: [1] mgcv_1.7-26 nlme_3.1-111 loaded via a namespace (and not attached): [1] grid_3.0.2 lattice_0.20-23 Matrix_1.1-4 1>
On 08/21/2014 04:29 PM, Simon Wood wrote:
Hi Andrew, Could you provide a bit more information, please. In particular the results of sessionInfo() and the code that caused this weird behaviour (+ an indication of dataset size). best, Simon On 21/08/14 12:53, Andrew Crane-Droesch wrote:
I am getting strange behavior when trying to fit models to large datasets using bam. I am working on a 4-core machine, but I think that there may be 2 physical cores that the computer uses as 4 cores in some sense that I don't understand. When I run the bam using makeCluster(3), the model runs on one core. But when I run it on makeCluster(2), top shoes me that three of my cores are taken up to full capacity, and my computer slows down or crashes. How can I get it to truly run on 2 cores? I'm on a thinkpad X230, running ubuntu. Thanks, Andrew
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
-------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2014-08-21 18:33:36.png Type: image/png Size: 576802 bytes Desc: not available URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20140821/41be1e87/attachment-0003.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2014-08-21 18:34:42.png Type: image/png Size: 387719 bytes Desc: not available URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20140821/41be1e87/attachment-0004.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2014-08-21 18:36:08.png Type: image/png Size: 388816 bytes Desc: not available URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20140821/41be1e87/attachment-0005.png>