bam (mgcv) not using the specified number of cores

Hi Andrew,

In some of the shots you sent then top was reporting several cores
working. I think the problem here may be to do with the way bam is
parallelized - at present only part of the computation is in parallel -
the model matrix QR decomposition part. The smoothing parameter
selection is still single cored (although we are working on that), so if
you watch top, you'll usually see multi-core and single core phases
alternating with each other. The strategy works best in n>>p situations
with few smoothing parameters.

For the case where you used 31 cores, there was a bug in earlier mgcv
versions in which it was assumed that when the model matrix is split
into chunks for processing, each chunk would have more rows than
columns. If you upgrade to the current mgcv version then this is fixed.
However using 31 cores is liable to actually be less efficient than
using fewer cores with the n to p (number of data to number of
coefficients) ratio that you seem to have. This is because the work
being done by each core is rather little, so that the overhead of
stitching the cores' work back together becomes too high. Using
'use.chol=TRUE' would reduce the overheads here (although it uses a
slightly less stable algorithm than the default).

best,
Simon


On 22/08/14 06:13, Andrew Crane-Droesch wrote:

Hi Simon,

(resending with all images as imgur so as to not bounce from list)

Thanks for the reply.  I've tried to reproduce the error, but I don't
know how to show output from `top` any other way than with screenshots,
so please excuse that.

Here are screenshots of what happens when I run with two
http://imgur.com/i26GKPo

and three
http://imgur.com/8SL7scy

cores.  In the former, it seems to be working on one core, and in the
latter, it appears to be working on three.  When reproducing the error,
I'm getting behavior that isn't entirely consistent -- sometimes it
"behaves" and operates on the asked-for number of cores, and other times
not.

I'm also attaching a screenshot
http://imgur.com/bJfuS6R
showing terminal output from a remote cluster when I run my full model
(N=67K) rather than a subset (N=7K) -- I get that error "Error in
qr.qty(qrx, f) : right-hand side should have 60650 not 118451 rows ..."
I suppose this is a memory overload problem?  Any suggestions on how to
get bam to not call for more memory than the node has available would be
welcome, though I suspect that is a supercomputing problem rather than a
mgcv problem.  I don't know much about memory management, except that R
doesn't do it explicitly.

Thanks,
Andrew

sessionInfo() for local machine:
1> sessionInfo()
R version 3.0.2 (2013-09-25)
Platform: x86_64-pc-linux-gnu (64-bit)

locale:
  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
  [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] parallel  stats     graphics  grDevices utils datasets  methods
[8] base

other attached packages:
[1] mgcv_1.7-26  nlme_3.1-111

loaded via a namespace (and not attached):
[1] grid_3.0.2      lattice_0.20-23 Matrix_1.1-4
1>
On 08/21/2014 04:29 PM, Simon Wood wrote:

Hi Andrew,

Could you provide a bit more information, please. In particular the
results of sessionInfo() and the code that caused this weird behaviour
(+ an indication of dataset size).

best,
Simon

On 21/08/14 12:53, Andrew Crane-Droesch wrote:

I am getting strange behavior when trying to fit models to large
datasets using bam.  I am working on a 4-core machine, but I think that
there may be 2 physical cores that the computer uses as 4 cores in some
sense that I don't understand.

When I run the bam using makeCluster(3), the model runs on one core.
But
when I run it on makeCluster(2), top shoes me that three of my cores
are
taken up to full capacity, and my computer slows down or crashes.

How can I get it to truly run on 2 cores?

I'm on a thinkpad X230, running ubuntu.

Thanks,
Andrew