Speed estimation for lmer?
On Thu, 11 Sep 2008, Douglas Bates wrote:
...I assume that this means that for iteration 0, those are the estimated parameters, and there are 36 of them. It appears to be completing an iteration every 15-20 seconds. The full interaction should estimate 8 variances and sum(1:7) = 28 covariances = 36 parameters total, so that checks out. 36 parameters is 3.6 times as many as fm2 required, so 312.860*3.6 = 1126, or about 18 minutes to fit...far less than it's taken.
36^2 / 10^2 = 13 times longer, or about 68 minutes to fit fm3...we'll see when we get there.
There are two ways in which this will take longer. Each iteration will take longer and it will probably take many more iterations. A 36 parameter optimization problem is a big problem. Optimization usually doesn't scale linearly. Worst case it is exponential in the length of the parameter vector. Most optimization algorithms can do better than that but the complexity is still on the order of p^3 or more.
(269 iterations over 106 minutes and counting). A jump from p^10=312 seconds to p^36 suggests that the model will probably never end up fit; p^3 suggests 243 minutes total (132 minutes from now).
However, I would check both with and without accelerated BLAS if you have a multi-core processor. Sometimes a multithreaded accelerated BLAS can make lmer run slower, not faster. This is because the calls to the BLAS are predominantly for small matrices and the communications overhead for multithreaded versions more than offsets the performance gain from using multiple cores.
Interesting...I hadn't thought of that. The processor is 4-core, and R was linked against the multithreaded BLAS. This may be covered in a faq somewhere, but is there an easy way to swap out the BLAS that R is accessing? Perhaps renaming my libraries is sufficient?
There is some discussion in the "Administration" manual about this. Rather than trying to hot swap the libraries I would probably reconfigure and recompile. Configure using --with-blas=no and --with-lapack=no.
This section in the R-admin manual confused me. Just to be clear, if I do this, then the BLAS library set in BLAS_LIBS *at runtime* will be used, or at compile time? Or are you suggesting I build another version of R which uses non-threaded ATLAS? Or a version of R that uses its own BLAS? --Adam