Making lme4 faster for specific case of sparse x

z <- data.frame(f=factor(sample(8,size=10000,replace=TRUE)))
m <- model.matrix(~f,data=z)
m2 <- Matrix::sparse.model.matrix(~f,data=z)
print(object.size(m),units="Mb")

print(object.size(m2),units="Mb")

On Tue, Aug 9, 2016 at 8:36 AM Patrick Miller <pmille13 at nd.edu> wrote:

Thanks for that clarification.  In my situation, the effect of each
predictor in X was allowed to vary by a single grouping variable. The lmer
formula is something like the following:

y ~ 1 + X1 + X2 + X3 + ... + ( 1 + X1 + X2 + X3 + ... | id)

Okay - that's not the same as X == Z but we'll let that slide.

It is extremely unlikely that you will be able to fit such a model and get
a meaningful result.  Suppose that you have p columns in the fixed-effects
model matrix, X ,and k levels of the id factor.  The covariance matrix of
the random effects will be p by p with p*(p + 1) / 2 distinct elements to
estimate.  It is difficult to estimate large covariance matrices with any
accuracy.  You would need k to be very, very large to have any hope of
doing so.

To make it worthwhile using a sparse representation of X you would need p
to be large - in the hundreds or thousands - which would leave you trying
to estimate tens of thousands of covariance parameters.

It is just not on.

If you feel you must fit this model because of the "keep it maximal" advice
of Barr et al. (2013), remember that they reached that conclusion on the
basis of a simulation of a model with one covariate.  That is, they were
comparing fitting 1 by 1 covariance matrix with fitting a 2 by 2 covariance
matrix.  To conclude on the basis of such a small simulation that everyone
must always use the maximal model, even when it would involve tens or
hundreds of covariance parameters, is quite a leap.

On Mon, Aug 8, 2016 at 6:08 PM, Douglas Bates <bates at stat.wisc.edu> wrote:

If X == Z don't you have problems with estimability?  It seems that mle

would always correspond to all random effects being zero.

Perhaps I misunderstand the situation.  Could you provide a bit more
detail on how it comes about that X == Z?

On Mon, Aug 8, 2016 at 5:01 PM Patrick Miller <pmille13 at nd.edu> wrote:

Hello,

For my dissertation, I'm working on extending boosted decision trees to
clustered data.

In one of the approaches I'm considering, I use *lmer* to estimate random
effects within each gradient descent iteration in boosting. As you might
expect, this is computationally intensive. However, my intuition is that
this step could be made faster because my use case is very specific.
Namely, in each iteration, *X = Z*, and *X* is a sparse matrix of 0s and
1s
(with an intercept).

I was wondering if anyone had suggestions or (theoretical) guidance on
this
problem. For instance, is it possible that this special case permits
faster
optimization via specific derivatives? I'm not expecting this to be
implemented in lmer or anything, and I'm happy to work out a basic
implementation myself for this case.

I've read the vignette on speeding up the performance of lmer, and
setting calc.derivs
= FALSE resulted in about a 15% performance improvement for free, which
was
great. I was just wondering if it was possible to go further.

Thanks in advance,

- Patrick

--
Patrick Miller
Ph.D. Candidate, Quantitative Psychology
University of Notre Dame

        [[alternative HTML version deleted]]

--
Patrick Miller
Ph.D. Candidate, Quantitative Psychology
University of Notre Dame

	[[alternative HTML version deleted]]