Skip to content
Prev 16512 / 20628 Next
Original Raw

comparing 2 models

Phillip Alday 
In addition to the previous suggestions (CV/LOO and AIC comparison),
there is also Vuong's closeness test for non nested models. I haven't
used it myself, but a quick search suggests that a fair amount of
information about its use and misuse (in e.g. checking for
zero-inflation in Poisson models) is available.

One small note here: PzAlpha and HRV are both continuous predictors, so
they both contribute the same number of degrees of freedom to a model,
so you can compare (log) likelihoods directly instead of using AIC or
BIC because the number of parameters will be the same. (You still can't
do the LRT because amongst other things the chisq df will be 0.) This
also suggests the parameter counting concern  is largely moot, as the
parameters and data are the same. (I'll let someone else more qualified
address the broader issue of parameter counting for AIC/BIC in mixed
models.)

There are also a number of parameter selection methods that work within
one model. In frequentist land, there's elastic net and LASSO, which are
implemented for mixed models in packages such as lmmen and glmmLasso.
There's two ways to use those methods here:

(1) cross validation on the penalty for number of parameters so that the
model selects the 'ideal' number of predictors
(2) setting a particularly penalty to achieve a desired number of
parameters and letting the model choose which ones those are

Both should be typically well supported in good elastic net and LASSO
implementations.

In Bayesian terms, you can achieve the same thing by appropriate choice
of priors. brms has LASSO and horseshoe priors and uses early the same
syntax as lme4:

sumModelInteraction4 <- brm(RT ~ 1 + (PzAlpha+ HRV+lnX+Drive)^3 +  (1 |
Pnum) + (1 | Trial), data = INFAST_Behavioralnew,
prior=set_prior('horseshoe',class='b'))

(The author of brms, Paul B?rkner, also contributes to this list and
might be able to say more about ways of addressing this issue.)

Finally, I see that you're using untransformed RTs in a linear/gaussian
model. There's a lot of debate on whether you should transform or use a
non linear/gaussian model (e.g. log transform, use the gamma transform
with identity or inverse link, etc.). I would check to see how well the
current model actually describes the data and see if non-gaussian models
or a suitable transformation improves fit before doing variable selection.

Best,
Phillip
On 04/25/2018 01:47 PM, Jonathan Judge wrote:

Thread (6 messages)

P Greenwood comparing 2 models Apr 24 Voeten, C.C. comparing 2 models Apr 25 Emmanuel Curis comparing 2 models Apr 25 Jonathan Judge comparing 2 models Apr 25 Phillip Alday comparing 2 models Apr 25 Ben Bolker comparing 2 models Apr 25