Advice on comparing non-nested random slope models

You might try a Vuong test. It's a likelihood ratio test that allows for
nonnested models.


On Mar 26, 2017 8:30 PM, "Paul Buerkner" <paul.buerkner at gmail.com> wrote:

Hi Craig,

in short, significance does not tell you anything about model fit. You may
find models to have the best fit without any particular predictor being
significant for this model. Similarily average "effect sizes" are not a
good indicator of model fit.

Information criteria are, in my opinion, the right way to go. For an
improved version of the AIC, I recommend going Bayesian and computing the
so called LOO (leave-one-out cross validation) or the WAIC (widely
applicable information criterion) as implemented in the R package loo. For
the bayesian GLMM model fitting (and convenient LOO computation), you could
use the R packages brms or rstanarm.

Best,
Paul

2017-03-26 22:15 GMT+02:00 Craig DeMars <cdemars at ualberta.ca>:

Hello,

This is a bit of a follow-up to a question last week on selecting among
GLMM models. Is there a recommended strategy for comparing non-nested,
random slope models? I have seen a similar question posted here
http://stats.stackexchange.com/questions/116935/comparing-non-nested-
models-with-aic but it doesn't seem to answer the problem - and maybe
there
is no "answer".  Zuur et al. (2010) discuss model selection but only in a
nested framework. Bolker et al. (2009) suggest AIC can be used in GLMMs

but

caution against boundary issues and don't specifically mention any issues
with comparing different random effects structures (as Zuur does).

The context of my question comes from an analysis where we have 5 *a
priori*
hypotheses describing different climate effects on juvenile recruitment

in

an ungulate species.  The data set has 21 populations (or herds) with
repeated annual measurements of recruitment and the climate variables
measured at the herd scale. To generate SE's that reflect herd as the
sampling unit, explanatory variables are specified as random slopes

within

herd (as recommended by Schielzeth & Forstmeier 2009; Year is also
specified as a random intercept).  Because there are only 21 herds,

models

are fairly simple with only 2-3 explanatory variables (3 may by pushing
it...????). I can't post the data but it isn't really relevant to the
question (I think).

Initially, we looked at AIC to compare models.  At the bottom of this
email, I have pasted the output from two models, each representing

separate

hypotheses, to illustrate "the problem".  The first model yields an AIC
value of 2210.7. The second model yields an AIC of 2479.5. Using AIC,

Model

1 would be the "best" model. However, examining the parameter estimates
within each model makes me think twice about declaring  Model 1 (or the
hypothesis it represents) as the most parsimonious explanation for the
data. In Model 1, two of the thee fixed effects estimates have small

effect

sizes and all estimates are "non-significant" (if one considers
p-values....). In Model 2, two of the three fixed effect estimates have
larger effect sizes are would be considered "significant.  Is this an
example of the difficulty in using AIC to compare non-nested mixed
models.....or am I missing something in my interpretation? I haven't come
across this type of result when model selecting among GLMs.

Any suggestions on how best to compare competing hypotheses represented

by

non-nested GLMMs? Should one just compare relative effect sizes of
parameter estimates among models?
Any help would be appreciated.

Thanks,
Craig

*Model 1:*
Generalized linear mixed model fit by maximum likelihood (Laplace
Approximation) ['glmerMod']
 Family: binomial  ( logit )
Formula: (Calves/Cows) ~ spr.indvi.ab + green.rate.ab + trend + (1 |

Year)

+      (spr.indvi.ab + green.rate.ab + trend | Herd)
   Data: bou.dat
Weights: Cows

     *AIC  *    BIC   logLik deviance df.resid
  *2210.7*   2265.0  -1090.3   2180.7      262

Scaled residuals:
    Min      1Q  Median      3Q     Max
-3.8700 -1.0800 -0.1057  1.0405  6.8353

Random effects:
 Groups Name          Variance Std.Dev. Corr
 Year   (Intercept)   0.10517  0.3243
 Herd   (Intercept)   0.29832  0.5462
        spr.indvi.ab  0.04331  0.2081    0.38
        green.rate.ab 0.03741  0.1934    0.68  0.62
        trend         0.62661  0.7916   -0.59  0.20 -0.46
Number of obs: 277, groups:  Year, 22; Herd, 21

Fixed effects:
              Estimate Std. Error z value Pr(>|z|)
(Intercept)   -1.62160    0.15798 -10.265   <2e-16 ***
spr.indvi.ab   0.04019    0.09793   0.410    0.682
green.rate.ab  0.04704    0.05555   0.847    0.397
trend         -0.29676    0.23092  -1.285    0.199
---
Signif. codes:  0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1

Correlation of Fixed Effects:
            (Intr) spr.n. grn.r.
spr.indvi.b -0.113
green.rat.b  0.347  0.438
trend       -0.606  0.349 -0.200

*Model 2:*
Generalized linear mixed model fit by maximum likelihood (Laplace
Approximation) ['glmerMod']
 Family: binomial  ( logit )
Formula: (Calves/Cows) ~ win.bb + tot.sn.ybb + trend + (1 | Year) + (
win.bb
+      tot.sn.ybb | Herd)
   Data: bou.dat
Weights: Cows

    * AIC*      BIC   logLik deviance df.resid
  *2479.5 *  2519.4  -1228.8   2457.5      266

Scaled residuals:
    Min      1Q  Median      3Q     Max
-4.5720 -1.1801 -0.1364  1.3704  8.3271

Random effects:
 Groups Name        Variance Std.Dev. Corr
 Year   (Intercept) 0.10694  0.3270
 Herd   (Intercept) 0.13496  0.3674
        win.bb      0.05351  0.2313   -0.13
        tot.sn.ybb  0.06200  0.2490    0.23  0.34
Number of obs: 277, groups:  Year, 22; Herd, 21

Fixed effects:
             Estimate Std. Error z value Pr(>|z|)
(Intercept) -1.851656   0.127702 -14.500  < 2e-16 ***
win.bb      -0.364019   0.101386  -3.590  0.00033 ***
tot.sn.ybb   0.275271   0.118111   2.331  0.01977 *
trend       -0.007568   0.115706  -0.065  0.94785
---
Signif. codes:  0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1

Correlation of Fixed Effects:
           (Intr) win.bb tt.sn.
win.bb      0.048
tot.sn.ybb  0.269  0.083
trend      -0.242 -0.269 -0.131
--
Craig DeMars, Ph.D.
Postdoctoral Fellow
Department of Biological Sciences
University of Alberta
Phone: 780-221-3971 <(780)%20221-3971> <(780)%20221-3971>

        [[alternative HTML version deleted]]

        [[alternative HTML version deleted]]