On Mon, Mar 27, 2017 at 4:36 AM, Poe, John <jdpo223 at g.uky.edu> wrote:
You might try a Vuong test. It's a likelihood ratio test that allows for
nonnested models.
On Mar 26, 2017 8:30 PM, "Paul Buerkner" <paul.buerkner at gmail.com> wrote:
Hi Craig,
in short, significance does not tell you anything about model fit. You may
find models to have the best fit without any particular predictor being
significant for this model. Similarily average "effect sizes" are not a
good indicator of model fit.
Information criteria are, in my opinion, the right way to go. For an
improved version of the AIC, I recommend going Bayesian and computing the
so called LOO (leave-one-out cross validation) or the WAIC (widely
applicable information criterion) as implemented in the R package loo. For
the bayesian GLMM model fitting (and convenient LOO computation), you could
use the R packages brms or rstanarm.
Best,
Paul
2017-03-26 22:15 GMT+02:00 Craig DeMars <cdemars at ualberta.ca>:
Hello,
This is a bit of a follow-up to a question last week on selecting among
GLMM models. Is there a recommended strategy for comparing non-nested,
random slope models? I have seen a similar question posted here
http://stats.stackexchange.com/questions/116935/comparing-non-nested-
models-with-aic but it doesn't seem to answer the problem - and maybe
there
is no "answer". Zuur et al. (2010) discuss model selection but only in a
nested framework. Bolker et al. (2009) suggest AIC can be used in GLMMs
caution against boundary issues and don't specifically mention any issues
with comparing different random effects structures (as Zuur does).
The context of my question comes from an analysis where we have 5 *a
priori*
hypotheses describing different climate effects on juvenile recruitment
an ungulate species. The data set has 21 populations (or herds) with
repeated annual measurements of recruitment and the climate variables
measured at the herd scale. To generate SE's that reflect herd as the
sampling unit, explanatory variables are specified as random slopes
herd (as recommended by Schielzeth & Forstmeier 2009; Year is also
specified as a random intercept). Because there are only 21 herds,
are fairly simple with only 2-3 explanatory variables (3 may by pushing
it...????). I can't post the data but it isn't really relevant to the
question (I think).
Initially, we looked at AIC to compare models. At the bottom of this
email, I have pasted the output from two models, each representing
hypotheses, to illustrate "the problem". The first model yields an AIC
value of 2210.7. The second model yields an AIC of 2479.5. Using AIC,
1 would be the "best" model. However, examining the parameter estimates
within each model makes me think twice about declaring Model 1 (or the
hypothesis it represents) as the most parsimonious explanation for the
data. In Model 1, two of the thee fixed effects estimates have small
sizes and all estimates are "non-significant" (if one considers
p-values....). In Model 2, two of the three fixed effect estimates have
larger effect sizes are would be considered "significant. Is this an
example of the difficulty in using AIC to compare non-nested mixed
models.....or am I missing something in my interpretation? I haven't come
across this type of result when model selecting among GLMs.
Any suggestions on how best to compare competing hypotheses represented
non-nested GLMMs? Should one just compare relative effect sizes of
parameter estimates among models?
Any help would be appreciated.
Thanks,
Craig
*Model 1:*
Generalized linear mixed model fit by maximum likelihood (Laplace
Approximation) ['glmerMod']
Family: binomial ( logit )
Formula: (Calves/Cows) ~ spr.indvi.ab + green.rate.ab + trend + (1 |
+ (spr.indvi.ab + green.rate.ab + trend | Herd)
Data: bou.dat
Weights: Cows
*AIC * BIC logLik deviance df.resid
*2210.7* 2265.0 -1090.3 2180.7 262
Scaled residuals:
Min 1Q Median 3Q Max
-3.8700 -1.0800 -0.1057 1.0405 6.8353
Random effects:
Groups Name Variance Std.Dev. Corr
Year (Intercept) 0.10517 0.3243
Herd (Intercept) 0.29832 0.5462
spr.indvi.ab 0.04331 0.2081 0.38
green.rate.ab 0.03741 0.1934 0.68 0.62
trend 0.62661 0.7916 -0.59 0.20 -0.46
Number of obs: 277, groups: Year, 22; Herd, 21
Fixed effects:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -1.62160 0.15798 -10.265 <2e-16 ***
spr.indvi.ab 0.04019 0.09793 0.410 0.682
green.rate.ab 0.04704 0.05555 0.847 0.397
trend -0.29676 0.23092 -1.285 0.199
---
Signif. codes: 0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1
Correlation of Fixed Effects:
(Intr) spr.n. grn.r.
spr.indvi.b -0.113
green.rat.b 0.347 0.438
trend -0.606 0.349 -0.200
*Model 2:*
Generalized linear mixed model fit by maximum likelihood (Laplace
Approximation) ['glmerMod']
Family: binomial ( logit )
Formula: (Calves/Cows) ~ win.bb + tot.sn.ybb + trend + (1 | Year) + (
win.bb
+ tot.sn.ybb | Herd)
Data: bou.dat
Weights: Cows
* AIC* BIC logLik deviance df.resid
*2479.5 * 2519.4 -1228.8 2457.5 266
Scaled residuals:
Min 1Q Median 3Q Max
-4.5720 -1.1801 -0.1364 1.3704 8.3271
Random effects:
Groups Name Variance Std.Dev. Corr
Year (Intercept) 0.10694 0.3270
Herd (Intercept) 0.13496 0.3674
win.bb 0.05351 0.2313 -0.13
tot.sn.ybb 0.06200 0.2490 0.23 0.34
Number of obs: 277, groups: Year, 22; Herd, 21
Fixed effects:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -1.851656 0.127702 -14.500 < 2e-16 ***
win.bb -0.364019 0.101386 -3.590 0.00033 ***
tot.sn.ybb 0.275271 0.118111 2.331 0.01977 *
trend -0.007568 0.115706 -0.065 0.94785
---
Signif. codes: 0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1
Correlation of Fixed Effects:
(Intr) win.bb tt.sn.
win.bb 0.048
tot.sn.ybb 0.269 0.083
trend -0.242 -0.269 -0.131
--
Craig DeMars, Ph.D.
Postdoctoral Fellow
Department of Biological Sciences
University of Alberta
Phone: 780-221-3971 <(780)%20221-3971> <(780)%20221-3971>
[[alternative HTML version deleted]]