AIC Comparison for MLM with Different Distributions
Hi Ben, Thank you again for your help before! We will be using other model assessments (including R^2) as well, but I wanted to check if the AICs can be compared for (hurdled) gamma models with (zero-inflated) beta models when fit in glmmTMB? Likewise, can the AICs be compared for (zero-inflated / hurdle) negative binomial and (zero-inflated) beta models? For the beta models, we have transformed raw counts/durations into percentages. Many thanks, K
On Sat, Mar 7, 2020 at 9:17 AM Ben Bolker <bbolker at gmail.com> wrote:
Only when the response variable is transformed.
[please keep r-sig-mixed-models in the cc: list if possible when
following up on questions ... ]
cheers
Ben Bolker
On 2020-03-07 12:16 p.m., Kate R wrote:
Hi Ben,
Thank you for your reply! Would I also apply the Jacobian correction to
the Gamma with log-link, or is it only used when the response variable
is transformed?
Many thanks again!
Katie
------------------------------
Message: 2
Date: Wed, 4 Mar 2020 09:55:55 -0500
From: Ben Bolker <bbolker at gmail.com <mailto:bbolker at gmail.com>>
To: r-sig-mixed-models at r-project.org
<mailto:r-sig-mixed-models at r-project.org>
Subject: Re: [R-sig-ME] AIC Comparison for MLM with Different
Distributions
Message-ID: <eae1791a-e56a-7c32-b872-f6fa93157857 at gmail.com
<mailto:eae1791a-e56a-7c32-b872-f6fa93157857 at gmail.com>>
Content-Type: text/plain; charset="utf-8"
I agree with Thierry's big-picture comment that you should
generally
use broader/qualitative criteria to decide on a model rather than
testing all possibilities. The only exception I can think of is if
you
are *only* interested in predictive accuracy (not in inference
[confidence intervals/p-values etc.]), and you make sure to use
cross-validation or a testing set to evaluate out-of-sample
predictive
error (although AIC *should* generally give a reasonable
approximation
to relative out-of-sample error).
Beyond that, if you still want to compute AIC (e.g. your
supervisor or
a reviewer is forcing to do it, and you don't think you're in a
position
to push back effectively):
* as long as you include the Jacobian correction when you transform
the predictor variable (i.e. #2), these log-likelihoods (and AICs)
should in principle be comparable (FWIW the robustness of the
derivation
of AIC is much weaker for non-nested models; Brian Ripley [of MASS
fame]
holds a minority opinion that one should *not* use AICs to compare
non-nested models)
* computing log-likelihoods/AICs by hand is in principle a good
idea,
but is often difficulty for multi-level models, as various integrals
or
approximations of integrals are involved. The lmer and glmer
likelihoods (1-4) are definitely comparable. To compare across
platforms
I often try to think of a simplified model that *can* be fitted in
both
platforms (e.g. in this case I think a proportional-odds ordinal
regression where the response has only two levels should be
equivalent
to a binomial model with cloglog link ...)
cheers
Ben Bolker
On 2020-03-03 5:29 p.m., Kate R wrote:
> Hi all,
>
> Thank you in advance for your time and consideration! I am a
> non-mathematically-inclined graduate student in communication just
learning
> multilevel modeling.
>
> I am trying to compare the AIC for 5 different models:
>
>
> 1. model.mn5 <- lmer(anxious ~ num.cm <http://num.cm> + num.pmc
+ (1|userid), data = df,
> REML = F)
> 2. model.mn5.log <- lmer(log(anxious) ~ num.cm <http://num.cm>
+ num.pmc + (1|userid),
> data = df, REML = F)
> 3. model.mn5.gamma.log <- glmer(anxious ~ num.cm
<http://num.cm> + num.pmc + (1|userid),
> data = df, family = Gamma(link="log"))
> 4. model.mn5.gamma.id <http://model.mn5.gamma.id> <-
glmer(anxious ~ num.cm <http://num.cm> + num.pmc + (1|userid),
> data = df, family = Gamma(link="identity"))
> 5. model.ord5 <- clmm(anxious ~ num.cm <http://num.cm> +
num.pmc + (1|userid), data =
> df, na.action = na.omit)
>
> (num.cm <http://num.cm> is the group mean and num.pmc is the
group-mean-centered score of
> the predictor)
>
> Despite many posts on various help forums, I understand that it's
possible
> to compare non-nested models with different distributions as long
as all
> terms, including constants, are retained (i.e. see Burnham &
Anderson, Ch
> 6.7 <https://www.springer.com/gp/book/9780387953649>), but that
different R
> packages or model classes might handle constants differently or use
> different algorithms (see point 7
> thus making it difficult to directly compare AIC values. To avoid
> this non-comparability pitfall, it was suggested in one post to
calculate
> your own log-likelihood (though I'm having trouble finding this
post again).
>
> Please could you help with the following:
>
> - What is the best practice for comparing the AICs for these 5
models?
> - What is the R-code for manually calculating the
log-likelihood and/or
> the AIC to retain all terms, including constants?
> - Can you compare ordinal models (clmm) with the continuous
models?
> - Do you recommend any other methods and/or packages for
comparing
> models with different distributions and/or links?
>
> Many thanks in advance for your time and consideration! I greatly
> appreciate any suggestions.
>
> Kind regards,
> K
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-mixed-models at r-project.org
<mailto:R-sig-mixed-models at r-project.org> mailing list