[R-meta] rma.glmm model selection

I would say that if the log likelihoods are not meaningfully comparable, then neither are the AIC or BIC values.

Speaking of these different models, in the 'devel' version of metafor, rma.glmm() now even allows for even more flexibility:

https://wviechtb.github.io/metafor/reference/rma.glmm.html

See especially the 'Note' section. One can now control the coding of the group variable and for "UM.RS" one can now allow for correlation between the random study effects and the random group effect (just as a side-note, to make our lives even more complicated having to choose among an even wider collection of potential modeling options).

Best,
Wolfgang

-----Original Message-----
From: Philippe Tadger [mailto:philippetadger at gmail.com]
Sent: Sunday, 12 September, 2021 22:54
To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] rma.glmm model selection

Hello Wolfgang, colleagues

Thanks for the answer!

I've been reading the article? (Jackson et al., 2017) were you explain such models
and provide codes for SAS/Stata/R.
I did run the SAS code with the sham data, and collect all goodness of fit for
models 2-6

         AIC   BIC   -2 x Likelihood   Model Name                          effects
Variance-covariance Matrix Other name
Model 4  80.74 80.25 62.74             modified Simmonds and Higgins model Fixed
UM.FS
Model 2  81.53 81.05 63.53             Simmonds and Higgins model          Fixed
Model 5  91.19 90.97 83.19             modified Simmonds and Higgins model Random
CV                         UM.RS
Model 6  93.18 92.91 83.18             Van Houwelingen bivariate model     Random
UN                         CM.AL
Model 3  93.76 93.54 85.76             Simmonds and Higgins model          Random
CV

I understand that the likelihood measure can not be used if the distributions or
models are not nested (models 6 & 7 or CM.AL and CM.EL), which is not the case as
you point it out.
Do you consider the AIC values also not "meaningful" to choose between models?

Sorry if there's is any typo in the self made table (trying to unify all the names
and features of each model)

Thanks in advance for you valuable time

On 12/09/2021 15:00, Viechtbauer, Wolfgang (SP) wrote:
Hi Philippe,

Good question. I doubt that a direct comparison of the likelihoods of these models
(or information criteria) is meaningful though. For example, UM.FS and UM.RS
differ both in terms of their fixed and random effects (since the whole point is
to use either fixed or random study effects). CM.AL and CM.EL differ even more
fundamentally, as they use other distributions.

The sentence that you quote is true, but the devil is, as always, in the details.
Here, I was thinking more in terms of: We have some specific model where we can
swap in or out certain random effects. Once you start swapping in and out fixed
and random effects at the same time and even switching distributions, then things
get a lot more tricky.

So, I really don't have any good suggestions at the moment.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of Philippe Tadger
Sent: Saturday, 11 September, 2021 12:42
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] rma.glmm model selection

Dear Meta community

I would like to ask for guidance on? how to do model selection in
metafor::rma.glmm. I'm thinking specifically in the comparison between:
UM.FS, UM.RS, CM.AL, CM.EL methods. Is it possible to conduct a goodness
of fit or alternative selection methods between the 4 models?

I saw the model selection here
https://www.metafor-
project.org/doku.php/tips:model_selection_with_glmulti_and_mumin
<https://www.metafor-
project.org/doku.php/tips:model_selection_with_glmulti_and_mumin>
for predictors that mention that is possible to do a similar approach
with rma.glmm. Also a specific phrase catch my attention: "one can also
consider model selection with respect to the random effects structure."

Thanks in advance for your help and time

--

Kind regards/Saludos cordiales
*Philippe Tadger*
ORCID <https://orcid.org/0000-0002-1453-4105>, Reseach Gate
<https://www.researchgate.net/profile/Philippe-Tadger>
Phone/WhatsApp: +32498774742