[R-meta] Questions about model averaging with complex multilevel meta-analytic model

-----Original Message-----
From: Margaret Slein [mailto:maslein at zoology.ubc.ca]
Sent: Saturday, 24 September, 2022 23:43
To: Viechtbauer, Wolfgang (NP)
Cc: r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] Questions about model averaging with complex multilevel
meta-analytic model

Hi Wolfgang,

Thank you, this has been most helpful. Another lingering question I had was in
regards to quantifying publication bias. I included the square root of the
sampling variance to assess this and in the best model, it has a very large
statistically significant effect. Is this approach, of including the square root
of the sampling variance in the global model with all other moderators,
appropriate and if so, how would I account for this publication bias in the rest
of my results?

I cannot conduct a trim and fill analysis with the function in metafor because of
the model's complex structure, but if there are other ways of trying to address
potential strong bias, I am interested in doing so.

Cheers,
Maggie

<*)))><> ?<*)))><> ? <*)))><> ?<*)))><>

Maggie Slein (she/her/hers)
PhD Student, O?Connor Lab
Department of Zoology
Unceded x?m??k??y??m (Musqueam)?territory
University of British Columbia

On Sep 24, 2022, at 11:41 AM, Viechtbauer, Wolfgang (NP)
<wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

Dear Maggie,

Some notes on this:

1) dredge() (and I assume the other functions from MuMIn as well) examines the
'nobs' attribute that logLik() returns to determine the sample size of the
various models. However, when using REML estimation, nobs = k - p, where p is the
number of model coefficients (for some technical reasons that are not important
right now). However, this leads dredge() to think that the sample size differs
across models where p differs.

In general, you should use method="ML" when comparing models that differ in terms
of their fixed effects.[1] In that case, nobs = k and this issue won't arise.

2) I would recommend to do all transformations (like mean centering or things
like sqrt(vi)) outside of the model call (so, beforehand).

3) You have *a lot* of fixed effects and even interactions. This will lead to
many models that dredge() needs to fit. This could take a looooong time. dredge()
has a 'cluster' argument for doing parallel processing, which you might consider
using if you have powerful enough hardware. Still, even then this could be a
rather daunting task.

4) I can confirm that dredge() works just fine with rma.mv() models. An example
with a similar model as you are fitting can be found here:

https://gist.github.com/wviechtb/891483eea79da21d057e60fd1e28856b

Best,
Wolfgang

[1] Actually, based on some research we did, REML might actually work:

Cinar, O., Umbanhowar, J., Hoeksema, J. D., & Viechtbauer, W. (2021). Using
information-theoretic approaches for model selection in meta-analysis. Research
Synthesis Methods, 12(4), 537?556. https://doi.org/10.1002/jrsm.1489

But we didn't examine complex models like you are using and I would still be very
cautious with using REML when doing so.


-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of Margaret Slein
Sent: Saturday, 24 September, 2022 18:58
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] Questions about model averaging with complex multilevel meta-
analytic model

Hi Wolfgang,

I am PhD student at UBC in Vancouver, Canada, currently working on a meta-
analysis. I have been trying to do model selection and model averaging using the
model.sel() and model.avg() functions from the MuMIn package with an rma.mv model
object, while also following your metafor help page using MuMIn and glmulti. I
have been unable to get any of my models to perform model selection or model
averaging because each model is being fit to different data. I have ensured there
are no missing values or NAs across the data frame.

Is it possible to do model averaging with the rma.mv function with 3 level random
effects, a phylogenetic correlation, and several moderators? There are currently
no examples I could find using model selection or averaging with this model
structure and I have had no luck on stack overflow.

Here is the full model I am trying to run:

full_mod<-rma.mv(yi=yi, V=V,
????????????????mods = ~ I(flux_range-mean(flux_range))*I(mean_temp_constant-
mean(mean_temp_constant))
????????????????+I(flux_range-mean(flux_range))*experiment_type
????????????????+I(mean_temp_constant-mean(mean_temp_constant))*experiment_type
????????????????+I(secondary_temp - mean(secondary_temp))*experiment_type
????????????????+duration_standard*experiment_type
????????????????+experiment_type*experiment_type
????????????????+exp_age*experiment_type
????????????????+size*experiment_type
????????????????+ecosystem*experiment_type
????????????????+trait_directionality*experiment_type
????????????????+sqrt(vi)*experiment_type, dfs="contain",
????????????????random = list( ~1 | phylo, ~1 | study_id, ~1 | response_id),
????????????????R = list(phylo=cor), test="t",
????????????????method="REML", data=dat_ES_final_2)

I have also tried using dredge() in MuMIn in addition to trying my own subset of
models with no luck but it takes days for dredge() to iterate and still yields an
error when trying to perform model averaging that none of the models are being
fit to the same data. Any suggestions or assistance you can provide on this would
be greatly appreciated.

Cheers,
Maggie

<*)))><> ?<*)))><> ??<*)))><> ?<*)))><>

Maggie Slein (she/her/hers)
PhD Student, O?Connor Lab
Department of Zoology
Unceded x?m??k??y??m (Musqueam) territory
University of British Columbia