[R-meta] Clarification on an answer

Dear Meta-analysis Experts,

I'm hoping to use the forwarded answer below in my model but need two
clarifications about the answer.

First, bullet point # 3, says that we fit the below model and then "compare
this model against the model that assumes homogeneous variance components
across subgroups."

rma.mv(yi, V = Vsub,
              mods = ~ tc_fac - 1,
              random = list(~ tc_fac | id, ~ tc_fac | id:esid), struct =
c("ID","ID"),
              data = cc_d, method = "REML")

But the above model already assumes homogeneous variance components across
subgroups, so I wonder if the intention was actually to set struct =
c("DIAG","DIAG")?

Second, the answer mentions "Based on the reported counts from your output,
it looks like there might be four publications (reporting a total of nine
studies) that have both levels."

I was wondering how to obtain the number of publications and their nested
number of studies that have both levels (so I can apply this to my model)?

Thank you for your time,
Yuhang
---------- Forwarded message ---------
From: James Pustejovsky via R-sig-meta-analysis
<r-sig-meta-analysis at r-project.org>
Date: Thu, Feb 9, 2023 at 8:14 PM
Subject: Re: [R-meta] Different estimates for multilevel meta-analysis
when using rma.mv with mods or subset
To: R Special Interest Group for Meta-Analysis
<r-sig-meta-analysis at r-project.org>

Hi Janis,

There are potentially two things going on here. First, when estimating
separate models within each subgroup, you allow the heterogeneity (variance
components) to differ by subgroup. This in turn leads to a different
weighting of the individual effect size estimates in the calculation of the
overall average effect sizes than the weighting used in the moderator
analysis. That difference in weighting might be enough to account for the
swings in the average ES for each category. To determine which model is
more appropriate, you could use fit statistics or a likelihood ratio test.
To get such information, you'll need to find a way to express the subgroup
analyses in terms of a single model. You can do this as follows:

1. Make a factor variable from targetcommitment so that you don't have to
repeatedly calculate it:
cc_d$tc_fac <- factor(cc_d$targetcommitment)

2. If you created the V matrix using metafor::vcalc(), set the subgroup
argument to the moderator variable:
Vsub <- vcalc(..., subgroup = cc_d$tc_fac, ...)

3. Fit a model using this new V matrix, specifying a random effects
structure that allows independent effects in each subgroup:
rma.mv(yi, V = Vsub,
              mods = ~ tc_fac - 1,
              random = list(~ tc_fac | id, ~ tc_fac | id:esid), struct =
c("ID","ID"),
              data = cc_d, method = "REML")
Note that I've omitted the middle level of random effects because there
doesn't seem to be any variance there after accounting for the id and esid
levels. The results of (3) should be identical (or nearly so) to the
results from the subgroup analysis. But now they're embedded within one
model, so you can get fit statistics or conduct a likelihood ratio test to
compare this model against the model that assumes homogeneous variance
components across subgroups.

The second thing that might be going on is that your data seems to include
a few publications that have both levels of the targetcommitment variable.
Based on the reported counts from your output, it looks like there might be
four publications (reporting a total of nine studies) that have both
levels. Because of this structure, the first model you use for moderator
analysis will calculated average effect size estimates for each level of
targetcommitment based in part on the effect size estimates *for the other
category* from the four publications / 9 studies that include both levels.
This has the effect of moving the averages towards each other---that is
-.175 and .241 get pulled inward to  -.155 and .188.

If you're particularly interested in figuring out the difference between
levels of targetcommitment, the question is then whether this "pulling in"
is a reasonable thing to do. You could consider this on a conceptual level:
is it reasonable to put special emphasis on the results from these four
publications in estimating the difference between levels of
targetcommitment? If there's not a clear conceptual rationale, then you
might consider fitting a slightly different model, which includes the
study-mean variable and the study-mean-centered variable as separate terms.
For targetcommitment, this would mean calculating the average of the dummy
variable (targetcommitment == 2) for each study (this gives you the first
predictor, call it tc_study) and then subtracting the average from the
original dummy variable (this gives you the second predictor, call it
tc_within). The model should then use the moderators:
~ 1 + tc_study + tc_within
Tanner-Smith and Tipton (2014; https://doi.org/10.1002/jrsm.1091) recommend
this as a generic strategy for meta-regression of dependent effect sizes.

James

On Wed, Feb 8, 2023 at 2:40 PM Janis Zickfeld via R-sig-meta-analysis <

[R-meta] Clarification on an answer

Thread (5 messages)