[R-meta] guidance for modeling SMCC type effect size

Dear Stefanou,

This has also come up many times on the list archived at:
https://stat.ethz.ch/pipermail/r-sig-meta-analysis/ and on metafor's
website at: http://www.metafor-project.org/doku.php/analyses:konstantopoulos2011#three-level_model.
I encourage you to explore these two resources to learn more about
these modeling details.

In short, in your case, all the sources of heterogeneity before "id"
related to heterogeneity amoge true effect size aggregates. For
example, true effect sizes aggregated at the study level bouncing
around the overall mean, and true effect sizes aggregated at the
condition level bouncing around their respective study means).
Instead, ~ 1 | id accounts for heterogeneity between individual true
effects (not aggregates of any kind) that are nested in the immediate
level that contains them (in your case that immediate level is
condition).

For example, if in study 1, you have two rows coded T1 for condition,
one representing an effect estimate obtained at interval_id == 0, the
other obtained at interval_id == 1, you can potentially assume that
these two observed estimates are estimating different population
values perhaps by the virtue of being measured at different intervals
(i.e., the only remaining feature that distinguishes between them).
This is as narrow as it gets to assign random effects to a
meta-analytic dataset (i.e., to each row)

If there are enough studies like study 1 in your data (e.g., multiple
T1 rows with different id values due to interval_id), then, the
variance component due to id represents the heterogeneity in a given
condition, just like the variance component due to condition
represents the heterogeneity in a given study.

Best,
Reza






Reza



On Tue, Sep 14, 2021 at 6:33 PM Stefanou Revesz
<stefanourevesz at gmail.com> wrote:

Very interesting! Thanks a lot! Just a very final question, what does
the ~ 1 | id part do or assume?

On Sun, Sep 12, 2021 at 6:13 PM Reza Norouzian <rnorouzian at gmail.com> wrote:

I wonder how conditions could have their own autoregressive structure?

At the study level, you could add *the same random-effect value* to
all the effects (i.e., rows) in each study. But within each study, at
the condition level, you could add *the same random-effect value* to
each set of effects (i.e., rows) that represent the same condition.

Therefore, the two (study-level and condition-level) auto-regressive
structures may lead to different estimates of correlation; one
representing the common correlation among the adjacent interval_id
levels across studies, the other representing the common correlation
among the adjacent interval_id levels across the conditions nested in
studies.

Likewise, the estimates of heterogeneity for SMCC effects at each
level of interval_id across the studies may be different from that
across the conditions nested in studies (assuming an HAR structure).

why that could be necessary?

There is nothing necessary, it's just an(other) assumption about the
structure of true effects, you can fit a model that employs such an
assumption and see if that improves the fit of the model to the data
relative to a model that doesn't utilize that assumption.

Best,
Reza




On Sun, Sep 12, 2021 at 4:43 PM Stefanou Revesz
<stefanourevesz at gmail.com> wrote:

Hi Reza,

Thank you so much! A quick follow-up, when you say "If in each study,
true SMCCs at all intervals across the conditions are assumed to have
their own auto-regressively correlated structure as well", I wonder
how conditions could have their own autoregressive structure and why
that could be necessary?

Thanks again!

On Sun, Sep 12, 2021 at 2:57 PM Reza Norouzian <rnorouzian at gmail.com> wrote:

Dear Stefanou,

The modeling functions like rma.mv(), rma.uni() etc. don't generally
depend on the type of effect size. That aside, just to make sure, did
you compute the change from each pre-test to each follow-up post-test,
or the change from each testing occasion to the following one (e.g.,
pre-test to post-test1, post-test1 to post-test2 ...)?

Regardless of how you define the intervals, if your studies include a
control (C) and multiple treatment conditions (T1, T2,...) that occur
across the studies, you can create a 'condition' variable to
distinguish between the control and the treatments' SMCCs (yi) in each
study over the intervals you have defined:

study condition  yi  interval_id id
1     T1        .1   0           1
1     T1        .3   1           2
1     T2        .7   0           3
1     T2        .2   1           4
1     C         .4   0           5
1     C         .5   1           6
2     T2        .6   0           7
2     C         .9   1           8

In which case, a starting point might be:

rma.mv(yi ~ condition*interval_id, V = Some_V_matrix, random = list(~
interval_id | study, ~ 1 | id), struct = "HAR")

This model assumes that in each study, true SMCCs at all intervals are
auto-regressively correlated with each other regardless of the
conditions they belong to. If in each study, true SMCCs at all
intervals across the conditions are assumed to have their own
auto-regressively correlated structure as well, then, you can
consider:

rma.mv(yi ~ condition*interval_id, V = Some_V_matrix, random = list(~
interval_id | study, ~ interval_id | interaction(study,condition), ~ 1
| id), struct = c("HAR","HAR") )

In both models, when the interaction term is decomposed to its simple
effects for each condition, you get the average SMCC for each
condition (e.g., T1, T2, or C) across the intervals for your studies.

For your second type of effect size (y_i = SMCC_T - SMCC_C, v_i =
v_{i_{SMCC_T}} + v_{i_{SMCC_C}}), pretty much everything I said above
applies. However, in this case, you're basically modeling some sort of
simple effect for each treatment's change vs the control's change
across the intervals for your studies. So, by fitting the above model
but using this type of effect size, you'll be asking how such simple
effects change over the interval you have considered.

I believe these metrics for effect size are nowadays less commonly
used, partly because you can use an SMD metric, which among other
things doesn't require direct knowledge of pre-post correlations for
their computation, model them using multivariate-multilevel models,
and present the results in perhaps more intuitive ways to your
audience.

Does that help?

Reza



Reza


On Sun, Sep 12, 2021 at 6:15 AM Stefanou Revesz
<stefanourevesz at gmail.com> wrote:

Dear Experts,

I need some guidance for modeling two types of effect sizes (for two
different meta-analyses) using the rma.mv() program in metafor.

First, I have computed the standardized mean change (SMCC) between
each pre-test and the follow-up post-tests in each of
multiple-treatment studies.

Second, I have computed the difference in standardized mean changes
(SMCC) between a treatment and a control group at each pre-test to
post-test intervals in each of multiple-treatment studies.

The reason I want to use rma.mv() is that each of my studies could
produce multiple of these types of effect sizes. But I wonder what
would be a starting point for modeling these two types of effect size?

Any help is appreciated,
Stefanou