Hi Nevo,
Responses inline below.
Kind Regards,
James
On Tue, Jul 25, 2023 at 1:37?AM Nevo Sagi <nevosagi8 at gmail.com> wrote:
I don't understand the rationale of using random effects at the
experiment level. Experiments in my meta-analysis are parallel to
observations in a conventional statistical analysis.
I think this analogy doesn't follow. Conventional statistical analysis
does have observation-level error terms (i.e., level-1 error)--it's just
included by default as part of the model. In meta-analytic models, these
errors are not included unless explicitly specified.
What is the meaning of using random effects at the observation level?
Observation-level random effects here are used to capture heterogeneity of
effects across the experiments nested within a study. Considering that
you're interested in looking at moderators that vary across the experiments
reported in the same reference, it seems useful to attend to heterogeneity
at this level as well.
In my understanding, by using random effects at the Reference level, I
already tell the model to look at within-reference variation.
This is not correct. Including reference-level random effects captures
_between-reference_ variation (or heterogeneity) of effects.
In fact, the reason I was thinking to omit the random effect is because
the model was over-sensitive to variation in effect size across moderator
levels within specific references, while I am more interested in the total
variation across the whole moderator spectrum, and therefore I want to
focus more on the between-reference variation.
Does that make sense?
I stand by my original recommendation to consider including
experiment-level heterogeneity here. Omitting the experiment-level
heterogeneity more-or-less corresponds to averaging the effect size
estimates together so that you have one effect per reference, which will
tend to conceal within-reference heterogeneity. In fact, if you are using a
model that does not include moderators / predictors that vary at the
experiment level (within reference), then the correspondence is exact.
Further details here: https://osf.io/preprints/metaarxiv/pw54r/