[R-meta] simulating three-level meta-analysis

Hi Filippo,

To add to Wolfgang's notes, here is a blog post where I walk through
simulating a meta-analysis of standardized mean differences:
https://www.jepusto.com/simulating-correlated-smds/
It covers a more general case than the 3-level meta-analysis, in which
effect size estimates from a given study may be correlated at the sampling
level, so it might be overkill for what you're doing.

With Wolfgang's notes, your code is a reasonable way to simulate a
three-level meta-analysis, especially if the purpose of your simulation is
exploratory or pedagogical. There are a few ways that I think it could be
improved in terms of its realism. These might be important if the
simulation is for methodological research--as David Hoaglin (2015;
https://doi.org/10.1002/jrsm.1146) argues, for research simulations, it's
important to consider the ways in which the meta-analysis model is a
simplification or idealization of plausible data-generating models. So, in
that spirit, here are a few suggestions:

* Drawing the standard errors from a uniform distribution on (0, 0.3)
implies a certain distribution of study sizes, which might or might not be
plausible for the context that you're thinking about. For a meta-analysis
of group differences with equally sized groups, a standard error of 0.3
implies a total sample size of about 44. Using a uniform distribution
implies that you will have many quite large studies:
study_sizes <- 4 / runif(1000, 0, 0.3)^2
summary(study_sizes)
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max.
      45       77      170    22732      830 11996245
An alternative approach would be to draw the standard errors from the
empirical distribution of standard errors in a previous conducted synthesis
that resembles the context you're interested in. Or, if there are studies
on the distribution of sample sizes or typical power levels of studies in
your area, you could use that to find a more realistic distribution of
sample sizes/standard errors from which to simulate.

* Your simulation involves generating a structure where every study has the
same number of effect sizes. This "balanced" structure is a best-case
scenario for the performance of the 3-level meta-analysis model. Simulating
studies with varying numbers of effect sizes would be more realistic and
might be important for performance. A simple way to do that is just
studies_per_paper <- sample(1:10, size = ni, replace = TRUE)
dat <- data.frame(paper = rep(1:ni, studies_per_paper), study =
1:sum(studies_per_paper))
table(dat$paper)
One step better would be to use a distribution that is motivated by
empirical data. You could do this by looking at the distribution of effect
sizes per study from one or more previously conducted syntheses that have
open data.

* Your simulation generates effect size estimates that are exactly normally
distributed. For most effect size metrics, this is a simplification that
works reasonably well when sample sizes are large but less well when sample
sizes are small, and which sometimes also depends on other features of the
studies (e.g., for odds ratios, the adequacy of the normality approximation
depends on the number of positive outcomes, not just the sample size). If
you're in a context with small studies or effect sizes that aren't truly
normally distributed, it would be more realistic to generate effect size
estimates following a three-step process:
1) simulate the true effect size parameters by adding up the random effects
at the paper level and study level;
2) simulate any other relevant study features (sample sizes, event rates in
the control condition); and then
3) simulate the effect size estimates based on a realistic data-generating
process, such as by sampling from a t distribution for standardized mean
differences or sampling from binomial distributions for log-odds ratios.
The blog post linked above demonstrates this process for correlated
standardized mean difference effect sizes.

Of course, this might be way more technical detail and way more nuance than
you need. I offer it just in case it's useful to you or others working on
simulations.

James

On Thu, Jun 16, 2022 at 3:11 AM Filippo Gambarota <

[R-meta] simulating three-level meta-analysis

Thread (3 messages)