-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of Viechtbauer, Wolfgang (NP) via R-sig-meta-analysis
Sent: Friday, 29 September, 2023 13:28
To: Kiet Huynh
Cc: Viechtbauer, Wolfgang (NP); R Special Interest Group for Meta-Analysis
Subject: Re: [R-meta] Score Normalization for Moderator Analysis in Meta-Analysis
I just noticed that the last question has remained unanswered:
Depends on what you mean by "need". To run such an analysis, assuming 'scale' is
just a two-level factor and you want to run a model with '~ factor(scale) *
pompmean', then you will need five effect sizes, two for the first and three for
the second level of 'scale'. That will give you just enough information to fit
such a model and estimate the amount of residual heterogeneity.
But I assume that this is not what you mean by "need". If you meant something
along the lines of 'having enough power', then I cannot give you an answer to
that question, because it is like asking: "I want to run a study - how many
subjects do I need?" (although turns out that the answer to that question is:
"three patients" -- https://www.youtube.com/watch?v=Hz1fyhVOjr4). To give an
informed answer to that question, one would have to do a power analysis:
Hedges, L. V., & Pigott, T. D. (2004). The power of statistical tests for
moderators in meta-analysis. Psychological Methods, 9(4), 426-445.
If you meant something along the lines of 'so that reviewers are not going to
complain that my sample size is too small', then one could refer to rules of
thumb like what you can find in the Cochrane Handbook:
https://training.cochrane.org/handbook/current/chapter-10#section-10-11-5-1
"It is very unlikely that an investigation of heterogeneity will produce useful
findings unless there is a substantial number of studies. Typical advice for
undertaking simple regression analyses: that at least ten observations (i.e. ten
studies in a meta-analysis) should be available for each characteristic modelled.
However, even this will be too few when the covariates are unevenly distributed
across studies."
To be clear, this is an entirely arbitrary rule (and one also finds suggestions
like '5 studies per characteristic'). Also, what exactly 'for each characteristic
modelled' means is not entirely clear, but say we interpret this as 'per model
coefficient'. The model above has 4 model coefficients (including the intercept),
so then we would need at least 40 effect sizes.
To be fair, this rule does relate somewhat to the issue of overfitting, since
more complex models require more data points to avoid overfitting. But even then,
one would have to articulate more precisely what exactly one is concerned about.
Best,
Wolfgang
-----Original Message-----
From: Kiet Huynh [mailto:kietduchuynh at gmail.com]
Sent: Thursday, 14 September, 2023 17:10
To: Viechtbauer, Wolfgang (NP)
Cc: R Special Interest Group for Meta-Analysis
Subject: Re: [R-meta] Score Normalization for Moderator Analysis in Meta-
Hi Wolfgang,
Thanks for the reminder about including links when cross posting.
I appreciate the helpful expiation for the proportion/percentage of maximum
possible' (POMP) score method for moderation analysis. Especially helpful was
tip on using the scale type to interact with the POMP score mean to determine if
the relationship between social support and the strength of the association
between LGBTQ+ discrimination and mental health differs depending on the scale
used. Do you have a sense of how many effect sizes would be needed for that?
Best,
Kiet