[R-meta] rule of thumb miminum number of studies per factor level meta-regression

Tue, Mar 29, 2022 3:35 AM

Hi Frank,

I would say that essentially all conclusions from moderator analyses (whether done by meta-regression, some kind of ANOVA-analogue, subgrouping, or some other technique) are hypothesis generating. Cooper calls this 'synthesis-generated evidence' (in contrast to study-generated evidence, which can be obtained by randomly assigning participants to different levels of a potential moderator of a treatment effect). His book 'Research synthesis and meta-analysis' discusses this distinction.

Best,
Wolfgang

-----Original Message-----
From: Frank van Boven [mailto:f.vanboven5 at gmail.com]
Sent: Monday, 28 March, 2022 13:13
To: Viechtbauer, Wolfgang (SP)
Cc: Lena Pollerhoff; r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] rule of thumb miminum number of studies per factor level
meta-regression

Dear all,

In reply to this explanation, I am wondering.
When subgrouping the studies (thus no meta-regression).
Would it be an option to limit the aim of the meta-analysis to only generate
hypotheses, irrespectfull the number of studies left in each subgroup?

Kind regards,

Frank van Boven

Op 28 mrt. 2022, om 12:04 heeft Viechtbauer, Wolfgang (SP)

<wolfgang.viechtbauer at maastrichtuniversity.nl> het volgende geschreven:

Dear Lena,

Just for the record, the '10 studies per covariate' rule comes from here:

https://training.cochrane.org/handbook/current/chapter-10#section-10-11-5-1

where it says:

"It is very unlikely that an investigation of heterogeneity will produce useful

findings unless there is a substantial number of studies. Typical advice for
undertaking simple regression analyses: that at least ten observations (i.e. ten
studies in a meta-analysis) should be available for each characteristic modelled.
However, even this will be too few when the covariates are unevenly distributed
across studies."

I have no idea where the 10 per covariate rule comes from (there is also no

reference in the Cochrane Handbook) and I am not aware of any empirical support
for it. I suspect it was just taken over from similar rules that have been
formulated in other contexts (e.g., regression models with primary data,
prediction models, factor analysis) where these rules have often been formulated
without much, if any, empirical support.

Given what it says in the Cochrane Handbook, one could read this to imply that

at least 10 studies per covariate are needed to 'produce useful findings'.
Without a definition of 'useful findings', I don't even know how to evaluate
whether such a rule is sensible or not.

I am not trying to rag on the Cochrane Handbook. The question about 'k per

moderator' (or k in general for a meta-analysis) is one of the questions that
*always* comes up in any course on meta-analysis I teach. It is a good question
and I have no good answer for it, except to mention that such rules exist (e.g.,
'10 per covariate'), but that they lack empirical support.

Analogously, I am not aware of any evidence-based guidelines with respect to

your 'k per level' question.

So, in the end, I am doing again the same thing as I always do when I get this

question, which is to provide no good answer.

Best,
Wolfgang

[R-meta] rule of thumb miminum number of studies per factor level meta-regression

Thread (5 messages)