Skip to content

[R-meta] rule of thumb miminum number of studies per factor level meta-regression

5 messages · Lena Pollerhoff, Wolfgang Viechtbauer, Frank van Boven +1 more

Lena Pollerhoff
# 
Dear list member, 

I am conducting meta-regressions in metafor at the moment and have a short question regarding rule of thumbs with respect to categorical predictors in meta-regression. While we are aware of one rule of thumb that meta-regressions should not be considered for fewer than ten studies per covariate (e.g., Cochrane Handbook), we were wondering whether such a rule of thumb also exists with respect to the minimum number of studies per factor level of a categorical variable?

In my case, I am conducting meta-regressions, where the number of studies per factor level are sometimes unevenly distributed: For example, k = 22, and I have one categorical predictor with three factor levels, with the first one represented by only one study, the second one by three studies, and the third one including 18 studies. 

Thanks in advance and have a nice day!
Lena Pollerhoff
Wolfgang Viechtbauer
# 
Dear Lena,

Just for the record, the '10 studies per covariate' rule comes from here:

https://training.cochrane.org/handbook/current/chapter-10#section-10-11-5-1

where it says:

"It is very unlikely that an investigation of heterogeneity will produce useful findings unless there is a substantial number of studies. Typical advice for undertaking simple regression analyses: that at least ten observations (i.e. ten studies in a meta-analysis) should be available for each characteristic modelled. However, even this will be too few when the covariates are unevenly distributed across studies."

I have no idea where the 10 per covariate rule comes from (there is also no reference in the Cochrane Handbook) and I am not aware of any empirical support for it. I suspect it was just taken over from similar rules that have been formulated in other contexts (e.g., regression models with primary data, prediction models, factor analysis) where these rules have often been formulated without much, if any, empirical support.

Given what it says in the Cochrane Handbook, one could read this to imply that at least 10 studies per covariate are needed to 'produce useful findings'. Without a definition of 'useful findings', I don't even know how to evaluate whether such a rule is sensible or not.

I am not trying to rag on the Cochrane Handbook. The question about 'k per moderator' (or k in general for a meta-analysis) is one of the questions that *always* comes up in any course on meta-analysis I teach. It is a good question and I have no good answer for it, except to mention that such rules exist (e.g., '10 per covariate'), but that they lack empirical support.

Analogously, I am not aware of any evidence-based guidelines with respect to your 'k per level' question.

So, in the end, I am doing again the same thing as I always do when I get this question, which is to provide no good answer.

Best,
Wolfgang
Frank van Boven
# 
Dear all,

In reply to this explanation, I am wondering.
When subgrouping the studies (thus no meta-regression).
Would it be an option to limit the aim of the meta-analysis to only generate hypotheses, irrespectfull the number of studies left in each subgroup?

Kind regards,

Frank van Boven
Michael Dewey
# 
Dear Lena

If you have one study per level then in the model you have one parameter 
purely for that study and that study will have high influence in the 
model which you should be able to confirm from influence(yourModelHere). 
If you have few studies per level then the situation is less stark but 
the influence statistics should be your friend in clarifying what happened.

Michael
On 28/03/2022 09:40, Lena Pollerhoff wrote:

  
    
  


  


      

    

    

      
      

        


  

        

      Wolfgang Viechtbauer
    

    

      
      #
    

  


  

      
Hi Frank,

I would say that essentially all conclusions from moderator analyses (whether done by meta-regression, some kind of ANOVA-analogue, subgrouping, or some other technique) are hypothesis generating. Cooper calls this 'synthesis-generated evidence' (in contrast to study-generated evidence, which can be obtained by randomly assigning participants to different levels of a potential moderator of a treatment effect). His book 'Research synthesis and meta-analysis' discusses this distinction.

Best,
Wolfgang