Skip to content

[R-meta] Meta-analyzing studies that failed to account for their nested data

4 messages · James Pustejovsky, Timothy MacKenzie

Original
Timothy MacKenzie
# 
Hello All,

I've noticed almost all the studies I have selected for meta-analysis
have ignored the nested structure of their data (subjects nested in
classrooms) and have conducted only single-level analyses.

I've extracted the condition-level summaries from those studies (i.e.,
Means and SDs for C vs. T groups).

But I'm wondering if I can/should make any adjustment to my
meta-regression model to account for the nested structure of the data
in those studies AND if not, whether such a situation poses a
limitation to my meta-analysis?

Thank you very much for your assistance,
Tim M
James Pustejovsky
# 
Hi Tim,

One important issue here is that the sampling variance of the effect size
estimate calculated from such a study will be inaccurate---possibly even an
order of magnitude smaller than it should be. If you ignore this, the
consequence will be to make the effect size estimates appear far more
precise than they actually are.

To properly correct the sampling variance estimate, you would need to know
the intra-class correlation describing the proportion of the total
variation in the outcome that is at the cluster level (in this case, what
fraction of the total variance is between classes?). If this isn't
reported, then it may be possible to develop a reasonable estimate based on
external information. The Cochrane Handbook describes how to correct the
sampling variance based on an imputed intra-class correlation:
https://training.cochrane.org/handbook/current/chapter-23#section-23-1
Hedges (2007; https://doi.org/10.3102/1076998606298043) and (2011;
https://doi.org/10.3102/1076998610376617) provides slightly more elaborate
methods that can be used if you have more details about the study designs.
Hedges and Hedberg's Variance Almanac (http://stateva.ci.northwestern.edu/)
is a helpful source for developing estimates of ICCs for educational
outcomes.

James
On Thu, Sep 30, 2021 at 4:58 PM Timothy MacKenzie <fswfswt at gmail.com> wrote:

            

      
      
      
    

        

      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Timothy MacKenzie
    

    

      
      #
    

  


  

      
Dear James,

Many thanks for this information. Certainly this is serious.

I should add that a few of the (newer) studies in my pool say that
they found their ICCs to be negligible and opted for the single-level
analyses (maybe I should not adjust the sampling variances in these
cases, correct?).

Also, I'm assuming that I can use these sampling variance adjustments
for quasi-experiments where schools/centers themselves haven't been
randomly recruited as well?

Thanks,
Tim M

        
On Thu, Sep 30, 2021 at 9:40 PM James Pustejovsky <jepusto at gmail.com> wrote:

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      James Pustejovsky
    

    

      
      #
    

  


  

      
For studies that claim to find negligible ICCs, I would guess that they
base this judgement either on a) failing to reject a test of ICC = 0 or b)
a rule of thumb. a) is not a good justification because with few classes,
the test will have little power. b) is arbitrary and even small ICCs (of
say 0.02 or 0.04) can be consequential for estimating the variance of the
effect size estimate. I would use the ICC adjustment regardless.

To your second question, yes these adjustments are also important for
quasi-experiments.

James

On Thu, Sep 30, 2021 at 10:53 PM Timothy MacKenzie <fswfswt at gmail.com>
wrote: