Skip to content

[R-meta] "Categorical" moderator varying within and between studies

3 messages · Simon Harmel, James Pustejovsky

James Pustejovsky
# 
Hi Simon,

Please keep the listserv cc'd so that others can benefit from these
discussions.

Unfortunately, I don't think there is any single answer to your
question---analytic strategies just depend too much on what your research
questions are and the substantive context that you're working in.

But speaking generally, the advantages of splitting predictors into within-
and between-study versions are two-fold. First is that doing this provides
an understanding of the structure of the data you're working with, in that
it forces one to consider *which* predictors have within-study variation
and *how much *variation there is (e.g., perhaps many studies have looked
at internalizing symptoms, many studies have looked at externalizing
symptoms, but only a few have looked at both types of outcomes in the same
sample). The second advantage is that within-study predictors have a
distinct interpretation from between-study predictors, and the within-study
version is often theoretically more interesting/salient. That's because
comparisons of effect sizes based on within-study variation hold constant
other aspects of the studies that could influence effect size (and that
could muddy the interpretation of the moderator).

Here is an example that comes up often in research synthesis projects.
Suppose that you're interested in whether participant sex moderates the
effect of some intervention. Most of the studies in the sample are of type
A, such that only aggregated effect sizes can be calculated. For these type
A studies, we are able to determine a) the average effect size across the
full sample (pooling across sex) and b) the sex composition of the sample
(e.g., % female). For a smaller number of studies of type B, we are able to
obtain dis-aggregated results for subgroups of male and female
participants. For these studies, we are able to determine a) the average
effect size for males and b) the average effect size for females, plus c)
the sex composition of each of the sub-samples (respectively 0% and 100%
female).

Without considering within/between variation in the predictor, a
meta-regression testing for whether sex is a moderator is:

Y_ij = b0 + b1 (% female)_ij + e_ij

The coefficient b1 describes how effect size magnitude varies across
samples that differ by 1% in the percent of females. But the estimate of
this coefficient pools information across studies of type A and studies of
type B, essentially assuming that the contextual effects (variance
explained by sample composition) are the same as the individual-level
moderator effects (how the intervention effect varies between males and
females).

Now, if we use the within/between decomposition, the meta-regression
becomes:

Y_ij = b0 + b1 (% female-within)_ij + b2 (% female-between)_j + e_ij

In this model, b1 will be estimated *using only the studies of type B*, as
an average of the moderator effects for the studies that provide
dis-aggregated data. And b2 will be estimated using studies of type A and
the study-level average % female in studies of type B. Thus b2 can be
interpreted as a pure contextual effect (variance explained by sample
composition). Why does this matter? It's because contextual effects usually
have a much murkier interpretation than individual-level moderator effects.
Maybe this particular intervention has been tested for several different
professions (e.g., education, nursing, dentistry, construction), and
professions that tend to have higher proportions of females are also those
that tend to be lower-status. If there is a positive contextual effect for
% female, then it might be that a) the intervention really is more
effective for females than for males or b) the intervention is equally
effective for males and females but tends to work better when used with
lower-status professions. Looking at between/within study variance in the
predictor lets us disentangle those possibilities, at least partially.

James
On Wed, Jun 3, 2020 at 9:27 AM Simon Harmel <sim.harmel at gmail.com> wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
  


  


    

    
2 days later

    

      
      

        


  

        

      Simon Harmel
    

    

      
      #
    

  


  

      
Many thanks James. A quick follow-up. The strategy that you described is a
general, regression modeling strategy, right? I mean even if we were
fitting a multi-level model, the fixed-effects part of the formula had to
include the same construction of (i.e., *b1 (% female-within)_ij + b2 (%
female-between)_j*) in it?

Thanks,
Simon

        
On Thu, Jun 4, 2020 at 9:42 AM James Pustejovsky <jepusto at gmail.com> wrote:

        

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      James Pustejovsky
    

    

      
      #
    

  


  

      
Yes, it?s general and also applies outside the context of meta-analysis. See for example Raudenbush & Bryk (2002) for a good discussion on centering and contextual effects in hierarchical linear models.