Skip to content

[R-meta] Estimating "overall effect" in meta-regression

6 messages · Luke Martinez, Wolfgang Viechtbauer

Luke Martinez
# 
Dear R-meta Community,

I'm meta-analyzing a group of pre-post studies. My first RQ is: what
is the "overall effect" of policy X on a dependent variable.

I know that I can fit an intercept-only model to answer this RQ.

But an intercept-only model estimates the average effect size across
BOTH pre-tests (before policy X implementation) and post-tests (after
policy X implementation), while pre-test effect sizes don't contain
any policy X effect.

Given that, is it appropriate to use an intercept-only model to answer
this RQ, or I actually need to use a time indicator as a moderator to
separate the pre- from post-test effect sizes to answer this RQ (in
which case the original RQ must change too)?

Thank you for helping me better conceptualize this basic question,
Luke
1 day later
Wolfgang Viechtbauer
# 
Dear Luke,

I don't understand the question, in part because it is not clear to me what kind of design the studies have and what kind of effect size measure you are using. Could you clarify this?

Best,
Wolfgang
Luke Martinez
# 
Dear Wolfgang,

The studies follow a pre-post-control design. The effect size measure used
is standardized mean difference (SMD).

I hope this clarifies my question.
Luke

sudy time  yi
1         0
1         1
2         0
2         1


On Mon, Jan 10, 2022, 5:43 AM Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

            

      
      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Wolfgang Viechtbauer
    

    

      
      #
    

  


  

      
For such a pre-post-control-group design design, one could compute two standardized mean differences (one for time 1 and one for time 2), or two standardized mean changes (one for the treatment and one for the control group), or one could compute the difference between two such estimates.

Based on the schema below, it looks like you are computing two standardized mean differences (one for time 1 and one for time 2). Note that the two standardized mean differences are not independent, since they are computed based on the same subjects. This aside, then one definitely should include time as a moderator, as it would make little sense to synthesize estimates from before and after the treatment into a single pooled effect.

Best,
Wolfgang

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Luke Martinez
    

    

      
      #
    

  


  

      
Thank you so much, Wolfgang. I gather from your response that any
additional moderators in such a meta-analysis must likewise interact
with time so we can separate the effect of such a moderator at each
different time point.

Thank you very much,
Luke

On Mon, Jan 10, 2022 at 9:15 AM Viechtbauer, Wolfgang (SP)

        
<wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Wolfgang Viechtbauer
    

    

      
      #
    

  


  

      
Yes, that would make sense.

Best,
Wolfgang