Skip to content

[R-meta] Calculating SMD from posttest only - is this possible?

7 messages · P. Roberto Bakker, Wolfgang Viechtbauer, James Pustejovsky +2 more

P. Roberto Bakker
# 
Hi everybody

I am performing a meta-analysis (with 'metafor' R-project) as described by
Becker (1988), and I compute the standardized mean change for a treatment
and control group.

I compute a (standardized) effect size measure for pretest posttest control
group designs, where the characteristic, response, or dependent variable
assessed in the individual studies is a quantitative variable (Morris 2008).

Now, two new articles are published recently with only the posttests.

Question: can SMDs still be calculated with only the posttest (and sd) of
treatment and control groups?

Thank you in advance, Roberto
2 days later
Wolfgang Viechtbauer
# 
Hi Roberto,

Yes. In fact, that's essentially the original definition of the SMD, that is, the difference between two groups at the posttest divided by the SD (at the posttest).

One can debate to what extent the standardized mean change (of a single group), the difference between two standardized mean changes (of two groups), and the standardized mean difference at the posttest (again, of two groups) can be combined in a single analysis. The canonical reference regarding this issue is:

Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7(1), 105-125.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of P. Roberto Bakker
Sent: Thursday, 03 October, 2019 18:54
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] Calculating SMD from posttest only - is this possible?

Hi everybody

I am performing a meta-analysis (with 'metafor' R-project) as described by
Becker (1988), and I compute the standardized mean change for a treatment
and control group.

I compute a (standardized) effect size measure for pretest posttest control
group designs, where the characteristic, response, or dependent variable
assessed in the individual studies is a quantitative variable (Morris 2008).

Now, two new articles are published recently with only the posttests.

Question: can SMDs still be calculated with only the posttest (and sd) of
treatment and control groups?

Thank you in advance, Roberto
1 day later
James Pustejovsky
# 
Roberto,

Just to add a bit to Wolfgang's response: I think it is helpful to consider
this question under the framework of risk of bias.

If an experiment is properly randomized and there is no attrition, then the
difference in post-test means is an unbiased estimator of the average
treatment effect (ATE). The same is true in a pre/post experiment: if the
randomization is properly conducted and there is not attrition, then the
difference in mean change scores is an unbiased estimator of the ATE
(because the difference in pre-test means has expectation zero). In other
words, both designs are used to estimate the same target parameter (albeit
with different degrees of precision).

Where discrepancies might arise is if the design isn't perfect, such as if
the randomization might have been faulty (due to lack of allocation
concealment or something) or if there is substantial attrition, either of
which could lead to bias in the estimate of the ATE. The pre-post design is
probably more robust to such imperfections because it adjusts for baseline
differences. The post-test only design is likely less robust because
there's no baseline covariates that could be used to adjust for differences
between groups at post-test. Considered in light, whether it's a good idea
to include post-test only designs depends on whether the post-test only
designs are at low risk of bias.

James

On Sat, Oct 5, 2019 at 4:46 PM Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

            

      
      
      
    

        

      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Guido Schwarzer
    

    

      
      #
    

  


  

      
Hi Roberto,

Just to add to the responses by Wolfgang and James.

The Cochrane Handbook warns against combining SMDs based on change 
scores (pre-post comparisons) and post-test comparisons (see 
https://training.cochrane.org/handbook/current/chapter-10#section-10-5-2). 
The reason is that the standard deviation used in the denominator of the 
SMD is different for change scores and post-tests.

Best wishes, Guido

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Michael Dewey
    

    

      
      #
    

  


  

      
Dear Roberto

You may have already thought of this but including a categorical 
moderator to distinguish the designs would be useful. Of course if it 
shows that the designs give different results you have the problem of 
explaining it but at least you know.

This is a supplement to the advice from W, J and G.

Michael

        
On 05/10/2019 22:40, Viechtbauer, Wolfgang (SP) wrote:

            

      
      
      
    

        

      
      
    

        

  
    
  


  


      

    

    

      
      

        


  

        

      Wolfgang Viechtbauer
    

    

      
      #
    

  


  

      
That is correct when using standardized mean change values using change score standardization -- and indeed, that's not something that should done. However, the standardized mean change using raw score standardization also standardizes based on the variability at a single time point and hence is in principle comparable to the standardized mean difference at the posttest (leaving aside other issues, such as time effects). I would still be very careful when doing this and I very much like Michael's suggestion to examine this via a moderator analysis.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of Guido Schwarzer
Sent: Monday, 07 October, 2019 14:16
To: P. Roberto Bakker; r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] Calculating SMD from posttest only - is this possible?

Hi Roberto,

Just to add to the responses by Wolfgang and James.

The Cochrane Handbook warns against combining SMDs based on change 
scores (pre-post comparisons) and post-test comparisons (see 
https://training.cochrane.org/handbook/current/chapter-10#section-10-5-2). 
The reason is that the standard deviation used in the denominator of the 
SMD is different for change scores and post-tests.

Best wishes, Guido

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      P. Roberto Bakker
    

    

      
      #
    

  


  

      
Hi Wolfgang, Guido, Michael, and James,

Your information and explanation are really helpful, interesting and
important. A lot of my questions now have a clear answer, and I
understand better what needs to be done. I will bring your information to
my meta-analysis group. As your information is published in the
R-sig-meta-analysis Archives, other people will benefit.

Thank you very much and best regards,
Roberto


Op ma 7 okt. 2019 om 13:13 schreef Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl>: