Skip to content

[R-meta] Standard Error of an effect size for use in longitudinal meta-analysis

6 messages · Simon Harmel, Wolfgang Viechtbauer

Simon Harmel
# 
Dear All,

Suppose I know that the likelihood function for an estimate of effect size
(called `dppc`) measuring the change in a "*control*" group from pre-test
to post-test in R language is given by:

    like1 <- function(x) dt(dppc*sqrt(nc), df = nc - 1, ncp = x*sqrt(nc))

where `dppc` is the observed estimate of effect size, and `nc` is the
"control" group's sample size.

Similarly, the likelihood function for an estimate of effect size (called `
dppt`) measuring the change in a "*treatment*" group from pre-test to
post-test in `R` language is given by:

    like2 <- function(x) dt(dppt*sqrt(nt), df = nt - 1, ncp = x*sqrt(nt))

where `dppt` is the observed estimate of effect size, and `nt` is the
"treatment" group's sample size.
`d_dif = dppt - dppc` (in `R`)?

Below, I tried to first get the likelihood function of `d_dif` and then get
the *Standard Deviation* of that likelihood function. In a sense, I assumed
I have a Bayesian problem with a "flat prior" and thus "SE" is the standard
deviation of the likelihood of `d_dif`.
-- Thank you, Simon

    library(distr)

    d_dif <- Vectorize(function(dppc, dppt, nc, nt){

     like1 <- function(x) dt(dppc*sqrt(nc), df = nc - 1, ncp = x*sqrt(nc))
     like2 <- function(x) dt(dppt*sqrt(nt), df = nt - 1, ncp = x*sqrt(nt))

      d1 <- distr::AbscontDistribution(d = like1, low1 = -15, up1 = 15,
withStand = TRUE)
      d2 <- distr::AbscontDistribution(d = like2, low1 = -15, up1 = 15,
withStand = TRUE)

     like.dif <- function(x) distr::d(d2 - d1)(x) ## Gives likelihood of
the difference i.e., `d_dif`

     Mean <- integrate(function(x) x*like.dif(x), -Inf, Inf)[[1]]
       SE <- sqrt(integrate(function(x) x^2*like.dif(x), -Inf, Inf)[[1]] -
Mean^2)

     return(c(SE = SE))
    })

     # EXAMPLE OF USE:
     d_dif(dppc = .2, dppt = .4, nc = 40, nt = 40)
Wolfgang Viechtbauer
# 
Dear Simon,

Based on what you describe, dppc and dppt appear to be standardized mean changes using what I would call "change-score standardization" (i.e., (m1 - m2)/SDd, where SDd is the standard deviation of the change scores).

I haven't really tried to figure out the details of what you are doing (and I am not familiar with the distr package), but the large-sample sampling variances of dppc and dppt are:

vc = 1/nc + dppc^2/(2*nc)

and

vt = 1/nt + dppt^2/(2*nt)

(or to be precise, these are the estimates of the sampling variances, since dppc and dppt have been plugged in for the unknown true values).

Hence, the SE of dppt - dppc is simply:

sqrt(vt + vc)

For the "example use" data, this is:

sqrt((1/40 + 0.2^2 / (2*40)) + (1/40 + 0.4^2 / (2*40)))

which yields 0.2291288.

Running your code yields 0.2293698. The former is a large-sample approximation while you seem to be using an 'exact' approach, so they are not expected to coincide, but should be close, which they are.

Best,
Wolfgang
Simon Harmel
# 
Dear Wolfang,

Many thanks for your confirmation! I have some follow-up questions.

(1) I believe "dppc" is biased and requires a correction, so does its
sampling variance, right?

(2) If "dppc" and "dppt" (along with their sampling variances) are each
biased and must be corrected, then, we don't need to again correct the
sampling variance of "d_dif" (i.e., dppt - dppc)?

(3) Do you see any advantage or disadvantage for using "d_dif" in
longitudinal meta-analysis where studies that have a control group?

[In terms of advantages:
 a- I think logistically using "d_dif" will reduce the number of effect
sizes that are otherwise usually computed from such studies by half,
 b- "d_dif"'s metric seems to better suit the repeated measures design used
in the primary studies,
 c- "d_dif" seems to allow for investigating (by using appropriate
moderators) the threats to the internal validity (regression to mean,
history, maturation, testing) say when the primary studies didn't use
random assignment of subjects (i.e., nonequivalent groups designs)

In terms disadvantages:
 a- I think obtaining "r", the correlation bet. pre- and post-tests to
compute "dppc" and "dppt" is a bit difficult,
 b- while "d_dif"'s metric seems to better suit the repeated measures
design used in the primary studies, most applied meta-analysts are not used
to such a metric.
]

On Tue, Aug 11, 2020 at 2:04 PM Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

            

      
      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Wolfgang Viechtbauer
    

    

      
      #
    

  


  

      
1) Yes, d = (m_1 - m_2) / SD_d (where m_1 and m_2 are the observed means at time 1 and 2 and SD_d is the standard deviation of the change scores) is a biased estimator of (mu_1-mu_2)/sigma_D. An approximatly unbiased estimator is given by:

g = (1 - 3/(4*(n-1) - 1) * d

See, for example:

Gibbons, R. D., Hedeker, D. R., & Davis, J. M. (1993). Estimation of effect size from a series of experiments involving paired comparisons. Journal of Educational Statistics, 18(3), 271-279.

where you can also find the exact correction factor.

Not sure what you mean by correcting the sampling variance (and which variance estimate you are referring to). You can find the equation for an unbiased estimator of the sampling variance of d and g in:

Viechtbauer, W. (2007). Approximate confidence intervals for standardized effect sizes in the two-independent and two-dependent samples design. Journal of Educational and Behavioral Statistics, 32(1), 39-60.

In particular, see equations 25 and 26.

2) I don't understand your question.

3) The idea of using 'd_dif' for computing an effect size for independent-groups pretest?posttest design is not new. See Gibbons et al. (1993) and:

Becker, B. J. (1988). Synthesizing standardized mean-change measures. British Journal of Mathematical and Statistical Psychology, 41(2), 257-278.

Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7(1), 105-125.

In this context, one also needs to consider whether one should compute d or g using raw-score or change-score standardization (the former being (m1-m2)/SD_1 where SD_1 is the standard deviations at the first time point). Becker (1988) discusses the use of raw-score standardization, while Gibbons et al. (1993) is focused on change-score standardization. In any case, I think you will find some useful discussions in these articles.

Best,
Wolfgang

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Simon Harmel
    

    

      
      #
    

  


  

      
Dear Wolfang,

Thank you very much for your insightful comments. Regarding my question #
2, I meant after correcting "dppc" and "dppt" (as well as their sampling
variances "vc" and "vt"), is there any need for further correction of
d_dif and sqrt(vt + vc)? I think no, right?

Now, am I wrong to think that your large sample "vc" and "vt" need to be
multiplied by "cfactor(n-1)^2" to become bias-free?

where cfactor = function(df) exp(lgamma(df/2)-log(sqrt(df/2)) -
lgamma((df-1)/2)) # df = n -1

Simon

On Wed, Aug 12, 2020 at 1:36 AM Viechtbauer, Wolfgang (SP) <

        
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

        

            

      
      
      
    

        

  
  


  


      

    

    
4 days later

    

      
      

        


  

        

      Wolfgang Viechtbauer
    

    

      
      #
    

  


  

      
If you have unbiased estimates y1 and y2, then y1-y2 is also unbiased, so no, there is no need to correct y1-y2.

If you also have unbiased estimates of the sampling variances, Var[y1] and Var[y2], then Var[y1] + Var[y2] is an unbiased estimate of the sampling variance of y1-y2 (note that this assumes that y1 and y2 are independent).

The large-sample variances of y1 and y2 are not unbiased, neither before or after multiplying them with cfactor(n-1)^2. The equation for an unbiased estimate of Var[y1] and Var[y2] is given in Viechtbauer (2007).

Best,
Wolfgang