Skip to content

[R-meta] effect size calculation with a null standard deviation

4 messages · Wolfgang Viechtbauer, James Pustejovsky, Marianne DEBUE

Original
Wolfgang Viechtbauer
# 
Dear Marianne,

I suspect that you did not receive any responses, since what you are describing are really unusual cases which have not been discussed in the literature (as far as I know).

I think you will just have to make a decision yourself how to handle these cases and be transparent about how you handled them.

Best,
Wolfgang
James Pustejovsky
# 
Hi Marianne,

I agree with Wolfgang that this is an unusual case and that figuring out
how to handle it requires some contextual judgements, which folks on the
mailing list aren't really in a position to advise on. That said, here are
two avenues that you might like to investigate further:

With only 3 observations in a given group, using a sample standard
deviation to calculate a standardized mean difference for the group will
give an EXTREMELY noisy estimate. Pooling the pre-test variance across
groups (as in the Morris "dppc2" estimator) might mitigate the problem a
bit--especially if only one of the groups is small and the other is larger.

More broadly, the fact that you're encountering these sorts of situations
makes me wonder whether it would be better to move to a different effect
size metric, such as the response ratio (Hedges, L. V., Gurevitch, J., &
Curtis, P. S. (1999). The meta?analysis of response ratios in experimental
ecology. *Ecology*, *80*(4), 1150-1156.). This might be a good way to go if
all of your outcomes are ratio scale measurements. My understanding is that
this is pretty common in ecology.

James


On Thu, Feb 18, 2021 at 2:49 AM Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

            

      
      
      
    

        

      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Marianne DEBUE
    

    

      
      #
    

  


  

      
Dear Wolfgang and James, 

Thank you for your answers. 

I also considered using the response ratio but I am still confronted to a problem of division by 0. There are indeed some species which are completely absent either before or after the intervention due to changes in ecological conditions so pre-Mean or post-Mean can also be equal to 0. 

I was wondering if I could slightly change the results of the studies where SDpre=0 by adding one individual in the pre and post-data, that is to say, instead of having N samples with 0 individual of a given species in the pre-data, considering N-1 samples with 0 individual and 1 sample with 1 individual, and similarly adding virtually one individual in one sample in the post-data. In this particular case, if I am not mistaken, the difference between means is unchanged and the new SDpre is equal to 1/N. 
Is it a conceivable solution? 

Regards, 
Marianne 



De: "James Pustejovsky" <jepusto at gmail.com> 
?: "Wolfgang Viechtbauer, SP" <wolfgang.viechtbauer at maastrichtuniversity.nl> 
Cc: "Marianne DEBUE" <marianne.debue at mnhn.fr>, "r-sig-meta-analysis" <r-sig-meta-analysis at r-project.org> 
Envoy?: Jeudi 18 F?vrier 2021 20:15:40 
Objet: Re: [R-meta] effect size calculation with a null standard deviation 

Hi Marianne, 
I agree with Wolfgang that this is an unusual case and that figuring out how to handle it requires some contextual judgements, which folks on the mailing list aren't really in a position to advise on. That said, here are two avenues that you might like to investigate further: 

With only 3 observations in a given group, using a sample standard deviation to calculate a standardized mean difference for the group will give an EXTREMELY noisy estimate. Pooling the pre-test variance across groups (as in the Morris "dppc2" estimator) might mitigate the problem a bit--especially if only one of the groups is small and the other is larger. 

More broadly, the fact that you're encountering these sorts of situations makes me wonder whether it would be better to move to a different effect size metric, such as the response ratio ( Hedges, L. V., Gurevitch, J., & Curtis, P. S. (1999). The meta?analysis of response ratios in experimental ecology. Ecology , 80 (4), 1150-1156.). This might be a good way to go if all of your outcomes are ratio scale measurements. My understanding is that this is pretty common in ecology. 

James

        
On Thu, Feb 18, 2021 at 2:49 AM Viechtbauer, Wolfgang (SP) < [ mailto:wolfgang.viechtbauer at maastrichtuniversity.nl | wolfgang.viechtbauer at maastrichtuniversity.nl ] > wrote:

        
Dear Marianne, 

I suspect that you did not receive any responses, since what you are describing are really unusual cases which have not been discussed in the literature (as far as I know). 

I think you will just have to make a decision yourself how to handle these cases and be transparent about how you handled them. 

Best, 
Wolfgang

            

      
      
      
    

        
_______________________________________________ 
R-sig-meta-analysis mailing list 
[ mailto:R-sig-meta-analysis at r-project.org | R-sig-meta-analysis at r-project.org ] 
[ https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis | https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis ]

  
  


  


      

    

    

      
      

        


  

        

      Wolfgang Viechtbauer
    

    

      
      #
    

  


  

      
Conceivable in the sense of "one can think it up"? Sure, you just did :)

Conceivable in the sense of "credible"? I don't think anybody knows.

It sounds to me though as if you are dealing with count data. Maybe measures such as standardized mean differences and response ratios (which are meant to be used for continuous variables) are not the most suitable choice in the first place. For counts, one could also consider using incidence rates (and differences/ratios thereof).

Best,
Wolfgang