Skip to content

[R-meta] Does trim and fill method correct for data falsification or lower quality of small studies?

6 messages · towhidi, Wolfgang Viechtbauer, Michael Dewey

Original
towhidi
# 
Dear all,

The asymmetry in a funnel plot can be caused by factors other than 
publication bias, such as data falsification or poorer quality in 
smaller trials. However, the Cochrane Handbook mentions that "the trim 
and fill method does not take into account reasons for funnel plot 
asymmetry other than publication bias".

I do not understand why it cannot account for data falsification or poor 
quality of small trials, assuming that these characteristics are 
associated with study size. For data falsification, the true observed 
effect size (before the fraudulent change in the data) for these studies 
converges on the true underlying effect size. But the falsified data 
move these data points to the right side, and, using the trim and fill 
method, this bias is neutralized by imputing their counterparts on the 
other side. Of course, the confidence intervals will be biased, because 
we are imputing data points that do not exist (which narrows the CI) and 
that the bias arose from data falsification or low quality has added to 
the estimated sampling variance (which widens the CI). Also, it changes 
the weights, especially in the random-effects model.

But, isn't the point estimate a corrected estimate, assuming that data 
falsification has caused the asymmetry?

The same argument may apply to the bias that arises from low-quality 
studies. However, if this is correct, I think that acknowledging this 
and interpreting the CIs with even more caution is more logical than 
assuming that the asymmetry is caused solely by publication bias and 
that misconduct and low quality of small studies have nothing to do with 
it.

Is this correct? Or I am missing something?

Thank you.
Wolfgang Viechtbauer
# 
Dear Ali,

Please see my responses below.

Best,
Wolfgang
... or unaccounted for moderators (that are correlated with study size) or more generally heterogeneity.
I searched through the handbook (https://training.cochrane.org/handbook/current) and couldn't find this quote. Where did you find this?
'Neutralized' sounds a bit too optimistic. If a study is imputed corresponding to the fraudulent study (which isn't guaranteed depending on how the funnel looks in general), it is going to be placed at 'est - delta', where 'est' is the pooled estimate at the end of the trim and fill procedure and 'delta' is the distance between est and the fraudulent study. If 'est' is larger than 0, then this would still leads to some bias, but it should indeed be reduced.
I would say yes. A simulation study that has examined the properties of various methods not only when there is publication bias but also under the use of 'questionable research practices' is:

Carter, E. C., Sch?nbrodt, F. D., Gervais, W. M. & Hilgard, J. (2019). Correcting for bias in psychology: A comparison of meta-analytic methods. Advances in Methods and Practices in Psychological Science, 2(2), 115-144. https://doi.org/10.1177/2515245919847196
towhidi
#
On 2022-05-03 11:45, Viechtbauer, Wolfgang (NP) wrote:

            

      
      
      
    

        
Dear Wolfgang,
Thank you for your response, and thank you for the article you 
mentioned.

The quoted text, "the trim and fill method does not take into account 
reasons for funnel plot asymmetry other than publication bias", was from 
the previous version of the Cochrane Handbook 
(https://handbook-5-1.cochrane.org/chapter_10/10_4_4_2_trim_and_fill.htm). 
The trim and fill subsection is removed from the current version.

Best,
Ali

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Wolfgang Viechtbauer
    

    

      
      #
    

  


  

      
Thanks for providing the source of this quote. 

Having read the entire text now, my interpretation of the statement ("Equally importantly, the trim and fill method does not take into account reasons for funnel plot asymmetry other than publication bias") is that this is not meant to exclude other forms of biases that could behave like publication bias (such as the use of questionable research practices), but in essence the method assumes that funnel plot asymmetry is automatically due to publication bias (or other forms of biases that behave like it), so if the asymmetry is due to something else (like an unknown moderator that is correlated with study size), it will falsely attribute that asymmetry to publication bias and falsely correct for it.

Best,
Wolfgang

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Michael Dewey
    

    

      
      #
    

  


  

      
One might also point out that the methods has been criticised for 
detecting publicaiton bias

@article{schwarzer10,
    author = {Schwarzer, G and Carpenter, J R and R\"ucker, G},
    title = {Empirical evaluation suggests {Copas} selection model
       preferable to trim--and--fill method for publication
       bias in meta--analysis},
    journal = {Journal of Clinical Epidemiology},
    year = {2010},
    volume = {63},
    pages = {282--288}
}

Michael

        
On 03/05/2022 23:11, towhidi wrote:

            

      
      
      
    

        

  
    
  


  


      

    

    

      
      

        


  

        

      towhidi
    

    

      
      #
    

  


  

      
On 2022-05-04 16:47, Michael Dewey wrote:

        

            

      
      
      
    

        
Dear Wolfgang
Thank you for elaborating on the text.

And Dear Micheal,
Thank you for your note. I intend to use different methods in my 
sensitivity analyses.

Best,
Ali