[R-meta] Does trim and fill method correct for data falsification or lower quality of small studies?

Dear Ali,

Please see my responses below.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis 
[mailto:r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of towhidi
Sent: Tuesday, 03 May, 2022 1:37
To: r sig meta-analysis list
Subject: [R-meta] Does trim and fill method correct for data 
falsification or
lower quality of small studies?

Dear all,

The asymmetry in a funnel plot can be caused by factors other than
publication bias, such as data falsification or poorer quality in
smaller trials.

... or unaccounted for moderators (that are correlated with study size) 
or more generally heterogeneity.

However, the Cochrane Handbook mentions that "the trim
and fill method does not take into account reasons for funnel plot
asymmetry other than publication bias".

I searched through the handbook 
(https://training.cochrane.org/handbook/current) and couldn't find this 
quote. Where did you find this?

I do not understand why it cannot account for data falsification or 
poor
quality of small trials, assuming that these characteristics are
associated with study size. For data falsification, the true observed
effect size (before the fraudulent change in the data) for these 
studies
converges on the true underlying effect size. But the falsified data
move these data points to the right side, and, using the trim and fill
method, this bias is neutralized by imputing their counterparts on the
other side.

'Neutralized' sounds a bit too optimistic. If a study is imputed 
corresponding to the fraudulent study (which isn't guaranteed depending 
on how the funnel looks in general), it is going to be placed at 'est - 
delta', where 'est' is the pooled estimate at the end of the trim and 
fill procedure and 'delta' is the distance between est and the 
fraudulent study. If 'est' is larger than 0, then this would still 
leads to some bias, but it should indeed be reduced.

Of course, the confidence intervals will be biased, because
we are imputing data points that do not exist (which narrows the CI) 
and
that the bias arose from data falsification or low quality has added 
to
the estimated sampling variance (which widens the CI). Also, it 
changes
the weights, especially in the random-effects model.

But, isn't the point estimate a corrected estimate, assuming that data
falsification has caused the asymmetry?

I would say yes. A simulation study that has examined the properties of 
various methods not only when there is publication bias but also under 
the use of 'questionable research practices' is:

Carter, E. C., Sch?nbrodt, F. D., Gervais, W. M. & Hilgard, J. (2019). 
Correcting for bias in psychology: A comparison of meta-analytic 
methods. Advances in Methods and Practices in Psychological Science, 
2(2), 115-144. https://doi.org/10.1177/2515245919847196

The same argument may apply to the bias that arises from low-quality
studies. However, if this is correct, I think that acknowledging this
and interpreting the CIs with even more caution is more logical than
assuming that the asymmetry is caused solely by publication bias and
that misconduct and low quality of small studies have nothing to do 
with
it.

Is this correct? Or I am missing something?

Thank you.

--
Ali Zia-Tohidi MSc
Clinical Psychology
University of Tehran