-----Original Message-----
From: towhidi [mailto:towhidi at ut.ac.ir]
Sent: Wednesday, 04 May, 2022 0:12
To: Viechtbauer, Wolfgang (NP)
Cc: r sig meta-analysis list
Subject: Re: [R-meta] Does trim and fill method correct for data falsification or
lower quality of small studies?
On 2022-05-03 11:45, Viechtbauer, Wolfgang (NP) wrote:
Dear Ali,
Please see my responses below.
Best,
Wolfgang
-----Original Message-----
From: R-sig-meta-analysis
[mailto:r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of towhidi
Sent: Tuesday, 03 May, 2022 1:37
To: r sig meta-analysis list
Subject: [R-meta] Does trim and fill method correct for data
falsification or
lower quality of small studies?
Dear all,
The asymmetry in a funnel plot can be caused by factors other than
publication bias, such as data falsification or poorer quality in
smaller trials.
... or unaccounted for moderators (that are correlated with study size)
or more generally heterogeneity.
However, the Cochrane Handbook mentions that "the trim
and fill method does not take into account reasons for funnel plot
asymmetry other than publication bias".
I do not understand why it cannot account for data falsification or
poor
quality of small trials, assuming that these characteristics are
associated with study size. For data falsification, the true observed
effect size (before the fraudulent change in the data) for these
studies
converges on the true underlying effect size. But the falsified data
move these data points to the right side, and, using the trim and fill
method, this bias is neutralized by imputing their counterparts on the
other side.
'Neutralized' sounds a bit too optimistic. If a study is imputed
corresponding to the fraudulent study (which isn't guaranteed depending
on how the funnel looks in general), it is going to be placed at 'est -
delta', where 'est' is the pooled estimate at the end of the trim and
fill procedure and 'delta' is the distance between est and the
fraudulent study. If 'est' is larger than 0, then this would still
leads to some bias, but it should indeed be reduced.
Of course, the confidence intervals will be biased, because
we are imputing data points that do not exist (which narrows the CI)
and
that the bias arose from data falsification or low quality has added
to
the estimated sampling variance (which widens the CI). Also, it
changes
the weights, especially in the random-effects model.
But, isn't the point estimate a corrected estimate, assuming that data
falsification has caused the asymmetry?
I would say yes. A simulation study that has examined the properties of
various methods not only when there is publication bias but also under
the use of 'questionable research practices' is:
Carter, E. C., Sch?nbrodt, F. D., Gervais, W. M. & Hilgard, J. (2019).
Correcting for bias in psychology: A comparison of meta-analytic
methods. Advances in Methods and Practices in Psychological Science,
2(2), 115-144. https://doi.org/10.1177/2515245919847196
The same argument may apply to the bias that arises from low-quality
studies. However, if this is correct, I think that acknowledging this
and interpreting the CIs with even more caution is more logical than
assuming that the asymmetry is caused solely by publication bias and
that misconduct and low quality of small studies have nothing to do
with
it.
Is this correct? Or I am missing something?
Thank you.
--
Ali Zia-Tohidi MSc
Clinical Psychology
University of Tehran
Dear Wolfgang,
Thank you for your response, and thank you for the article you
mentioned.
The quoted text, "the trim and fill method does not take into account
reasons for funnel plot asymmetry other than publication bias", was from
the previous version of the Cochrane Handbook
(https://handbook-5-1.cochrane.org/chapter_10/10_4_4_2_trim_and_fill.htm).
The trim and fill subsection is removed from the current version.
Best,
Ali
--
Ali Zia-Tohidi MSc
Clinical Psychology
University of Tehran