[R-meta] methods for assessing publication bias while accounting for dependency
Hi all, To add the Copas selection model to the models already suggested: This model combines the usual random effects model with a selection model that models how the probability of publication depends on both a study's effect size and its standard error. In a sensitivity analysis it is investigated how effect estimates are expected to change with increasing level of selection. A goodness-of-fit test provides a plausible selection level, along with a corrected effect size, given the data. The Copas selection model is implemented in R package metasens https://cran.r-project.org/web/packages/metasens/ For the implementation, see Carpenter et al. https://www.jclinepi.com/article/S0895-4356(08)00348-X/fulltext . For a simulation study, see https://onlinelibrary.wiley.com/doi/10.1002/bimj.201000151 . Best, Gerta Am 28.02.2022 um 21:44 schrieb James Pustejovsky:
In addition to Wolfgang's and Lukasz's suggestions, I would add that I find the Mathur and Vanderweele approach pretty compelling. It is not exactly a "bias adjustment" technique (as Trim and Fill or PET/PEESE purport to be) but rather a sensitivity analysis, which examines hypothetical questions such as: * Supposing that statistically significant results are at most X times more likely to be published than non-significant results, what is the maximum degree of bias that would be expected in the overall average effect size estimate? * How strong would the selective publication process need to be to reduce the overall average effect size estimate to no more than Y? An interesting implication of their results is that there are scenarios where an overall average effect size cannot possibly be reduced to null, even with very extreme forms of selective publication. James On Mon, Feb 28, 2022 at 2:28 PM Lukasz Stasielowicz < lukasz.stasielowicz at uni-osnabrueck.de> wrote:
Dear Brendan, unsurprisingly Wolfgang was faster than me so I'll just add one more reference (with further references) if your curious about the problems of some methods (e.g. trim and fill) even in a basic two-level meta-analysis: Carter, E. C., Sch?nbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019). Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods. Advances in Methods and Practices in Psychological Science, 115?144. https://doi.org/10.1177/2515245919847196 One other possibility to address publication bias when dealing with dependent effect sizes is to conduct a moderator analysis comparing journal articles with other sources (e.g. conference proceedings, dissertations). If one is willing to assume that the latter are more similar to unpublished literature than journal articles then the results of this moderator analysis approximate the mangnitude of publication bias. Obviously, it is only some kind of sensitivity analysis and not the perfect estimate of publication bias. Best, Lukasz -- Lukasz Stasielowicz Osnabr?ck University Institute for Psychology Research methods, psychological assessment, and evaluation Seminarstra?e 20 49074 Osnabr?ck (Germany) Am 28.02.2022 um 19:45 schrieb r-sig-meta-analysis-request at r-project.org:
Send R-sig-meta-analysis mailing list submissions to
r-sig-meta-analysis at r-project.org
To subscribe or unsubscribe via the World Wide Web, visit
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis or, via email, send a message with subject or body 'help' to r-sig-meta-analysis-request at r-project.org You can reach the person managing the list at r-sig-meta-analysis-owner at r-project.org When replying, please edit your Subject line so it is more specific than "Re: Contents of R-sig-meta-analysis digest..." Today's Topics: 1. Re: methods for assessing publication bias while accounting for dependency (Viechtbauer, Wolfgang (SP)) 2. Re: Heterogeneity and moderated mediation (Michael Dewey) 3. Re: Meta-analysis of prevalence data: back-transformation and polytomous data (Viechtbauer, Wolfgang (SP)) 4. Re: Importing Correlations from PDF to table format (Kiet Huynh) ---------------------------------------------------------------------- Message: 1 Date: Mon, 28 Feb 2022 13:31:46 +0000 From: "Viechtbauer, Wolfgang (SP)" <wolfgang.viechtbauer at maastrichtuniversity.nl> To: Brendan Hutchinson <Brendan.Hutchinson at anu.edu.au>, "r-sig-meta-analysis at r-project.org" <r-sig-meta-analysis at r-project.org> Subject: Re: [R-meta] methods for assessing publication bias while accounting for dependency Message-ID: <2377cc39202643a0ac5d87a34fce3cda at UM-MAIL3214.unimaas.nl> Content-Type: text/plain; charset="iso-8859-1" Dear Brendan, Using the 'regression method' approach could also be regarded as a form
of sensitivity analysis, when focusing on the model intercept as an estimate of the 'adjusted' effect (as in the PET/PEESE methods). In fact, if I recall the findings from various simulation studies, this seems to work better than the trim and fill method.
One can also aggregate the estimates to the study level (or to whatever
level needed so that the resulting aggregated values can be assumed to be independent) and then run methods that assume independence on these aggregated data (including trim and fill).
Another recent method by James Pustejovsky:
Some other relevant readings: Fern?ndez-Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N.,
Onghena, P. & Van den Noortgate, W. (2021). Detecting selection bias in meta-analyses with multiple outcomes: A simulation study. The Journal of Experimental Education, 89(1), 125-144. https://doi.org/10.1080/00220973.2019.1582470
Rodgers, M. A. & Pustejovsky, J. E. (2021). Evaluating meta-analytic
methods to detect selective reporting in the presence of dependent effect sizes. Psychological Methods, 26(2), 141-160. https://doi.org/10.1037/met0000300
P.S.: Please use meaningful post titles to make the mailing list
archives more useful.
Best, Wolfgang
-----Original Message----- From: R-sig-meta-analysis [mailto:
r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of Brendan Hutchinson Sent: Friday, 25 February, 2022 14:15 To: r-sig-meta-analysis at r-project.org Subject: [R-meta] (no subject) Dear mailing list, I have a couple of minor questions regarding methods for assessing
publication
bias while accounting for dependency. To my understanding, there is no means of running a publication bias
analysis,
such as trim and fill, with a multilevel meta-analytic model in R (or a
model in
which dependency issues need be accounted for). I am aware that one can
use a
regression method, such as regressing the standard error onto the
summary
estimate, within a multi-level model (this is fairly straightforward
using
rma.mv(), for example). However, what about methods for assessing the
robustness
of findings, if publication bias is a concern (such as trim and fill),
while also
accounting for dependency? The best I have found is a recent package "PublicationBias" by Mathur
and
VanderWeele (10.1111/rssc.12440). I am wondering if anyone has any recommendations for particular
methods, R
packages, or readings? Thanks so much! Brendan
------------------------------
Message: 2
Date: Mon, 28 Feb 2022 14:05:08 +0000
From: Michael Dewey <lists at dewey.myzen.co.uk>
To: Amy Zadow <Amy.Zadow at unisa.edu.au>, R meta
<r-sig-meta-analysis at r-project.org>
Subject: Re: [R-meta] Heterogeneity and moderated mediation
Message-ID: <e560f46d-9887-d498-8c01-fa63b87fae24 at dewey.myzen.co.uk>
Content-Type: text/plain; charset="windows-1252"; Format="flowed"
It is hard to comment in detail as we do not have any information about
the meta-analysis you ran. Are there two separate analyses, one for
groups and one for individuals, two separate data-sets, one for groups
and one for individuals, or one analysis using a multi-level
meta-analysis? Presumably that is all replicated four times for each PSC
(whatever that is) but that could equally be another level in the
multi-level mode.
Would the nature of the research environment and study design have
caused you to believe that heterogeneity was expected or unlikely?
Michael
On 28/02/2022 06:50, Amy Zadow wrote:
Hello, I am seeking advice about my current results ? any comments/criticism/advice around the heterogeneity statistics would be very helpful Also I would be keen to conduct a moderated mediation but not sure where to start. Any advice/ recommended resources or code would be much appreciated. Many thanks, Amy
> _______________________________________________ > R-sig-meta-analysis mailing list > R-sig-meta-analysis at r-project.org
<mailto:R-sig-meta-analysis at r-project.org>
>
[[alternative HTML version deleted]]
_______________________________________________ R-sig-meta-analysis mailing list R-sig-meta-analysis at r-project.org <mailto:
R-sig-meta-analysis at r-project.org>
http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient
Virus-free. www.avg.com <
http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
_______________________________________________ R-sig-meta-analysis mailing list R-sig-meta-analysis at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
_______________________________________________ R-sig-meta-analysis mailing list R-sig-meta-analysis at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
[[alternative HTML version deleted]]
_______________________________________________ R-sig-meta-analysis mailing list R-sig-meta-analysis at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
Dr. rer. nat. Gerta R?cker, Dipl.-Math. Guest Scientist Institute of Medical Biometry and Statistics, Faculty of Medicine and Medical Center - University of Freiburg Zinkmattenstr. 6a, D-79108 Freiburg, Germany Mail: ruecker at imbi.uni-freiburg.de Homepage: https://www.uniklinik-freiburg.de/imbi-en/employees.html?imbiuser=ruecker