-----Original Message-----
From: R-sig-meta-analysis <r-sig-meta-analysis-bounces at r-project.org> On Behalf
Of Will Hopkins via R-sig-meta-analysis
Sent: Thursday, March 28, 2024 01:47
To: 'R Special Interest Group for Meta-Analysis' <r-sig-meta-analysis at r-
project.org>
Cc: Will Hopkins <willthekiwi at gmail.com>
Subject: Re: [R-meta] Calculation of p values in selmodel
Thanks for the response, Wolfgang. I was using the usual p values for the
two-sided nil-hypothesis test calculated via the t statistic for a mean
change in a normally distributed continuous variable. I simulated
publication bias by excluding a defined proportion (e.g., 90%) of study
estimates that were NOT(P<0.05 AND positive), i.e., by excluding a
proportion of study estimates that were P>0.05 OR negative. In other words,
given the simulated means and sampling variability, very occasionally a
statistically significant negative effect was included. I think this is the
way publication based on significance would work: when everyone is expecting
positive effects, the occasional significant negative effect would be as
unlikely to get published as a non-significant effect. If I had excluded all
significant negative effects, and I then wanted to simulate only minor
publication bias (e.g., exclude only 10% of non-significant effects), it
would be unrealistic to still exclude ALL significant negative effects, so I
would have to start making an arbitrary decision about including some
significant negative effects. Anyway, there were very few significant
negative effects in the simulations. Selmodel worked fine for adjusting the
bias associated with steps=(0.025) in the simulations. When the only change
I made was to add the pval= option, the adjusted mean effects showed more
bias, not less, as I described in my earlier posting. So I dunno what's
going on.
Will
-----Original Message-----
From: R-sig-meta-analysis <r-sig-meta-analysis-bounces at r-project.org> On
Behalf Of Viechtbauer, Wolfgang (NP) via R-sig-meta-analysis
Sent: Thursday, March 28, 2024 1:17 AM
To: R Special Interest Group for Meta-Analysis
<r-sig-meta-analysis at r-project.org>
Cc: Viechtbauer, Wolfgang (NP)
<wolfgang.viechtbauer at maastrichtuniversity.nl>
Subject: Re: [R-meta] Calculation of p values in selmodel
In the simulation study I did, I did not find any noteworthy difference when
I used the standard Wald-type test p-values versus those from a 'proper'
t-test (for a meta-analysis of standardized mean differences). Note that
when supplying p-values, they should correspond in type (one-sided versus
two-sided and in the proper direction when one-sided) as what is being
used/assumed in the selection model. Otherwise, results would be total
nonsense. Not sure what kind of p-values you were supplying to the function.
Best,
Wolfgang