[R-meta] An issue with selmodel( type="step")

Thu, Mar 28, 2024 4:57 AM

There was another update (4.5-13) where I also added a new argument to selmodel() called 'decreasing' -- when set to TRUE, it forces the delta values to be a monotonically decreasing function of the p-values. And I just pushed another update, but this one was mostly just to update metafor to version 4.6-0 for a new CRAN release.

If you simulated only significant estimates, then one should heed the warning that is issued by the function that results from step function models with empty intervals should be treated with caution.

As for the beta selection model -- it isn't easy to make this model fit (as you noticed) due to some numerical issues when working with a beta distribution. When there are p-values very close to 0, then this can lead to selection weights that want to drift off to infinity. There is a note about this in the docs:

https://wviechtb.github.io/metafor/reference/selmodel.html#note-1

So one has to use numerical fixes to avoid this, which isn't ideal.

As for heterogeneity -- yes, it is difficult to distinguish selection effects and heterogeneity. Random-effects selection models are notoriously difficult to fit and typically require a very large number of studies to yield estimates that are somewhat usable/stable.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis <r-sig-meta-analysis-bounces at r-project.org> On Behalf
Of Will Hopkins via R-sig-meta-analysis
Sent: Thursday, March 28, 2024 08:56
To: 'R Special Interest Group for Meta-Analysis' <r-sig-meta-analysis at r-
project.org>
Cc: Will Hopkins <willthekiwi at gmail.com>
Subject: Re: [R-meta] An issue with selmodel( type="step")

Cool, thanks. I presume enough of those updates were already in the metafor
I downloaded last week (4.5-12), otherwise it wouldn't have run. Is there
another update I should try?

FYI, I tried selmodel(..., type="step", steps=(0.025)) with a similar
simulation of 2500 meta-analyses, all estimates significant, but this time
with small-moderate true heterogeneity (SD=1.5) rather than trivial-small
(SD=0.5). It's still working, although not that well: some remaining bias;
coverage is not very good, in spite of the wide confidence intervals, much
wider than those before adjustment; overall it's better than PEESE. I then
tried type="beta" with the first 100 of the simulations. Again it took more
than 2 hours, and only half of the sims had confidence limits for the fixed
effects. Overall beta would be better, if there was some way to coax it to
give more confidence limits.

Others have already noted how heterogeneity reduces the effectiveness of
adjustment for publication bias, so it's obviously important to include
study and subject characteristics as predictors to try to reduce
heterogeneity to a trivial value.

Will

-----Original Message-----
From: R-sig-meta-analysis <r-sig-meta-analysis-bounces at r-project.org> On
Behalf Of Viechtbauer, Wolfgang (NP) via R-sig-meta-analysis
Sent: Thursday, March 28, 2024 1:10 AM
To: R Special Interest Group for Meta-Analysis
<r-sig-meta-analysis at r-project.org>
Cc: Viechtbauer, Wolfgang (NP)
<wolfgang.viechtbauer at maastrichtuniversity.nl>
Subject: Re: [R-meta] An issue with selmodel( type="step")

I am working on some updates to selmodel() and one of those changes is that
the function now continues to run even if an interval contains no p-values.
As discussed previously, the corresponding delta estimate will then either
try to drift to 0 or to infinity. Results should be treated with caution (as
noted in the output from the function).

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis <r-sig-meta-analysis-bounces at r-project.org>
On Behalf Of Will Hopkins via R-sig-meta-analysis
Sent: Tuesday, March 26, 2024 20:47
To: 'R Special Interest Group for Meta-Analysis'
<r-sig-meta-analysis at r- project.org>
Cc: Will Hopkins <willthekiwi at gmail.com>
Subject: [R-meta] An issue with selmodel( type="step")

Wolfgang, according to the documentation "there must be at least one
observed p-value within each interval to fit this model. If this is
not the case, an error will be issued." When I tried it with
steps=(0.025) for a simulated meta-analysis in which all estimates
were significant (p<0.05), it issued an error ("One or more intervals
do not contain any observed p-values"), but the analysis nevertheless
produced a result. When I ran it with 2500 such simulations, it
produced point estimates for the fixed effects with 2456 simulations,
and confidence limits with 2399. See below for a typical result. The
selection model results show 0.0000 for the estimated probability of
non-significant p values, as expected, so how is it able to make

adjustments?

Type="steps" seems to be about as good as type="beta" for bias and
coverage with this particular set of study characteristics, but the
beta type runs so slowly that I have only used it for 100 simulations for

comparison so far.

It took more than an hour, and only 51/100 produced confidence limits
for the fixed effects, so in these respects the steps type is
definitely better).

Will

Mixed-Effects Model (k = 22; tau^2 estimator: ML)

tau^2 (estimated amount of residual heterogeneity): 0.2541 (SE = 0.2288)
tau (square root of estimated tau^2 value):         0.5041

Test for Residual Heterogeneity:
LRT(df = 1) = 4.9738, p-val = 0.0257

Test of Moderators (coefficients 1:2):
QM(df = 2) = 94.2813, p-val < .0001

Model Results:

               estimate      se    zval    pval    ci.lb   ci.ub
xxx$SexFemale    2.9331  0.3022  9.7046  <.0001   2.4359  3.4302  ***
xxx$SexMale      1.1294  0.7376  1.5312  0.1257  -0.0838  2.3425

Test for Selection Model Parameters:
LRT(df = 1) = 22.8488, p-val < .0001

Selection Model Results:

                     k  estimate   se  zval  pval  ci.lb  ci.ub
0     < p <= 0.025  22    1.0000  ---   ---   ---    ---    ---
0.025 < p <= 1       0    0.0000   NA    NA    NA     NA     NA

[R-meta] An issue with selmodel( type="step")

Thread (4 messages)