Skip to content

[R-meta] An issue with selmodel( type="step")

4 messages · Will Hopkins, Wolfgang Viechtbauer

Will Hopkins
# 
Wolfgang, according to the documentation "there must be at least one
observed p-value within each interval to fit this model. If this is not the
case, an error will be issued." When I tried it with steps=(0.025) for a
simulated meta-analysis in which all estimates were significant (p<0.05), it
issued an error ("One or more intervals do not contain any observed
p-values"), but the analysis nevertheless produced a result. When I ran it
with 2500 such simulations, it produced point estimates for the fixed
effects with 2456 simulations, and confidence limits with 2399. See below
for a typical result. The selection model results show 0.0000 for the
estimated probability of non-significant p values, as expected, so how is it
able to make adjustments?

 

Type="steps" seems to be about as good as type="beta" for bias and coverage
with this particular set of study characteristics, but the beta type runs so
slowly that I have only used it for 100 simulations for comparison so far.
It took more than an hour, and only 51/100 produced confidence limits for
the fixed effects, so in these respects the steps type is definitely
better). 

 

Will

 

Mixed-Effects Model (k = 22; tau^2 estimator: ML)

 

tau^2 (estimated amount of residual heterogeneity): 0.2541 (SE = 0.2288)

tau (square root of estimated tau^2 value):         0.5041

 

Test for Residual Heterogeneity:

LRT(df = 1) = 4.9738, p-val = 0.0257

 

Test of Moderators (coefficients 1:2):

QM(df = 2) = 94.2813, p-val < .0001

 

Model Results:

 

               estimate      se    zval    pval    ci.lb   ci.ub      

xxx$SexFemale    2.9331  0.3022  9.7046  <.0001   2.4359  3.4302  *** 

xxx$SexMale      1.1294  0.7376  1.5312  0.1257  -0.0838  2.3425      

 

Test for Selection Model Parameters:

LRT(df = 1) = 22.8488, p-val < .0001

 

Selection Model Results:

 

                     k  estimate   se  zval  pval  ci.lb  ci.ub   

0     < p <= 0.025  22    1.0000  ---   ---   ---    ---    ---   

0.025 < p <= 1       0    0.0000   NA    NA    NA     NA     NA
Wolfgang Viechtbauer
# 
I am working on some updates to selmodel() and one of those changes is that the function now continues to run even if an interval contains no p-values. As discussed previously, the corresponding delta estimate will then either try to drift to 0 or to infinity. Results should be treated with caution (as noted in the output from the function).

Best,
Wolfgang
Will Hopkins
# 
Cool, thanks. I presume enough of those updates were already in the metafor
I downloaded last week (4.5-12), otherwise it wouldn't have run. Is there
another update I should try?

FYI, I tried selmodel(..., type="step", steps=(0.025)) with a similar
simulation of 2500 meta-analyses, all estimates significant, but this time
with small-moderate true heterogeneity (SD=1.5) rather than trivial-small
(SD=0.5). It's still working, although not that well: some remaining bias;
coverage is not very good, in spite of the wide confidence intervals, much
wider than those before adjustment; overall it's better than PEESE. I then
tried type="beta" with the first 100 of the simulations. Again it took more
than 2 hours, and only half of the sims had confidence limits for the fixed
effects. Overall beta would be better, if there was some way to coax it to
give more confidence limits.

Others have already noted how heterogeneity reduces the effectiveness of
adjustment for publication bias, so it's obviously important to include
study and subject characteristics as predictors to try to reduce
heterogeneity to a trivial value.

Will

-----Original Message-----
From: R-sig-meta-analysis <r-sig-meta-analysis-bounces at r-project.org> On
Behalf Of Viechtbauer, Wolfgang (NP) via R-sig-meta-analysis
Sent: Thursday, March 28, 2024 1:10 AM
To: R Special Interest Group for Meta-Analysis
<r-sig-meta-analysis at r-project.org>
Cc: Viechtbauer, Wolfgang (NP)
<wolfgang.viechtbauer at maastrichtuniversity.nl>
Subject: Re: [R-meta] An issue with selmodel( type="step")

I am working on some updates to selmodel() and one of those changes is that
the function now continues to run even if an interval contains no p-values.
As discussed previously, the corresponding delta estimate will then either
try to drift to 0 or to infinity. Results should be treated with caution (as
noted in the output from the function).

Best,
Wolfgang
adjustments?
comparison so far.
_______________________________________________
R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org To
manage your subscription to this mailing list, go to:
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
Wolfgang Viechtbauer
# 
There was another update (4.5-13) where I also added a new argument to selmodel() called 'decreasing' -- when set to TRUE, it forces the delta values to be a monotonically decreasing function of the p-values. And I just pushed another update, but this one was mostly just to update metafor to version 4.6-0 for a new CRAN release.

If you simulated only significant estimates, then one should heed the warning that is issued by the function that results from step function models with empty intervals should be treated with caution.

As for the beta selection model -- it isn't easy to make this model fit (as you noticed) due to some numerical issues when working with a beta distribution. When there are p-values very close to 0, then this can lead to selection weights that want to drift off to infinity. There is a note about this in the docs:

https://wviechtb.github.io/metafor/reference/selmodel.html#note-1

So one has to use numerical fixes to avoid this, which isn't ideal.

As for heterogeneity -- yes, it is difficult to distinguish selection effects and heterogeneity. Random-effects selection models are notoriously difficult to fit and typically require a very large number of studies to yield estimates that are somewhat usable/stable.

Best,
Wolfgang