[R-meta] Assessing selection bias / multivariate meta-analysis
Hi Pia, You can ignore the warning messages that you're getting. (We haven't yet worked out how to suppress them in the bootstrapping code.) Your code looks good to me except for one subtle but potentially consequential issue. Based on the multivariate summary meta-analyses, it looks like you have a strongly negative average effect size. If the effects are coded so that negative values represent improvement, then this needs to be taken into account when fitting the selection models. The models implemented in metaselection are based on one-side p-values, where the null is mu <= 0 and the alternative is mu > 0 (i.e., positive effects are improvements). We have not yet implemented an option to change the direction of the null and alternative hypotheses (although it's high on my to-do list). In the mean time, there are a few alternative ways that you could modify the existing code to fit a more appropriate model. Either: a) Recode effect sizes so that positive values correspond to improvement (i.e., take yi = -yi) and interpret the beta estimate accordingly. or b) Change the threshold of the step function to steps = .975 and interpret the lambda (selection parameter) estimate as the relative probability that a statistically significant (and negative) effect size is reported compared to the probability that a non-significant or counter-therapeutic effect size is reported. For instance, if lambda = 4 then this means that a statistically significant, therapeutic effect is four times more likely to be reported than a non-significant or non-therapeutic effect. Again, you only need do (a) or (b) but not both. It's possible that making this change will shift the results in a meaningful way because the revised model will be capturing a more plausible form of selective reporting. James On Thu, Nov 28, 2024 at 11:24?AM Pia-Magdalena Schmidt <
pia-magdalena.schmidt at uni-bonn.de> wrote:
Dear James & Wolfgang, dear all, I have tried the metaselection package for some of my meta-analyses and would be very grateful if you could have a look at the results to see if it looks plausible to you? James, you implemented several models, did I understand correctly that you recommend using the modified 3PSM with bootstrapping? Furthermore, I received warnings after fitting the mod_3PSM_boot (see below). May I ignore them? Results from fitted models with rma.mv, cluster robust & mod_3PSM bootstrapping for two analyses: 1. 1.1 Results rma.mv res_spv <- rma.mv(yi=ES_all_spv$yi, V, random = ~ 1 | id_database/effect_id, data = dat) Multivariate Meta-Analysis Model (k = 45; method: REML) logLik Deviance AIC BIC AICc -52.7333 105.4667 111.4667 116.8192 112.0667 Variance Components: estim sqrt nlvls fixed factor sigma^2.1 0.0619 0.2488 30 no id_database sigma^2.2 0.2018 0.4493 45 no id_database/effect_id Test for Heterogeneity: Q(df = 44) = 187.0060, p-val < .0001 Model Results: estimate se zval pval ci.lb ci.ub -1.0496 0.1073 -9.7845 <.0001 -1.2599 -0.8394 *** 1.2 Results rma.mv cluster robust res_spv.robust <- robust(res_spv, cluster = id_database, clubSandwich = TRUE) Multivariate Meta-Analysis Model (k = 45; method: REML) logLik Deviance AIC BIC AICc -52.7333 105.4667 111.4667 116.8192 112.0667 Variance Components: estim sqrt nlvls fixed factor sigma^2.1 0.0619 0.2488 30 no id_database sigma^2.2 0.2018 0.4493 45 no id_database/effect_id Test for Heterogeneity: Q(df = 44) = 187.0060, p-val < .0001 Number of estimates: 45 Number of clusters: 30 Estimates per cluster: 1-3 (mean: 1.50, median: 1) Model Results: estimate se? tval? df? pval? ci.lb? ci.ub? -1.0496 0.1071 -9.7997 24.28 <.0001 -1.2706 -0.8287 *** 1) results based on cluster-robust inference (var-cov estimator: CR2, approx t-test and confidence interval, df: Satterthwaite approx) 1.3 Results mod_3PSM bootstrapping set.seed(20240916) system.time( mod_3PSM_boot <- selection_model( data = dat, yi = ES_all_spv$yi, sei = sei, cluster = id_database, selection_type = "step", steps = .025, CI_type = "percentile", bootstrap = "multinomial", R = 1999 ) ) print(mod_3PSM_boot, transf_gamma = TRUE, transf_zeta = TRUE) user system elapsed 269.861 0.799 270.760 Warning messages: 1: In optimx.run(par, optcfg$ufn, optcfg$ugr, optcfg$uhess, lower, : Hessian is reported non-symmetric with asymmetry ratio 1.24952908365386e-11 2: Hessian forced symmetric 3: In optimx.run(par, optcfg$ufn, optcfg$ugr, optcfg$uhess, lower, : Hessian is reported non-symmetric with asymmetry ratio 5.66259792559018e-11 4: Hessian forced symmetric 5: In optimx.check(par, optcfg$ufn, optcfg$ugr, optcfg$uhess, lower, : Parameters or bounds appear to have different scalings. This can cause poor performance in optimization. It is important for derivative free methods like BOBYQA, UOBYQA, NEWUOA. 6: In optimx.check(par, optcfg$ufn, optcfg$ugr, optcfg$uhess, lower, : Parameters or bounds appear to have different scalings. This can cause poor performance in optimization. It is important for derivative free methods like BOBYQA, UOBYQA, NEWUOA.
print(mod_3PSM_boot, transf_gamma = TRUE, transf_zeta = TRUE)
param Est SE percentile_lower percentile_upper beta -1.06e+00 0.1703 -1.63e+00 -8.09e-01 tau2 3.07e-01 0.1781 9.40e-02 2.20e+00 lambda1 8.07e+13 0.0604 1.13e+12 4.87e+14 2. 2.1 Results rma.mv res_LOR <- rma.mv(yi=ES_LOR_spv$yi, V, random = ~ 1 | id_database/effect_id, data = dat) Multivariate Meta-Analysis Model (k = 19; method: REML) logLik Deviance AIC BIC AICc -14.8934 29.7867 35.7867 38.4579 37.5010 Variance Components: estim sqrt nlvls fixed factor sigma^2.1 0.0000 0.0000 13 no id_database sigma^2.2 0.1889 0.4347 19 no id_database/effect_id Test for Heterogeneity: Q(df = 18) = 88.4226, p-val < .0001 Model Results: estimate se zval pval ci.lb ci.ub -0.8414 0.1257 -6.6954 <.0001 -1.0877 -0.5951 *** 2.2 Results rma.mv cluster robust res_LOR.robust <- robust(res_LOR, cluster = id_database, clubSandwich = TRUE) Multivariate Meta-Analysis Model (k = 19; method: REML) logLik Deviance AIC BIC AICc -14.8934 29.7867 35.7867 38.4579 37.5010 Variance Components: estim sqrt nlvls fixed factor sigma^2.1 0.0000 0.0000 13 no id_database sigma^2.2 0.1889 0.4347 19 no id_database/effect_id Test for Heterogeneity: Q(df = 18) = 88.4226, p-val < .0001 Number of estimates: 19 Number of clusters: 13 Estimates per cluster: 1-3 (mean: 1.46, median: 1) Model Results: estimate se? tval? df? pval? ci.lb? ci.ub? -0.8414 0.1178 -7.1438 9.08 <.0001 -1.1075 -0.5754 *** 1) results based on cluster-robust inference (var-cov estimator: CR2, approx t-test and confidence interval, df: Satterthwaite approx) 2.3 Results mod_3PSM bootstrapping set.seed(20240916) system.time( mod_3PSM_boot <- selection_model( data = dat, yi = ES_LOR_spv$yi, sei = sei, cluster = id_database, selection_type = "step", steps = .025, CI_type = "percentile", bootstrap = "multinomial", R = 1999 ) ) print(mod_3PSM_boot, transf_gamma = TRUE, transf_zeta = TRUE) user system elapsed 262.519 0.324 262.812 Warning messages: 1: In optimx.run(par, optcfg$ufn, optcfg$ugr, optcfg$uhess, lower, : Hessian is reported non-symmetric with asymmetry ratio 9.13585399793946e-13 2: Hessian forced symmetric 3: In optimx.run(par, optcfg$ufn, optcfg$ugr, optcfg$uhess, lower, : Hessian is reported non-symmetric with asymmetry ratio 1.06505934682683e-11 4: Hessian forced symmetric
print(mod_3PSM_boot, transf_gamma = TRUE, transf_zeta = TRUE)
param Est SE percentile_lower percentile_upper beta -8.13e-01 0.1227 -1.14e+00 -6.07e-01 tau2 1.56e-01 0.0892 1.87e-02 3.67e-01 lambda1 1.12e+13 0.0566 1.36e+11 2.31e+14 Best, Pia On Do, 21 Nov 2024 17:18:28 +0000 Viechtbauer, Wolfgang (NP) via R-sig-meta-analysis <r-sig-meta-analysis at r-project.org> wrote:
Thanks for providing additional info about metaselection. I think this package will be a very important tool in the meta-analytic toolbox! I was inspired by your recent blog post: https://jepusto.com/posts/beta-density-selection-models/ and added the truncated beta selection model also to selmodel(). I took a very quick peak at your code and I think you are using analytic gradients, which helps to speed up / stabilize the model fitting. But it's nice to have two parallel implementations to cross-check the results. Best, Wolfgang
-----Original Message----- From: James Pustejovsky <jepusto at gmail.com> Sent: Thursday, November 21, 2024 15:10 To: R Special Interest Group for Meta-Analysis <r-sig-meta-analysis at r- project.org> Cc: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer at maastrichtuniversity.nl> Subject: Re: [R-meta] Assessing selection bias / multivariate meta-analysis Was going to chime in about the metaselection package (https://github.com/jepusto/metaselection)---it's still under development but the core functionality and documentation is in place. The package implements the bootstrapped selection model as demonstrated in my blog post (https://jepusto.com/posts/cluster-bootstrap-selection-model/), but with a much easier interface and faster calculation; it also implements selection models with cluster-robust standard errors, though these seem to be not as accurate as bootstrapping. Folks are welcome to give the package a try and to reach out with questions or potential bugs if you run into anything. We are working on a paper describing the methods implemented in the package and reporting pretty extensive simulations about their performance. My student Man Chen (now on the faculty at UT Austin) has studied a whole bunch of the available methods for selective reporting bias correction, looking specifically at how they perform in meta-analyses with dependent effect sizes, and proposing adaptations of some of the methods to better acknowledge dependency. Our working paper (on this is here: https://osf.io/preprints/metaarxiv/jq52s Pia asked about a few other possible techniques: - The Egger's test / PET-PEESE approach with cluster-robust variance estimation is reasonable but, as Wolfgang noted, it is not specifically diagnostic about missing studies vs. missing effects. If the effect sizes nested within a given study tend to have similar standard errors, then it will mostly be picking up on association between study sample size and study-level average effect size. And of course, it also has the limitation that this small-study association can be caused by things other than selective reporting. - Mathur & Vanderweele's sensitivity analysis is quite useful, though it does not provide an estimate of the severity of selective reporting (instead, it provides information about the degree of potential bias assuming a specific level of selection). - For 3PSM, the cluster-bootstrap technique implemented in the metaselection package is a way to deal with dependent effects, so it is no longer necessary to use ad hoc approaches like ignoring dependence, aggregating to the study level, or selecting a single effect per study. James On Thu, Nov 21, 2024 at 6:37?AM Viechtbauer, Wolfgang (NP) via R-sig-meta- analysis <mailto:r-sig-meta-analysis at r-project.org> wrote: And I just stumbled across this: https://github.com/jepusto/metaselection James, don't hide all your good work from us! Best, Wolfgang
-----Original Message----- From: R-sig-meta-analysis
<mailto:r-sig-meta-analysis-bounces at r-project.org> On Behalf
Of Viechtbauer, Wolfgang (NP) via R-sig-meta-analysis Sent: Thursday, November 21, 2024 13:21 To: R Special Interest Group for Meta-Analysis
<r-sig-meta-analysis at r-
http://project.org> Cc: Viechtbauer, Wolfgang (NP)
<mailto:wolfgang.viechtbauer at maastrichtuniversity.nl>
Subject: Re: [R-meta] Assessing selection bias /
multivariate meta-analysis
Dear Pia, Generally, I don't think there really is any method
that is going to be a great
choice here. The 'Egger sandwich' (i.e., an Egger type
regression model using
cluster-robust inference methods) is a decent option,
since it logically
generalizes the standard Egger regression method to
this context, but it is
unclear what kind of bias/selection effect this may
pick up (missing studies,
missing estimates within studies, a combination
thereof).
Yes, for the 3PSM, you would have to either ignore the
dependencies or select
one estimate per study (and maybe repeat the latter a
large number of times for
different subsets). I assume you are familiar with these papers. If not,
they are directly relevant:
Rodgers, M. A., & Pustejovsky, J. E. (2021).
Evaluating meta-analytic methods to
detect selective reporting in the presence of
dependent effect sizes.
Psychological Methods, 26(2), 141-160.
Fern?ndez-Castilla, B., Declercq, L., Jamshidi, L.,
Beretvas, S. N., Onghena,
P., & Van den Noortgate, W. (2021). Detecting
selection bias in meta-analyses
with multiple outcomes: A simulation study. The
Journal of Experimental
Education, 89(1), 125-144.
Nakagawa, S., Lagisz, M., Jennions, M. D., Koricheva,
J., Noble, D. W. A.,
Parker, T. H., S?nchez-T?jar, A., Yang, Y., & O'Dea,
R. E. (2022). Methods for
testing publication bias in ecological and
evolutionary meta-analyses. Methods
in Ecology and Evolution, 13(1), 4-21.
I think James is working on some methods related to
this topic:
Best, Wolfgang
-----Original Message----- From: R-sig-meta-analysis
<mailto:r-sig-meta-analysis-bounces at r-project.org> On
Behalf
Of Pia-Magdalena Schmidt via R-sig-meta-analysis Sent: Wednesday, November 20, 2024 21:58 To: mailto:r-sig-meta-analysis at r-project.org Cc: Pia-Magdalena Schmidt
<mailto:pia-magdalena.schmidt at uni-bonn.de>
Subject: [R-meta] Assessing selection bias /
multivariate meta-analysis
Dear all, Although this topic has been discussed several times
and I read the archives
and referenced papers, I?m still not sure how to
assess and possibly correct
for selection bias in multivariate meta-analyses. I used the metafor package and ran meta-analyses
with SMCC as effect size
(all studies used within-designs) and fitted
http://rma.mv models as several
studies report more than one effect size.
Furthermore, I used cluster-robust
methods to examine the robustness of the models. For a subset of my data, I used meta-regressions
with one continuous
moderator. All effect sizes are from published journal
articles. The range of included
studies is between 30 and 6 with a number of effect
sizes between 45 and 10.
Since I want to take the dependencies into account,
I would not use funnel
plots or trim and fill. I wonder if using Egger's
regression test adjusted
for http://rma.mv as well as PET-PEESE and perhaps
the sensitivity analysis
suggested by Mathur & Vanderweele (2020) as well as
3PSM would be a
reasonable way to go? Although the latter would only
use one effect size per
study or an aggregated effect size, right? I would be very grateful for any recommendations! Best, Pia Below is an excerpt from my code: ES_all <- escalc(measure="SMCC", m1i= m1i, sd1i=
sd1i, ni = ni, m2i= m2i,
sd2i= sd2i, pi= pi, ri = ri, data= dat) V <- vcalc(vi=ES_all$vi, cluster=id_database, obs =
effect_id, rho =0.605,
data=dat) res <- http://rma.mv(yi=ES_all$yi, V, random = ~ 1 |
id_database/effect_id, data =
dat) res.robust <- robust(res, cluster = id_database,
clubSandwich = TRUE)
# subset res_LOR <- http://rma.mv(yi=ES_LOR$yi, V, random = ~
1 | id_database/effect_id,
mods = ~ dose, data = dat)
_______________________________________________ R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org To manage your subscription to this mailing list, go to: https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis