[R-meta] metafor::rma-function: Statistically significant interaction, but increased tau2 – and how to get the slope from the output
Dear Sabrina Just a side issue but the journals I review for usually take the line that deviating from the protocol is fine as long as you say you have done so and why you did it so readers can tell whether it was just data-driven. Michael
On 02/08/2019 14:25, Sabrina Mai Nielsen wrote:
Thanks a lot for very fast response!
1) Ok, I am happy to hear that it is a known issue and not necessarily caused by errors in my coding.
2) Perfect, that makes sense. Thanks a lot for making that clear for me.
3) Thanks a lot for the suggestions - I will try those models out! For my study, however, I have already protocolized the model I am using, so I may have to stick with that.
- For the model that allows tau^2 to be different for every level of id, I do get the same as with the 'subset' model, as you said - cool!
- The multilevel model adding random effects for meta-analyses was definitely among our considerations, however, we found it rather complex (e.g. by resulting in two variance components, sigma^2.1 and sigma^2.2) and we ended up choosing fixed effects for meta-analyses.
- The profile plots are peaking nicely for the two first plots (?^2 and tau1^2), but not for the last two plots.
Thanks again
Best,
Sabrina
-----Oprindelig meddelelse-----
Fra: Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer at maastrichtuniversity.nl>
Sendt: 1. august 2019 22:06
Til: Sabrina Mai Nielsen <sabrina.mai.nielsen at regionh.dk>; r-sig-meta-analysis at r-project.org
Emne: RE: metafor::rma-function: Statistically significant interaction, but increased tau2 ? and how to get the slope from the output
Hi Sabrina,
1) While conceptually, tau^2 should never increase when adding moderators to a meta-regression model, this can happen in practice (as in your case). It's a bit weird when this happens, especially when the moderators added (or interactions in your case) are actually significant. I can't tell you why this happens in your case specifically.
2) The slope for id 8 is 0.0114 + (-0.0718). When you select only the studies for id 8, then you allow tau^2 to be different in that subset (and hence different from tau^2 for the entire collection) and that can cause such a discrepancy. And it is just a coincidence that -0.0718 is closer to -0.0940. And yes, it can happen that the slopes in the subsets are not significant, but that some of the *differences* between the slope for id 1 and the slopes for the other ids are.
You can also fit a model where you allow tau^2 to be different for every level of id. That would be:
d$row <- 1:nrow(d)
rma.mv(logOR, logOR.var, mods = ~ CF1_Women..IMP * id, random = ~ factor(id) | row, struct="DIAG", data=d, btt=21:38, verbose=TRUE)
It could take a while to fit that model, since 19 different values of tau^2 will need to be estimated (verbose=TRUE will keep you updated on the progress). Once the model converges, you should find that the results are now (essentially) identical to what you find if you fit the 'subset' models, such as:
rma(logOR, logOR.var, mods= ~ CF1_Women..IMP, data=d, subset=id==8)
3) You might want to take another approach here, since the data have a multilevel structure (with studies nested within meta-analyses). A multilevel model adding random effects for meta-analyses and studies within meta-analyses can be fitted with:
rma.mv(logOR, logOR.var, mods = ~ CF1_Women..IMP, random = ~ 1 | id/row, data=d)
This model assumes that the slope for CF1_Women..IMP is identical across meta-analyses (so it is analogous to the model without interactions). To examine if the slope varies across meta-analyses, you could use a model with random slopes:
rma.mv(logOR, logOR.var, mods = ~ CF1_Women..IMP, random = ~ CF1_Women..IMP | id, struct="GEN", data=d)
This fits a model with random intercepts and slopes for meta-analyses (with a var-cov structure that allows intercepts and slopes to be correlated). Note: The "GEN" structure is currently undocumented, as I still need to make a few of the method functions work with it. anova() for a model comparison should work though.
You might even want to try:
res <- rma.mv(logOR, logOR.var, mods = ~ CF1_Women..IMP, random = list(~ CF1_Women..IMP | id, ~ 1 | row), struct="GEN", data=d) res
which also adds random effects for studies within meta-analyses. I am not entirely sure right now if this is an overparameterized model, so you would want to profile all variance/correlation components of this model with:
profile(res)
This could take a while, so be patient. All profile plots should have a peak at the estimate. If one or more of the profile likelihoods are flat around the estimate, then you know that the model is overparameterized.
If you pursue these models, I would be interested in some feedback as to how this worked out (and how those profile plots look).
Best,
Wolfgang
-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of Sabrina Mai Nielsen
Sent: Thursday, 01 August, 2019 20:52
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] metafor::rma-function: Statistically significant interaction, but increased tau2 ? and how to get the slope from the output
I am having troubles making sense of a REML-based meta-regression with an interaction between a factor and numeric variable, fitted with rma.
In my study, I have included RCTs from 19 meta-analyses to investigate the association between %women in the RCTs and the effect sizes (i.e. logOR).
The slope for %women, without interaction, is not significant (see below, M_REML0, p=0.6858), and I am now investigating the interaction between %women (CF1_Women..IMP) and the ID for the meta-analysis (i.e. if the slope for %women varies in the different meta-analyses).
1) The interaction seem significant when testing for moderators, but the between-trial variation, tau^2, does not decrease as expected. Why is that?
(When I try method="ML" instead of REML, then tau^2 does decrease as expected; see anova(M0,M1) below)
M_REML0 (without interaction), tau^2 = 0.2076
M_REML1 (with an interaction), tau^2 = 0.2127, Test of Moderators (interaction terms) p-val = 0.0044 (see full outputs below)
2) I am in doubt how to get the estimate for the slopes ? e.g. for meta-analysis with ID 8.
estimate se zval pval ci.lb ci.ub
intrcpt -0.3567 1.0739 -0.3322 0.7398 -2.4616 1.7481
CF1_Women..IMP 0.0114 0.0139 0.8244 0.4097 -0.0157 0.0386
(...)
CF1_Women..IMP:id8 -0.0718 0.0214 -3.3572 0.0008 -0.1137 -0.0299 ***
(see full output below)
I would think it would be calculated as CF1_Women..IMP + CF1_Women..IMP:id8, i.e. 0.0114 +(-0.0718) = -0.0604 (with corresponding 95%CI of -0.1294 to 0.0087). Is that correct?
Or is the slope for ID 8 simply the estimate reported for CF1_Women..IMP:id8, i.e. (-0.0718, 95%CI -0.1137 to -0.0299)?
I am asking because when I do a meta-regression with only RCTs for ID 8, I get -0.0940 (95%CI -0.1864 to -0.0016; output not shown), which is more similar to the CF1_Women..IMP:id8 estimate.
Also, when I calculate the slopes for all the meta-analysis IDs, none of them are significantly different from 0, despite the interaction term is significant. I guess that is possible, but it makes me doubt the way I calculated the slopes.
Thanks a lot in advance! Your help is very appreciated!
Best,
Sabrina
=======A FEW RELEVANT OUTPUTS=======
M_REML0<-rma(logOR, logOR.var, mods= ~CF1_Women..IMP + id, method="REML", data=d, btt=2) M_REML1<-rma(logOR, logOR.var, mods= ~CF1_Women..IMP * id, method="REML", data=d, btt=c(21:38)) M_REML0
Mixed-Effects Model (k = 212; tau^2 estimator: REML)
tau^2 (estimated amount of residual heterogeneity): 0.2076 (SE = 0.0357)
tau (square root of estimated tau^2 value): 0.4556
I^2 (residual heterogeneity / unaccounted variability): 66.60%
H^2 (unaccounted variability / sampling variability): 2.99
R^2 (amount of heterogeneity accounted for): 63.70%
Test for Residual Heterogeneity:
QE(df = 192) = 545.9074, p-val < .0001
Test of Moderators (coefficient 2):
QM(df = 1) = 0.1637, p-val = 0.6858
Model Results:
estimate se zval pval ci.lb ci.ub
intrcpt 0.4156 0.3105 1.3383 0.1808 -0.1931 1.0242
CF1_Women..IMP 0.0013 0.0032 0.4046 0.6858 -0.0049 0.0075
id2 0.0672 0.3965 0.1695 0.8654 -0.7099 0.8443
id3 0.5059 0.2521 2.0066 0.0448 0.0117 1.0000 *
id4 0.1753 0.2739 0.6402 0.5221 -0.3615 0.7121
id5 0.2610 0.2558 1.0201 0.3077 -0.2404 0.7624
id6 0.0351 0.2763 0.1272 0.8988 -0.5065 0.5768
id7 0.8416 0.3464 2.4295 0.0151 0.1626 1.5205 *
id8 0.1775 0.2723 0.6519 0.5145 -0.3562 0.7112
id9 -0.3137 0.2684 -1.1690 0.2424 -0.8398 0.2123
id10 -0.8378 0.2723 -3.0765 0.0021 -1.3715 -0.3040 **
id11 0.2878 0.2106 1.3665 0.1718 -0.1250 0.7006
id12 0.1932 0.3449 0.5603 0.5753 -0.4827 0.8692
id13 0.1538 0.2959 0.5199 0.6031 -0.4261 0.7337
id14 -1.0020 0.3432 -2.9195 0.0035 -1.6747 -0.3293 **
id15 -0.6162 0.2739 -2.2497 0.0245 -1.1531 -0.0794 *
id16 -1.5704 0.3394 -4.6269 <.0001 -2.2356 -0.9052 ***
id17 -1.4576 0.2942 -4.9552 <.0001 -2.0341 -0.8811 ***
id18 -0.0457 0.4967 -0.0921 0.9266 -1.0193 0.9278
id19 -0.3440 0.3283 -1.0479 0.2947 -0.9874 0.2994
---
Signif. codes: 0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1
M_REML1
Mixed-Effects Model (k = 212; tau^2 estimator: REML)
tau^2 (estimated amount of residual heterogeneity): 0.2127 (SE = 0.0378)
tau (square root of estimated tau^2 value): 0.4612
I^2 (residual heterogeneity / unaccounted variability): 67.50%
H^2 (unaccounted variability / sampling variability): 3.08
R^2 (amount of heterogeneity accounted for): 62.80%
Test for Residual Heterogeneity:
QE(df = 174) = 488.8555, p-val < .0001
Test of Moderators (coefficients 21:38):
QM(df = 18) = 37.5693, p-val = 0.0044
Model Results:
estimate se zval pval ci.lb ci.ub
intrcpt -0.3567 1.0739 -0.3322 0.7398 -2.4616 1.7481
CF1_Women..IMP 0.0114 0.0139 0.8244 0.4097 -0.0157 0.0386
id2 8.6017 8.6515 0.9942 0.3201 -8.3549 25.5583
id3 0.7355 1.3494 0.5451 0.5857 -1.9093 3.3804
id4 0.2098 1.2889 0.1628 0.8707 -2.3164 2.7360
id5 1.7079 1.1849 1.4414 0.1495 -0.6144 4.0302
id6 -3.6331 2.2106 -1.6435 0.1003 -7.9658 0.6995
id7 -5.4865 2.7547 -1.9917 0.0464 -10.8857 -0.0874 *
id8 5.2745 1.5782 3.3421 0.0008 2.1812 8.3677 ***
id9 0.4174 1.2215 0.3417 0.7326 -1.9766 2.8114
id10 -0.1295 1.2015 -0.1078 0.9142 -2.4844 2.2253
id11 0.2269 1.7098 0.1327 0.8944 -3.1242 3.5779
id12 -0.7519 2.3333 -0.3222 0.7473 -5.3250 3.8212
id13 -0.0701 3.4723 -0.0202 0.9839 -6.8758 6.7355
id14 -0.1842 2.0870 -0.0882 0.9297 -4.2747 3.9063
id15 0.9002 1.3974 0.6442 0.5194 -1.8386 3.6389
id16 6.4300 7.6265 0.8431 0.3992 -8.5176 21.3776
id17 0.7142 2.1172 0.3373 0.7359 -3.4353 4.8638
id18 -0.4641 1.5171 -0.3059 0.7597 -3.4376 2.5095
id19 1.2194 4.9584 0.2459 0.8057 -8.4989 10.9376
CF1_Women..IMP:id2 -0.1243 0.1269 -0.9795 0.3274 -0.3730 0.1244
CF1_Women..IMP:id3 -0.0026 0.0177 -0.1480 0.8824 -0.0374 0.0322
CF1_Women..IMP:id4 0.0012 0.0174 0.0672 0.9464 -0.0329 0.0353
CF1_Women..IMP:id5 -0.0207 0.0157 -1.3162 0.1881 -0.0515 0.0101
CF1_Women..IMP:id6 0.0533 0.0308 1.7332 0.0831 -0.0070 0.1136 .
CF1_Women..IMP:id7 0.0690 0.0313 2.2031 0.0276 0.0076 0.1304 *
CF1_Women..IMP:id8 -0.0718 0.0214 -3.3572 0.0008 -0.1137 -0.0299 ***
CF1_Women..IMP:id9 -0.0095 0.0162 -0.5858 0.5580 -0.0413 0.0223
CF1_Women..IMP:id10 -0.0092 0.0156 -0.5907 0.5547 -0.0399 0.0214
CF1_Women..IMP:id11 0.0004 0.0217 0.0167 0.9867 -0.0422 0.0430
CF1_Women..IMP:id12 0.0112 0.0290 0.3852 0.7001 -0.0456 0.0679
CF1_Women..IMP:id13 0.0042 0.0496 0.0856 0.9318 -0.0929 0.1014
CF1_Women..IMP:id14 -0.0108 0.0281 -0.3837 0.7012 -0.0658 0.0443
CF1_Women..IMP:id15 -0.0207 0.0186 -1.1133 0.2656 -0.0570 0.0157
CF1_Women..IMP:id16 -0.0834 0.0777 -1.0733 0.2831 -0.2356 0.0689
CF1_Women..IMP:id17 -0.0252 0.0239 -1.0543 0.2918 -0.0721 0.0217
CF1_Women..IMP:id18 0.0186 0.0274 0.6814 0.4956 -0.0350 0.0723
CF1_Women..IMP:id19 -0.0203 0.0637 -0.3189 0.7498 -0.1452 0.1046
---
Signif. codes: 0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1
M0<-rma(logOR, logOR.var, mods= ~CF1_Women..IMP + id, method="ML", data=d) M1<-rma(logOR, logOR.var, mods= ~CF1_Women..IMP * id, method="ML", data=d, btt=c(21:38)) anova(M0,M1)
df AIC BIC AICc logLik LRT pval QE tau^2 R^2
Full 39 515.2691 646.1760 533.4086 -218.6346 488.8555 0.1182
Reduced 21 520.4230 590.9113 525.2862 -239.2115 41.1539 0.0014 545.9074 0.1510 21.75%
_______________________________________________ R-sig-meta-analysis mailing list R-sig-meta-analysis at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis --- This email has been checked for viruses by AVG. https://www.avg.com