On March 29, 2019 at 12:10 PM "Viechtbauer, Wolfgang (SP)" <wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
(I get so excited when I talk about this stuff that I (again) hit 'Send' too soon! Sorry -- continued properly below.)
Yes, this is a good reference.
I will try to address a few of your questions.
1) What does it means if the pseudo R^2 statistic is negative?
R^2 type measures in meta-analysis are typically computed as proportional reductions in variance components (that reflect heterogeneity) when predictors/moderators are added to the model. However, for the models we typically use for a meta-analysis, there is no guarantee that the variance components always decrease (or remain unchanged) when we add predictors. So, it can happen that a variance component increases, in which case the R^2 value is technically negative. An example:
library(metafor)
dat <- escalc(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, data=dat.bcg)
res0 <- rma(yi, vi, data=dat)
res1 <- rma(yi, vi, mods = ~ alloc, data=dat)
res1
# proportional 'reduction' in tau^2 (how R^2 is computed here)
(res0$tau2 - res1$tau2) / res0$tau2
If you look at 'res1', R^2 is given as 0%, since negative R^2 values are simply set to 0.
So what does this mean? I would say it simply means that there is no reduction in the estimate of tau^2, so 'alloc' does not account for any heterogeneity.
But also keep in mind that the value of R^2 is an estimate that can be quite inaccurate esp. if the number of studies is small (https://onlinelibrary.wiley.com/doi/abs/10.1111/bmsp.12002). One may need maybe 30 or even more studies to get a fairly decent estimate (but don't quote me on that number).
Another important point here is that we compute R^2 based on how much *heterogeneity* is accounted for, not how much of the *total variance* is accounted for. So, for example:
res1 <- rma(yi, vi, mods = ~ ablat, data=dat)
res1
This suggests that ~76% of the heterogeneity is accounted for by 'ablat' (but again, this estimate could be way off with k=13). But the total variance in this example is tau^2 plus the amount of sampling variability in the estimates (the 'vi' values). Since each study has a different amount of sampling variability, it isn't super clear how we then should think of the total variance, but one commonly used approach is to compute a 'typical' sampling variance and use that to define the total variance. For example, Higgins and Thompson (2002) suggest to compute the 'typical' sampling variance with:
k <- res1$k
wi <- 1/dat$vi
vt <- (k-1) / (sum(wi) - sum(wi^2)/sum(wi))
vt
Others (https://academic.oup.com/aje/article/150/2/206/55325) have suggested using the harmonic mean, which would be:
1/mean(wi)
Regardless, we can then use tau^2 + vt as the total variance. Then how much of the *total variance* does 'ablat' account?
((res0$tau2 + vt) - (res1$tau2 + vt)) / (res0$tau2 + vt)
That yields about 70%. Why don't we do that? For one, we like larger R^2 values :) But study-level moderators cannot account for sampling variability, so one could also argue it is a bit unfair to see how much of the total variability is accounted for.
2) How can we think of variance accounted for by random effects?
Another issue in this discussion is the question of how to think of variance accounted for by random effects. It makes no sense to ask how much heterogeneity is accounted for by tau^2 here, since that is always 100%. It makes sense to ask how much of the total variability is due to the random effects. In the context of model 'res0', that would be:
res0$tau2 / (res0$tau2 + vt)
If you look at res0, you will see that this is I^2.
For model 'res1', things are a bit more tricky. One approach would be as follow. We again define res0$tau2 + vt as the total variability (note I am using res0$tau2 here, so the amount of heterogeneity before we include moderators, plus sampling variability). Then:
res1$tau2 / (res0$tau2 + vt)
indicates how much of this total variability is due to the random effects, that is, residual heterogeneity (about 22%). And:
vt / (res0$tau2 + vt)
indicates how much of this total variability is due to sampling variability (about 8%). And by subtraction, we then have:
1 - res1$tau2 / (res0$tau2 + vt) - vt / (res0$tau2 + vt)
which indicates how much of the total variability is due to 'ablat' (about 70%).
This is *not* something that has been done or suggested before, but makes sense and is completely defensible.
I hope this helps.
Best,
Wolfgang