Skip to content
Prev 1692 / 5636 Next

[R-meta] large increase of the QM value after applying robust on rma.mv

James Pustejovsky 
Emily,

Thanks for posting such a clear and detailed description of your analysis. I think the issue may be that you are fitting a model with three hierarchical levels of random effects, but then you are clustering the standard errors at an intermediate level (level 2 of 3). This will not generally work (in the sense of producing valid estimates of the SEs for average effect sizes or meta-regression coefficients). The reason is that cluster-robust standard errors (as implemented in metafor and in the clubSandwich package) are based on the assumption that each cluster is independent. This assumption will not hold if you are clustering by study but then using a model that posits dependence between different studies conducted on the same species. Using cluster-robust SEs in this situation is akin to ignoring the dependence at the top level.

I would speculate that this issue might be why you are getting the implausibly large QM values when you use cluster-robust SEs. If the moderator varies at the species level, the species-level variance component will matter a lot in how much variance can be explained by the moderator. But then if the SEs ignore the species-level variation, it might appear as if the moderator has explained it all away. 

To figure out what to do from here, I think the first thing is to determine the level at which each of the moderators vary and count the number of highest-level units for each unique value of the moderator. For example, a study-level moderator with two levels might have 4 species with level A only, 9 species with level B only, and 3 with both A and B (i.e., there are studies of each type for each of these 3 species). Cluster-robust SEs are based on only between-cluster variation, so there have to be an adequate number of clusters at each level of the moderator. If there?s just one species with a given moderator value, then there?s no to cluster by species (short of digging up some more studies).

James

Thread (2 messages)

James Pustejovsky large increase of the QM value after applying robust on rma.mv Sep 2 Lea BRIARD large increase of the QM value after applying robust on rma.mv Sep 3