[R-meta] Confusion about how to use UN structure

Hi Simon,

Like Wolfgang, I don't know of any simulation studies that look at this
situation, so it's hard to say what the best approach would be. But I'll
offer a few (somewhat speculative) observations:

First, to your question about the meaning of the single rho returned when
using "HCS" or "CS", I would *not* characterize it as an average
correlation among all pairs of levels. It's the (restricted) maximum
likelihood estimate of the correlation between random effects for different
moderator categories, under whatever structure is assumed. Suppose there's
adequate data to estimate a correlation between categories A and B, but
little or no data about the correlations between categories A and C or
between B and C. Then we can't really say anything empirical about the AC
or BC correlations. All we can do is make assumptions. But perhaps HCS or
CS is reasonable as a very rough "working" model, which at least
acknowledges that there is dependence between effects drawn from different
moderator categories within the same study.

Second, the choice of whether to fix a rho value to zero has implications
for how the average effects for a given moderator category are estimated.
Consider a scenario with a three-level moderator and with the slightly
simpler meta-regression specification where you just have intercept terms
for each moderator category (but no other predictors):
 rma.mv(eff_size ~ UN_MOD -1, V = V, struct = "UN", random = ~ UN_MOD |
study, data = data)
If you set rho_AC and rho_BC to zero, then the average effect for category
A is going to get estimated using only the effect size estimates from
category A. The estimated average effect should be equivalent (up to
numerical error) to the effect you get from a random effects meta-analysis
using only the category A effects. The effect size estimates from the other
two categories don't influence the results at all.

On the other hand, if you set rho_AC to a non-zero value (or make the
assumption that it is equivalent to a different correlation, such as
rho_AB), then the model will "borrow strength" from the effect size
estimates in category C for any study that has both A and C results, and so
the average effect for category A will be influenced (at least a little
bit) by the ES estimates in category C. The degree of influence depends on
how many studies include categories A and C together, and on the size of
the assumed rho_AC. If you set *all* of the correlations to zero (or
equivalently, use struct = "DIAG"), then the average effect estimates are
based only on the ES estimates for that category---that is, they're based
only on the direct evidence, with no borrowing of strength across
categories. If you were to estimate rho_AB but set rho_AC and rho_BC to
zero, then you would get some borrowing of strength between categories A
and B but none with category C. In my "expanding the range of working
models" paper, we call the model where all correlations are zero the
"subgroup correlated effects" working model because it is equivalent to
running separate models on each category of the moderator.

So to figure out how to proceed, you might consider whether the idea of
borrowing of strength across categories of the moderator makes practical
sense. If it does, then I would probably go with HCS or CS (paired with RVE
for insurance against mis-specification of the random effects structure).
If it seems like a weird thing to do, then I would go with the subgroup
correlated effects model that uses only the direct evidence. Of course, you
could also report both, which would have the benefit of allowing others to
judge the extent to which the findings are influenced by borrowing of
strength across categories.

James

[R-meta] Confusion about how to use UN structure

Thread (12 messages)