[R-meta] robust variance estimation with small number of elements within the cluster

Hi Diego,

clubSandwich uses cluster-robust (or "sandwich") variance estimators.
These estimators quantify the uncertainty in an average effect (or fixed
effect in a meta-regression model) *using only the between-cluster
variation* in effect sizes. Therefore, they simply will not work if all of
the information about a given category of effect sizes comes from a single
cluster.

As a rough heuristic, you can think of cluster-robust variance estimators
as involving the following. Consider your example where you have a
multi-level mixed model that has a categorical moderator and omits the
intercept. The model coefficients are therefore interpreted as average
effect sizes for each category of the moderator. Imagine aggregating the
effect sizes to the level of the study, so that there is only one average
effect size per category per study. Denote the aggregated effect for
category c in study j as T_cj. Suppose that there are k_c studies that
include effect sizes in category c. The overall average effect for category
c is then going to be, roughly, a weighted average of the study-level
aggregated effects for category c:

beta_c = sum_{j=1}^{k} w_cj T_cj

for some weights w_cj. (This is a rough approximation because, if you're
using a multilevel model, the estimate beta_c will actually also involve
the average effect sizes for categories other than c. But let's ignore that
wrinkle for purposes of building intuition.)

The robust estimator of the sampling variance of beta_c is going to be a
weighted version of the sample variance of the T_cj's:

V_c = sum_{j=1}^k (w_{cj})^2 (T_cj - beta_c)^2

If the weights are roughly equal, then the robust estimator ends up having
the even simpler form

V_c = S_c^2 / k_c,

where S_c^2 is the sample variance of the T_cj's. As you can see from
this, if there is only one study that includes effect size estimates in
category c, then S_c^2 will be zero by definition, but it can't be correct
that there is no uncertainty in the average effect size. If there's only
one observation, there's no information available to estimate the
uncertainty in the average. The cluster-robust variance estimator just
won't work.

James

On Thu, Aug 26, 2021 at 3:36 AM Diego Grados Bedoya <
diegogradosb at gmail.com> wrote:

Dear all,

I would like to report the parameters obtained using a robust variance
estimation of a multilevel mixed model (including a categorical moderator)
without the intercept (sample size is 42). The categorical variable has 17
levels of which 7 of them only have 1 study (due to the nature of the
intervention). I am using as a cluster the study ID since some studies
contributed with more than 1 effect size. Using the clubSandwhich R
package, 4 out of the 7 levels of the moderator reported NA for the df and
CIs values. I think the reason is that the estimates of these levels
precisely coincide with the effect sizes obtained from the studies (se=0).
I wonder if someone has faced something similar? (the robust function from
metafor package is reporting the same values of the estimates).

Any hint is more than welcome,

Diego

        [[alternative HTML version deleted]]