[R-meta] Does clubSandwich::coef_test() handle crossed random-effects?

Ah I see. In my previous message, I think I misinterpreted what was going
on. Based on what you've just described, it sounds like you actually do not
have crossed random effects. Rather, you've got a hierarchical structure
where effect sizes are nested within studies and studies are nested within
scales. If you do not specify a clustering variable, clubSandwich defaults
to clustering based on the highest level of random effects, so in your case
you were clustering on the scale ID. (You could confirm this by running
coef_test(m1, "CR2", cluster = data$scale), to see if the results are
identical to coef_test(m1, "CR2").) If there are only a few unique values
of the scale variable (I would guess maybe around 6 or 7?) then this would
also explain the low degrees of freedom.

So, in addition to the approach described in my previous message, a
further alternative would be to treat scale as a set of fixed effects
(i.e., dummy variables for each level of scale). and then cluster on
studyID. Strictly speaking, this means that you would not be generalizing
beyond the observed set of scales. But if you only observe 6 unique levels,
trying to do so may be asking for more than the data can support.

James

On Sun, Oct 3, 2021 at 8:41 AM Farzad Keyhan <f.keyhaniha at gmail.com>
wrote:

Hi James and Wolfgang,

Thank you both. Please note that 'scale' doesn't vary within each
'study'. Thus, to generalize beyond 'scale' levels, the only way was to add
it as a crossed random-effect.

James, does that make sense to construct a V based only on 'study'
cluster using 'impute_covariance_matrix' but then conduct sensitivity on
'r'?

Thanks,
Fred


On Sun, Oct 3, 2021, 8:25 AM James Pustejovsky <jepusto at gmail.com> wrote:

Hi Farzad,

clubSandwich does not support clustering in models with crossed random
effects.

There are generalizations of CRVE that work under crosses dependence
structures (see Cameron, Gelbach, & Miller, 2011; Cameron & Miller, 2015).
But the small-sample corrections implemented in clubSandwich have yet to be
worked out for multi-way clustering. Thus, these generalizations only work
when there are sufficiently large numbers of clusters *in each clustering
dimension*.

Based on currently available methods and software, I would suggest
sticking with model-based inference (as implemented in rma.mv) if your
interest is in a model with crossed random effects. That means you?d either
have to do the work to develop a defensible V matrix and/or conduct
sensitivity analysis for whatever assumptions go into constructing the V
matrix.

James

Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2011). Robust inference
with multiway clustering. *Journal of Business & Economic Statistics*,
*29*(2), 238-249.

Cameron, A. C., & Miller, D. L. (2015). A practitioner?s guide to
cluster-robust inference. *Journal of human resources*, *50*(2),
317-372.

On Oct 3, 2021, at 4:20 AM, Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

?Hi Fred,

The dfs you see are computed based on a Satterthwaite approximation.
Depending on the data structure, the dfs could very well be much lower than
the number of studies/estimates.

This aside, a limitation of the cluster robust approach is that it can
only capture dependencies that occur within the groups defined by the
clustering variable. Or put in a different way, for the approach to work,
there need to be a sufficient (for some definition of "sufficient") number
of independent groups within the dataset. When introducing crossed random
effects or other random effects that can create dependencies across the
entire dataset (like spatial correlation or correlation due to phylogeny in
meta-analyses in ecology and evolution involing multiple species), then
there might only be a single (or very few) subsets in the data that are
independent and then the cluster robust approach cannot really be used
anymore.

Best,
Wolfgang

-----Original Message-----

From: R-sig-meta-analysis [mailto:
r-sig-meta-analysis-bounces at r-project.org] On

Behalf Of Farzad Keyhan

Sent: Sunday, 03 October, 2021 7:35

To: R meta

Subject: [R-meta] Does clubSandwich::coef_test() handle crossed
random-effects?


Hello Wolfgang and James,


I have 51 studies, but when I run an unconditional model (below),

coef_test() outputs a df of 4.76 (makes no sense).


I suspected that coef_test() is not picking up my highest-level

cluster (study). So, I made that explicit in the call. And then, I got

an error saying "random effects are not nested".


First, does clubSandwich handle models with crossed random-effects? If

not, is there an alternative code for rma.mv() models to accomplish

similar things (I wanted to do egger's test as well)--and of course

the small sample correction may not be available there.


Thanks, Fred


# Note 'scale' is crossed with 'study'

m1 <- rma.mv(yi, V, random = list(~1|study/es, ~1|scale), data=data)


coef_test(m1, "CR2")

  Coef. Estimate    SE t-stat d.f. p-val (Satt) Sig.

1 intrcpt    0.506 0.193   2.63 4.76       0.0489    *


## I made cluster explicit but get an error:

coef_test(m1, "CR2", cluster = data$study)

*** Error: Random effects are not nested within clustering