[R-meta] rma, sandwich correction and very small data sets

Dear Valeria,

Unless you have very good reasons to assume that estimates within studies are homogeneous, you should always add a random effect at the estimate level to the model. See:

http://www.metafor-project.org/doku.php/analyses:konstantopoulos2011

and esp. the "A Common Mistake in the Three-Level Model" section.

So, I would do:

wb$ID_estimate <- 1:nrow(wb)

random = list(~ 1 | ID_estimate, ~ 1 | ID_study, ~ 1 | ID_database)

Also, if you use data=wb, you do not need wb$ in the model call.

Finally, SE_Influence sounds like this is a variable for the standard errors. The second argument of rma.mv() is for specifying the sampling *variances* (or an entire var-cov matrix).

So, to summarize:

eff1 <- rma.mv(yi=EFFECT_SIZE_Influence, V=SE_Influence^2, 
               random = list(~ 1 | ID_estimate, ~ 1 | ID_study, ~ 1 | ID_database), 
               tdist=TRUE, data=wb)

However, with the number of levels you show, I would indeed be worried about fitting such a complex model with so little data. You won't get precise estimates of the variance components and hence they can be all over the place.

Also, cluster-robust inference methods work asymptotically, that is, when the number of levels of the clustering variable gets large. With 5 or 7 levels for 'databases', I would say we are rather far away from 'asymptotically'. The clubSandwich package you are using for this includes small-sample corrections which should help a but, but I would still question the use of such methods with such low k at the clustering level. Maybe James Pustejovsky (the author of clubSandwich) can chime in here.

As for combining the results of multiple (independent) meta-analyses, see:

http://www.metafor-project.org/doku.php/tips:comp_two_independent_estimates

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org]
On Behalf Of Valeria Ivaniushina
Sent: Monday, 07 December, 2020 18:00
To: R meta
Subject: [R-meta] rma, sandwich correction and very small data sets

Dear colleagues,

I do 3-level meta-analysis with a small number of studies and a small
number of clusters.
1st level - model, 2nd level - study, 3rd level - database.

The effect I am interested in can be specified in different ways. Experts
in the field advised me to make separate meta analyses for each
specification and then combine the results, kind of meta-meta.

I have several questions:

1)  Is this a correct code?
First I do REML:
eff1 <- rma.mv(yi=wb$EFFECT_SIZE_Influence,
V=wb$SE_Influence,random = list(~1 | ID_study, ~1 | ID_database),
tdist=TRUE, data=wb)

Then with this object I use sandwich, to get cluster-robust standard
errors, clustering at the highest level of nesting:
coef_test(eff1, vcov = "CR2",cluster = wb$ID_database)

2)  I am worried that the numbers of clusters are too small -- are the
results reliable?
eff1: 17 models, 12 studies, 5 databases
eff2: 8 models, 5 studies, 5 databases
eff3: 11 models, 9 studies, 7 databases

3) Variance distribution is vastly different between three models - what
does it tell me?
eff1: 1st level 4%, 2nd level 0%, 3rd level 96%
eff2: 1st level 100%, 2nd level 0%, 3rd level 0%
eff3: 1st level 15%, 2nd level 0%, 3rd level 85%

4) How can I combine the results of three meta-analyses?

Best,
Valeria