[R-meta] 3-level meta with robust errors

Dear James,
Thank you!

Attached are the code and the database.
And here is some results
summary(based_inf)
Multivariate Meta-Analysis Model (k = 17; method: REML)
  logLik  Deviance       AIC       BIC      AICc
-14.2991   28.5983   34.5983   36.9161   36.5983

Variance Components:
            estim    sqrt  nlvls  fixed       factor
sigma^2.1  0.0000  0.0000     12     no     ID_study
sigma^2.2  4.1629  2.0403      5     no  ID_database

Test for Heterogeneity:
Q(df = 16) = 48.1411, p-val < .0001

Model Results:
estimate      se    tval    pval   ci.lb   ci.ub
  2.0905  0.9704  2.1542  0.0468  0.0333  4.1478  *
coef_test(based_inf, vcov = "CR2",
+            cluster = wb$ID_database)
    Coef. Estimate    SE t-stat d.f. p-val (Satt) Sig.
1 intrcpt     2.09 0.954   2.19 3.91       0.0952    .

I think I found out where our mistake was.

The sandwich correction doesn't calculate Conf Intervals, so we calculated
them using formula: SE*1.96
Stupid, I know.
Still, even now I am not sure how to correctly calculate CI here - could
you please explain?

And another question
There are several methods for outliers detection: Cook distance,
residuals, hat values.  Rather often a study is problematic with one method
but OK with others. Are there any guidelines which studies should be
removed   -- i.e., when at least two methods indicate it as outliers?

Best,
Valeria

Valeria,

These are indeed perplexing results. Based on the information you've
provided, it's hard to say what could be going on. Could you provide
examples of the code you're using and the results of your analyses? Doing
so will help to identify potential problems or coding errors.

Kind Regards,
James

On Tue, Dec 1, 2020 at 10:45 AM Valeria Ivaniushina <
v.ivaniushina at gmail.com> wrote:

Dear list members,

We are conducting several meta-analyses using the metafor package in R
(Viechtbauer 2010) because of 3-level data structure, followed by
sandwich-type estimator with a small-sample adjustment to get cluster
robust standard errors.

There are some things that puzzle me, and I hope to get answers from the
community.

1. We calculate 95% CI for our mean effect size, and p-value is calculated
as a part of the output. While CI always indicate significant mean effect
size, p-values are often > 0.05
- Should I report both CI and p-value?
- How to interpret such discrepancy?

2. When I draw a forest plot for a meta-analysis of 8 models, I can see
that 95% CIs for every coefficient contain zero (for example, -0.40 -
0.84). However, the 95% CI for the mean coefficient is well above zero
(0,28 - 0,45).  How is it possible?

3.  Theoretically, the data has a 3-level structure (model; article;
database). But sometimes I see that there is no variance on one or two of
the levels. Should I repeat the analysis with only 2 or 1 level, according
to the variance distribution?

Best, Valeria

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-meta-analysis mailing list
R-sig-meta-analysis at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20201202/7db8b4d4/attachment.html>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: script for avSim_SH.R
Type: application/octet-stream
Size: 2941 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20201202/7db8b4d4/attachment.obj>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: MetaAnallysis avSim.sav
Type: application/octet-stream
Size: 2955 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20201202/7db8b4d4/attachment-0001.obj>

[R-meta] 3-level meta with robust errors

Thread (5 messages)