Message-ID: <00a8565b428d4f008013124834c26628@UM-MAIL3214.unimaas.nl>
Date: 2020-12-03T08:03:38Z
From: Wolfgang Viechtbauer
Subject: [R-meta] 3-level meta with robust errors
In-Reply-To: <CAEJcFY1e9-8hGUGJXHQho1USXc8atK1W_i54igJXTcNa2CGvdw@mail.gmail.com>
Dear Valeria,
You can use conf_int() from clubSandwich to get CIs.
As for detecting outliers: I am not aware of any such rules (that are actually validated).
Best,
Wolfgang
>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org]
>On Behalf Of Valeria Ivaniushina
>Sent: Wednesday, 02 December, 2020 17:03
>To: James Pustejovsky; R meta
>Subject: Re: [R-meta] 3-level meta with robust errors
>
>ATTACHMENT(S) REMOVED: ATT00001.txt | MetaAnallysis avSim.sav | script for
>avSim_SH.R
>
>Dear James,
>Thank you!
>
>Attached are the code and the database.
>And here is some results
>> summary(based_inf)
>Multivariate Meta-Analysis Model (k = 17; method: REML)
>? logLik ?Deviance ? ? ? AIC ? ? ? BIC ? ? ?AICc
>-14.2991 ? 28.5983 ? 34.5983 ? 36.9161 ? 36.5983
>
>Variance Components:
>? ? ? ? ? ? estim ? ?sqrt ?nlvls ?fixed ? ? ? factor
>sigma^2.1 ?0.0000 ?0.0000 ? ? 12 ? ? no ? ? ID_study
>sigma^2.2 ?4.1629 ?2.0403 ? ? ?5 ? ? no ?ID_database
>
>Test for Heterogeneity:
>Q(df = 16) = 48.1411, p-val < .0001
>
>Model Results:
>estimate ? ? ?se ? ?tval ? ?pval ? ci.lb ? ci.ub
>? 2.0905 ?0.9704 ?2.1542 ?0.0468 ?0.0333 ?4.1478 ?*
>
>
>> coef_test(based_inf, vcov = "CR2",
>+ ? ? ? ? ? ?cluster = wb$ID_database)
>? ? Coef. Estimate ? ?SE t-stat d.f. p-val (Satt) Sig.
>1 intrcpt ? ? 2.09 0.954 ? 2.19 3.91 ? ? ? 0.0952 ? ?.
>
>I think I found out where our mistake was.
>
>The sandwich?correction doesn't calculate Conf Intervals, so we calculated
>them using formula: SE*1.96
>Stupid, I know.
>Still, even now I am not sure how to correctly calculate CI here - could you
>please explain?
>
>And another question
>There are several methods for outliers detection: Cook distance,? residuals,
>hat values.? Rather often a study is problematic with one method but OK with
>others. Are there any guidelines which studies should be removed? ?-- i.e.,
>when at least two methods indicate it as outliers?
>
>Best,
>Valeria
>
>On Tue, Dec 1, 2020 at 9:10 PM James Pustejovsky <jepusto at gmail.com> wrote:
>Valeria,
>
>These are indeed perplexing results. Based on the information you've
>provided, it's hard to say what could be going on. Could you provide
>examples of the code you're using and the results of your analyses? Doing so
>will help to identify potential problems or coding errors.
>
>Kind Regards,
>James
>
>On Tue, Dec 1, 2020 at 10:45 AM Valeria Ivaniushina
><v.ivaniushina at gmail.com> wrote:
>Dear list members,
>
>We are conducting several meta-analyses using the metafor package in R
>(Viechtbauer 2010) because of 3-level data structure, followed by
>sandwich-type estimator with a small-sample adjustment to get cluster
>robust standard errors.
>
>There are some things that puzzle me, and I hope to get answers from the
>community.
>
>1. We calculate 95% CI for our mean effect size, and p-value is calculated
>as a part of the output. While CI always indicate significant mean effect
>size, p-values are often > 0.05
>- Should I report both CI and p-value?
>- How to interpret such discrepancy?
>
>2. When I draw a forest plot for a meta-analysis of 8 models, I can see
>that 95% CIs for every coefficient contain zero (for example, -0.40 -
>0.84). However, the 95% CI for the mean coefficient is well above zero
>(0,28 - 0,45).? How is it possible?
>
>3.? Theoretically, the data has a 3-level structure (model; article;
>database). But sometimes I see that there is no variance on one or two of
>the levels. Should I repeat the analysis with only 2 or 1 level, according
>to the variance distribution?
>
>Best, Valeria