[R-meta] Outlier and influential case analysis for multilevel meta-analysis with RVE

Am 24.08.2024 um 14:40 schrieb James Pustejovsky via R-sig-meta-analysis <r-sig-meta-analysis at r-project.org>:

I think it makes sense to do the analysis of outliers and influential cases
before applying RVE. One way to think about this approach is that you are
examining the assumptions _of the working model_, to understand the extent
to which those assumptions are reasonable, even if you will later use RVE
to protect against model misspecification.

I think this approach is advantageous because it gives access to a richer
set of diagnostic tools, whereas the other approach is just a single
rule-of-thumb (one which I don't think has a strong statistical rationale
in the first place).

A further challenge here that I don't think has been addressed thoroughly
in the meta-analysis methods literature is how to think about outliers in
the multilevel context. When there is both between-study and within-study
variation, one could imagine there being outlying studies, outlying effect
sizes with respect to the overall distribution, or outlying effect sizes
relative to the distribution of effects within the same study. Perhaps
others on the list know of guidance about how to diagnose these features.

Best,
James

On Fri, Aug 23, 2024 at 6:09?AM Maximilian Steininger via
R-sig-meta-analysis <r-sig-meta-analysis at r-project.org> wrote:

Dear all,

I am conducting a three-level meta-analysis where I have different
dependency structures in my data. I model the dependency by approximating
the var-cov matrix, followed by estimating a three-level model and then I
apply robust variance estimation to compute my outcome (as suggested e.g.
here:
https://wviechtb.github.io/metafor/reference/misc-recs.html#general-workflow-for-meta-analyses-involving-complex-dependency-structures
<
https://wviechtb.github.io/metafor/reference/misc-recs.html#general-workflow-for-meta-analyses-involving-complex-dependency-structures

)

I wanted to do some sensitivity analysis on the model by running outlier
and influential diagnostics. However, most of the proposed diagnostics do
not work on "robust.rma" objects.

So far I did some model diagnostics by calculating cook's distance and hat
values for my robust model (see e.g.,
https://wviechtb.github.io/metafor/reference/influence.rma.mv.html <
https://wviechtb.github.io/metafor/reference/influence.rma.mv.html>). But
as far as I am concerned these "only" give me information on influential
cases and not outliers.

What is the best approach to check for outliers when using robust models?
Are the two options below a sensible approach to check for outliers?

According to this source a possible but rather conservative approach is to
label all studies as outliers that have confidence intervals that do not
overlap with the confidence interval of the pooled effect. (see:
https://cjvanlissa.github.io/Doing-Meta-Analysis-in-R/detecting-outliers-influential-cases.html
<
https://cjvanlissa.github.io/Doing-Meta-Analysis-in-R/detecting-outliers-influential-cases.html

).

Is it a feasible option to perform outlier diagnostics for the non-robust
model as suggested e.g. by Viechtbauer & Cheung (2010; 10.1002/jrsm.11). My
approach here would be to identify outliers based on the non-robust model
--> exclude the outliers --> rerun the whole analysis without the outliers
(i.e., approximate var-cov matrix, estimate three-level model, apply robust
variance estimation for the subset of studies).
Or are there other, more elegant ways to do this?

Best and many thanks!
??

Mag. Maximilian Steininger
 PhD candidate

 Social, Cognitive and Affective Neuroscience Unit
 Faculty of Psychology
 University of Vienna

 Liebiggasse 5
 1010 Vienna, Austria

 e: maximilian.steininger at univie.ac.at
 w: http://scan.psy.univie.ac.at


       [[alternative HTML version deleted]]

	[[alternative HTML version deleted]]