[R-meta] Outlier and influential case analysis for multilevel meta-analysis with RVE
I think it makes sense to do the analysis of outliers and influential cases before applying RVE. One way to think about this approach is that you are examining the assumptions _of the working model_, to understand the extent to which those assumptions are reasonable, even if you will later use RVE to protect against model misspecification. I think this approach is advantageous because it gives access to a richer set of diagnostic tools, whereas the other approach is just a single rule-of-thumb (one which I don't think has a strong statistical rationale in the first place). A further challenge here that I don't think has been addressed thoroughly in the meta-analysis methods literature is how to think about outliers in the multilevel context. When there is both between-study and within-study variation, one could imagine there being outlying studies, outlying effect sizes with respect to the overall distribution, or outlying effect sizes relative to the distribution of effects within the same study. Perhaps others on the list know of guidance about how to diagnose these features. Best, James On Fri, Aug 23, 2024 at 6:09?AM Maximilian Steininger via
R-sig-meta-analysis <r-sig-meta-analysis at r-project.org> wrote:
Dear all, I am conducting a three-level meta-analysis where I have different dependency structures in my data. I model the dependency by approximating the var-cov matrix, followed by estimating a three-level model and then I apply robust variance estimation to compute my outcome (as suggested e.g. here: https://wviechtb.github.io/metafor/reference/misc-recs.html#general-workflow-for-meta-analyses-involving-complex-dependency-structures < https://wviechtb.github.io/metafor/reference/misc-recs.html#general-workflow-for-meta-analyses-involving-complex-dependency-structures
)
I wanted to do some sensitivity analysis on the model by running outlier and influential diagnostics. However, most of the proposed diagnostics do not work on "robust.rma" objects. So far I did some model diagnostics by calculating cook's distance and hat values for my robust model (see e.g., https://wviechtb.github.io/metafor/reference/influence.rma.mv.html < https://wviechtb.github.io/metafor/reference/influence.rma.mv.html>). But as far as I am concerned these "only" give me information on influential cases and not outliers. What is the best approach to check for outliers when using robust models? Are the two options below a sensible approach to check for outliers? According to this source a possible but rather conservative approach is to label all studies as outliers that have confidence intervals that do not overlap with the confidence interval of the pooled effect. (see: https://cjvanlissa.github.io/Doing-Meta-Analysis-in-R/detecting-outliers-influential-cases.html < https://cjvanlissa.github.io/Doing-Meta-Analysis-in-R/detecting-outliers-influential-cases.html
).
Is it a feasible option to perform outlier diagnostics for the non-robust model as suggested e.g. by Viechtbauer & Cheung (2010; 10.1002/jrsm.11). My approach here would be to identify outliers based on the non-robust model --> exclude the outliers --> rerun the whole analysis without the outliers (i.e., approximate var-cov matrix, estimate three-level model, apply robust variance estimation for the subset of studies). Or are there other, more elegant ways to do this? Best and many thanks! ?? Mag. Maximilian Steininger PhD candidate Social, Cognitive and Affective Neuroscience Unit Faculty of Psychology University of Vienna Liebiggasse 5 1010 Vienna, Austria e: maximilian.steininger at univie.ac.at w: http://scan.psy.univie.ac.at [[alternative HTML version deleted]]
_______________________________________________ R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org To manage your subscription to this mailing list, go to: https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis