[R-meta] Multilevel meta-analysis, mutually exclusive levels
Erik, This is an interesting question. I would like to offer a perspective that contrasts a bit with the approach you are pursuing. If I understand your problem correctly, your aim is to investigate whether follow-up time (or some other predictor) is associated with effect magnitude, while controlling for the possible confounding effects of the type of outcome. In this case, I think it is acceptable and even desirable to include dummy variables for each outcome category, even though this leads to a model with many predictors. It's not that you'd want to interpret the coefficients on all of the outcome category dummies, it's just that they need to be in the model to control for possible spurious association. This approach is the simplest and I would argue safest way to examine the focal moderator, because the estimated association between follow-up time and effect size is based entirely on patterns *within* outcome categories. The random effects approach that you've described (and that Wolfgang elaborated) might be valid as well, but it entails a further, potentially strong assumption that the random outcome effects are independent of the study's follow-up time (i.e., that the random effects are independent of the predictors). Based on the description of the problem you're investigating, it seems like this assumption might not be that reasonable. If it is wrong, then the focal coefficient estimate may be biased (to an extent that depends on the strength of association between the random effects and the predictor, among other factors) because it is estimated based on a combination of associations both within *and between* outcome categories, rather than solely within outcome categories. James On Tue, Jan 2, 2018 at 4:36 AM, Viechtbauer Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
Dear Erik, Random effects do not have to be nested -- there can also be crossed random effects. See, for example, chapter 12 (Models for cross-classified random effects) in Raudenbush and Bryk (2002). In your case, you might consider adding random effects for studies, random effects for the effects within studies, and random effects for the outcomes (not nested within studies!). So, assuming the data are coded like this: study effect outcome lag yi vi --------------------------------- 1 1 A 1 2 D 2 1 A 2 2 B 2 3 C 3 1 B 4 1 B 4 2 D then this would be something along the lines of (if you are using metafor): rma.mv(yi, vi, mods = ~ lag, random = list(~ 1 | study/effect, ~ 1 | outcome), data=dat) Not sure what you mean by 'participant level' -- unless you have the raw data, then there is no participant level. Also, in studies examining multiple outcomes, I assume the same participants were measured with respect to the various outcomes. In that case, the sampling errors of the effects (yi values) are not independent, so you would need to account for this, either by computing the non-diagonal V matrix (which would require knowing or obtaining 'guestimates' of the correlations between the outcomes) or using cluster-robust inference methods. See the archives for lots of discussions on this. Best, Wolfgang -----Original Message----- From: R-sig-meta-analysis [mailto:r-sig-meta-analysis- bounces at r-project.org] On Behalf Of E. van der Meulen Sent: Tuesday, 19 December, 2017 11:38 To: r-sig-meta-analysis at r-project.org Subject: [R-meta] Multilevel meta-analysis, mutually exclusive levels Dear all, I have been struggling with a specific problem in conducting a multilevel meta-analysis. I will discuss my problem in more detail below. However, in essence the question is the following: in a multilevel structure for meta-analysis can a single study fall under multiple branches of a level that is higher than the study level. In more detail: The study context: I am conducting a review on the predictive validity of a personality concept of soldiers (concept X) n the development of a range of mental health and functioning problems. In short the research question is: too which degree can the concept X predict mental health status of soldiers. Concept X has been longitudinally associated to a wide variety of mental health outcomes: PTSD, depression, alcohol abuse, anxiety problems, etc. The first part of the review is to conduct a meta-analysis for these different concept X and mental health outcomes associations separately without any MULTILEVEL meta-analysis. A second part is to test a series of moderators using multilevel meta-analysis, and that is where the problems start. I expect to find an effect of for example the time-lag between the measurement of concept X and mental health outcomes (the longer the time-lag, the lower the effect size). However, to make such analyses meaningful, I can only use the aggregate of all effect sizes available, regardless of the type of mental health outcome (otherwise I would have too few effect sizes). The multilevel moderator analysis problem: as longitudinal associations of concept X and different outcomes can not (and are not) similar, the assessment of the time-lag moderator might yield spurious relationships. For example, when concept X is correlated with PTSD at .10 and with depression on .30, but on the same account the typical time-lag in studies on PTSD is 3 years, and depression is only one week. Then a result could be that the time-lag is a significant moderator, however, due to the huge difference in time-lag and effect size magnitude in PTSD an depression studies, it is not the time-lag but the outcome measure that causes this significant effect. The most tangible manner in overcoming this problem would be to correct for the type of outcome measure (i.e. depression or PTSD as a predictor). However, this would lead to a high number of predictors (75 effect sizes and about 18 predictors). Therefore I made a multilevel model with four levels, from highest to lowest: level 4) mental health outcome, level 3) individual study, level 2) individual effect sizes and level 1) participant level. (In which the level 1 through 3 are similar to most 'typical' multilevel meta-analysis.) However, a consequence of this multilevel model is that several studies fall under multiple branches of level 4, as they conduct analyses on concept X associations with more than 1 mental health outcome (e.g. PTSD AND depression). In a typical multilevel analogy, consider a multilevel construct of classrooms, schools and school districts. In this a typical multilevel model with include a highest level of school districts (I and II), under which you might have several schools A, B and C, and several classrooms within each school (A1, A2, A3, B1, C1). In my model I would have the problem that my model includes the following: Schooldistrict level I --> School A and B ---> Classrooms A3 and B1 Schooldistrict Level II --> School A and C ---> Classrooms A1, A2, and C1 As you can see school A (or in my actual problem, the school is a study) falls both under Schooldistrict level I and II. However, the classrooms are only used a single time (or effect sizes only appear once). Is that a problem, that these categories are not mutually exclusive? Or even violating an assumption? My own model would like something like this: Mental health outcome (PTSD) ---> Study A and B ---> Effect size A1, A2 and B1 (all effect sizes of associations between PTSD and concept X). Mental health outcome (depression) --> Study A and C ---> Effect sizes A3 and C1 (all effect sizes of associations between depression and concept X). I could really use some input on this! Kind regards, Erik van der Meulen
_______________________________________________ R-sig-meta-analysis mailing list R-sig-meta-analysis at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis