Hello,
I am part of a team, with Prof Peter John at King?s College London and Dr Florian Foos and Ceren Cinar at the London School of Economics, that is working on a meta-analysis of get-out-the-vote (GOTV) interventions. Our project looks at whether GOTV?s effect is larger on people with low vs high voting propensity. We had a question about how to specify the random effect?s structure of our model.
Our setup is as follows. We have estimates for the ITT of many GOTV interventions, computed comparing turnout between individuals randomized to some treatment group and a control group. Importantly, for each treatment, we have effect estimates separately for individuals with high vs medium vs low voting propensities. Many experiments have multiple treatment arms (which often differ on the particular get-out-the-vote message used), so effectively we have effect estimates for each voting propensity-experiment-arm triplet, with arms nested within experiments. Since we have subjects of each voting propensity on (almost) all arms, for any experiment with (say) two treatment arms, we have an estimate of control vs treatment 1 and control vs treatment 2 for each voting propensity separately. Using simulated data, our dataset has the following structure:
df <- structure(list(effect = c(0.155803932130353, 0.172093700115289, 0.228564716936809, -0.0029411764705886, 0.226006191950464, 0.24365585512081, 0.205042523718298, 0.529411764705883, 0.418300653594771, 0.333333333333333, 0.357142857142857, 0.333333333333333, 0.227386316387913, 0.110813804839315, -0.0158629160414587, 0.362962962962963, 0.266666666666667, 0.127272727272728), arm = structure(c(1L, 1L, 1L, 1L, 2L, 1L, 2L, 1L, 2L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L), .Label = c("1", "2", "3"), class = "factor"), experiment = c("A", "A", "A", "B", "B", "B", "B", "B", "B", "C", "C", "C", "C", "C", "C", "C", "C", "C"), voting_prop = c("low", "medium", "high", "low", "low", "medium", "medium", "high", "high", "low", "low", "low", "medium", "medium", "medium", "high", "high", "high")), row.names = c(NA, -18L), class = "data.frame")
Sampling errors are not independent across arms within voting propensity-experiment pairs, since the same control group (which is specific to the voting propensity-experiment pair) is used to estimate the ITT of different treatment arms. We use the Gleser & Olkin (2009) method to estimate the variance-covariance matrix of sampling errors, which for the example above is:
V <- structure(c(0.00527759653574149, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0.000272246907504789, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0012451834193348, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0268238335114407,
0.0144993694656436, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0.0144993694656436, 0.0274724895138511, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.000219964986797141,
0.000110094264225202, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0.000110094264225202, 0.000220356782547541, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0227385071675729,
0.0084216693213233, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0.0084216693213233, 0.0163754681247953, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0523485584710074, 0.0294460641399417,
0.0294460641399417, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0.0294460641399417, 0.0441690962099125, 0.0294460641399417,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0294460641399417,
0.0294460641399417, 0.0523485584710074, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.000477394970026329, 0.00023697669984828,
0.000236976699848284, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0.00023697669984828, 0.000475661779613111, 0.000236976699848285,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.000236976699848284,
0.000236976699848285, 0.000476154421676615, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0147396599742279, 0.00947549569771795,
0.00947549569771793, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0.00947549569771795, 0.0189509913954359, 0.00947549569771794,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.00947549569771793,
0.00947549569771794, 0.015936060946162), .Dim = c(18L, 18L))
We are interested in testing whether the effect of get-out-the-vote interventions differs across voting propensities. We are planning to use the metafor package. But we are not sure of what would be a reasonable way to specify the random effect structure of the model to properly capture how true effects are correlated. We think the best approach is to estimate the following multilevel meta-analysis model that captures that arms are nested within experiments, such that true effects of different arms in the same experiment may resemble (e.g. because they were tested in the same election and country):
rma.mv(effect, random = list(~ 1 | experiment/arm), data = df, V = V, mods = ~ voting_prop)
Does the model above look reasonable? Or do we also need to allow true effects to be correlated within voting propensity groups across arms? We think this is not necessary but are not sure. If it is, we are not really sure about which of the approaches below is the best way to specify this. Note voting propensities are not really nested within experiment-arms (effectively this is a cross-classified structure).
rma.mv(effect, random = list(~ 1 | experiment/arm/voting_prop), data = df, V = V, mods = ~ voting_prop)
df$arm.in.experiment <- paste0(df$experiment, df$arm)
rma.mv(effect, random = list(~ 1 | experiment, ~ arm.in.experiment | voting_prop), data = df, V = V, mods = ~ voting_prop)
Any guidance you could provide would be very much welcome.
Francisco Tom?s-Valiente Jord?
ETH Z?rich
[R-meta] Specifying the random effect structure of our multilevel meta-analysis in metafor
5 messages · Tomas-Valiente Jorda Francisco, James Pustejovsky, Wolfgang Viechtbauer
Hi Francisco, Your question is ultimately an empirical one because the best model is one that captures the actual sources of variation in effects, so I don't think you'll find a definitive answer. I'll offer a couple of observations. First, considering that your research aims seem to focus on variation across VPGs, I think that this suggests prioritizing modeling the variation across VPGs rather than treating it solely as a fixed factor (as in your first model, where you have random effects for arms nested within studies but not for VPGs). Second, it sounds like your data structure consist of multiple studies, each with two factors: voting propensity group (VPG) and experimental arm. Thus, in principle you could have a model in which effects for each unique combination of VPG and arm vary and co-vary across studies, as in: df$arm.by.vpg <- paste0(df$arm, df$voting_prop) Model A: random = ~ arm.by.vpg | experiment, struct = "UN" This is assuming that the arm labels (1, 2, 3 in your data) have the same meaning across studies. Further, this model might be quite difficult to fit if there are a lot of unique arms (e.g., 3 arms, 3 VPG categories leads to a 9-dimensional multivariate effect for each study). So, perhaps you could simplify the model by imposing some further structure. For instance, you could treat the arms and VPGs as following an additive compound-symmetric structure: df$vpg.in.experiment <- paste0(df$experiment, df$voting_prop) Model B: random = list(~ 1 | experiment, ~ 1 | vpg.in.experiment, ~ 1 | arm.in.experiment) or equivalent: Model B: random = list(~ 1 | experiment / voting_prop, ~ 1 | arm.in.experiment) Here, you've got study-level random effects, within-study effects for each VPG, and within-study effects for each arm. In principle you could also include the interaction between VPG and arm in study: Model C: random = list(~ 1 | experiment / voting_prop / arm, ~ 1 | arm.in.experiment) The assumptions I would be concerned with in this model are that it treats the within-study variation as homogeneous for each VPG and homogeneous across studies. This might not be ideal if, for instance, the study-to-study variation in effects is minimal for the low VPG but substantial for the high VPG. Loosening this assumption can be done with something like Model D: random = list(~ VPG | experiment, ~ 1 | arm.in.experiment), struct = "UN" which is a compromise between the full multivariate model (Model A) and the compound symmetry model (Model B). Third, in the example data you sent, each study includes low-, medium-, and high-VPG groups. If that is true in your real data, then all's well. If not---as in, if some studies include only high-VPG and other studies include only low and medium VPG---then you might also want to consider centering the voting_prop predictor within study. Specifically, you'd create dummy variables for each category and then center each dummy variable by study. Centering in this way means that the differences between VPG levels will be estimated using the direct, head-to-head comparison of effects within each study. Without centering, the differences between VPG levels will be estimated based in part on cross-study differences, which are at greater risk of confounding due to factors (methodological or otherwise) that vary from study to study. See Tanner-Smith & Tipton (2014; https://doi.org/10.1002/jrsm.1091) for further discussion. Fourth, and predictably for me, this is another good instance to consider using robust variance estimation (RVE) methods for inference. You've got a complicated data structure and potential concern that the random effects specification could be mis-specified in some way (such as by failing to capture a relevant source of study-to-study variation). RVE is essentially an insurance policy against model misspeciification. Even if you don't exactly nail the random effects down perfectly, you'll still have protection for inference about the fixed effects of voting_prop levels. See Pustejovsky & Tipton (2022; https://doi.org/10.1007/s11121-021-01246-3) for details. James On Mon, Jan 23, 2023 at 5:05 AM Tomas-Valiente Jorda Francisco <
tomasf at student.ethz.ch> wrote:
Hello,
I am part of a team, with Prof Peter John at King?s College London and Dr
Florian Foos and Ceren Cinar at the London School of Economics, that is
working on a meta-analysis of get-out-the-vote (GOTV) interventions. Our
project looks at whether GOTV?s effect is larger on people with low vs high
voting propensity. We had a question about how to specify the random
effect?s structure of our model.
Our setup is as follows. We have estimates for the ITT of many GOTV
interventions, computed comparing turnout between individuals randomized to
some treatment group and a control group. Importantly, for each treatment,
we have effect estimates separately for individuals with high vs medium vs
low voting propensities. Many experiments have multiple treatment arms
(which often differ on the particular get-out-the-vote message used), so
effectively we have effect estimates for each voting
propensity-experiment-arm triplet, with arms nested within experiments.
Since we have subjects of each voting propensity on (almost) all arms, for
any experiment with (say) two treatment arms, we have an estimate of
control vs treatment 1 and control vs treatment 2 for each voting
propensity separately. Using simulated data, our dataset has the following
structure:
df <- structure(list(effect = c(0.155803932130353, 0.172093700115289,
0.228564716936809, -0.0029411764705886, 0.226006191950464,
0.24365585512081, 0.205042523718298, 0.529411764705883, 0.418300653594771,
0.333333333333333, 0.357142857142857, 0.333333333333333, 0.227386316387913,
0.110813804839315, -0.0158629160414587, 0.362962962962963,
0.266666666666667, 0.127272727272728), arm = structure(c(1L, 1L, 1L, 1L,
2L, 1L, 2L, 1L, 2L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L), .Label = c("1",
"2", "3"), class = "factor"), experiment = c("A", "A", "A", "B", "B", "B",
"B", "B", "B", "C", "C", "C", "C", "C", "C", "C", "C", "C"), voting_prop =
c("low", "medium", "high", "low", "low", "medium", "medium", "high",
"high", "low", "low", "low", "medium", "medium", "medium", "high", "high",
"high")), row.names = c(NA, -18L), class = "data.frame")
Sampling errors are not independent across arms within voting
propensity-experiment pairs, since the same control group (which is
specific to the voting propensity-experiment pair) is used to estimate the
ITT of different treatment arms. We use the Gleser & Olkin (2009) method to
estimate the variance-covariance matrix of sampling errors, which for the
example above is:
V <- structure(c(0.00527759653574149, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0.000272246907504789, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0012451834193348, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0268238335114407,
0.0144993694656436, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0.0144993694656436, 0.0274724895138511, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.000219964986797141,
0.000110094264225202, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0.000110094264225202, 0.000220356782547541, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0227385071675729,
0.0084216693213233, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0.0084216693213233, 0.0163754681247953, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0523485584710074,
0.0294460641399417,
0.0294460641399417, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0.0294460641399417, 0.0441690962099125, 0.0294460641399417,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0294460641399417,
0.0294460641399417, 0.0523485584710074, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.000477394970026329, 0.00023697669984828,
0.000236976699848284, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0.00023697669984828, 0.000475661779613111, 0.000236976699848285,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.000236976699848284,
0.000236976699848285, 0.000476154421676615, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0147396599742279,
0.00947549569771795,
0.00947549569771793, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0.00947549569771795, 0.0189509913954359, 0.00947549569771794,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.00947549569771793,
0.00947549569771794, 0.015936060946162), .Dim = c(18L, 18L))
We are interested in testing whether the effect of get-out-the-vote
interventions differs across voting propensities. We are planning to use
the metafor package. But we are not sure of what would be a reasonable way
to specify the random effect structure of the model to properly capture how
true effects are correlated. We think the best approach is to estimate the
following multilevel meta-analysis model that captures that arms are nested
within experiments, such that true effects of different arms in the same
experiment may resemble (e.g. because they were tested in the same election
and country):
rma.mv(effect, random = list(~ 1 | experiment/arm), data = df, V = V,
mods = ~ voting_prop)
Does the model above look reasonable? Or do we also need to allow true
effects to be correlated within voting propensity groups across arms? We
think this is not necessary but are not sure. If it is, we are not really
sure about which of the approaches below is the best way to specify this.
Note voting propensities are not really nested within experiment-arms
(effectively this is a cross-classified structure).
rma.mv(effect, random = list(~ 1 | experiment/arm/voting_prop), data =
df, V = V, mods = ~ voting_prop)
df$arm.in.experiment <- paste0(df$experiment, df$arm)
rma.mv(effect, random = list(~ 1 | experiment, ~ arm.in.experiment |
voting_prop), data = df, V = V, mods = ~ voting_prop)
Any guidance you could provide would be very much welcome.
Francisco Tom?s-Valiente Jord?
ETH Z?rich
[[alternative HTML version deleted]]
_______________________________________________ R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org To manage your subscription to this mailing list, go to: https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
Dear James, Thanks a lot for your response. Arm labels do not have the same meaning across studies, so Model A is not suitable for us, but B, C and D all seem reasonable. Also, some arms will not have observations in all three voting propensity groups, so we may need to use the suggestion you offer on this point. Thanks for all of this. One clarification question on the RVE approach you suggest. Your proposal is that (as per the code below) we estimate our model with metafor::rma.mv using some assumed variance-covariance matrix and that for hypothesis testing we use clubSandwich's coef_test or coef_int or Wald_test, right? Or did I misunderstand you? V2 <- impute_covariance_matrix(vi=diag(V), cluster=df$experiment, r=0.6) model <- rma.mv(effect, random = list(~ 1 | experiment / voting_prop, ~ 1 | arm.in.experiment), data = df, V = V2, mods = ~ voting_prop) coef_test(model, vcov="CR2") Best, Paco
From: James Pustejovsky <jepusto at gmail.com>
Sent: 23 January 2023 16:07:14
To: Tomas-Valiente Jorda Francisco
Cc: r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] Specifying the random effect structure of our multilevel meta-analysis in metafor
Sent: 23 January 2023 16:07:14
To: Tomas-Valiente Jorda Francisco
Cc: r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] Specifying the random effect structure of our multilevel meta-analysis in metafor
Hi Francisco, Your question is ultimately an empirical one because the best model is one that captures the actual sources of variation in effects, so I don't think you'll find a definitive answer. I'll offer a couple of observations. First, considering that your research aims seem to focus on variation across VPGs, I think that this suggests prioritizing modeling the variation across VPGs rather than treating it solely as a fixed factor (as in your first model, where you have random effects for arms nested within studies but not for VPGs). Second, it sounds like your data structure consist of multiple studies, each with two factors: voting propensity group (VPG) and experimental arm. Thus, in principle you could have a model in which effects for each unique combination of VPG and arm vary and co-vary across studies, as in: df$arm.by.vpg <- paste0(df$arm, df$voting_prop) Model A: random = ~ arm.by.vpg | experiment, struct = "UN" This is assuming that the arm labels (1, 2, 3 in your data) have the same meaning across studies. Further, this model might be quite difficult to fit if there are a lot of unique arms (e.g., 3 arms, 3 VPG categories leads to a 9-dimensional multivariate effect for each study). So, perhaps you could simplify the model by imposing some further structure. For instance, you could treat the arms and VPGs as following an additive compound-symmetric structure: df$vpg.in.experiment <- paste0(df$experiment, df$voting_prop) Model B: random = list(~ 1 | experiment, ~ 1 | vpg.in.experiment, ~ 1 | arm.in.experiment) or equivalent: Model B: random = list(~ 1 | experiment / voting_prop, ~ 1 | arm.in.experiment) Here, you've got study-level random effects, within-study effects for each VPG, and within-study effects for each arm. In principle you could also include the interaction between VPG and arm in study: Model C: random = list(~ 1 | experiment / voting_prop / arm, ~ 1 | arm.in.experiment) The assumptions I would be concerned with in this model are that it treats the within-study variation as homogeneous for each VPG and homogeneous across studies. This might not be ideal if, for instance, the study-to-study variation in effects is minimal for the low VPG but substantial for the high VPG. Loosening this assumption can be done with something like Model D: random = list(~ VPG | experiment, ~ 1 | arm.in.experiment), struct = "UN" which is a compromise between the full multivariate model (Model A) and the compound symmetry model (Model B). Third, in the example data you sent, each study includes low-, medium-, and high-VPG groups. If that is true in your real data, then all's well. If not---as in, if some studies include only high-VPG and other studies include only low and medium VPG---then you might also want to consider centering the voting_prop predictor within study. Specifically, you'd create dummy variables for each category and then center each dummy variable by study. Centering in this way means that the differences between VPG levels will be estimated using the direct, head-to-head comparison of effects within each study. Without centering, the differences between VPG levels will be estimated based in part on cross-study differences, which are at greater risk of confounding due to factors (methodological or otherwise) that vary from study to study. See Tanner-Smith & Tipton (2014; https://doi.org/10.1002/jrsm.1091) for further discussion. Fourth, and predictably for me, this is another good instance to consider using robust variance estimation (RVE) methods for inference. You've got a complicated data structure and potential concern that the random effects specification could be mis-specified in some way (such as by failing to capture a relevant source of study-to-study variation). RVE is essentially an insurance policy against model misspeciification. Even if you don't exactly nail the random effects down perfectly, you'll still have protection for inference about the fixed effects of voting_prop levels. See Pustejovsky & Tipton (2022; https://doi.org/10.1007/s11121-021-01246-3) for details. James On Mon, Jan 23, 2023 at 5:05 AM Tomas-Valiente Jorda Francisco <tomasf at student.ethz.ch<mailto:tomasf at student.ethz.ch>> wrote: Hello, I am part of a team, with Prof Peter John at King?s College London and Dr Florian Foos and Ceren Cinar at the London School of Economics, that is working on a meta-analysis of get-out-the-vote (GOTV) interventions. Our project looks at whether GOTV?s effect is larger on people with low vs high voting propensity. We had a question about how to specify the random effect?s structure of our model. Our setup is as follows. We have estimates for the ITT of many GOTV interventions, computed comparing turnout between individuals randomized to some treatment group and a control group. Importantly, for each treatment, we have effect estimates separately for individuals with high vs medium vs low voting propensities. Many experiments have multiple treatment arms (which often differ on the particular get-out-the-vote message used), so effectively we have effect estimates for each voting propensity-experiment-arm triplet, with arms nested within experiments. Since we have subjects of each voting propensity on (almost) all arms, for any experiment with (say) two treatment arms, we have an estimate of control vs treatment 1 and control vs treatment 2 for each voting propensity separately. Using simulated data, our dataset has the following structure: df <- structure(list(effect = c(0.155803932130353, 0.172093700115289, 0.228564716936809, -0.0029411764705886, 0.226006191950464, 0.24365585512081, 0.205042523718298, 0.529411764705883, 0.418300653594771, 0.333333333333333, 0.357142857142857, 0.333333333333333, 0.227386316387913, 0.110813804839315, -0.0158629160414587, 0.362962962962963, 0.266666666666667, 0.127272727272728), arm = structure(c(1L, 1L, 1L, 1L, 2L, 1L, 2L, 1L, 2L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L), .Label = c("1", "2", "3"), class = "factor"), experiment = c("A", "A", "A", "B", "B", "B", "B", "B", "B", "C", "C", "C", "C", "C", "C", "C", "C", "C"), voting_prop = c("low", "medium", "high", "low", "low", "medium", "medium", "high", "high", "low", "low", "low", "medium", "medium", "medium", "high", "high", "high")), row.names = c(NA, -18L), class = "data.frame") Sampling errors are not independent across arms within voting propensity-experiment pairs, since the same control group (which is specific to the voting propensity-experiment pair) is used to estimate the ITT of different treatment arms. We use the Gleser & Olkin (2009) method to estimate the variance-covariance matrix of sampling errors, which for the example above is: V <- structure(c(0.00527759653574149, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.000272246907504789, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0012451834193348, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0268238335114407, 0.0144993694656436, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0144993694656436, 0.0274724895138511, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.000219964986797141, 0.000110094264225202, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.000110094264225202, 0.000220356782547541, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0227385071675729, 0.0084216693213233, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0084216693213233, 0.0163754681247953, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0523485584710074, 0.0294460641399417, 0.0294460641399417, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0294460641399417, 0.0441690962099125, 0.0294460641399417, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0294460641399417, 0.0294460641399417, 0.0523485584710074, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.000477394970026329, 0.00023697669984828, 0.000236976699848284, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.00023697669984828, 0.000475661779613111, 0.000236976699848285, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.000236976699848284, 0.000236976699848285, 0.000476154421676615, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0147396599742279, 0.00947549569771795, 0.00947549569771793, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.00947549569771795, 0.0189509913954359, 0.00947549569771794, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.00947549569771793, 0.00947549569771794, 0.015936060946162), .Dim = c(18L, 18L)) We are interested in testing whether the effect of get-out-the-vote interventions differs across voting propensities. We are planning to use the metafor package. But we are not sure of what would be a reasonable way to specify the random effect structure of the model to properly capture how true effects are correlated. We think the best approach is to estimate the following multilevel meta-analysis model that captures that arms are nested within experiments, such that true effects of different arms in the same experiment may resemble (e.g. because they were tested in the same election and country): rma.mv<http://rma.mv>(effect, random = list(~ 1 | experiment/arm), data = df, V = V, mods = ~ voting_prop) Does the model above look reasonable? Or do we also need to allow true effects to be correlated within voting propensity groups across arms? We think this is not necessary but are not sure. If it is, we are not really sure about which of the approaches below is the best way to specify this. Note voting propensities are not really nested within experiment-arms (effectively this is a cross-classified structure). rma.mv<http://rma.mv>(effect, random = list(~ 1 | experiment/arm/voting_prop), data = df, V = V, mods = ~ voting_prop) df$arm.in.experiment <- paste0(df$experiment, df$arm) rma.mv<http://rma.mv>(effect, random = list(~ 1 | experiment, ~ arm.in.experiment | voting_prop), data = df, V = V, mods = ~ voting_prop) Any guidance you could provide would be very much welcome. Francisco Tom?s-Valiente Jord? ETH Z?rich [[alternative HTML version deleted]] _______________________________________________ R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org<mailto:R-sig-meta-analysis at r-project.org> To manage your subscription to this mailing list, go to: https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
Hi Paco, Responses below. James On Mon, Jan 23, 2023 at 4:25 PM Tomas-Valiente Jorda Francisco <
tomasf at student.ethz.ch> wrote:
One clarification question on the RVE approach you suggest. Your proposal is that (as per the code below) we estimate our model with metafor::rma.mv using some assumed variance-covariance matrix and that for hypothesis testing we use clubSandwich's coef_test or coef_int or Wald_test, right? Or did I misunderstand you? V2 <- impute_covariance_matrix(vi=diag(V), cluster=df$experiment, r=0.6) model <- rma.mv(effect, random = list(~ 1 | experiment / voting_prop, ~ 1 | arm.in.experiment), data = df, V = V2, mods = ~ voting_prop) coef_test(model, vcov="CR2")
Correct. You can also accomplish the same thing using the metafor::robust() function, which calls clubSandwich under the hood: model_robust <- robust(model, cluster = experiment, clubSandwich = TRUE) For tests of hypotheses involving multiple parameters, you'll still need to use clubSandwich::Wald_test() or wildmeta::Wald_test_cwb().
A few more comments from me ... Best, Wolfgang
Hi Paco, Responses below. James On Mon, Jan 23, 2023 at 4:25 PM Tomas-Valiente Jorda Francisco < tomasf at student.ethz.ch> wrote:
One clarification question on the RVE approach you suggest. Your proposal is that (as per the code below) we estimate our model with metafor::rma.mv using some assumed variance-covariance matrix and that for hypothesis testing we use clubSandwich's coef_test or coef_int or Wald_test, right? Or did I misunderstand you? V2 <- impute_covariance_matrix(vi=diag(V), cluster=df$experiment, r=0.6)
Don't use impute_covariance_matrix() if you have already done what you mentioned in your first post: "Sampling errors are not independent across arms within voting propensity-experiment pairs, since the same control group (which is specific to the voting propensity-experiment pair) is used to estimate the ITT of different treatment arms. We use the Gleser & Olkin (2009) method to estimate the variance-covariance matrix of sampling errors [...]" So you already have an appropriately calculated V matrix, so no need to make some rougher approximation of it!
model <- rma.mv(effect, random = list(~ 1 | experiment / voting_prop, ~ 1 | arm.in.experiment), data = df, V = V2, mods = ~ voting_prop) coef_test(model, vcov="CR2")
Correct. You can also accomplish the same thing using the metafor::robust() function, which calls clubSandwich under the hood: model_robust <- robust(model, cluster = experiment, clubSandwich = TRUE) For tests of hypotheses involving multiple parameters, you'll still need to use clubSandwich::Wald_test() or wildmeta::Wald_test_cwb().
The anova() function in metafor (which can be used to test multiple parameter) is actually set up to call Wald_test() appropriately (when passing a "robust.rma" object where clubSandwich was TRUE to it).