Dear Listmembers,
I am planning to conduct a meta-analysis based on data from a series of factorial survey studies. The factorial survey studies I'd like to analyze correspond to within-subject experimental designs with multiple treatment factors. However, there seem to be very few, if any, meta-analyses using data from factorial survey studies or related designs. I thus lack examples corresponding to the design I need to ask some questions about below. So here's a brief description of my data in general terms:
A factorial survey, also known as a vignette study or factorial vignette design, is a research method used to study how individuals make judgments or decisions in complex situations by presenting them with a series of hypothetical scenarios (vignettes) that vary systematically across multiple factors or attributes. For example, respondents might be asked how likely they would vote for a candidate of a political party who is
- male vs. female, (factor 1)
- has a leftwing vs. rightwing orientation (factor 2)
- and ?is perceived to be very empathic/somewhat empathic/not empathic (factor 3).
This results in a 2 x 2 x 3 factorial design with 12 different vignettes.
Let?s assume that in the original data, the factor levels were measured using dummy variables. The results are unstandardised regression slopes plus standard errors. Accordingly, I'd like to meta-analyse 4 unstandardised regression slopes:
- the slope of the dummy variable for factor A (the average effect of ?male? vs. ?female?),
- the slope of the dummy variable for factor B (the average effect of "rightwing" vs. leftwing orientation), plus
- the two slopes of the two dummy variables for factor C (?somewhat empathic? vs ?very empathic? and ?not empathic? vs ?very empathic?).
With one dependent variable, all studies were conducted as full or fractional factorial within-subjects designs. Therefore, I believe it would be best to account for any potential non-independence in the data, given that each respondent rated multiple (up to 12) vignettes. What I find a bit challenging here is that there is not just one average effect to examine, but four in total. Furthermore, these four average slopes result from three orthogonal factors. I (think I) have been able to combine clubSandwich and metafor to examine this data - but I would be very happy to hear if you think my approach is adequate, or if there are alternative ways to analyse this data?
Here's an illustration/sample of the data ? restricted to 5 studies only, the ?real? dataset will comprise of approx. 20 studies. Notice that I arranged all coefficient values in one column:
Data (df) for illustration:
study
coeff_name
coeff_value
v
coeff_index
study_1
cgenderMale
-0.07
0.000113
1
study_1
cRightwing
0.17
0.000344
2
study_1
cSomewhatEmpathic
-0.08
0.000160
3
study_1
cNotEmpathic
-0.12
0.000171
4
study_2
cgenderMale
-0.08
0.000179
5
study_2
cRightwing
0.18
0.000504
6
study_2
cSomewhatEmpathic
-0.06
0.000309
7
study_2
cNotEmpathic
-0.14
0.000435
8
study_3
cgenderMale
-0.06
0.000097
9
study_3
cRightwing
0.16
0.000319
10
study_3
cSomewhatEmpathic
-0.07
0.000150
11
study_3
cNotEmpathic
-0.11
0.000152
12
study_4
cgenderMale
-0.03
0.000107
13
study_4
cRightwing
0.15
0.000452
14
study_4
cSomewhatEmpathic
-0.05
0.000198
15
study_4
cNotEmpathic
-0.10
0.000196
16
study_5
cgenderMale
-0.04
0.000123
17
study_5
cRightwing
0.15
0.000366
18
study_5
cSomewhatEmpathic
-0.11
0.000193
19
study_5
cNotEmpathic
-0.18
0.000205
20
And here's the code I used:
library(metafor)
library(clubSandwich)
# Create sampling variance covariance matrix
V_mat <- impute_covariance_matrix(Testdata$v, cluster = df$study, r = 0.6)
# model in metafor
mod <- rma.mv(coeff_value ~ 0 + coeff_name, V = V_mat, #coeff_name contains all coefficients as factor levels
random = ~ 1 | study / coeff_index, #coefficients are nested in studies
data = df)
# clustered SEs and CIs
conf_int(mod, vcov = "CR2")
And here are the results:
Coef.
Estimate
SE
d.f.
Lower_95%CI
Upper_95%CI
coeff_namecgenderMale
-0.0553
0.00924
3.99
-0.081
-0.0296
coeff_namecRightwing
0.1623
0.0057
3.97
0.146
0.1782
coeff_namecSomewhatEmpathic
-0.0739
0.01044
3.97
-0.103
-0.0448
coeff_namecNotEmpathic
-0.1289
0.01465
3.92
-0.17
-0.0879
The code seems to works. But as I outlined at the beginning, it is not clear to if one might arrange the data in long format with all coefficients of interest in one column as I did. I thus highly appreciate your advice and any comments ?
Best,
Elmar Schlueter