Dear all, I am performing a meta-analysis for the first time, and I would be grateful if you could give me some recommendations / suggestions. I apologize for this long description: I have 20 different papers and, because some papers have more than one experiment, 23 experiments are considered, which correspond to 23 different samples. The complexity of my case is that each experiment includes different domains, and each domain includes several tests. These domains are latent variables in the individual studies (tests are the indicators). I'm interested in studying the effects in five different domains. Thus, for each domain, I usually collected more than one test type measure. Finally, each sample includes an experimental and a control group. The effects were estimated using Hedges' g (adjusted) and then calculating the difference y between groups. In these calculations, I followed this valuable example: http://www.metafor-project.org/doku.php/analyses:morris2008 Unfortunately, papers do not report correlations/covariances between test measures and, so, for each y, I only have the estimated variance v. So, my database looks like: ID ----- Paper --------- Sample ------- Domain ------- Test ------- y ---- v ---- etc. 01 ------- P1 ---------- S1 ----------- D1 ---------- T1 -------- ... 02 ------- P1 ---------- S1 ----------- D1 ---------- T2 -------- ... 03 ------- P1 ---------- S1 ----------- D2 ---------- T3 -------- ... 04 ------- P1 ---------- S1 ----------- D4 ---------- T4 -------- ... 05 ------- P1 ---------- S1 ----------- D4 ---------- T5 -------- ... 06 ------- P1 ---------- S1 ----------- D4 ---------- T6 -------- ... 07 ------- P2 ---------- S2 ----------- D2 ---------- T3 -------- ... 08 ------- P2 ---------- S2 ----------- D2 ---------- T7 -------- ... 09 ------- P2 ---------- S2 ----------- D2 ---------- T8 -------- ... 10 ------- P2 ---------- S2 ----------- D3 ---------- T9 -------- ... 11 ------- P2 ---------- S2 ----------- D4 ---------- T4 -------- ... 12 ------- P2 ---------- S2 ----------- D4 ---------- T5 -------- ... 13 ------- P2 ---------- S2 ----------- D5 ---------- T0 -------- ... 14 ------- P3 ---------- S3 ----------- D1 ---------- T1 -------- ... 15 ------- P3 ---------- S4 ----------- D1 ---------- T1 -------- ... 16 ------- P3 ---------- S4 ----------- D2 ---------- T3 -------- ... ..... I would like to assess the significance of these five domains' effects as well as their correlation, using a meta-analyst approach. D1 is the central domain. Thus, I thought about performing a multivariate multilevel random meta-analysis model and then using the correlation matrix between true effects provided in the output. Since I have no sampling covariances, I used the "impute-the-correlation" strategy, by James Pustejovsky, together with robust and sensitive analysis. Because I'm interested in the domains (latent variables), the R code I'm using is the following: Vlist <- impute_covariance_matrix(vi = dat$v, cluster = dat$Sample, r = 0.2) # Followed by r=.4, r=.6 and r=.8 meta1 <- rma.mv(y ~ 0 + Domain, V=Vlist, random = ~ Domain | Sample, struct = "UN", data = dat) meta1r <- robust(meta1, cluster = dat$Sample); summary(meta1r); r1<-cov2cor(meta1r$vb); R1 <- round(c1,2) Does it make sense to you? Do you recommend corrections? Introducing 2 moderators in the model, I got interesting results with this "impute-the-correlation"+robust+sensitive strategy (ranging r from .2 to .8), namely, the significance of D1 true effect. However, the "impute-the-correlation" assumes equal correlations across studies and across outcomes. I've been searching in literature, and I found some papers reporting a high correlation between D2, D3, D4, D5: r=-6-.7 between pairs, and r=.7-.8 intra-pairs. Their correlation with D1 is lower, and even lower for one of the pairs. Given this different range of correlations among the outcomes, do you think that the results I got with this strategy are truthful? Or do you think it is better to perform the analysis in different steps, for different subsets of variables in the multivariate multilevel model, for example, considering two at a time, or D1 with each pair separately? Thank you very much for your attention!
[R-meta] "impute-the-correlation + robustness + sensitive analysis" strategy
4 messages · James Pustejovsky, Célia Sofia Moreira, Michael Dewey
Celia, If you have reasonable prior information about the correlation between certain tests, which suggests that some tests are more highly correlated then others, then I would definitely recommend using that information. In principle, you can still use the "impute-the-correlation" strategy even while assuming unequal correlations between certain tests. For instance, as you suggested, you might assume r = 0.7 for the inter-correlation between T1 through T4, but then assume r = 0.2 for the correlation between these measures and T5. For sensitivity analysis, you could vary these correlations by adding/subtracting 0.1 or 0.2 from each (off-diagonal) cell. Another useful sensitivity analysis would be to assume zero correlations between tests from different domains and then also use the struct = "DIAG" argument in rma.mv. This amounts to estimating separate (marginal) models for each domain. If you get very different average effect estimates than you do with the full multivariate model, then it would indicate that the results are going to be fairly sensitive to the assumption you make about the cross-domain correlations. Of course, implementing these approaches takes a bit more work. One approach would be to create the variance-covariance matrices for each study "by hand," and then store them in a list that can be fed into rma.mv. This is tedious but might it be the easiest way to go. I have an idea for how to make the impute_covariance_matrix() function more helpful for your use-case, but it will be a week or two before I can get to it. James On Tue, Jan 23, 2018 at 6:16 AM, C?lia Sofia Moreira <
celiasofiamoreira at gmail.com> wrote:
Dear all, I am performing a meta-analysis for the first time, and I would be grateful if you could give me some recommendations / suggestions. I apologize for this long description: I have 20 different papers and, because some papers have more than one experiment, 23 experiments are considered, which correspond to 23 different samples. The complexity of my case is that each experiment includes different domains, and each domain includes several tests. These domains are latent variables in the individual studies (tests are the indicators). I'm interested in studying the effects in five different domains. Thus, for each domain, I usually collected more than one test type measure. Finally, each sample includes an experimental and a control group. The effects were estimated using Hedges' g (adjusted) and then calculating the difference y between groups. In these calculations, I followed this valuable example: http://www.metafor-project.org/doku.php/analyses:morris2008 Unfortunately, papers do not report correlations/covariances between test measures and, so, for each y, I only have the estimated variance v. So, my database looks like: ID ----- Paper --------- Sample ------- Domain ------- Test ------- y ---- v ---- etc. 01 ------- P1 ---------- S1 ----------- D1 ---------- T1 -------- ... 02 ------- P1 ---------- S1 ----------- D1 ---------- T2 -------- ... 03 ------- P1 ---------- S1 ----------- D2 ---------- T3 -------- ... 04 ------- P1 ---------- S1 ----------- D4 ---------- T4 -------- ... 05 ------- P1 ---------- S1 ----------- D4 ---------- T5 -------- ... 06 ------- P1 ---------- S1 ----------- D4 ---------- T6 -------- ... 07 ------- P2 ---------- S2 ----------- D2 ---------- T3 -------- ... 08 ------- P2 ---------- S2 ----------- D2 ---------- T7 -------- ... 09 ------- P2 ---------- S2 ----------- D2 ---------- T8 -------- ... 10 ------- P2 ---------- S2 ----------- D3 ---------- T9 -------- ... 11 ------- P2 ---------- S2 ----------- D4 ---------- T4 -------- ... 12 ------- P2 ---------- S2 ----------- D4 ---------- T5 -------- ... 13 ------- P2 ---------- S2 ----------- D5 ---------- T0 -------- ... 14 ------- P3 ---------- S3 ----------- D1 ---------- T1 -------- ... 15 ------- P3 ---------- S4 ----------- D1 ---------- T1 -------- ... 16 ------- P3 ---------- S4 ----------- D2 ---------- T3 -------- ... ..... I would like to assess the significance of these five domains' effects as well as their correlation, using a meta-analyst approach. D1 is the central domain. Thus, I thought about performing a multivariate multilevel random meta-analysis model and then using the correlation matrix between true effects provided in the output. Since I have no sampling covariances, I used the "impute-the-correlation" strategy, by James Pustejovsky, together with robust and sensitive analysis. Because I'm interested in the domains (latent variables), the R code I'm using is the following: Vlist <- impute_covariance_matrix(vi = dat$v, cluster = dat$Sample, r = 0.2) # Followed by r=.4, r=.6 and r=.8 meta1 <- rma.mv(y ~ 0 + Domain, V=Vlist, random = ~ Domain | Sample, struct = "UN", data = dat) meta1r <- robust(meta1, cluster = dat$Sample); summary(meta1r); r1<-cov2cor(meta1r$vb); R1 <- round(c1,2) Does it make sense to you? Do you recommend corrections? Introducing 2 moderators in the model, I got interesting results with this "impute-the-correlation"+robust+sensitive strategy (ranging r from .2 to .8), namely, the significance of D1 true effect. However, the "impute-the-correlation" assumes equal correlations across studies and across outcomes. I've been searching in literature, and I found some papers reporting a high correlation between D2, D3, D4, D5: r=-6-.7 between pairs, and r=.7-.8 intra-pairs. Their correlation with D1 is lower, and even lower for one of the pairs. Given this different range of correlations among the outcomes, do you think that the results I got with this strategy are truthful? Or do you think it is better to perform the analysis in different steps, for different subsets of variables in the multivariate multilevel model, for example, considering two at a time, or D1 with each pair separately? Thank you very much for your attention! [[alternative HTML version deleted]]
_______________________________________________ R-sig-meta-analysis mailing list R-sig-meta-analysis at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
Dear James, Thank you very much for your quick reply. You write that "assume r = 0.7 for the inter-correlation between T1 through T4, but then assume r = 0.2 for the correlation between these measures and T5." However, I can not assume two different correlations simultaneously with the impute_the_covariance function. So, I presume you are suggesting to run the "impute_the_covariance + rma.mv" successively for r=.2,..., r=.7 (steps of .1 or .2), isn't is? In affirmative case, is the R code I wrote - in my previous message - the adequate? About creating the variance-covariance matrix by hand is a very complex procedure because correlations across domains vary from paper to paper, and I would have to consider a reasonable number of matrices resulting from different correlations combinations. I would be grateful if you could provide me a generalization of the impute_the_covariance function to my use-case. I will definitely wait. Thank you! Please let me know when it's ready :) Kind regards, celia 2018-01-23 15:44 GMT+00:00 James Pustejovsky <jepusto at gmail.com>:
Celia, If you have reasonable prior information about the correlation between certain tests, which suggests that some tests are more highly correlated then others, then I would definitely recommend using that information. In principle, you can still use the "impute-the-correlation" strategy even while assuming unequal correlations between certain tests. For instance, as you suggested, you might assume r = 0.7 for the inter-correlation between T1 through T4, but then assume r = 0.2 for the correlation between these measures and T5. For sensitivity analysis, you could vary these correlations by adding/subtracting 0.1 or 0.2 from each (off-diagonal) cell. Another useful sensitivity analysis would be to assume zero correlations between tests from different domains and then also use the struct = "DIAG" argument in rma.mv. This amounts to estimating separate (marginal) models for each domain. If you get very different average effect estimates than you do with the full multivariate model, then it would indicate that the results are going to be fairly sensitive to the assumption you make about the cross-domain correlations. Of course, implementing these approaches takes a bit more work. One approach would be to create the variance-covariance matrices for each study "by hand," and then store them in a list that can be fed into rma.mv. This is tedious but might it be the easiest way to go. I have an idea for how to make the impute_covariance_matrix() function more helpful for your use-case, but it will be a week or two before I can get to it. James On Tue, Jan 23, 2018 at 6:16 AM, C?lia Sofia Moreira < celiasofiamoreira at gmail.com> wrote:
Dear all, I am performing a meta-analysis for the first time, and I would be grateful if you could give me some recommendations / suggestions. I apologize for this long description: I have 20 different papers and, because some papers have more than one experiment, 23 experiments are considered, which correspond to 23 different samples. The complexity of my case is that each experiment includes different domains, and each domain includes several tests. These domains are latent variables in the individual studies (tests are the indicators). I'm interested in studying the effects in five different domains. Thus, for each domain, I usually collected more than one test type measure. Finally, each sample includes an experimental and a control group. The effects were estimated using Hedges' g (adjusted) and then calculating the difference y between groups. In these calculations, I followed this valuable example: http://www.metafor-project.org/doku.php/analyses:morris2008 Unfortunately, papers do not report correlations/covariances between test measures and, so, for each y, I only have the estimated variance v. So, my database looks like: ID ----- Paper --------- Sample ------- Domain ------- Test ------- y ---- v ---- etc. 01 ------- P1 ---------- S1 ----------- D1 ---------- T1 -------- ... 02 ------- P1 ---------- S1 ----------- D1 ---------- T2 -------- ... 03 ------- P1 ---------- S1 ----------- D2 ---------- T3 -------- ... 04 ------- P1 ---------- S1 ----------- D4 ---------- T4 -------- ... 05 ------- P1 ---------- S1 ----------- D4 ---------- T5 -------- ... 06 ------- P1 ---------- S1 ----------- D4 ---------- T6 -------- ... 07 ------- P2 ---------- S2 ----------- D2 ---------- T3 -------- ... 08 ------- P2 ---------- S2 ----------- D2 ---------- T7 -------- ... 09 ------- P2 ---------- S2 ----------- D2 ---------- T8 -------- ... 10 ------- P2 ---------- S2 ----------- D3 ---------- T9 -------- ... 11 ------- P2 ---------- S2 ----------- D4 ---------- T4 -------- ... 12 ------- P2 ---------- S2 ----------- D4 ---------- T5 -------- ... 13 ------- P2 ---------- S2 ----------- D5 ---------- T0 -------- ... 14 ------- P3 ---------- S3 ----------- D1 ---------- T1 -------- ... 15 ------- P3 ---------- S4 ----------- D1 ---------- T1 -------- ... 16 ------- P3 ---------- S4 ----------- D2 ---------- T3 -------- ... ..... I would like to assess the significance of these five domains' effects as well as their correlation, using a meta-analyst approach. D1 is the central domain. Thus, I thought about performing a multivariate multilevel random meta-analysis model and then using the correlation matrix between true effects provided in the output. Since I have no sampling covariances, I used the "impute-the-correlation" strategy, by James Pustejovsky, together with robust and sensitive analysis. Because I'm interested in the domains (latent variables), the R code I'm using is the following: Vlist <- impute_covariance_matrix(vi = dat$v, cluster = dat$Sample, r = 0.2) # Followed by r=.4, r=.6 and r=.8 meta1 <- rma.mv(y ~ 0 + Domain, V=Vlist, random = ~ Domain | Sample, struct = "UN", data = dat) meta1r <- robust(meta1, cluster = dat$Sample); summary(meta1r); r1<-cov2cor(meta1r$vb); R1 <- round(c1,2) Does it make sense to you? Do you recommend corrections? Introducing 2 moderators in the model, I got interesting results with this "impute-the-correlation"+robust+sensitive strategy (ranging r from .2 to .8), namely, the significance of D1 true effect. However, the "impute-the-correlation" assumes equal correlations across studies and across outcomes. I've been searching in literature, and I found some papers reporting a high correlation between D2, D3, D4, D5: r=-6-.7 between pairs, and r=.7-.8 intra-pairs. Their correlation with D1 is lower, and even lower for one of the pairs. Given this different range of correlations among the outcomes, do you think that the results I got with this strategy are truthful? Or do you think it is better to perform the analysis in different steps, for different subsets of variables in the multivariate multilevel model, for example, considering two at a time, or D1 with each pair separately? Thank you very much for your attention! [[alternative HTML version deleted]]
_______________________________________________ R-sig-meta-analysis mailing list R-sig-meta-analysis at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
Dear Celia One thing which you could do while you are waiting is to do the imputations assuming all the correlations are equal. If you do this for a range of plausible values it may be that the substantive result of your meta-analysis is unaffected and so you can be confident in your interpretation. Most techniques are much more sensitive to some inputs than to others. Michael
On 23/01/2018 20:35, C?lia Sofia Moreira wrote:
Dear James, Thank you very much for your quick reply. You write that "assume r = 0.7 for the inter-correlation between T1 through T4, but then assume r = 0.2 for the correlation between these measures and T5." However, I can not assume two different correlations simultaneously with the impute_the_covariance function. So, I presume you are suggesting to run the "impute_the_covariance + rma.mv" successively for r=.2,..., r=.7 (steps of .1 or .2), isn't is? In affirmative case, is the R code I wrote - in my previous message - the adequate? About creating the variance-covariance matrix by hand is a very complex procedure because correlations across domains vary from paper to paper, and I would have to consider a reasonable number of matrices resulting from different correlations combinations. I would be grateful if you could provide me a generalization of the impute_the_covariance function to my use-case. I will definitely wait. Thank you! Please let me know when it's ready :) Kind regards, celia 2018-01-23 15:44 GMT+00:00 James Pustejovsky <jepusto at gmail.com>:
Celia, If you have reasonable prior information about the correlation between certain tests, which suggests that some tests are more highly correlated then others, then I would definitely recommend using that information. In principle, you can still use the "impute-the-correlation" strategy even while assuming unequal correlations between certain tests. For instance, as you suggested, you might assume r = 0.7 for the inter-correlation between T1 through T4, but then assume r = 0.2 for the correlation between these measures and T5. For sensitivity analysis, you could vary these correlations by adding/subtracting 0.1 or 0.2 from each (off-diagonal) cell. Another useful sensitivity analysis would be to assume zero correlations between tests from different domains and then also use the struct = "DIAG" argument in rma.mv. This amounts to estimating separate (marginal) models for each domain. If you get very different average effect estimates than you do with the full multivariate model, then it would indicate that the results are going to be fairly sensitive to the assumption you make about the cross-domain correlations. Of course, implementing these approaches takes a bit more work. One approach would be to create the variance-covariance matrices for each study "by hand," and then store them in a list that can be fed into rma.mv. This is tedious but might it be the easiest way to go. I have an idea for how to make the impute_covariance_matrix() function more helpful for your use-case, but it will be a week or two before I can get to it. James On Tue, Jan 23, 2018 at 6:16 AM, C?lia Sofia Moreira < celiasofiamoreira at gmail.com> wrote:
Dear all, I am performing a meta-analysis for the first time, and I would be grateful if you could give me some recommendations / suggestions. I apologize for this long description: I have 20 different papers and, because some papers have more than one experiment, 23 experiments are considered, which correspond to 23 different samples. The complexity of my case is that each experiment includes different domains, and each domain includes several tests. These domains are latent variables in the individual studies (tests are the indicators). I'm interested in studying the effects in five different domains. Thus, for each domain, I usually collected more than one test type measure. Finally, each sample includes an experimental and a control group. The effects were estimated using Hedges' g (adjusted) and then calculating the difference y between groups. In these calculations, I followed this valuable example: http://www.metafor-project.org/doku.php/analyses:morris2008 Unfortunately, papers do not report correlations/covariances between test measures and, so, for each y, I only have the estimated variance v. So, my database looks like: ID ----- Paper --------- Sample ------- Domain ------- Test ------- y ---- v ---- etc. 01 ------- P1 ---------- S1 ----------- D1 ---------- T1 -------- ... 02 ------- P1 ---------- S1 ----------- D1 ---------- T2 -------- ... 03 ------- P1 ---------- S1 ----------- D2 ---------- T3 -------- ... 04 ------- P1 ---------- S1 ----------- D4 ---------- T4 -------- ... 05 ------- P1 ---------- S1 ----------- D4 ---------- T5 -------- ... 06 ------- P1 ---------- S1 ----------- D4 ---------- T6 -------- ... 07 ------- P2 ---------- S2 ----------- D2 ---------- T3 -------- ... 08 ------- P2 ---------- S2 ----------- D2 ---------- T7 -------- ... 09 ------- P2 ---------- S2 ----------- D2 ---------- T8 -------- ... 10 ------- P2 ---------- S2 ----------- D3 ---------- T9 -------- ... 11 ------- P2 ---------- S2 ----------- D4 ---------- T4 -------- ... 12 ------- P2 ---------- S2 ----------- D4 ---------- T5 -------- ... 13 ------- P2 ---------- S2 ----------- D5 ---------- T0 -------- ... 14 ------- P3 ---------- S3 ----------- D1 ---------- T1 -------- ... 15 ------- P3 ---------- S4 ----------- D1 ---------- T1 -------- ... 16 ------- P3 ---------- S4 ----------- D2 ---------- T3 -------- ... ..... I would like to assess the significance of these five domains' effects as well as their correlation, using a meta-analyst approach. D1 is the central domain. Thus, I thought about performing a multivariate multilevel random meta-analysis model and then using the correlation matrix between true effects provided in the output. Since I have no sampling covariances, I used the "impute-the-correlation" strategy, by James Pustejovsky, together with robust and sensitive analysis. Because I'm interested in the domains (latent variables), the R code I'm using is the following: Vlist <- impute_covariance_matrix(vi = dat$v, cluster = dat$Sample, r = 0.2) # Followed by r=.4, r=.6 and r=.8 meta1 <- rma.mv(y ~ 0 + Domain, V=Vlist, random = ~ Domain | Sample, struct = "UN", data = dat) meta1r <- robust(meta1, cluster = dat$Sample); summary(meta1r); r1<-cov2cor(meta1r$vb); R1 <- round(c1,2) Does it make sense to you? Do you recommend corrections? Introducing 2 moderators in the model, I got interesting results with this "impute-the-correlation"+robust+sensitive strategy (ranging r from .2 to .8), namely, the significance of D1 true effect. However, the "impute-the-correlation" assumes equal correlations across studies and across outcomes. I've been searching in literature, and I found some papers reporting a high correlation between D2, D3, D4, D5: r=-6-.7 between pairs, and r=.7-.8 intra-pairs. Their correlation with D1 is lower, and even lower for one of the pairs. Given this different range of correlations among the outcomes, do you think that the results I got with this strategy are truthful? Or do you think it is better to perform the analysis in different steps, for different subsets of variables in the multivariate multilevel model, for example, considering two at a time, or D1 with each pair separately? Thank you very much for your attention! [[alternative HTML version deleted]]
_______________________________________________ R-sig-meta-analysis mailing list R-sig-meta-analysis at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
[[alternative HTML version deleted]]
_______________________________________________ R-sig-meta-analysis mailing list R-sig-meta-analysis at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis