Dear C?lia, If I understand you correctly, the general question here is how to analyze the relationship between two effects measured on the same subjects. So, for each study, we have [y_i1, y_i2] and corresponding sampling variances [v_i1, v_i2]. Ideally, we also have the covariance between [y_i1, y_i2], so we have the 2x2 var-cov matrix of the sampling errors for each study. In that case, one can fit a multivariate (or rather: bivariate) model along the lines of Berkey et al. (1998): http://www.metafor-project.org/doku.php/analyses:berkey1998 And then, based on the var-cov matrix of the random effects, we can estimate the correlation of the underlying true effects (the correlation is given directly in the output). One can even estimate the regression line that describes the linear relationship between the underlying true effects. See, for example: van Houwelingen, H. C., Arends, L. R., & Stijnen, T. (2002). Advanced methods in meta-analysis: Multivariate approach and meta-regression. Statistics in Medicine, 21(4), 589-624. Page 601 is the most relevant here. If you do not know the covariances (and one would assume them to be 0), then this approach is going to give you biased results, so I would not recommended this (and cluster robust methods are not going to help you here). An easier approach would be to simply treat one of the effects as your outcome and the other as a predictor. Technically, this isn't quite right, since the predictor is measured with error. This will lead to bias of the underlying true relationship (e.g., https://en.wikipedia.org/wiki/Errors-in-variables_models). Things are a bit more complex compared to the 'standard' regression context, since the amount of error actually varies across studies, but it's the same fundamental issue. I haven't given this a lot of thought, but I would assume that the bias will again be negative, so the strength of the relationship will tend to be underestimated. Unless somebody has a better idea, one could argue that if a relationship is still found, then this provides evidence that a relationship does exist, although the actual strength is uncertain. Best, Wolfgang -----Original Message----- From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of C?lia Sofia Moreira Sent: Wednesday, 10 January, 2018 1:04 To: r-sig-meta-analysis at r-project.org Subject: Re: [R-meta] Meta-analysis when sampling covariance matrices are missing Dear all, I have been reading some of the messages in the list related to my problem, and I realized that ?unknown correlations? is an old topic. Special thanks to Prof. Wolfgang, James Pustejovsky, and Isabel Schlegel in https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2017-August/000127.html. Really helpful! I also understood that my attempt to perform the multilevel model in my previous message was wrong. Please, forget the previous questions. My data are multivariate and so outcomes should be analyzed as such. However, I only have sampling means and SD, and thus, only variances of the effects (SMD) are available. So, I will input a covariance matrix (clubSandwich) and/or RVE (robumeta), always with the indispensable rma.mv (metafor). Nevertheless, I would like to investigate a regression between two effects. However, due to the previous limitation, I have no idea if it is possible, and, in affirmative case, how to do it. Thus, any recommendations / suggestions will be much appreciated. In negative case, can the unstructured correlation matrix obtained with the rma.mv output be used to assess the strength of the relationship between these effects? Please, tell me your opinion! Kind regards
[R-meta] Meta-analysis when sampling covariance matrices are missing
3 messages · Viechtbauer Wolfgang (STAT), Philipp Doebler, Célia Sofia Moreira
Dear Wolfgang and C?lia, it is an interesting thought to see this as an errors in variables problem. Your intuition that there is a negative bias can probably be made more precise: From the literature on attenutation of variances, covariances and regression coefficients for insufficient reliability, I would assume the absolute value is biased towards zero, i.e. a negative bias in the absolute value of the regression coefficient. In this sense, I would agree with Wolfgang: If one manages to find a significant (but attenuated) relationship, then there is one. I would conjecture that the factor should be similar to Var(d)/(Var(d) + \bar nu^2)), where \bar nu^2 is the average squared standard error, or around the square-root of this for a standardized regression coefficient. Caution that people show results in this case assuming that the variance is homogeneous, but it would surprise me if results differed dramatically in the heterogeneous case. Kind wishes, Philipp On Tue, Jan 16, 2018 at 6:18 PM, Viechtbauer Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
Dear C?lia, If I understand you correctly, the general question here is how to analyze the relationship between two effects measured on the same subjects. So, for each study, we have [y_i1, y_i2] and corresponding sampling variances [v_i1, v_i2]. Ideally, we also have the covariance between [y_i1, y_i2], so we have the 2x2 var-cov matrix of the sampling errors for each study. In that case, one can fit a multivariate (or rather: bivariate) model along the lines of Berkey et al. (1998): http://www.metafor-project.org/doku.php/analyses:berkey1998 And then, based on the var-cov matrix of the random effects, we can estimate the correlation of the underlying true effects (the correlation is given directly in the output). One can even estimate the regression line that describes the linear relationship between the underlying true effects. See, for example: van Houwelingen, H. C., Arends, L. R., & Stijnen, T. (2002). Advanced methods in meta-analysis: Multivariate approach and meta-regression. Statistics in Medicine, 21(4), 589-624. Page 601 is the most relevant here. If you do not know the covariances (and one would assume them to be 0), then this approach is going to give you biased results, so I would not recommended this (and cluster robust methods are not going to help you here). An easier approach would be to simply treat one of the effects as your outcome and the other as a predictor. Technically, this isn't quite right, since the predictor is measured with error. This will lead to bias of the underlying true relationship (e.g., https://en.wikipedia.org/wiki/ Errors-in-variables_models). Things are a bit more complex compared to the 'standard' regression context, since the amount of error actually varies across studies, but it's the same fundamental issue. I haven't given this a lot of thought, but I would assume that the bias will again be negative, so the strength of the relationship will tend to be underestimated. Unless somebody has a better idea, one could argue that if a relationship is still found, then this provides evidence that a relationship does exist, although the actual strength is uncertain. Best, Wolfgang -----Original Message----- From: R-sig-meta-analysis [mailto:r-sig-meta-analysis- bounces at r-project.org] On Behalf Of C?lia Sofia Moreira Sent: Wednesday, 10 January, 2018 1:04 To: r-sig-meta-analysis at r-project.org Subject: Re: [R-meta] Meta-analysis when sampling covariance matrices are missing Dear all, I have been reading some of the messages in the list related to my problem, and I realized that ?unknown correlations? is an old topic. Special thanks to Prof. Wolfgang, James Pustejovsky, and Isabel Schlegel in https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2017-August/000127.html . Really helpful! I also understood that my attempt to perform the multilevel model in my previous message was wrong. Please, forget the previous questions. My data are multivariate and so outcomes should be analyzed as such. However, I only have sampling means and SD, and thus, only variances of the effects (SMD) are available. So, I will input a covariance matrix (clubSandwich) and/or RVE (robumeta), always with the indispensable rma.mv (metafor). Nevertheless, I would like to investigate a regression between two effects. However, due to the previous limitation, I have no idea if it is possible, and, in affirmative case, how to do it. Thus, any recommendations / suggestions will be much appreciated. In negative case, can the unstructured correlation matrix obtained with the rma.mv output be used to assess the strength of the relationship between these effects? Please, tell me your opinion! Kind regards
_______________________________________________ R-sig-meta-analysis mailing list R-sig-meta-analysis at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
Prof. Dr. Philipp Doebler Technische Universit?t Dortmund Fakult?t Statistik Vogelpothsweg 87 44227 Dortmund Tel.: +49 231-755 8259 Fax: +49 231-755 3918 doebler at statistik.tu-dortmund.de www.statistik.tu-dortmund.de/1261.html Wichtiger Hinweis: Die Information in dieser E-Mail ist vertraulich. Sie ist ausschlie?lich f?r den Adressaten bestimmt. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, unterrichten Sie bitte den Absender und vernichten Sie diese Mail. Vielen Dank. Unbeschadet der Korrespondenz per E-Mail, sind unsere Erkl?rungen ausschlie?lich final rechtsverbindlich, wenn sie in herk?mmlicher Schriftform (mit eigenh?ndiger Unterschrift) oder durch ?bermittlung eines solchen Schriftst?cks per Telefax erfolgen. Important note: The information included in this e-mail is confidential. It is solely intended for the recipient. If you are not the intended recipient of this e-mail please contact the sender and delete this message. Thank you. Without prejudice of e-mail correspondence, our statements are only legally binding when they are made in the conventional written form (with personal signature) or when such documents are sent by fax. [[alternative HTML version deleted]]
6 days later
Dear Prof. Wolfgang and Prof. Doebler, Thank you very much for your valuable recommendations to my problem. However, given some complexities in 'my' outcomes, it would be difficult to apply the "errors-in-variables" model. Moreover, without a reliable published result regarding the bias and the significance of the relationship, it would be difficult to publish my results. So, I will try a different approach, which I will explain in another message. Once more, thank you very much for your attention and your interesting comments. I think you have a theorem to prove about your joint conjecture on the bias and the significance of the relationship in the errors-in-variables model! Kind regards, celia 2018-01-16 19:09 GMT+00:00 Philipp Doebler <doebler at statistik.tu-dortmund.de
:
Dear Wolfgang and C?lia, it is an interesting thought to see this as an errors in variables problem. Your intuition that there is a negative bias can probably be made more precise: From the literature on attenutation of variances, covariances and regression coefficients for insufficient reliability, I would assume the absolute value is biased towards zero, i.e. a negative bias in the absolute value of the regression coefficient. In this sense, I would agree with Wolfgang: If one manages to find a significant (but attenuated) relationship, then there is one. I would conjecture that the factor should be similar to Var(d)/(Var(d) + \bar nu^2)), where \bar nu^2 is the average squared standard error, or around the square-root of this for a standardized regression coefficient. Caution that people show results in this case assuming that the variance is homogeneous, but it would surprise me if results differed dramatically in the heterogeneous case. Kind wishes, Philipp On Tue, Jan 16, 2018 at 6:18 PM, Viechtbauer Wolfgang (SP) < wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
Dear C?lia, If I understand you correctly, the general question here is how to analyze the relationship between two effects measured on the same subjects. So, for each study, we have [y_i1, y_i2] and corresponding sampling variances [v_i1, v_i2]. Ideally, we also have the covariance between [y_i1, y_i2], so we have the 2x2 var-cov matrix of the sampling errors for each study. In that case, one can fit a multivariate (or rather: bivariate) model along the lines of Berkey et al. (1998): http://www.metafor-project.org/doku.php/analyses:berkey1998 And then, based on the var-cov matrix of the random effects, we can estimate the correlation of the underlying true effects (the correlation is given directly in the output). One can even estimate the regression line that describes the linear relationship between the underlying true effects. See, for example: van Houwelingen, H. C., Arends, L. R., & Stijnen, T. (2002). Advanced methods in meta-analysis: Multivariate approach and meta-regression. Statistics in Medicine, 21(4), 589-624. Page 601 is the most relevant here. If you do not know the covariances (and one would assume them to be 0), then this approach is going to give you biased results, so I would not recommended this (and cluster robust methods are not going to help you here). An easier approach would be to simply treat one of the effects as your outcome and the other as a predictor. Technically, this isn't quite right, since the predictor is measured with error. This will lead to bias of the underlying true relationship (e.g., https://en.wikipedia.org/wiki/ Errors-in-variables_models). Things are a bit more complex compared to the 'standard' regression context, since the amount of error actually varies across studies, but it's the same fundamental issue. I haven't given this a lot of thought, but I would assume that the bias will again be negative, so the strength of the relationship will tend to be underestimated. Unless somebody has a better idea, one could argue that if a relationship is still found, then this provides evidence that a relationship does exist, although the actual strength is uncertain. Best, Wolfgang -----Original Message----- From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bo unces at r-project.org] On Behalf Of C?lia Sofia Moreira Sent: Wednesday, 10 January, 2018 1:04 To: r-sig-meta-analysis at r-project.org Subject: Re: [R-meta] Meta-analysis when sampling covariance matrices are missing Dear all, I have been reading some of the messages in the list related to my problem, and I realized that ?unknown correlations? is an old topic. Special thanks to Prof. Wolfgang, James Pustejovsky, and Isabel Schlegel in https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2017- August/000127.html. Really helpful! I also understood that my attempt to perform the multilevel model in my previous message was wrong. Please, forget the previous questions. My data are multivariate and so outcomes should be analyzed as such. However, I only have sampling means and SD, and thus, only variances of the effects (SMD) are available. So, I will input a covariance matrix (clubSandwich) and/or RVE (robumeta), always with the indispensable rma.mv (metafor). Nevertheless, I would like to investigate a regression between two effects. However, due to the previous limitation, I have no idea if it is possible, and, in affirmative case, how to do it. Thus, any recommendations / suggestions will be much appreciated. In negative case, can the unstructured correlation matrix obtained with the rma.mv output be used to assess the strength of the relationship between these effects? Please, tell me your opinion! Kind regards
_______________________________________________ R-sig-meta-analysis mailing list R-sig-meta-analysis at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
-- Prof. Dr. Philipp Doebler Technische Universit?t Dortmund Fakult?t Statistik Vogelpothsweg 87 <https://maps.google.com/?q=Vogelpothsweg+87+44227+Dortmund&entry=gmail&source=g> 44227 Dortmund <https://maps.google.com/?q=Vogelpothsweg+87+44227+Dortmund&entry=gmail&source=g> Tel.: +49 231-755 8259 <+49%20231%207558259> Fax: +49 231-755 3918 <+49%20231%207553918> doebler at statistik.tu-dortmund.de www.statistik.tu-dortmund.de/1261.html Wichtiger Hinweis: Die Information in dieser E-Mail ist vertraulich. Sie ist ausschlie?lich f?r den Adressaten bestimmt. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, unterrichten Sie bitte den Absender und vernichten Sie diese Mail. Vielen Dank. Unbeschadet der Korrespondenz per E-Mail, sind unsere Erkl?rungen ausschlie?lich final rechtsverbindlich, wenn sie in herk?mmlicher Schriftform (mit eigenh?ndiger Unterschrift) oder durch ?bermittlung eines solchen Schriftst?cks per Telefax erfolgen. Important note: The information included in this e-mail is confidential. It is solely intended for the recipient. If you are not the intended recipient of this e-mail please contact the sender and delete this message. Thank you. Without prejudice of e-mail correspondence, our statements are only legally binding when they are made in the conventional written form (with personal signature) or when such documents are sent by fax.