[R-meta] Any metafor resource for meta-analysis with individual data
Dear Lukasz, Thank you for sharing your perspective. I should certainly learn more about IPD meta-analysis. My main goal was to understand whether any form of meta-analysis (e.g., with IPD) might have any methodological advantage over pulling across the replications using standard mixed-effects modeling. I should add that these 10 replication experiments are almost exactly the same and if there are differences they are solely due to the items' characteristics on the pre- and post-tests across the replications as well as the participants' characteristics taking those tests across the replications. I'll study IPD meta-analysis to better understand the parameters that IPD meta-analysis estimates and how they differ in substance and efficiency from the ones estimated by an equivalent standard mixed-effects model. Thanks again, Luke On Mon, Mar 21, 2022 at 7:29 AM Lukasz Stasielowicz
<lukasz.stasielowicz at uni-osnabrueck.de> wrote:
Dear Luke, whether aggregation is sensible depends on the research question. In the replication context one could certainly argue that examining differences between studies through a standard meta-analysis might be interesting. In fact many initatives have done this, e.g. Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B., Albohn, D. N., Allard, E. S., Benning, S. D., Blouin-Hudon, E.-M., Bulnes, L. C., Caldwell, T. L., Calin-Jageman, R. J., Capaldi, C. A., Carfagno, N. S., Chasten, K. T., Cleeremans, A., Connell, L., DeCicco, J. M., ? Zwaan, R. A. (2016). Registered replication report: Strack, Martin, & Stepper (1988). Perspectives on Psychological Science, 11(6), 917?928. https://doi.org/10.1177/1745691616674458 However, including raw data can have some advantages (e.g. including individual-level moderators and not only summary statistiscs: age of person 1, age of person 2 etc. vs mean age of the sample). Thus, one could conduct an individual participant data meta-analysis. If you're not familiar with IPD meta-analysis then I would recommend taking a look at the resources provided by Richard Riley (Code, Videos etc.): https://www.ipdma.co.uk/ Best wishes, Lukasz -- Lukasz Stasielowicz Osnabr?ck University Institute for Psychology Research methods, psychological assessment, and evaluation Seminarstra?e 20 49074 Osnabr?ck (Germany) Am 20.03.2022 um 12:00 schrieb r-sig-meta-analysis-request at r-project.org:
Send R-sig-meta-analysis mailing list submissions to
r-sig-meta-analysis at r-project.org
To subscribe or unsubscribe via the World Wide Web, visit
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis or, via email, send a message with subject or body 'help' to r-sig-meta-analysis-request at r-project.org You can reach the person managing the list at r-sig-meta-analysis-owner at r-project.org When replying, please edit your Subject line so it is more specific than "Re: Contents of R-sig-meta-analysis digest..." Today's Topics: 1. Re: Can z-transformed R-squared be used as an effect size? (Viechtbauer, Wolfgang (SP)) 2. Any metafor resource for meta-analysis with individual data (Luke Martinez) ---------------------------------------------------------------------- Message: 1 Date: Sat, 19 Mar 2022 11:09:41 +0000 From: "Viechtbauer, Wolfgang (SP)" <wolfgang.viechtbauer at maastrichtuniversity.nl> To: "r-sig-meta-analysis at r-project.org" <r-sig-meta-analysis at r-project.org> Subject: Re: [R-meta] Can z-transformed R-squared be used as an effect size? Message-ID: <4a693fee87b447d4abc929d29696e3d0 at UM-MAIL3214.unimaas.nl> Content-Type: text/plain; charset="us-ascii" I sent a response yesterday along the same lines, but it never made it through to the list. Very strange. Maybe the spam catcher was trying to tell me to stop posting so much ... In any case, just for the record, let me try to post my response one more time: ############################## Hi Matthew, And for once, there is also a pretty quick answer: Doing this is nonsense. The r-to-z transformation is a variance-stabilizing transformation for Pearson product-moment correlation coefficients. It does not work the same way for squared correlations or R^2 values. Also, the variance of an r-to-z transformed R^2 value is not just 1/(n-3) as for r-to-z transformed Pearson product-moment correlation coefficients. So yes, your intuition that this is problematic is absolutely right. ############################## Let's see if this one goes through. Best, Wolfgang -----Original Message----- From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of Michael Dewey Sent: Friday, 18 March, 2022 17:06 To: Matthew Yates; r-sig-meta-analysis at r-project.org Subject: Re: [R-meta] Can z-transformed R-squared be used as an effect size? Dear Matthew I share your concerns and doubts about this. I would hesitate to go out on a limb over it but if you are reviewing it for a journal I would suggest politely telling the authors that you find it hard to justify and ask them to provide a reference or other wise justify it. I do not make confidential comments to the editors on principle but if I did I would tell them that unless the authors can justify it the paper demands instant rejection. Michael On 17/03/2022 17:47, Matthew Yates wrote: Hello SIG-meta folks, I have (what I think is) a pretty quick question. I'm currently conducting a peer-review of a meta-analysis in my field. The authors of this manuscript elected to use fisher-Z-transformed R-squared values (note: NOT Pearson correlation coefficients, but their squared-values) as their 'effect size' statistic, and then calculated variance for the z-transformed R-squared values as for a typical Pearson correlation coefficient (r). Is this a valid statistical effect size? The z-transformation, as I understand it, was developed specifically for Pearson correlation coefficients, so this strikes me as potentially problematic - the z-transformation, itself, is meant for variables that can span -1 to 1, so the underlying distribution of the transformed variables (0 to 1 for r-squared values) are inherently different. Similarly, estimating the variances based on the sample size (n) of the z- transformed R-squared values again strikes me as potentially problematic as well. As far as I can tell, I think pretty much all of the studies being analysed were bivariate linear regressions, so there isn't an issue with non-linear relationships, covariates, etc (I saw another post on here asking that question....). I've just never seen this done before in a meta-analysis, or read of it in any literature, guides, etc. on how to conduct a meta-analysis. Most people typically just use the z-transformed Pearson correlation coefficients, rather than the R-squared values! I've done a few meta-analyses myself, so am familiar with general techniques but would not consider myself an expert/specialist (most of mine were pretty basic). However, this strikes me as potentially problematic, and I was wondering what others with more statistical expertise in meta-analytic techniques might think of this issue. Any input would be appreciated. Thanks, Dr. Matthew C. Yates Post-doctoral Researcher Great Lakes Institute for Environmental Research (GLIER) University of Windsor 2990 Riverside Dr W, Windsor, ON N9C 1A2 (514) 919 5613 Website: https://matthewyates6.wixsite.com/ecologist ------------------------------ Message: 2 Date: Sat, 19 Mar 2022 23:26:00 -0500 From: Luke Martinez <martinezlukerm at gmail.com> To: R meta <r-sig-meta-analysis at r-project.org> Subject: [R-meta] Any metafor resource for meta-analysis with individual data Message-ID: <CAJnTYHGAH+RDR=LXG2y1ZU1z+oMNGG+v+MNUC6b76p_674ohMw at mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hello All, Imagine a single-group experiment (same group pre- and post-tested) has been replicated ten times with independent groups of participants by the same lab: R O X O ==> 10 times In each experiment, participants answer a multiple-choice test at the pre- and the post-test. So, participants' responses on each test item are binary (correct, not correct). So, we have access to the raw data. In this situation, is there any advantage for meta-analyzing these 10 replication experiments? Can metafor do this? I'd highly appreciate any resources/guidance that I could access to learn more about software, or methodology surrounding this issue. Sincerely, Luke ------------------------------ Subject: Digest Footer _______________________________________________ R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org To manage your subscription to this mailing list, go to: https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis ------------------------------ End of R-sig-meta-analysis Digest, Vol 58, Issue 30 ***************************************************