Hi Pengzhen,
Oh dear, your intuition is obviously correct. If we exclude
non-significant correlations, then we will overestimate the correlation.
In other words, we would get a biased effect estimate.
This issue is addressed in many meta-analytic textbooks, e.g.
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R.
(2021). Introduction to meta-analysis. John Wiley & Sons.
The authors offer a few free chapters, some of which could be useful in
your case.
https://introduction-to-meta-analysis.com/download/c01.pdf
For example, Figure 1.1 (p. 4) clearly shows that studies with
insignificant p-values are included in the meta-analysis.
The book also contains a chapter, "Vote counting - a new name for an old
problem," which has a nice example showing that even when all individual
studies have large p-values, the meta-analytic estimate can be
statistically significant. By combining the individual studies, we
increase the power and can detect even small effects.
Best,
Lukasz
Lukasz Stasielowicz
Osnabr?ck University
Institute for Psychology
Research methods, psychological assessment, and evaluation
Lise-Meitner-Stra?e 3
49076 Osnabr?ck (Germany)
Twitter: https://twitter.com/l_stasielowicz
Tel.: +49 541 969-7735
On 08.05.2024 03:30, r-sig-meta-analysis-request at r-project.org wrote:
> Send R-sig-meta-analysis mailing list submissions to
> r-sig-meta-analysis at r-project.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> or, via email, send a message with subject or body 'help' to
> r-sig-meta-analysis-request at r-project.org
>
> You can reach the person managing the list at
> r-sig-meta-analysis-owner at r-project.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of R-sig-meta-analysis digest..."
>
>
> Today's Topics:
>
> 1. Online course: Meta-analysis in R (info at physalia-courses.org)
> 2. The P value of correlation coefficent in meta-analysis
> (Pengzhen Huang)
> 3. Correcting gain effects in nested studies (Zhouhan Jin)
> 4. Re: Correcting gain effects in nested studies (James Pustejovsky)
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 7 May 2024 21:06:47 +0200 (CEST)
> From: "info at physalia-courses.org" <info at physalia-courses.org>
> To: r-sig-meta-analysis at r-project.org
> Subject: [R-meta] Online course: Meta-analysis in R
> Message-ID: <1715108807.553710121 at webmail.jimdo.com>
> Content-Type: text/plain; charset="utf-8"
>
>
> Dear all,
> There are only 2 seats left for our upcoming online course, META-ANALYSIS IN R.
> Dates: May 13-16, 2024
> Online: Accessible internationally
> This course covers:
> Systematic review and meta-analysis process
> Statistical analysis methods and interpretation
> Model diagnostics and sensitivity analyses
> Practical exercises with real meta-analytic datasets
> Prerequisites include basic statistical knowledge and familiarity with R. Resources for R preparation are provided.
> Course website: [ https://www.physalia-courses.org/courses-workshops/metain-r/ ]( https://www.physalia-courses.org/courses-workshops/metain-r/ )
>
> Best regards,
>
> Carlo
>
>
>
> --------------------
>
> Carlo Pecoraro, Ph.D
>
>
> Physalia-courses DIRECTOR
>
> info at physalia-courses.org
>
> mobile: +49 17645230846
>
>
>
>
>
> [[alternative HTML version deleted]]
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Wed, 8 May 2024 03:47:15 +0800 (GMT+08:00)
> From: "Pengzhen Huang" <maiqi1317 at 163.com>
> To: =?UTF-8?Q?r-sig-meta-analysis=40r-pr=E2=80=A6?=
> <r-sig-meta-analysis at r-project.org>
> Subject: [R-meta] The P value of correlation coefficent in
> meta-analysis
> Message-ID: <505c373a.1c5.18f54988e70.Coremail.maiqi1317 at 163.com>
> Content-Type: text/plain; charset="utf-8"
>
> Dear Community,
>
>
> I submitted a meta-analysis paper months ago and am now dealing with the reviewers' comments. In my research, the Pearson correlation coefficients are considered as effect size and put into the meta-analysis, and we regard the coefficients representing to what extent two variables are correlated with each other.
>
>
> On this point, one of reviewers argues that "as not all r values are significant, it does not make sense to put these non-significant correlation coefficients into the analysis".
>
>
> I?m not sure how to reply to this reviewer?s comment. But I guess this may be a common issue in meta-analysis. May I have some advice from you or could you tell me some references I should read through?
>
>
> I would greatly appreciate any suggestions you can provide!
>
>
>
>
> All the best,
> Pengzhen
>
>
>
>
>
>
> [[alternative HTML version deleted]]
>
>
>
> ------------------------------
>
> Message: 3
> Date: Wed, 8 May 2024 00:14:11 +0000
> From: Zhouhan Jin <zjin65 at uwo.ca>
> To: R Special Interest Group for Meta-Analysis
> <r-sig-meta-analysis at r-project.org>
> Subject: [R-meta] Correcting gain effects in nested studies
> Message-ID: <d68c25db-a085-4e10-9e4b-d23de85a24ee at Spark>
> Content-Type: text/plain; charset="utf-8"
>
> Hello All,
>
> Hedges (2007) provides formulas for adjusting SMD effects (g) and their SEs for when primary studies have a nested design (below).
>
> But I want to compute gain effects (ex. SMCC in metafor::escalc) from my nested studies, not SMDs.
>
> So, how can I adjust my SMCCs and their SEs for nestedness in the primary studies?
>
>
> adjusted_g = g * sqrt(1 - ((2 * (n_bar - 1) * icc) /
> (n_cluster * n_bar - 2)))
>
> adjusted_SE = ((Nt+Nc)/(Nt*Nc))*(1 + ((n_bar- 1)*icc)) +
> ( g^2 * (
> (((N_tot -2)*(1-icc)^2 ) + (n_bar*(N_tot - 2*n_bar)*icc^2) +
> (2* (N_tot - 2*n_bar) * icc * (1 - icc)) ) /
> ((2* (N_tot-2)) * ( (N_tot-2) - (2* (n_bar-1)*icc) ))
> ) )
>
> Thanks a lot!
>
> Best wishes,
>
> Zhouhan
>
>
> [[alternative HTML version deleted]]
>
>
>
> ------------------------------
>
> Message: 4
> Date: Tue, 7 May 2024 20:30:33 -0500
> From: James Pustejovsky <jepusto at gmail.com>
> To: Zhouhan Jin <zjin65 at uwo.ca>
> Cc: R Special Interest Group for Meta-Analysis
> <r-sig-meta-analysis at r-project.org>
> Subject: Re: [R-meta] Correcting gain effects in nested studies
> Message-ID:
> <CAFUVuJyQmjrJJMFz3EZHGT+rR76YqKr1Zv6AVbvFqLcf+v_AOg at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> See Taylor, Pigott, and Williams (2022;
> https://doi.org/10.3102/0013189X211051319) for how to handle
> cluster-randomized trials that involve gain scores or covariate adjustment.
> They provide a shiny app too. The technical details are also described in
> Appendix E of the What Works Clearinghouse handbook (Version 5;
> https://ies.ed.gov/ncee/WWC/Docs/referenceresources/Final_WWC-HandbookVer5_0-0-508.pdf).
> See pp. 173-174
>
> The methods described in these sources are consistent with the "general
> recipe" for standardized mean difference estimates as described here:
> https://www.jepusto.com/alternative-formulas-for-the-smd/
>
> James
>
> On Tue, May 7, 2024 at 7:14?PM Zhouhan Jin <zjin65 at uwo.ca> wrote:
>
>> Hello All,
>>
>> Hedges (2007) provides formulas for adjusting SMD effects (g) and their
>> SEs for when primary studies have a nested design (below).
>>
>> But I want to compute gain effects (ex. SMCC in metafor::escalc) from my
>> nested studies, not SMDs.
>>
>> So, how can I adjust my SMCCs and their SEs for nestedness in the primary
>> studies?
>>
>>
>> adjusted_g = g * sqrt(1 - ((2 * (n_bar - 1) * icc) /
>> (n_cluster * n_bar - 2)))
>>
>> adjusted_SE = ((Nt+Nc)/(Nt*Nc))*(1 + ((n_bar- 1)*icc)) +
>> ( g^2 * (
>> (((N_tot -2)*(1-icc)^2 ) + (n_bar*(N_tot - 2*n_bar)*icc^2) +
>> (2* (N_tot - 2*n_bar) * icc * (1 - icc)) ) /
>> ((2* (N_tot-2)) * ( (N_tot-2) - (2* (n_bar-1)*icc) ))
>> ) )
>>
>> Thanks a lot!
>>
>> Best wishes,
>>
>> Zhouhan
>>
>>
>
> [[alternative HTML version deleted]]
>
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
>
> ------------------------------
>
> End of R-sig-meta-analysis Digest, Vol 84, Issue 10
> ***************************************************
Hi Guido and Lukasz,
Great thanks for the details! This issue was already extensively discussed in these publications, especially in the meta-analysis book. Thank you for helping working things out!
Cheers!
Pengzhen
---- Replied Message ----
| From | Lukasz Stasielowicz<lukasz.stasielowicz at uni-osnabrueck.de> |
| Date | 5/8/2024 03:25 |
| To | <r-sig-meta-analysis at r-project.org> |
| Cc | <maiqi1317 at 163.com> |
| Subject | Re: The P value of correlation coefficent in meta-analysis |
Hi Pengzhen,
Oh dear, your intuition is obviously correct. If we exclude
non-significant correlations, then we will overestimate the correlation.
In other words, we would get a biased effect estimate.
This issue is addressed in many meta-analytic textbooks, e.g.
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R.
(2021). Introduction to meta-analysis. John Wiley & Sons.
The authors offer a few free chapters, some of which could be useful in
your case.
https://introduction-to-meta-analysis.com/download/c01.pdf
For example, Figure 1.1 (p. 4) clearly shows that studies with
insignificant p-values are included in the meta-analysis.
The book also contains a chapter, "Vote counting - a new name for an old
problem," which has a nice example showing that even when all individual
studies have large p-values, the meta-analytic estimate can be
statistically significant. By combining the individual studies, we
increase the power and can detect even small effects.
Best,
Lukasz
--
Lukasz Stasielowicz
Osnabr?ck University
Institute for Psychology
Research methods, psychological assessment, and evaluation
Lise-Meitner-Stra?e 3
49076 Osnabr?ck (Germany)
Twitter: https://twitter.com/l_stasielowicz
Tel.: +49 541 969-7735
On 08.05.2024 03:30, r-sig-meta-analysis-request at r-project.org wrote:
Send R-sig-meta-analysis mailing list submissions to
r-sig-meta-analysis at r-project.org
To subscribe or unsubscribe via the World Wide Web, visit
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
or, via email, send a message with subject or body 'help' to
r-sig-meta-analysis-request at r-project.org
You can reach the person managing the list at
r-sig-meta-analysis-owner at r-project.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of R-sig-meta-analysis digest..."
Today's Topics:
1. Online course: Meta-analysis in R (info at physalia-courses.org)
2. The P value of correlation coefficent in meta-analysis
(Pengzhen Huang)
3. Correcting gain effects in nested studies (Zhouhan Jin)
4. Re: Correcting gain effects in nested studies (James Pustejovsky)
----------------------------------------------------------------------
Message: 1
Date: Tue, 7 May 2024 21:06:47 +0200 (CEST)
From: "info at physalia-courses.org" <info at physalia-courses.org>
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] Online course: Meta-analysis in R
Message-ID: <1715108807.553710121 at webmail.jimdo.com>
Content-Type: text/plain; charset="utf-8"
Dear all,
There are only 2 seats left for our upcoming online course, META-ANALYSIS IN R.
Dates: May 13-16, 2024
Online: Accessible internationally
This course covers:
Systematic review and meta-analysis process
Statistical analysis methods and interpretation
Model diagnostics and sensitivity analyses
Practical exercises with real meta-analytic datasets
Prerequisites include basic statistical knowledge and familiarity with R. Resources for R preparation are provided.
Course website: [ https://www.physalia-courses.org/courses-workshops/metain-r/ ]( https://www.physalia-courses.org/courses-workshops/metain-r/ )
Best regards,
Carlo
--------------------
Carlo Pecoraro, Ph.D
Physalia-courses DIRECTOR
info at physalia-courses.org
mobile: +49 17645230846
------------------------------
Message: 2
Date: Wed, 8 May 2024 03:47:15 +0800 (GMT+08:00)
From: "Pengzhen Huang" <maiqi1317 at 163.com>
To: =?UTF-8?Q?r-sig-meta-analysis=40r-pr=E2=80=A6?=
<r-sig-meta-analysis at r-project.org>
Subject: [R-meta] The P value of correlation coefficent in
meta-analysis
Message-ID: <505c373a.1c5.18f54988e70.Coremail.maiqi1317 at 163.com>
Content-Type: text/plain; charset="utf-8"
Dear Community,
I submitted a meta-analysis paper months ago and am now dealing with the reviewers' comments. In my research, the Pearson correlation coefficients are considered as effect size and put into the meta-analysis, and we regard the coefficients representing to what extent two variables are correlated with each other.
On this point, one of reviewers argues that "as not all r values are significant, it does not make sense to put these non-significant correlation coefficients into the analysis".
I?m not sure how to reply to this reviewer?s comment. But I guess this may be a common issue in meta-analysis. May I have some advice from you or could you tell me some references I should read through?
I would greatly appreciate any suggestions you can provide!
All the best,
Pengzhen
------------------------------
Message: 3
Date: Wed, 8 May 2024 00:14:11 +0000
From: Zhouhan Jin <zjin65 at uwo.ca>
To: R Special Interest Group for Meta-Analysis
<r-sig-meta-analysis at r-project.org>
Subject: [R-meta] Correcting gain effects in nested studies
Message-ID: <d68c25db-a085-4e10-9e4b-d23de85a24ee at Spark>
Content-Type: text/plain; charset="utf-8"
Hello All,
Hedges (2007) provides formulas for adjusting SMD effects (g) and their SEs for when primary studies have a nested design (below).
But I want to compute gain effects (ex. SMCC in metafor::escalc) from my nested studies, not SMDs.
So, how can I adjust my SMCCs and their SEs for nestedness in the primary studies?
adjusted_g = g * sqrt(1 - ((2 * (n_bar - 1) * icc) /
(n_cluster * n_bar - 2)))
adjusted_SE = ((Nt+Nc)/(Nt*Nc))*(1 + ((n_bar- 1)*icc)) +
( g^2 * (
(((N_tot -2)*(1-icc)^2 ) + (n_bar*(N_tot - 2*n_bar)*icc^2) +
(2* (N_tot - 2*n_bar) * icc * (1 - icc)) ) /
((2* (N_tot-2)) * ( (N_tot-2) - (2* (n_bar-1)*icc) ))
) )
Thanks a lot!
Best wishes,
Zhouhan
------------------------------
Message: 4
Date: Tue, 7 May 2024 20:30:33 -0500
From: James Pustejovsky <jepusto at gmail.com>
To: Zhouhan Jin <zjin65 at uwo.ca>
Cc: R Special Interest Group for Meta-Analysis
<r-sig-meta-analysis at r-project.org>
Subject: Re: [R-meta] Correcting gain effects in nested studies
Message-ID:
<CAFUVuJyQmjrJJMFz3EZHGT+rR76YqKr1Zv6AVbvFqLcf+v_AOg at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
See Taylor, Pigott, and Williams (2022;
https://doi.org/10.3102/0013189X211051319) for how to handle
cluster-randomized trials that involve gain scores or covariate adjustment.
They provide a shiny app too. The technical details are also described in
Appendix E of the What Works Clearinghouse handbook (Version 5;
https://ies.ed.gov/ncee/WWC/Docs/referenceresources/Final_WWC-HandbookVer5_0-0-508.pdf).
See pp. 173-174
The methods described in these sources are consistent with the "general
recipe" for standardized mean difference estimates as described here:
https://www.jepusto.com/alternative-formulas-for-the-smd/
James
On Tue, May 7, 2024 at 7:14?PM Zhouhan Jin <zjin65 at uwo.ca> wrote:
Hello All,
Hedges (2007) provides formulas for adjusting SMD effects (g) and their
SEs for when primary studies have a nested design (below).
But I want to compute gain effects (ex. SMCC in metafor::escalc) from my
nested studies, not SMDs.
So, how can I adjust my SMCCs and their SEs for nestedness in the primary
studies?
adjusted_g = g * sqrt(1 - ((2 * (n_bar - 1) * icc) /
(n_cluster * n_bar - 2)))
adjusted_SE = ((Nt+Nc)/(Nt*Nc))*(1 + ((n_bar- 1)*icc)) +
( g^2 * (
(((N_tot -2)*(1-icc)^2 ) + (n_bar*(N_tot - 2*n_bar)*icc^2) +
(2* (N_tot - 2*n_bar) * icc * (1 - icc)) ) /
((2* (N_tot-2)) * ( (N_tot-2) - (2* (n_bar-1)*icc) ))
) )
Thanks a lot!
Best wishes,
Zhouhan
------------------------------
Subject: Digest Footer
_______________________________________________
R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org
To manage your subscription to this mailing list, go to:
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
------------------------------
End of R-sig-meta-analysis Digest, Vol 84, Issue 10
***************************************************