Dear All, I conducted several meta-analyses. I used the R package "metafor" as well as the free open-source software OpenMetaAnalyst for conducting the analyses. Now, I am writing up the results and face some uncertainty with regard to the statistics. 1) I have a p-value of .000 for several effect sizes (Hedge's g). Also for Higgins' I^2 (heterogeneity) I sometimes get this value for p. I would tend to write p < .001 and possibly footnote the .000. Is this possible or what would you suggest in such a case? Simply report it as p < .001? 2) As a measure of heterogeneity, I interpret Higgins' I^2. I interpret the values above 25% as low heterogeneity (following Higgins et al. 2003). In some of the analyses, I get a value of 0%. I would now have to write that there is no heterogeneity. However, I consider this value almost impossible, since there is always a certain variance between the studies. Since I had assumed heterogeneity between the studies, I had also performed the calculations on the basis of a random-effects model. I am also not sure whether values of 25, 50 and 75 % have to be considered as cut-off values or whether values in between can also be interpreted as for example "small to medium" for I^2=45 %. This question also arises for me with Hedge's g. 3) To investigate a possible publication bias, I ran Egger's regression test in R for some of the analyses. I would like to report the regression intercept as well and assume that the intercept is the ?b? (provided together with CI) I get in the output. Would it be correct to report this value as beta^ with index 1? In the literature I found that in most meta-analyses only report the p-value for Egger's test, however the recommendation is made to report the intercept as well. Now I would like to report this regression intercept correctly. I would be very grateful if you could help me out with my questions. (Unfortunately,) I am the first to conduct a meta-analysis in my lab and this topic has not been covered in my studies so far.
[R-meta] Some questions: effect sizes, heterogeneity and Egger's test
2 messages · Teresa Luther, James Pustejovsky
Hi Teresa, Responses below. James
On Mon, Sep 13, 2021 at 9:54 AM Teresa Luther <Teresa.Luther at gmx.de> wrote:
Dear All, I conducted several meta-analyses. I used the R package "metafor" as well as the free open-source software OpenMetaAnalyst for conducting the analyses. Now, I am writing up the results and face some uncertainty with regard to the statistics. 1) I have a p-value of .000 for several effect sizes (Hedge's g). Also for Higgins' I^2 (heterogeneity) I sometimes get this value for p. I would tend to write p < .001 and possibly footnote the .000. Is this possible or what would you suggest in such a case? Simply report it as p < .001?
Yes, reporting as p < .001 is sensible.
2) As a measure of heterogeneity, I interpret Higgins' I^2. I interpret the values above 25% as low heterogeneity (following Higgins et al. 2003). In some of the analyses, I get a value of 0%. I would now have to write that there is no heterogeneity. However, I consider this value almost impossible, since there is always a certain variance between the studies. Since I had assumed heterogeneity between the studies, I had also performed the calculations on the basis of a random-effects model. I am also not sure whether values of 25, 50 and 75 % have to be considered as cut-off values or whether values in between can also be interpreted as for example "small to medium" for I^2=45 %. This question also arises for me with Hedge's g.
I would recommend reporting and focusing your interpretation on the estimate of tau, the between-study heterogeneity parameter, rather than on I^2. The interpretation of I^2 depends on the distribution of sample sizes in your primary studies, so it doesn't make much sense at all to use decontextualized benchmarks based on I^2. For a more detailed explanation of this reasoning and some additional suggestions about how to interpret heterogeneity, check out the following article: Borenstein, M., Higgins, J. P., Hedges, L. V., & Rothstein, H. R. (2017). Basics of meta?analysis: I2 is not an absolute measure of heterogeneity. *Research synthesis methods*, *8*(1), 5-18.
3) To investigate a possible publication bias, I ran Egger's regression test in R for some of the analyses. I would like to report the regression intercept as well and assume that the intercept is the ?b? (provided together with CI) I get in the output. Would it be correct to report this value as beta^ with index 1? In the literature I found that in most meta-analyses only report the p-value for Egger's test, however the recommendation is made to report the intercept as well. Now I would like to report this regression intercept correctly. I don't think there is a standard symbol or notation for the intercept
from Egger's regression test. The more important thing is to give an accurate description of the coefficient, as the estimated average effect size in a study with zero sampling error, based on a model that includes the standard error as a linear predictor. In recent versions of metafor, the regtest() function reports this estimate and describes it as the "Limit Estimate (as sei -> 0)".