[R-meta] Question on funnel plot interpretation
Dear Gabriel Before you continue searching for small study effects I think a search for possible moderators of the effects would be more worthwhile. If you believe that the purpose of meta-analysis is to synthesise then the spread of values you have calls that goal into question. If you believ that the purpose of meta-analysis is to understand why effects differ then searching for possible moderators must be more helpful than looking for small study efects. Michael
On 05/07/2023 12:31, Gabriel Cotlier wrote:
Ok, so in my case would it be better? to use for instance?either inverse
n or alternatively?variance,? is this correct?
Regarding the high variability in the range of my Fisher's z transformed
values you noticed before.
This variability could be due to the fact that the values come
from?different?geographies, use of varied treatments?and
different?methods?as well?
Is there a quantitative?measure or way that can help to give account of
such variability or explain?it?
Thanks a lot again for your advice and guidance.
Kind regards,
Gabriel
On Wed, Jul 5, 2023 at 2:20?PM Yefeng Yang <yefeng.yang1 at unsw.edu.au
<mailto:yefeng.yang1 at unsw.edu.au>> wrote:
There is a misunderstanding. Fishers Zr's sampling variance (or
error)? is not correlated with Zr. Correlation coefficent r?is
correlated with is variance or error. So, if you transformed r to
Zr, as you said in early email, variance should be a better measure
of precision contrast to inverse n.
Best,
Yefeng
------------------------------------------------------------------------
*From:* Gabriel Cotlier <gabiklm01 at gmail.com
<mailto:gabiklm01 at gmail.com>>
*Sent:* Wednesday, 5 July 2023 21:10
*To:* Yefeng Yang <yefeng.yang1 at unsw.edu.au
<mailto:yefeng.yang1 at unsw.edu.au>>
*Cc:* R Special Interest Group for Meta-Analysis
<r-sig-meta-analysis at r-project.org
<mailto:r-sig-meta-analysis at r-project.org>>; Michael Dewey
<lists at dewey.myzen.co.uk <mailto:lists at dewey.myzen.co.uk>>
*Subject:* Re: [R-meta] Question on funnel plot interpretation
Thanks a lot Yefeng
Regards,
Gabriel
On Wed, Jul 5, 2023 at 2:03?PM Yefeng Yang <yefeng.yang1 at unsw.edu.au
<mailto:yefeng.yang1 at unsw.edu.au>> wrote:
If you are doing an Egger's test using rma.mv <http://rma.mv>,
use SE of Zr estimates as the predictor and look at the slope's
estimate and corresponding test. Also, add other important
predictors that might cause variations.
Best,
Yefeng
------------------------------------------------------------------------
*From:* Gabriel Cotlier <gabiklm01 at gmail.com
<mailto:gabiklm01 at gmail.com>>
*Sent:* Wednesday, 5 July 2023 20:35
*To:* Yefeng Yang <yefeng.yang1 at unsw.edu.au
<mailto:yefeng.yang1 at unsw.edu.au>>
*Cc:* R Special Interest Group for Meta-Analysis
<r-sig-meta-analysis at r-project.org
<mailto:r-sig-meta-analysis at r-project.org>>; Michael Dewey
<lists at dewey.myzen.co.uk <mailto:lists at dewey.myzen.co.uk>>
*Subject:* Re: [R-meta] Question on funnel plot interpretation
Dear Michael and Yefeng?,
Thank you very much for the interesting observations and
orientation provided.
- Regarding?the?strangeness of primary studies.
Yes indeed, it is something I noticed before that some studies
have a very low value or almost no correlation (0.001) while
others have a very high value almost maximum possible (close to 1).
?All correlations included (n = 149)? are the result of
Fisher's z-to-r transformation, and original values of r also in
some cases present such extreme?values.
The primary dataset of correlations were in some cases given in
the screened studies, but for the vast majority of
the?correlations were calculated by myself as Pearsons's
product-moment correlation employing the values reported by the
different studies. These correlations (r) were later transformed
to z by means of Fisher's /r-to-z/ transform.
Studies are very diverse, coming from?different?geographies,
with varied types of treatments and applying?different?methods.
I do not know the reason for such variability?of the range of
the correlations, but it would be interesting to have a test or
quantitative way to give account of such variation in the range.
- Regarding?the? the subjectivity of the interpretation?of the
funnel plot and that I cannot?use the function regtest()since is
I am using rma.mv <http://rma.mv>() object,?I also run numerical
test for publication bias employing different?predictors:
1. sampling variance
2. inverse of sampling variance
3. standard?error
However since for each of the cases/predictors used (see below)
I got a full model result, I assume --may be wrongly--that the
value I should take as the numerical?estimation?of the
publication bised is the "intercept" of the model, is this correct?
Is there a given range that might serve as a proxy indicator of
potential?publication bias?
Code and model's results?below.
Thanks a?lot.
Kind regards,
Gabriel
## NUMERICAL TEST FOR PUBLICATION BIAS
## extending Egger's test to more complex models.
## "regression test for funnel plot asymmetry".
## 1. using : the sampling variance
PubB<-rma.mv <http://rma.mv>(yi = yi,
? ? ? ? ? ? ?V = vi,
? ? ? ? ? ? ?mods = ?vi,
? ? ? ? ? ? ?random = ~ 1 | Article / Sample_ID,
? ? ? ? ? ? ?data = dat,
? ? ? ? ? ? ?method = "REML")
PubB
# Multivariate Meta-Analysis Model (k = 149; method: REML)
#
# Variance Components:
#
# ? ? ? ? ? ?estim ? ?sqrt ?nlvls ?fixed ? ? ? ? ? ? factor
# sigma^2.1 ?0.8908 ?0.9438 ? ? 72 ? ? no ? ? ? ? ? ?Article
# sigma^2.2 ?2.1970 ?1.4822 ? ?149 ? ? no ?Article/Sample_ID
#
# Test for Residual Heterogeneity:
# QE(df = 147) = 24617.3110, p-val < .0001
#
# Test of Moderators (coefficient 2):
# QM(df = 1) = 1.4381, p-val = 0.2304
#
# Model Results:
#
# ? ? ? ? ?estimate ? ? ?se ? ? zval ? ?pval ci.lb
<http://ci.lb> ? ci.ub
# intrcpt ? ?0.5620 ?0.2584 ? 2.1743 ?0.0297 ? ?0.0554 ?1.0685 ?*
# mods ? ? ?-6.7997 ?5.6701 ?-1.1992 ?0.2304 ?-17.9130 ?4.3135
#
# ---
# ? Signif. codes: ?0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1
## 2. using : the inverse of the sampling variance
PubB_1<-rma.mv <http://rma.mv>(yi = yi,
? ? ? ? ? ? ?V = vi,
? ? ? ? ? ? ?mods = ?1/vi,
? ? ? ? ? ? ?random = ~ 1 | Article / Sample_ID,
? ? ? ? ? ? ?data = dat,
? ? ? ? ? ? ?method = "REML")
PubB_1
# Multivariate Meta-Analysis Model (k = 149; method: REML)
#
# Variance Components:
#
# ? ? ? ? ? ?estim ? ?sqrt ?nlvls ?fixed ? ? ? ? ? ? factor
# sigma^2.1 ?0.9176 ?0.9579 ? ? 72 ? ? no ? ? ? ? ? ?Article
# sigma^2.2 ?2.1797 ?1.4764 ? ?149 ? ? no ?Article/Sample_ID
#
# Test for Residual Heterogeneity:
# ? QE(df = 147) = 23656.2997, p-val < .0001
#
# Test of Moderators (coefficient 2):
# QM(df = 1) = 1.4469, p-val = 0.2290
#
# Model Results:
#
# ? ? ? ? estimate ? ? ?se ? ?zval ? ?pval ci.lb <http://ci.lb>
? ci.ub
# intrcpt ? ?0.0957 ?0.2595 ?0.3689 ?0.7122 ?-0.4129 ?0.6044
# mods ? ? ? 0.0040 ?0.0034 ?1.2029 ?0.2290 ?-0.0025 ?0.0106
#
# ---
# ? Signif. codes: ?0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1
## 3. using : standard errors (square-root of the sampling
variances)
PubB_2<-rma.mv <http://rma.mv>(yi = yi,
? ? ? ? ? ? ? ?V = vi,
? ? ? ? ? ? ? ?mods = ?sqrt(vi),
? ? ? ? ? ? ? ?random = ~ 1 | Article / Sample_ID,
? ? ? ? ? ? ? ?data = dat,
? ? ? ? ? ? ? ?method = "REML")
PubB_2
# Multivariate Meta-Analysis Model (k = 149; method: REML)
#
# Variance Components:
#
# ? ? ? ? ? ?estim ? ?sqrt ?nlvls ?fixed ? ? ? ? ? ? factor
# sigma^2.1 ?0.8991 ?0.9482 ? ? 72 ? ? no ? ? ? ? ? ?Article
# sigma^2.2 ?2.1952 ?1.4816 ? ?149 ? ? no ?Article/Sample_ID
#
# Test for Residual Heterogeneity:
# QE(df = 147) = 24489.6313, p-val < .0001
#
# Test of Moderators (coefficient 2):
# QM(df = 1) = 1.2528, p-val = 0.2630
#
# Model Results:
#
# ? ? ? ? ? estimate ? ? ?se ? ? zval ? ?pval ci.lb
<http://ci.lb> ? ci.ub
# intrcpt ? ?0.7801 ?0.4374 ? 1.7834 ?0.0745 ?-0.0772 ?1.6375 ?.
# mods ? ? ?-2.6473 ?2.3652 ?-1.1193 ?0.2630 ?-7.2829 ?1.9883
#
# ---
# ? Signif. codes: ?0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1
On Wed, Jul 5, 2023 at 12:45?PM Yefeng Yang
<yefeng.yang1 at unsw.edu.au <mailto:yefeng.yang1 at unsw.edu.au>> wrote:
Dear Gabriel
Apart from?Michael observation on data checking before
analyses (which is always a good practice), I add one.
The funnel plot is just a visual check of publication bias.
So the observations based on the funnel plots are inevitably
subjective - I mean you think that is "randomly scattering",
while others might think not. In contrast, Egger's test is a
more objective way to test the asymmetry of a funnel plot.
Regarding how to do it, you can try to find them in the
archives associated with this mailing list.
Please be noted that whether none funnel plot and Egger's
test can indicate publication bias directly. But it is
common to assume the asymmetry of a funnel plot is caused by
publication bias (or more precisely. small study effects),
after accounting for heterogeneity.
Best,
Yefeng
------------------------------------------------------------------------
*From:* R-sig-meta-analysis
<r-sig-meta-analysis-bounces at r-project.org
<mailto:r-sig-meta-analysis-bounces at r-project.org>> on
behalf of Michael Dewey via R-sig-meta-analysis
<r-sig-meta-analysis at r-project.org
<mailto:r-sig-meta-analysis at r-project.org>>
*Sent:* Wednesday, 5 July 2023 19:31
*To:* R Special Interest Group for Meta-Analysis
<r-sig-meta-analysis at r-project.org
<mailto:r-sig-meta-analysis at r-project.org>>
*Cc:* Michael Dewey <lists at dewey.myzen.co.uk
<mailto:lists at dewey.myzen.co.uk>>; Gabriel Cotlier
<gabiklm01 at gmail.com <mailto:gabiklm01 at gmail.com>>
*Subject:* Re: [R-meta] Question on funnel plot interpretation
Dear Gabriel
My interpretation looking at your plots is that you have a
very strange
set of primary studies. If the x-axis is really the z
transformation of
r then some of? the r are .999 and some 0.001 which seems
worthy of
investigation before looking further.
Michael
On 05/07/2023 09:14, Gabriel Cotlier via R-sig-meta-analysis
wrote:
> Hello all,
>
> I have produced?a funnel plot on the basis of an rma.mv <http://rma.mv>
> <http://rma.mv>()?objectapplied to all the data set together
( not
> subsetting? using moderators ) as follows:
>
> image.png
>
>
> When looking at the figure I tried to think that maybe one of the
> following two interpretations could be the correct?one:
>
> a.? There is a kind of random scattering of the effect sizes, therefore
> no symmetry is found and thus publication bised is observed.
> b.? Given the randomness of the effect sizes distribution?covering the
> plot's space?unevenly?there is not a clear pattern that can indicate
> publication bias is observed.
>
> Is any of this interpretation?the correct one?
>
> Thanks a lot.
> Kind?regards,
> Gabriel
>
>
> #### CODE. ######
> funnel_all <- rma.mv <http://rma.mv> <http://rma.mv <http://rma.mv>>(yi,
>? ? ? ? ? ? ? ? ? ? ? ?vi,
>? ? ? ? ? ? ? ? ? ? ? ?random = ~ 1 | Article / Sample_ID,
>? ? ? ? ? ? ? ? ? ? ? ?data=dat)
> png(file = "funnel.png",
>? ? ? width = 250,
>? ? ? height = 200,
>? ? ? res = 600,
>? ? ? units = "mm")
> # par(mfrow = c(2, 1))
>
> # full data
> f1 <- funnel(funnel_all,
>? ? ? ? ? ? ? ?yaxis = "seinv",
>? ? ? ? ? ? ? ?level = c(90, 95, 99),
>? ? ? ? ? ? ? ?ylim = c(1, 20),
>? ? ? ? ? ? ? ?shade = c("white", "gray55", "gray75"),
>? ? ? ? ? ? ? ?refline = 0,
>? ? ? ? ? ? ? ?legend = TRUE)
> mtext("A", side = 3, line = 0, adj = -0.13, cex = 2)
>
> dev.off()
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>> Virus-free.www.avg.com <http://Virus-free.www.avg.com> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>>
>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org
<mailto:R-sig-meta-analysis at r-project.org>
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
_______________________________________________
R-sig-meta-analysis mailing list @
R-sig-meta-analysis at r-project.org
<mailto:R-sig-meta-analysis at r-project.org>
To manage your subscription to this mailing list, go to:
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
<https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>