Skip to content

[R-meta] Meta-Analysis using different correlation coefficients

3 messages · Lukasz Stasielowicz, Lena Pollerhoff

Lukasz Stasielowicz
# 
Dear Lena,

since you haven't received a response yet, I will address some points:

It is important to define a target effect size for the meta-analysis, 
e.g. Pearson correlation r. If other effect sizes are reported in the 
primary studies then one could transform them or use the raw data to 
compute the target effect size.

If Pearson correlation is the target effect size and you have 
point-biserial correlations then no transformation is necessary because 
it is mathematically equivalent to Pearson correlation.

Spearman correlation and Pearson correlation will not always lead to 
similar values, as the latter is less likely to detect nonlinear 
(monotonic) relationships. Therefore, transforming Spearman correlations 
to Pearson correlations could ensure, that the effect size values can be 
interpreted the same way. In this particular case one could use arcsin 
formulas:
de Winter, J. C. F., Gosling, S. D., & Potter, J. (2016). Comparing the 
Pearson and Spearman correlation coefficients across distributions and 
sample sizes: A tutorial using simulations and empirical data. 
Psychological Methods, 21(3), 273?290. doi.org/10.1037/met0000079
You'll find formulas for other transformations in textbooks and other 
journal articles.

While many meta-analysts tend to use all available information and apply 
all possible transformations (d --> r, Spearman --> Pearson, OR --> r), 
there are some people, who would point out that under certain 
circumstances not all transformations are valid (e.g., d --> r  can the 
distinction between control group and experimental group be thought as a 
part of an underlying continuous distribution or is it a natural 
dichotomy?).
However, some people would argue that every effect size is useful (in 
particular, if the meta-analysis is small).
Sometimes meta-analysts conduct a moderator analysis and compare 
converted effect sizes (e.g., d-->r) with effect sizes that didn't 
require converting (e.g. r), in order to check the influence of pooling 
different designs/effects.

Multiple effect sizes: If there are multiple studies with multiple 
effect sizes then a three-level meta-analysis could be useful. 
Alternatively, one could use cluster-robust variance estimation, e.g. 
cran.r-project.org/web/packages/clubSandwich/vignettes/meta-analysis-with-CRVE.html
If there is only one sample with multiple effects then choosing one 
effect size (e.g., the typical operationalization of the constructs, the 
most valid operationalization) could be an option. Alternatively, one 
could compute a mean effect size, as you have mentioned in your message.


I hope it answers at least some of your questions. Good luck with your 
project!


Best,
Lukasz
6 days later
Lena Pollerhoff
# 
Dear Lukasz,

Thank you so much for answering, we really appreciate the effort and your time. 

We had also done some research, which lead us to the following thread from Wolfgang Viechtbauer: https://chat.stackexchange.com/rooms/60238/discussion-between-mark-white-and-wolfgang <https://chat.stackexchange.com/rooms/60238/discussion-between-mark-white-and-wolfgang>, and further to his paper (Jacobs & Viechtbauer) regarding point-biserial and biserial correlations (?Estimation of the biserial correlation and its sampling variance for use in meta-analysis?) from 2016 (https://onlinelibrary.wiley.com/doi/full/10.1002/jrsm.1218 <https://onlinelibrary.wiley.com/doi/full/10.1002/jrsm.1218>).

Based on this we were a little worried about including point-biserial and Pearson?s product-moment correlations in the same meta-analysis without transforming the point-biserial coefficients? E.g., in the paper they are saying ?Unlike the point-biserial correlation coefficient, biserial coefficients can therefore be integrated with product-moment correlation coefficients in the same meta-analysis?, which lead us to assume (contrary to your suggestion) that point-biserial and product-moment correlation coefficients should not be included in the same meta-analysis without further transformation? But the paper also exclusively treats cases where a variable was *artificially* dichotomized (and still analyzed with point-biserial correlation instead of biserial correlation). 

But you are suggesting that in case the point-biserial correlation is based on a naturally occurring binary variable (e.g., gender), we can integrate it with Pearson's poruduct-moment correlation in the same analysis?

Thank you so much in advance and best wishes
Lena

  
  


  


      

    

    

      
      

        


  

        

      Lukasz Stasielowicz
    

    

      
      #
    

  


  

      
Dear Lena,

the order of the sentences in my original reply could be confusing, so 
just to clarify:
I meant that it is generally possible to transform effect sizes and many 
meta-analysts choose this option. However, I wasn't suggesting that this 
approach is better/equivalent/inferior. If you choose this option, then 
point-biserial correlation is mathematically equivalent to Pearson 
correlation. Hence, there would be no need to transform the correlation.

However, I've mentioned that there are arguments against transforming 
effect sizes. I should have added that it also means that one could 
argue that point-biserial correlation and Pearson correlation shouldn't 
be considered together in one meta-analysis. In general, "when in doubt 
follow Wolfgang's advice" is a good heuristic. In particular, if you 
don't have a strong opinion on a certain topic.

There are many decisions that can be criticized during peer-review in 
primary studies (why ANOVA and not multi-level models? why multi-level 
models and not structural equation modelling?) and meta-analysis (e.g., 
dealing with dependent effect sizes, inclusion of regression 
coefficients when sythesizing correlations, inclusion of studies without 
weighting/post-stratification). While there are some decisions on which 
most of us would agree there are also areas where the opinions may vary 
(or: some reviewers may have strong opinions and other can be 
indifferent). Transformation/inclusion of other effect sizes belongs to 
the second category.

In your case, the safest solution is to restrict meta-analysis to only 
one type of effect sizes (e.g. Pearson correlation). You coud cite the 
article (Jacobs & Viechtbauer) to justify exclusion of other effect sizes.
However, if this means that say only 5 studies meet your inclusion 
criteria, then one could argue that including/transforming all effect 
sizes is justifiable (more information, moderator analyses etc). One 
could conduct a moderator analysis (effect size type, see previous 
message) in order to test the robustness of the results and address 
potential concerns of reviewers/readers.
There are many meta-analysts, who would consider all effect sizes 
irrespective of the number of available studies but it is risky as some 
reviewers could criticize the lack of justification. Providing a 
justification could help, but in (rare?) cases it will be rejected.


Best,
Lukasz