Skip to content

[R-meta] Metafor results tau^2 and R^2

7 messages · Dustin Lee, Wolfgang Viechtbauer, Gerta Ruecker +1 more

Dustin Lee
# 
Dear all,

I am currently conducting a meta regression in which we are examining the
role of temporal effects (year of study) in the relationship between
organizational attitudes and job performance. Using a mixed-effects model
using ML estimation, our analyses have thus far produced results that do
not appear to be irregular.

Our problem: With one relationship the analysis is showing the following:
tau^2 (estimated amount of residual heterogeneity):     0 (SE = 0.0152)
tau (square root of estimated tau^2 value):             0
I^2 (residual heterogeneity / unaccounted variability): 0.00%
H^2 (unaccounted variability / sampling variability):   1.00
R^2 (amount of heterogeneity accounted for):            100.00%

However, the significance of the effect of 'year of study' is significant
along with the omnibus Q_M statistic. While I inherently understand this is
due to the way in which these values (R^2, tau^2, I^2, etc.) are calculated
and that it may be due to the smaller than ideal sample size (k =32) as
suggested by L?pez?L?pez and colleagues (2014). I am unsure on how these
findings should be reported, particularly the 100% R^2 with the significant
predictor 'year of study' result.

Thank you for any assistance you may be able to provide.

All the best,

Dustin

Reference:
L?pez?L?pez, J. A., Mar?n?Mart?nez, F., S?nchez?Meca, J., Van den
Noortgate, W., & Viechtbauer, W. (2014). Estimation of the predictive power
of the model in mixed?effects meta?regression: A simulation study. *British
Journal of Mathematical and Statistical Psychology*, *67*(1), 30-48.
Gerta Ruecker
# 
Dear Dustin,

The results you report show that in this analysis there was no 
between-study heterogeneity found at all. As explained in the message, 
all measures given are measures of heterogeneity, also R^2. You find all 
definitions in Higgins JP, Thompson SG. Quantifying heterogeneity in a 
meta-analysis. Stat Med. 2002;21(11):1539-1558. doi:10.1002/sim.1186.

R^2 should not be confused with the coefficient of determination (which 
is also often denoted R^2). It is unusual to report the heterogeneity 
measure R^2 in a study report; most authors would report tau, tau^2 or I^2.

See also R?cker G, Schwarzer G, Carpenter JR, Schumacher M. Undue 
reliance on I(2) in assessing heterogeneity may mislead. /BMC Med Res 
Methodol/. 2008;8:79. Published 2008 Nov 27. doi:10.1186/1471-2288-8-79.

Best,

Gerta

Am 08.08.2020 um 22:13 schrieb Dustin Lee:

  
  


  


      

    

    

      
      

        


  

        

      Wolfgang Viechtbauer
    

    

      
      #
    

  


  

      
Hi All,

R^2 in the output of metafor is *not* R^2 from Higgins et al. (2002). It is in fact a (pseudo) coefficient of determination that goes back to Raudenbush (1994). It estimates how much of the (total) heterogeneity is accounted for by the moderator(s) included in the model. If the *residual* amount of heterogeneity (i.e., the unaccounted for heterogeneity) is 0 after including the moderator(s) in the model, then R^2 is going to be 100% (i.e., all of the heterogeneity has been accounted for). One would in fact expect then that the moderator (or set of moderators) is significant -- it would actually be a bit odd if a moderator accounts for all of the heterogeneity, but fails to be significant (although one could probably construct an example where this is the case). And reporting R^2 is definitely useful, although should be cautiously interpreted given that R^2 can be rather inaccurate when k is small (as discussed in L?pez?L?pez et al., 2014).

Best,
Wolfgang

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Gerta Ruecker
    

    

      
      #
    

  


  

      
Dear Wolfgang,

Thank you for clarifying this. I really thought it was the Higgins R^2, 
as it stands in the neighborhood of I^2 and H^2 and also as in the given 
case also its value 1 is plausible (however, in fact , Higgins's R^2 
would not be expressed in percent).

I confused these two R^2s, and I might not be the only person confusing 
these. Do you see a way to avoid this misconception, for example by 
mentioning Raudenbush in the output text?

Best,

Gerta

Am 09.08.2020 um 12:57 schrieb Viechtbauer, Wolfgang (SP):

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Wolfgang Viechtbauer
    

    

      
      #
    

  


  

      
Hi Gerta,

I would have figured the description in the parentheses (amount of heterogeneity accounted for) makes it clear that this is not the "R^2" from Higgins et al. (2002). help(print.rma) also documents the meaning of R^2 in the output. I wonder how many people actually know the "Higgins' R^2", given that I^2 has pretty much come out as the 'winner' from the 2002 paper that everybody reports.

Best,
Wolfgang

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Gerta Ruecker
    

    

      
      #
    

  


  

      
Hi Wolfgang,

Yes. I had read the Higgins 2002 paper a long time ago and knew that 
there was an R^2, but had forgotten how this was defined and what it 
meant. And *just because of that* my mistake arose:

  * R^2 is given by metafor next to I^2 and H^2 (by the way: who knows H^2?)
  * R^2 was 1 in the given example (and not larger)
  * I (probably like many others) didn't know Raudenbush's R^2.

There are simply too many R^2s around (and too few letters in the 
alphabet ...).

Best,

Gerta

Am 10.08.2020 um 12:20 schrieb Viechtbauer, Wolfgang (SP):

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Michael Dewey
    

    

      
      #
    

  


  

      
I suppose that, at the risk of cluttering up the output, both could be 
reported. Then we can spend a happy few hours answering questions on 
this list about what the difference is.

Michael

        
On 10/08/2020 12:52, Gerta Ruecker wrote: