Skip to content

Calibration score for survival probability

3 messages · Eleni Rapsomaniki, Frank E Harrell Jr

Eleni Rapsomaniki
# 
Good afternoon!

I need to evaluate the goodness-of-fit (aka calibration) for survival probability estimates from a Cox model. 
I tried to use 'calibrate' in the Design package but I'm not sure if it should/would produce what I need (ie a chi-sq type statistic with a table of expected vs observed probabilities). Any other functions I should be aware of?

Also, has anybody come across an implementation of the statistic described in:
"A global goodness of fit statistic for Cox regression models" by Parzen & Lpisitz, Biometrics 55, 1999 

Many thanks in advance

Eleni Rapsomaniki

Research Associate
Strangeways Research Laboratory
Department of Public Health and Primary Care

University of Cambridge
?
Frank E Harrell Jr
#
Eleni Rapsomaniki wrote:
Eleni,

The Design package, and its replacement, the rms package, produces 
calibration curves but no chi-square test because we do not have a 
corresponding method for that.  Formal tests are overused in this 
context anyway.  An index such as the maximum or 90th percentile of 
absolute calibration error are often more useful.  I have learned 
however that any statistical method that categorizes continuous 
variables (in this case, the predictions or the covariate space) is 
arbitrary and has many other problems.  The calibrate functions in the 
rms package have a new option to obtain smooth calibration curves 
without grouping, by fitting spline hazard models during validation.

Note that if you have done any model/variable selection you have to 
re-run such model building from scratch for each resample of the data, 
or the calibration plot will be over optimistic.  calibrate() makes this 
automatic if doing backward stepdown variable selection.  Many 
statisticians make the mistake of only "validating" the final selected 
model, which can only be done by one-time data splitting (which requires 
  tens of thousands of observations to perform adequately).

Frank

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Eleni Rapsomaniki
    

    

      
      #
    

  


  

      
Dear Prof. Harrell,
Thank you very much for your prompt and very helpful response. I guess
that since a global statistic such as a chi-sq test is not applicable in
this case, the calibration curve itself from the calibration() is the
most informative alternative (most graphical methods reveal more
information than a single statistic anyway!). I will try the updated
version in the rms package to compare.

Best Wishes 

Eleni Rapsomaniki
 
Research Associate
Strangeways Research Laboratory
Department of Public Health and Primary Care
University of Cambridge
 

-----Original Message-----
From: Frank E Harrell Jr [mailto:f.harrell at vanderbilt.edu] 
Sent: 23 November 2009 13:01
To: Eleni Rapsomaniki
Cc: r-help at r-project.org
Subject: Re: [R] Calibration score for survival probability

        
Eleni Rapsomaniki wrote:

            

      
      
      
    

        
probability estimates from a Cox model.

            

      
      
      
    

        
it should/would produce what I need (ie a chi-sq type statistic with a
table of expected vs observed probabilities). Any other functions I
should be aware of?

            

      
      
      
    

        
described in:

            

      
      
      
    

        
Parzen & Lpisitz, Biometrics 55, 1999

            

      
      
      
    

        
Eleni,

The Design package, and its replacement, the rms package, produces 
calibration curves but no chi-square test because we do not have a 
corresponding method for that.  Formal tests are overused in this 
context anyway.  An index such as the maximum or 90th percentile of 
absolute calibration error are often more useful.  I have learned 
however that any statistical method that categorizes continuous 
variables (in this case, the predictions or the covariate space) is 
arbitrary and has many other problems.  The calibrate functions in the 
rms package have a new option to obtain smooth calibration curves 
without grouping, by fitting spline hazard models during validation.

Note that if you have done any model/variable selection you have to 
re-run such model building from scratch for each resample of the data, 
or the calibration plot will be over optimistic.  calibrate() makes this

automatic if doing backward stepdown variable selection.  Many 
statisticians make the mistake of only "validating" the final selected 
model, which can only be done by one-time data splitting (which requires

  tens of thousands of observations to perform adequately).

Frank

            

      
      
      
    

        

      
      
    

        
http://www.R-project.org/posting-guide.html