Hello, I'd need to compute the calibration error of posterior class probabilities p(y|x) estimated by using rpart as classification tree. Namely, I train rpart on a dataset D and then use predict(... type="prob") to estimate p(y|x). I've found the possibility to do that in the ROCR package, but I cannot find a link to a paper/book which explains the details of the implemented algorithm. Do you know of any reference where I can find the details of the algorithm that computes the calibration error implemented in ROCR (apart from ROCR's source code)? Is there any other function/package I can use to compute the calibration error? thank you, regards, Roberto
Probabilities calibration error & ROCR
2 messages · Roberto Perdisci, Tobias Sing
Roberto,
On 1/24/07, Roberto Perdisci <roberto.perdisci at gmail.com> wrote:
[...]. Do you know of any reference where I can find the details of the algorithm that computes the calibration error implemented in ROCR (apart from ROCR's source code)?
we use it as defined in Caruana & Niculescu-Mizil: "Data mining in metric space: An empirical evaluation of supervised learning performance criteria". Knowledge Discovery and Data Mining (KDD) 2006. http://www.cs.cornell.edu/~caruana/perfs.rocai04.revised.rev1.ps Also, have a look at ?performance. HTH, Tobias