ROCR package finding maximum accuracy and optimal cutoff point
Found the solution to my own question. To find the false positive rate and the false negative rate that correspond to a certain cutoff point using the ROCR package, one can do the following (for sure there is simpler ways, but this works): library(ElemStatLearn) library(rpart) data(spam) ################################## # create a train and test sets # ################################## index<- 1:nrow(spam) testindex <- sample(index, trunc(length(index)/3)) testset <- spam[testindex, ] trainset <- spam[-testindex, ] rpart.model <- rpart(spam ~ ., data = trainset) # training model ################################## # use ROCR to calculate accuracy # # fp,fn,tp,tn rates # ################################## library(ROCR) rpart.pred2 <- predict(rpart.model, testset)[,2] #testing model pred<-prediction(rpart.pred2,testset[,58]) #prediction using rocr perf.acc<-performance(pred,"acc") #find list of accuracies perf.fpr<-performance(pred,"fpr") # find list of fp rates perf.fnr<-performance(pred,"fnr") # find list of fn rates acc.rocr<-max(perf.acc at y.values[[1]]) # accuracy using rocr #find cutoff list for accuracies cutoff.list.acc <- unlist(perf.acc at x.values[[1]]) #find optimal cutoff point for accuracy optimal.cutoff.acc<-cutoff.list.acc[which.max(perf.acc at y.values[[1]])] #find optimal cutoff fpr, as numeric because a list is returned optimal.cutoff.fpr<-which(perf.fpr at x.values[[1]]==as.numeric(optimal.cutoff.acc)) # find cutoff list for fpr cutoff.list.fpr <- unlist(perf.fpr at y.values[[1]]) # find fpr using rocr fpr.rocr<-cutoff.list.fpr[as.numeric(optimal.cutoff.fpr)] #find optimal cutoff fnr optimal.cutoff.fnr<-which(perf.fnr at x.values[[1]]==as.numeric(optimal.cutoff.acc)) #find list of fnr cutoff.list.fnr <- unlist(perf.fnr at y.values[[1]]) #find fnr using rocr fnr.rocr<-cutoff.list.fnr[as.numeric(optimal.cutoff.fnr)] Now acc.rocr, fpr.rocr, fnr.rocr will give you the accuracy, fpr, and fnr percentages
Saeed Abu Nimeh wrote:
If we use the ROCR package to find the accuracy of a classifier pred <- prediction(svm.pred, testset[,2]) perf.acc <- performance(pred,"acc") Do we find the maximum accuracy as follows (is there a simplier way?):
max(perf.acc at x.values[[1]])
Then to find the cutoff point that maximizes the accuracy do we do the following (is there a simpler way):
cutoff.list <- unlist(perf.acc at x.values[[1]]) cutoff.list[which.max(perf.acc at y.values[[1]])]
If the above is correct how is it possible to find the average false positive and negative rates from the following perf.fpr <- performance(pred, "fpr") perf.fnr <- performance(pred, "fnr") The dataset that consists of two columns; score and a binary response, similar to this: 2.5, 0 -1, 0 2, 1 6.3, 1 4.1, 0 3.3, 1 Thanks, Saeed --- R 2.8.1 Win XP Pro SP2 ROCR package v1.0-2 e1071 v1.5-19