Skip to content

aucRoc in caret package [SEC=UNCLASSIFIED]

7 messages · Li Jin, Max Kuhn, David Winsemius

Original
David Winsemius
# 
Using AUC for discrete predictor variables with inly two levels  
doesn't seem very sensible. What are you planning to to with this  
measure?
Li Jin
# 
Please note that predicted1 and predicted2 are two sets of predictions instead of predictors. As you can see the predictions with only two levels, 1 is for hard and 2 for soft. I need to assess which one is more accurate. Hope this is clear now. Thanks.
Jin

-----Original Message-----
From: David Winsemius [mailto:dwinsemius at comcast.net] 
Sent: Thursday, 2 June 2011 10:55 AM
To: Li Jin
Cc: R-help at r-project.org
Subject: Re: [R] aucRoc in caret package [SEC=UNCLASSIFIED]

Using AUC for discrete predictor variables with inly two levels  
doesn't seem very sensible. What are you planning to to with this  
measure?
David Winsemius
#
On Jun 1, 2011, at 9:24 PM, <Jin.Li at ga.gov.au> <Jin.Li at ga.gov.au> wrote:

            

      
      
      
    

        
Yes, I (very clearly I think) saw that.

            

      
      
      
    

        
So how big do you want to dig your hole? AUC is not designed to be a  
score for categorical variables. It's designed for a continuous  
predictor. The only information in your two-way classification of  
dichotomous states is in the off-axis values.... 11 to naught versus  
11 to 2. Other than that you have total agreement.  Not much to work on.

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Max Kuhn
    

    

      
      #
    

  


  

      
David,

The ROC curve should really be computed with some sort of numeric data
(as opposed to classes). It varies the cutoff to get a continuum of
sensitivity and specificity values. ?Using the classes as 1's and 2's
implies that the second class is twice the value of the first, which
doesn't really make sense.

Try getting the class probabilities for predicted1 and predicted2 and
use those instead.

Thanks,

Max

        
On Wed, Jun 1, 2011 at 9:24 PM, <Jin.Li at ga.gov.au> wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        
--

Max

  
  


  


      

    

    

      
      

        


  

        

      David Winsemius
    

    

      
      #
    

  


  

      
On Jun 1, 2011, at 10:41 PM, Max Kuhn wrote:

        

            

      
      
      
    

        
Yes. You should be addressing this to Jin. I have been trying with  
little success to explain this.

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Li Jin
    

    

      
      #
    

  


  

      
Hi All,
Thanks for the clarification. Now, perhaps I should use kappa instead.
Since my predictions are in 1 and 2, there are no numeric predictions. To my surprise, when I applied kappa and auc to the data, their values are highly correlated, with only an exception when there are perfect predictions for one or both classes. 
Are there any other accuracy measurements applicable to such predictions with two unbalanced classes?
Thanks,
Jin
-----Original Message-----
From: Max Kuhn [mailto:mxkuhn at gmail.com] 
Sent: Thursday, 2 June 2011 12:41 PM
To: Li Jin
Cc: dwinsemius at comcast.net; R-help at r-project.org
Subject: Re: [R] aucRoc in caret package [SEC=UNCLASSIFIED]

David,

The ROC curve should really be computed with some sort of numeric data
(as opposed to classes). It varies the cutoff to get a continuum of
sensitivity and specificity values. ?Using the classes as 1's and 2's
implies that the second class is twice the value of the first, which
doesn't really make sense.

Try getting the class probabilities for predicted1 and predicted2 and
use those instead.

Thanks,

Max

        
On Wed, Jun 1, 2011 at 9:24 PM, <Jin.Li at ga.gov.au> wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        
--

Max