statistical significance of accuracy increase in classification

Tue, Feb 24, 2009 9:48 AM

Hi again,
 
Looking more into test statistics i realized that maybe i can use the power.prop.test to see if the difference between the 2 accuracies are zero or not. Do you have any comments about that? Also, should i considered kappa statistics also a kind of proportion and use the same test? If this does not violate any important hypothesis then ....
 
power.prop.test(n = 146, p1 = 0.7877, p2 = 0.8014, strict = TRUE)
 
     Two-sample comparison of proportions power calculation 
              n = 146
             p1 = 0.7877
             p2 = 0.8014
      sig.level = 0.05
          power = 0.0596356
    alternative = two.sided
 NOTE: n is number in *each* group 

which just tells that the difference in accuracies are barely different .... since the p.value = 0.06> 0.05
 
For Kappa statistics it will be:
 
power.prop.test(n = 146, p1 = 0.3675, p2 = 0.4315, strict = TRUE)
 
     Two-sample comparison of proportions power calculation 
              n = 146
             p1 = 0.3675
             p2 = 0.4315
      sig.level = 0.05
          power = 0.1999816
    alternative = two.sided
 NOTE: n is number in *each* group 

Any comments are really appreciated,
 
Monica

----------------------------------------

From: pisicandru at hotmail.com
To: r-help at r-project.org
CC: max.kuhn at pfizer.com
Subject: [R] statistical significance of accuracy increase in classification
Date: Tue, 24 Feb 2009 16:22:41 +0000


Hi everyone,

I would like to test for the statistical significance(for what it worth ...) in increasing classification accuracy and kappa statistics from different land classifications. The classifications were done using other software (like eCognition and See5), but the results were "sampled" at locations where i have the "reference" class known. So using package "caret" i did the confusion matrix. For now i am interested in the overall results which give the overall classification accuracy and kappa statistics among others. Depending which classification i test, i have some small increase inaccuracy and a little larger increase in kappa statistics. I wonder if there is a way to do a statistical significance test for the accuracy and kappa increase between the 2 classifications.

Data example and some code:

library(caret)

ref <- c(15, 13, 13, 13, 13, 15, 14, 14, 14, 15, 13, 13, 13, 15, 13, 13, 13, 15, 13, 13, 13, 13, 13, 13, 13,13, 14, 13, 13, 13, 13, 13, 13, 13, 15, 13, 13, 15, 13, 15, 13, 13, 15, 13, 13, 13, 13, 13, 13, 13,13, 13, 13, 13, 13, 15, 13, 13, 13, 13, 13, 13, 15, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13,13, 14, 13, 13, 13, 13, 13, 14, 14, 15, 15, 13, 13, 13, 13, 13, 15, 13, 13, 13, 13, 13, 13, 13, 13,13, 13, 14, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 15, 13, 13, 13, 13, 13, 13, 13,13, 13, 13, 13, 13, 13, 13, 14, 13, 13, 13, 13, 13, 13, 15, 13, 13, 13, 13, 13, 13)

class1 <- c(14, 14, 13, 13, 13, 15, 13, 14, 15, 14, 14, 13, 14, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 14, 13,13, 13, 13, 13, 13, 13, 13, 13, 13, 15, 13, 14, 13, 13, 14, 13, 13, 15, 13, 13, 13, 13, 13, 13, 13,13, 13, 15, 21, 13, 15, 13, 21, 13, 13, 14, 13, 15, 13, 15, 13, 13, 14, 13, 13, 13, 13, 13, 13, 13,13, 14, 14, 13, 13, 13, 13, 15, 15, 15, 15, 13, 13, 13, 13, 13, 5, 13, 15, 13, 13, 13, 13, 13, 13,15, 13, 15, 14, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13,13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13)

class2 <- c(14, 15, 13, 13, 13, 15, 13, 14, 15, 15, 14, 13, 14, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 14, 13,13, 13, 13, 13, 13, 13, 13, 13, 13, 15, 13, 14, 13, 13, 15, 13, 13, 15, 14, 13, 13, 13, 13, 13, 13,13, 13, 15, 13, 13, 15, 13, 21, 13, 13, 13, 13, 15, 13, 15, 15, 13, 14, 13, 13, 13, 13, 13, 13, 15,13, 14, 14, 13, 13, 13, 13, 15, 14, 15, 15, 13, 14, 13, 13, 13, 15, 13, 15, 13, 13, 13, 13, 13, 13,15, 13, 15, 14, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 22, 13, 13, 13, 13, 13, 13, 13,13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13)

ref1 <- factor(ref, levels = c(5, 13, 14, 15, 21, 22))
pred1 <- factor(class1, levels = c(5, 13, 14, 15, 21, 22))
pred2 <- factor(class2, levels = c(5, 13, 14, 15, 21, 22))

t1 <- table(pred1, ref1)
t2 <- table(pred2, ref1)

cm1 <- confusionMatrix(t1)
cm1$overall

cm2 <- confusionMatrix(t2)
cm2$overall

As you see the increase in accuracy is very small, but the increase in kappa is a little bit more substantial. Is this increase statistical significant?

Thanks for any help,

Monica

_________________________________________________________________

_________________________________________________________________
It?s the same Hotmail?. If by ?same? you mean up to 70% faster.

statistical significance of accuracy increase in classification

Thread (7 messages)