cohen kappa for two-way table

2 messages · Dylan Beaudette, Jim Lemon

Sun, Dec 10, 2006 4:00 PM #

Greetings,

I am a bit confused by the results returned by the functions:
cohen.kappa {concord}
classAgreement {e1071}

when using a two-way table.


for example, if I have an matrix A, and a similar matrix B (same
dimensions), then:

matrix A and B can be found:
http://casoilresource.lawr.ucdavis.edu/drupal/files/a_40.txt
http://casoilresource.lawr.ucdavis.edu/drupal/files/b_40.txt

A <- matrix(unlist( read.table('a_40.txt'), use.names=FALSE), ncol=14)
B <- matrix(unlist( read.table('b_40.txt'), use.names=FALSE), ncol=14)

# compute cohen's kappa, default settings:
cohen.kappa(table(A,B))
Kappa test for nominally classified data
9 categories - 90 methods
kappa (Cohen) = 0.97353 , Z = 45.4465 , p = 0
kappa (Siegel) = -0.00744097 , Z = -0.0794501 , p = 0.531663
kappa (2*PA-1) = 0.947061


# compute cohen's kappa - type = counts
cohen.kappa(table(A,B), type='counts')

Different row sums, a no-classification category was added.

Kappa test for nominally classified data
91 categories - 22 methods
kappa (Siegel) = 0.168593 , Z = 2.50298 , p = 0.00615762
kappa (2*PA-1) = 0.71485

it seems like the second method (type='counts') is the correct way to
use a contingency table... but am i correct?

Secondly, when using the classAgreements() function I get different numbers:

classAgreement(table(A,B))
$diag
[1] 0.03296703

$kappa
[1] 0.02180419

$rand
[1] 0.9874325

$crand
[1] 0.7648124



Perhaps I am mis-reading the relevant manual pages. Can anyone shed
some light on the proper use, and therfore interpretation of these two
methods - when using a contingency table as input?

Thanks in advance,

Dylan

Jim Lemon

Mon, Dec 11, 2006 12:23 AM #

Dylan Beaudette wrote:

If I interpret this correctly, you are considering the numbers in A and 
B to be nominal variables representing levels of some data attribute.
What cohen.kappa wants is:

1(type=counts) a matrix of counts where each cell represents the number 
of methods (rows are different methods) that classified the data objects 
into levels (columns are different levels).
2(type=scores) a matrix of scores where each cell represents the level 
assigned by a method (rows again) to a data object (columns are data 
objects here).

If your numbers are counts, you can get a sensible kappa for _each_ matrix.
If your numbers are scores, you can get a sensible kappa by specifying 
type=scores for _each_ matrix
As far as I can see, you are computing a sparsely populated table of the 
two matrices and passing that, and I don't see how that can give a 
meaningful result. I suspect that the numbers might be counts in the 
first place, but they are being used as scores.

I'll let someone who knows more about the classAgreement function deal 
with this.

Jim