Cohen's Kappa for beginners

Hi,
I've got two vectors with ratings from two coders, like this:

x<-c("red", "yellow", "blue", "red") #coder number 1
y<-c("red", "blue", "blue", "red") #coder number 2

I want to find Cohen's Kappa using the wkappa function in the psych
package. The only example in the docs is using a matrix, which I'm
afraid I don't understand--I don't know how to get there from what I've
got. Could anyone point me in the right direction? Thanks!

On 05/25/2010 06:01 PM, Jason Priem wrote:

Hi,
I've got two vectors with ratings from two coders, like this:

x<-c("red", "yellow", "blue", "red") #coder number 1
y<-c("red", "blue", "blue", "red") #coder number 2

I want to find Cohen's Kappa using the wkappa function in the psych
package. The only example in the docs is using a matrix, which I'm
afraid I don't understand--I don't know how to get there from what I've
got. Could anyone point me in the right direction? Thanks!

Hi Jason,
You're not alone. I tried to work out how to run this and it took a 
while. Both kappa2 in the "irr" package:

kappa2(cbind(x,y))

and classAgreement in the "e1071" package:

classAgreement(table(x,y))

produce a kappa of 0.6. I was unable to work out how to use the wkappa 
function in the "psych" package:

wkappa(table(x,y))

The above led to a kappa of -0.8. Although wkappa is presented as a 
test of the reliability of ratings of nominal level data, the example 
uses numeric data.

Jim

Hi,

It doesn't seem happy with non-numeric data in the data frame version.
Maybe a recode would work?

 > x1 <- c(1, 2, 3, 1)
 > y1 <- c(1, 3, 3, 1)
 > cohen.kappa(data.frame(x = x1, y = y1))

Call: cohen.kappa1(x = x, w = w, n.obs = n.obs, alpha = alpha)

Cohen Kappa and Weighted Kappa correlation coefficients and confidence
boundaries
lower estimate upper
unweighted kappa 0.098 0.60 1.1
weighted kappa 0.601 0.86 1.1

The unweighted kappa is .60.

I also tried to create the p x p table version, thinking that it would
be unhappy with less than a p x p table. The example dataset produces a
3 x 2 table, so this pads it out to a 3 x 3:

 > x <- factor(x)
 > y <- factor(y, levels = c("blue", "red", "yellow"))
 > table(x,y)

y
x blue red yellow
blue 1 0 0
red 0 2 0
yellow 1 0 0

 > cohen.kappa(table(x,y))

Call: cohen.kappa1(x = x, w = w, n.obs = n.obs, alpha = alpha)

Cohen Kappa and Weighted Kappa correlation coefficients and confidence
boundaries
lower estimate upper
unweighted kappa 0.098 0.6 1.10
weighted kappa -0.693 0.0 0.69

Number of subjects = 4>

The unweighted kappa is the same as above and as found by Jim with other
packages.

Scot

On 5/25/2010 7:31 AM, Jim Lemon wrote:

On 05/25/2010 06:01 PM, Jason Priem wrote:

Hi,
I've got two vectors with ratings from two coders, like this:

x<-c("red", "yellow", "blue", "red") #coder number 1
y<-c("red", "blue", "blue", "red") #coder number 2

I want to find Cohen's Kappa using the wkappa function in the psych
package. The only example in the docs is using a matrix, which I'm
afraid I don't understand--I don't know how to get there from what I've
got. Could anyone point me in the right direction? Thanks!

Hi Jason,
You're not alone. I tried to work out how to run this and it took a
while. Both kappa2 in the "irr" package:

kappa2(cbind(x,y))

and classAgreement in the "e1071" package:

classAgreement(table(x,y))

produce a kappa of 0.6. I was unable to work out how to use the wkappa
function in the "psych" package:

wkappa(table(x,y))

The above led to a kappa of -0.8. Although wkappa is presented as a
test of the reliability of ratings of nominal level data, the example
uses numeric data.

Jim

 ck <- cohen.kappa(cbind(x,y))   
 ck

 ck$agree     #show the table of matches

cohen.kappa(cbind(x,y)) works for me.

 -Peter Ehlers

On 2010-05-25 7:42, Scot W. McNary wrote:

Hi,

It doesn't seem happy with non-numeric data in the data frame version.
Maybe a recode would work?

 > x1 <- c(1, 2, 3, 1)
 > y1 <- c(1, 3, 3, 1)
 > cohen.kappa(data.frame(x = x1, y = y1))

Call: cohen.kappa1(x = x, w = w, n.obs = n.obs, alpha = alpha)

Cohen Kappa and Weighted Kappa correlation coefficients and confidence
boundaries
lower estimate upper
unweighted kappa 0.098 0.60 1.1
weighted kappa 0.601 0.86 1.1

The unweighted kappa is .60.

I also tried to create the p x p table version, thinking that it would
be unhappy with less than a p x p table. The example dataset produces a
3 x 2 table, so this pads it out to a 3 x 3:

 > x <- factor(x)
 > y <- factor(y, levels = c("blue", "red", "yellow"))
 > table(x,y)

y
x blue red yellow
blue 1 0 0
red 0 2 0
yellow 1 0 0

 > cohen.kappa(table(x,y))

Call: cohen.kappa1(x = x, w = w, n.obs = n.obs, alpha = alpha)

Cohen Kappa and Weighted Kappa correlation coefficients and confidence
boundaries
lower estimate upper
unweighted kappa 0.098 0.6 1.10
weighted kappa -0.693 0.0 0.69

Number of subjects = 4>

The unweighted kappa is the same as above and as found by Jim with other
packages.

Scot

On 5/25/2010 7:31 AM, Jim Lemon wrote:

On 05/25/2010 06:01 PM, Jason Priem wrote:

Hi,
I've got two vectors with ratings from two coders, like this:

x<-c("red", "yellow", "blue", "red") #coder number 1
y<-c("red", "blue", "blue", "red") #coder number 2

I want to find Cohen's Kappa using the wkappa function in the psych
package. The only example in the docs is using a matrix, which I'm
afraid I don't understand--I don't know how to get there from what I've
got. Could anyone point me in the right direction? Thanks!

Hi Jason,
You're not alone. I tried to work out how to run this and it took a
while. Both kappa2 in the "irr" package:

kappa2(cbind(x,y))

and classAgreement in the "e1071" package:

classAgreement(table(x,y))

produce a kappa of 0.6. I was unable to work out how to use the wkappa
function in the "psych" package:

wkappa(table(x,y))

The above led to a kappa of -0.8. Although wkappa is presented as a
test of the reliability of ratings of nominal level data, the example
uses numeric data.

Jim

x<-c("red", "yellow", "blue", "red")  #coder number 1
y<-c("red", "blue", "blue", "red") #coder number 2 
ck <- cohen.kappa(cbind(x,y))   

Thanks for you quick responses, all!  Bill, it looks like that's just what I
want, but I'm not sure where you're getting the cohen.kappa() function. The
only function I could find by that name is in the concord library, and it
gives me:

x<-c("red", "yellow", "blue", "red")  #coder number 1
y<-c("red", "blue", "blue", "red") #coder number 2
ck<- cohen.kappa(cbind(x,y))

Error in counts[i, j]<- sum(scores[i, ] == score.levels[j], na.rm = TRUE) :
   subscript out of bounds
In addition: Warning messages:
1: NAs introduced by coercion
2: NAs introduced by coercion

wkappa in the psych lib gives me
Error in tr(x) : m must be a square matrix

Thanks for you quick responses, all!  Bill, it looks like that's just what I
want, but I'm not sure where you're getting the cohen.kappa() function. The
only function I could find by that name is in the concord library, and it
gives me:

 x<-c("red", "yellow", "blue", "red")  #coder number 1
 y<-c("red", "blue", "blue", "red") #coder number 2
 ck <- cohen.kappa(cbind(x,y))  

Error in counts[i, j] <- sum(scores[i, ] == score.levels[j], na.rm = TRUE) :
  subscript out of bounds
In addition: Warning messages:
1: NAs introduced by coercion
2: NAs introduced by coercion

wkappa in the psych lib gives me
Error in tr(x) : m must be a square matrix

--
View this message in context: 
http://r.789695.n4.nabble.com/Cohen-s-Kappa-for-beginners-tp2229658p2230348.html
Sent from the R help mailing list archive at Nabble.com.