Skip to content

troubleshooting data structure to run krippendorff's alpha

7 messages · Hallie Kamesch, Jeff Newmiller, Jim Lemon

Original
Hallie Kamesch
# 
Hi -
I'm trying to run Krippendorff's alpha for data consisting of 4 subjects
rated on 6 events each by three raters.  The ratings are interval ratio
scale data.

I've rearranged my data into a 3 x 24  of ratersXevents. (per this
discussion on CrossValidated: (
https://stats.stackexchange.com/questions/255164/inter-rater-reliability-for-binomial-repeated-ratings-from-two-or-more-raters/256144#256144)
).

This is the code I've used:
library(irr)
dat <- read.csv(file.choose(), header = TRUE)
head(dat)
kripp.alpha(dat, method=c("ratio"))
#### error message: Error in sort.list(y) : 'x' must be atomic for
'sort.list'
Have you called 'sort' on a list?
kripp.alpha(dat,"ratio")
#### error message: Error in sort.list(y) : 'x' must be atomic for
'sort.list'
Have you called 'sort' on a list?

I read rhelp on sort, but I'm still confused.  Please help!
Thank you!

PS
I arranged my data in that matrix based upon this comment and response from
the CrossValidated posting forum (
https://stats.stackexchange.com/questions/255164/inter-rater-reliability-for-binomial-repeated-ratings-from-two-or-more-raters/256144#256144),
but my question above was rejected there.
Jeff Newmiller
# 
I don't understand most of what you wrote, but when you say "matrix" you are mistaken. A matrix is NOT the same thing as a data frame, which is what you get when you call read.csv(). Read 

RShowDoc("R-intro")

Sections 5 and 6... A data frame is a list of column vectors, while a matrix is a vector with a dimension attribute.

You can use the as.matrix function to convert a data frame to a matrix.
On January 27, 2019 9:08:02 AM PST, Hallie Kamesch <hallie.kamesch at gmail.com> wrote:

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Jim Lemon
    

    

      
      #
    

  


  

      
Hi Halllie,
As Jeff noted, a data frame is not a matrix (it is a variety of list),
so that looks like your problem.

hkdf<-data.frame(sample(3:5,4,TRUE),sample(1:3,4,TRUE),sample(2:4,4,TRUE),
 sample(3:5,4,TRUE),sample(1:3,4,TRUE),sample(2:4,4,TRUE))
library(irr)
kripp.alpha(hkdf)
kripp.alpha(as.matrix(hkdf))

Jim

        
On Mon, Jan 28, 2019 at 6:04 PM Hallie Kamesch <hallie.kamesch at gmail.com> wrote:

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Hallie Kamesch
    

    

      
      #
    

  


  

      
Hi all, 
Thank you for your responses. You are correct that it is not a matrix. I used the incorrect term. 
I meant I put my data in a spreadsheet with three rows and 24 columns. 

Sent from my iPhone

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Hallie Kamesch
    

    

      
      #
    

  


  

      
Thank you Jim, for the code, and thank you Jeff for the tutorial PDF.  I've
read through the sections and I appreciate the help.
I'm in way over my head - I don't even understand enough of the vocabulary
to ask my question correctly.
Jim, in your code, I ended up with an entry of 4 observations of 6
variables. I understand how that happened now since I read your code - that
helped very much.
My only problem, that I can't figure out, is how to make it so I have 3
raters with 4 observations of 6 variables.
I really am trying to educate myself enough to not waste your time:  I've
?data.frame, ?sample, ?matrix, ?$names, ?attributes, etc... I read the
sections in Jeff's PDF, and the tutorials on datamentor, I'm just not
finding how to do this. I'm sorry this is such a newbie question.
thank you for your time,
hallie

On Mon, Jan 28, 2019 at 1:21 PM Hallie Kamesch <hallie.kamesch at gmail.com>
wrote:

            

      
      
      
    

        

      
      
    

        

      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Jim Lemon
    

    

      
      #
    

  


  

      
Hi Hallie,
If I understand your email correctly, you have four repeated
observations by the three raters of the same six variables. This is a
tougher problem and I can't solve it at the moment. I'll return to
this later and see if I can offer a solution.


Jim

        
On Wed, Jan 30, 2019 at 3:56 AM Hallie Kamesch <hallie.kamesch at gmail.com> wrote:

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Jim Lemon
    

    

      
      #
    

  


  

      
Hi Hallie,
I tried both the "cccUst" and "cccvc" functions in the "cccrm"
package. While I can get what looks like sensible statistics with the
following example, I am not sure that it can be interpreted as you
wish. For one thing, it assumes that the concordance will be the same
on all variables. I was not able to get a statistic on each variable
separately. Perhaps someone who is more familiar with the package can
offer better advice. Also, check the "hkdf" data frame to ensure that
it looks like your data.

hkdf<-data.frame(rater=rep(1:3,each=24),occasion=rep(rep(1:4,each=6),3),
 var=rep(rep(1:6,4),3),obs=runif(72))
library(cccrm)
cccUst(hkdf,"obs","rater","occasion")
cccvc(hkdf,"obs","rater","occasion")

Jim

        
On Wed, Jan 30, 2019 at 1:01 PM Jim Lemon <drjimlemon at gmail.com> wrote: