Skip to content
Prev 13680 / 20628 Next
Original Raw

Using lmer to determine inter-rater reliability

Kevin E. Thorpe 
Hello.

I have a data frame with following structure.

 > str(vision)
'data.frame':	268 obs. of  9 variables:
  $ Child          : Factor w/ 67 levels "C01-05","C01-10",..: 43 43 43 
43 44 44 44 44 42 42 ...
  $ Test           : Factor w/ 4 levels "1","2","3","4": 1 2 3 4 1 2 3 4 
1 2 ...
  $ Rater          : Factor w/ 4 levels "F","L","P","S": 4 1 4 1 1 4 1 4 
4 1 ...
  $ Binoc          : int  100 100 100 100 0 0 0 0 40 0 ...
  $ Yield          : int  100 100 100 80 100 20 50 30 100 100 ...
  $ Tries          : int  5 5 5 6 5 10 10 10 5 5 ...
  $ Result         : Factor w/ 5 levels "Pass","ReferBI",..: 1 1 1 1 3 2 
3 2 3 3 ...
  $ ResultCollapsed: Factor w/ 3 levels "Pass","Refer",..: 1 1 1 1 2 2 2 
2 2 2 ...
  $ Test1          : Factor w/ 16 levels "F:1","F:2","F:3",..: 13 2 15 4 
1 14 3 16 13 2 ...

In these data, each subject is rated by 2 (of 4) raters twice. The Test1 
variable was created from Test and Rater with
(Rater:Test)[drop=TRUE] to explicitly create the nesting.

I then fit the following model.

 > binoc.lmer1 <- lmer(Binoc~1+(1|Child) + (1|Rater) + 
(1|Test1),data=vision)
 > binoc.lmer1
Linear mixed model fit by REML ['lmerMod']
Formula: Binoc ~ 1 + (1 | Child) + (1 | Rater) + (1 | Test1)
    Data: vision
REML criterion at convergence: 2592.62
Random effects:
  Groups   Name        Std.Dev.
  Child    (Intercept) 29.226
  Test1    (Intercept)  2.292
  Rater    (Intercept)  5.823
  Residual             26.330
Number of obs: 264, groups:  Child, 66; Test1, 16; Rater, 4
Fixed Effects:
(Intercept)
       51.68


Now my questions.

1. Have I fit the right model?

2. If so, would the right estimate for the rater ICC be
	Rater/(Rater + Residual)
    or
	(Rater + Test1)/(Rater + Test1 + Residual)

3. Would Test1/(Test1 + Residual) give an estimate of intra-rater 
reliability?

Thanks for you time.

Kevin