Hello all, I'm doing a generalizability reliability study of teacher performance ratings of classroom observations similar to this: http://www.metproject.org/downloads/MET_Reliability_of_Classroom_Observations_Research_Paper.pdf On page 8 the authors detail the variance decomposition in a table. (I'm pasting it in here but the formatting gets all messed up. Sorry.) Source Description T Teacher variance or ?true score? variance. The ?signal? that is separable from ?error.? I Variance due to items. Some items are more difficult than others. R Variance due to raters. Some raters are more difficult than others. L:T Variance due to lessons. Confounded with teacher score dependence upon lessons. T x I Some teachers score higher on certain items. T x R Some raters score higher certain teachers. I x R Some raters score higher certain items. T x I x R Some raters score higher certain teachers on certain items. I x (L:T) Some items receive higher scores on certain lessons. Cofounded with teacher score dependence. (L:T) x R Some raters score certain lessons higher. Confounded with teacher score dependence. (L:T) x I x R, e Error variance, confounded with teacher score dependence on items, raters, and lessons. I have tried to duplicate this but I'm not sure if I'm specifying the interactions correctly. Here's my lmer call. (tid.f is the teacher identifier, obsid is the rater, obsorder is the lesson identifier, and comp.f is what they call the item in the above table.) lme7 <- lmer(rating ~ (1|tid.f) + (1|obsid) + (1|comp.f) + (1|tid.f:obsorder.f) + (1|tid.f:comp.f) + (1|obsid:obsorder.f) + (1|comp.f:obsorder.f) + (1|tid.f:obsid) + (1|tid.f:comp.f:obsid), data=ratings, REML=FALSE) Does this look right? Anyone have any advice? Thanks.
Stuart Luppescu <slu at ccsr.uchicago.edu>