Skip to content

How to deal with outcomes assessed by raters?

6 messages · Chris Howden, David Duffy, Joseph Bulbulia +1 more

Original
Chris Howden
# 
I'm no expert, but I believe that with only 4 judges that's not enough to
get an accurate estimate of the variability associated with judges.

So U may be better to include them as fixed effects with 4 levels.

That said if U just want a random intercept for each judge and don't want
an accurate measure of their variance it may still be OK to include them
as a random effect? But I'm a little unclear on this point myself.

Chris Howden B.Sc. (Hons) GStat.
Founding Partner
Evidence Based Strategic Development, IP Commercialisation and Innovation,
Data Analysis, Modelling and Training
(mobile) 0410 689 945
(fax) +612 4782 9023
chris at trickysolutions.com.au




Disclaimer: The information in this email and any attachments to it are
confidential and may contain legally privileged information.?If you are
not the named or intended recipient, please delete this communication and
contact us immediately.?Please note you are not authorised to copy, use or
disclose this communication or any attachments without our consent.
Although this email has been checked by anti-virus software, there is a
risk that email messages may be corrupted or infected by viruses or other
interferences. No responsibility is accepted for such interference. Unless
expressly stated, the views of the writer are not those of the company.
Tricky Solutions always does our best to provide accurate forecasts and
analyses based on the data supplied, however it is possible that some
important predictors were not included in the data sent to us. Information
provided by us should not be solely relied upon when making decisions and
clients should use their own judgement.


-----Original Message-----
From: r-sig-mixed-models-bounces at r-project.org
[mailto:r-sig-mixed-models-bounces at r-project.org] On Behalf Of Joseph
Bulbulia
Sent: Tuesday, 16 April 2013 3:07 PM
To: r-sig-mixed-models at r-project.org
Subject: [R-sig-ME] How to deal with outcomes assessed by raters?

Hi all,

Id like to model emotional dynamics in a highly arousing firewalk ritual.


Four judges rated images from 42 participants for arousal and valence. The
predictor variables are ritual phase and role.

Question 1
Any thoughts about how best to handle the rater assessments?

Specifically, is it nuts to explicitly include a component for raters in
the random component of the model?

E.g.

library(MCMCglmm)
prior.fw.0 = list(
  B = list(mu=rep(0,4),V = diag(4)*1e+10),
  R = list(V =diag(2), fix = 1),
  G = list(G1 = list(V = diag(2), n = 2, alpha.mu = c(0,0), alpha.V =
diag(2)*1000),
           G2 = list (V = diag(2),n = 2, alpha.mu = c(0,0), alpha.V =
diag(2)*1000),
           G3 = list (V = diag(2),  fix=1)))


firemodel.test <-MCMCglmm(cbind(arousal, valence) ~ trait:role:trait:phase
-1,
                          random = ~us(trait):phase:id
                          + idh(trait):event:id
                          + idh(trait):rater,
                          rcov= ~ idh(trait):units,
                          family = rep("ordinal",2),
                          data=Firewalkdata, burnin=5000,
                          thin = 10,
                          nitt=20000,
                          prior=prior.fw.0)



Thanks everyone. Very grateful for and advice.

Joseph


Disclaimer
I'm new to GLMMs,so apologies if this doesn't make sense.

Data sample below
(Only 20 data points, just to get a sense of the structure)

Firewalkdata <- structure(list(obs = c("1", "2", "3", "4", "5", "6", "7",
"8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19",
"20"), id = structure(c(24L, 4L, 26L, 37L, 32L, 3L, 20L, 9L, 3L, 2L, 5L,
19L, 23L, 28L, 29L, 8L, 3L, 18L, 40L, 26L), .Label = c("a", "b", "c", "d",
"e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "pa", "pb",
"pc", "pd", "pe", "pf", "pg", "ph", "pi", "pj", "pk", "pl", "pm", "pn",
"po", "pp", "q", "r", "s", "t", "u", "v", "w", "x", "y"), class =
"factor"), phase = c(1, 2, 5, 5, 2, 3, 1, 4, 5, 2, 5, 1, 5, 4, 3, 4, 5, 3,
2, 4), event = structure(c(11L, 4L, 13L, 21L, 22L, 3L, 4L, 9L, 3L, 2L, 5L,
3L, 9L, 16L, 17L, 8L, 3L, 2L, 24L, 13L), .Label = c("a", "b", "c", "d",
"e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s",
"t", "u", "v", "w", "x", "y"), class = "factor"), role = structure(c(2L,
1L, 2L, 1L, 2L, 1L, 2L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 1L, 1L, 2L, 1L,
2L), .Label = c("FW", "PS"), class = "factor"), dyad = structure(c(2L, 2L,
2L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 2L),
.Label = c("n", "y"), class = "factor"), gender = structure(c(1L, 2L, 2L,
2L, 2L, 2L, 1L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 2L, 2L, 1L, 2L, 2L),
.Label = c("f", "m"), class = "factor"), rater = structure(c(4L, 1L, 3L,
1L, 3L, 4L, 1L, 2L, 4L, 3L, 4L, 3L, 4L, 1L, 1L, 4L, 4L, 1L, 4L, 4L),
.Label = c("rat1", "rat2", "rat3", "rat4"), class = "factor"),
    arousal = c(4, 5, NA, 6, 6, 6, 4, 4, 7, 4, 3, 7, 5, 5, 5,
    6, 5, NA, 4, 6), valence = c(4, 3, NA, 2, 6, 1, 2, 6, 3,
    5, 5, 7, 5, 2, 2, 1, 4, NA, 4, 6)), .Names = c("obs", "id", "phase",
"event", "role", "dyad", "gender", "rater", "arousal", "valence"),
row.names = c(3977L, 83L, 2996L, 525L, 3134L, 3213L, 726L, 1267L, 3221L,
2134L, 3273L, 2801L, 3975L, 944L, 964L, 3344L, 3223L, 688L, 3758L, 4041L),
class = "data.frame")


IMAGE SAMPLE (for the curious)
https://www.dropbox.com/s/xmbci5814h73i0l/4_MF_e5.png

GRAPH bootstrapped means
https://www.dropbox.com/s/509sgh5zqxq18tn/Figure_FireWalk.pdf

Crude overview of the design
https://www.dropbox.com/s/0o0a9kkrh5ttsd2/plot.plan.emotions_firewalk.pdf




Joseph Bulbulia
Senior Lecturer, Religious Studies
Faculty of Humanities and Social Sciences Victoria University, New Zealand
+64 21 95 94 23
http://www.metaphysicalclub.com
David Duffy
#
On Tue, 16 Apr 2013, Joseph Bulbulia wrote:

            

      
      
      
    

        
So what do the inter-rater agreements look like?  I presume rater is 
actually a nuisance variable?  The path model I would usually use would 
have phase and role acting on the averaged-over-raters a and v 
scores (measurement model bit).

Just 2c, David Duffy.

  
  


  


      

    

    

      
      

        
      

    

    
2 days later

    

      
      

        


  

        

      Andrew Robinson
    

    

      
      #
    

  


  

      
Hi Joseph,

thanks for this detailed summary.  Based on my understanding, I think
that it is defensible to include the raters as random effects in the
model, and I think that doing so provides a more faithful
representation of your experimental design than would excluding them.

Definitely not nuts.

On the niggles: I'm not sure what exactly you mean by "averaging over
the ratings".  It sounds risky to me.

Cheers

Andrew

On Tue, Apr 16, 2013 at 7:50 PM, Joseph Bulbulia

        
<joseph.bulbulia at icloud.com> wrote: