Kind regards,
Bernard
-----Original Message-----
From: pierre.de.villemereuil at mailoo.org
<pierre.de.villemereuil at mailoo.org>
Sent: Friday, June 15, 2018 4:05 PM
To: r-sig-mixed-models at r-project.org
Cc: Ben Bolker <bbolker at gmail.com>; Doran, Harold <HDoran at air.org>;
Bernard Liew <B.Liew at bham.ac.uk>
Subject: Re: [R-sig-ME] [FORGED] Re: Using variance components of lmer
for ICC computation in reliability study
Hi,
However, I'm not sure how one would go about computing an ICC from
ordinal data
I've never used the package "ordinal", but if it's anything like the "ordinal" family of MCMCglmm, then computing an ICC on the liability scale would be fairly easy, as one would just proceed as always and simply add the so-called "link variance" corresponding to the chosen link function (1 for probit, (pi^2)/3 for logit). E.g. for a given variance component Vcomp and a probit link:
ICC = Vcomp / (sum(variance components of the model) + 1)
However, computing an ICC on the data scale would be much more difficult as it is actually multivariate...
I think in the case when such scores were used, having the estimate on the liability scale make sense though, as the binning is more due to our inability of finely measuring this scale rather than an actual property of the system.
Cheers,
Pierre.
Le vendredi 15 juin 2018, 03:27:54 CEST Ben Bolker a ?crit :
More generally, the best way to fit this kind of model is to use an
*ordinal* model, which assumes the responses are in increasing
sequence but does not assume the distance between levels (e.g. 1 vs
2,
2 vs 3 ...) is uniform. However, I'm not sure how one would go
about computing an ICC from ordinal data ... (the 'ordinal' package
is the place to look for the model-fitting procedures). Googling it
finds some stuff, but it seems that it doesn't necessarily apply to
complex designs ...
https://stats.stackexchange.com/questions/3539/inter-rater-reliabili
ty
-for-ordinal-or-interval-data
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/
On Thu, Jun 14, 2018 at 6:58 PM, Doran, Harold <HDoran at air.org> wrote:
That?s a helpful clarification, Rolf. However, with gaussian
normal errors in the linear model, we can?t *really* assume they
would asymptote at 1 or 10. My suspicion is that these are
likert-style ordered counts of some form, although the OP should
clarify. In which case, the 1 or 10 are limits with censoring, as
true values for some measured trait could exist outside those
boundaries (and I suspect the model is forming predicted values outside of 1 or 10).
On 6/14/18, 6:33 PM, "Rolf Turner" <r.turner at auckland.ac.nz> wrote:
On 15/06/18 05:35, Doran, Harold wrote:
Well no, you?re specification is not right because your variable
is not continuous as you note. Continuous means it is a real
number between -Inf/Inf and you have boundaries between 1 and 10.
So, you should not be using a linear model assuming the outcome is continuous.
I think that the foregoing is a bit misleading. For a variable to
be continuous it is not necessary for it to have a range from
-infinity to infinity.
The OP says that dv "is a continuous variable (scale 1-10)". It
is not clear to me what this means. The "obvious"/usual meaning
or interpretation would be that dv can take (only) the (positive
integer) values 1, 2, ..., 10. If this is so, then a continuous
model is not appropriate. (It should be noted however that people
in the social sciences do this sort of thing --- i.e. treat
discrete variables as continuous --- all the time.)
It is *possible* that dv can take values in the real interval
[1,10], in which case it *is* continuous, and a "continuous model"
is indeed appropriate.
The OP should clarify what the situation actually is.
cheers,
Rolf Turner
--
Technical Editor ANZJS
Department of Statistics
University of Auckland
Phone: +64-9-373-7599 ext. 88276
On 6/14/18, 11:16 AM, "Bernard Liew" <B.Liew at bham.ac.uk> wrote:
Dear Community,
I am doing a reliability study, using the methods of
https://www.ncbi.nlm.nih.gov/pubmed/28505546. I have a question
on the lmer formulation and the use of the variance components.
Background: I have 20 subjects, 2 fixed raters, 2 testing
sessions, and
10 trials per sessions. my dependent variable is a continuous
variable (scale 1-10). Sessions are nested within each
subject-assessor combination. I desire a ICC (3) formulation of
inter-rater and inter-session reliability from the variance components.
My lmer model is:
lmer (dv ~ rater + (1|subj) + (1|subj:session), data = df)
Question:
1. is the model formulation right? and is my interpretation
of the variance components for ICC below right?
2. inter-rater ICC = var (subj) / (var(subj) + var
(residual)) # I read that the variation of raters will be lumped with the residual
3. inter-session ICC =( var (subj) + var (residual)) /( var
(subj) + var (subj:session) + var (residual)) some simulated
data:
df = expand.grid(subj = c(1:20), rater = c(1:2), session =
c(1:2), trial = c(1:10)) df$vas = rnorm (nrow (df_sim), mean =
3, sd = 1.5)
I appreciate the kind response.