Message-ID: <C0EF851A-68B9-40EE-88D8-69DA1416675B@uos.de>
Date: 2009-02-26T14:18:32Z
From: Stefan Evert
Subject: statistical significance of accuracy increase in classification
In-Reply-To: <6731304c0902260514u24096bcfi1cff1712d7014d49@mail.gmail.com>
On 26 Feb 2009, at 14:14, Max Kuhn wrote:
>> Do you know about any good reference that discusses kappa for
>> classification and maybe CI for kappa???
You might also want to take a look at this survey article on kappa and
its alternatives:
Artstein, Ron and Poesio, Massimo (2008). Survey article: Inter-coder
agreement for computational linguistics. Computational Linguistics,
34(4), 555?596.
which you can download from
http://www.aclweb.org/anthology-new/J/J08/
Alternatives to the standard Fleiss-Cohen asymptotic confidence
intervals in the elementary 2x2 case are discussed in
Lee, J.J., Tu, Z. N.:"A Better Confidence for Kappa on Measuring
Agreement Between Two Raters with Binary Outcomes" Journal of
Computational and Graphical Statistics, 3:301-321, 1994.
which is available from JSTOR:
http://www.jstor.org/stable/1390914
An S implementation of their approximations can be downloaded here:
http://lib.stat.cmu.edu/S/kappa
I've started to evaluate the accuracy of these approximations with
simulation experiments some time ago, but haven't found the time to
follow up on it.
Hope this helps,
Stefan