comparing classification methods: 10-fold cv or leaving-one-out ?

Leave-one-out is very inaccurate for some methods, notably trees, but fine 
for some others (e.g. LDA) if used with a good measure of accuracy.

Hint: there is a very large literature on this, so read any good book on 
classification to find out what is known.

Hi
what would you recommend to compare classification methods such as LDA,
classification trees (rpart), bagging, SVM, etc:

10-fold cv (as in Ripley p. 346f)

Not a valid reference:  did you mean Venables & Ripley (2000, p.346f)?
Try reading Ripley (1996), for example.

or

leaving-one-out (as e.g. implemented in LDA)?

my data-set is not that huge (roughly 200 entries)

That's rather small to compare error rates on.

Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

I would recommend reading the following:  Dietterich, T. G., (1998). 
Approximate Statistical Tests for Comparing Supervised Classification 
Learning Algorithms. Neural Computation, 10 (7) 1895-1924. 
http://web.engr.oregonstate.edu/~tgd/publications/index.html

The issues in comparing methods are subtle and difficult.  With such a 
small data set I would be a little surprised if you could get any result 
that are truly statistically significant, especially if your goal is to 
compare among good non-linear methods (i.e., in which there are unlikely to 
huge differences because of model misspecification).  However, because the 
issues are subtle, it is easy to get results that appear significant...

hope this helps,

Tony Plate

Hi
what would you recommend to compare classification methods such as LDA,
classification trees (rpart), bagging, SVM, etc:

10-fold cv (as in Ripley p. 346f)

or

leaving-one-out (as e.g. implemented in LDA)?

my data-set is not that huge (roughly 200 entries)

many thanks for a hint

Christoph
--
Christoph Lehmann <christoph.lehmann at gmx.ch>

______________________________________________
R-help at stat.math.ethz.ch mailing list
https://www.stat.math.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html

Tony Plate   tplate at acm.org