statistical significance of accuracy increase in classification

Do you know about any good reference that discusses kappa for classification and maybe CI for kappa???

I don't, but googling on kappa and confusion matrix etc should get you
there. Kappa works very well when the true classes are skewed. For
example, if 10% of you samples are class A and 90% class B, you can
optimize accuracy by calling everything B.

Max