We've mostly gotten out of the area where I know enough statistically to
speak with confidence, but I'll risk some lumps anyway...
I always thought that the idea of retaining a portion of the data for
validation was a good idea. I asked David Anderson about this personally
and
he said he couldn't see any reason to do that. Using likelihood, he
thought
the best approach was to use all the data to determine the best model.
I'm pretty muddy on the difference between selecting a good model with AIC
(which is sometimes referred to as being predictive in nature) and what is
meant by post-hoc validation of predictive ability (aside from testing on
another data set). I've often seen the "leave-one-out" approach used to
"validate" a model. If anyone has a good reference that differentiates the
two with an example, I'd really appreciate it.