-----Mensaje original-----
De: Bert Gunter [mailto:gunter.berton@] Enviado el: jueves, 26 de
enero de 2012 21:20
Para: Rub?n Roa
CC: Ben Bolker; Frank Harrell
Asunto: Re: [R] How do I compare 47 GLM models with 1 to 5
interactions and unique combinations?
On Wed, Jan 25, 2012 at 11:39 PM, Rub?n Roa <rroa@> wrote:
I think we have gone through this before.
First, the destruction of all aspects of statistical inference is not
at stake, Frank Harrell's post notwithstanding.
Second, checking all pairs is a way to see for _all pairs_ which
model outcompetes which in terms of predictive ability by -2AIC or
more. Just sorting them by the AIC does not give you that if no model
is better than the next best by less than 2AIC.
Third, I was not implying that AIC differences play the role of
significance tests. I agree with you that model selection is better
not understood as a proxy or as a relative of significance tests procedures.
Incidentally, when comparing many models the AIC is often inconclusive.
If one is bent on selecting just _the model_ then I check numerical
optimization diagnostics such as size of gradients, KKT criteria, and
other issues such as standard errors of parameter estimates and the
correlation matrix of parameter estimates.
-- And the mathematical basis for this claim is .... ??
--
Ruben: In my area of work (building/testing/applying mechanistic
nonlinear models of natural and economic phenomena) the issue of
numerical optimization is a very serious one. It is often the case
that a really good looking model does not converge properly (that's
why ADMB is so popular among us). So if the AIC is inconclusive, but
one AIC-tied model yields reasonably looking standard errors and low
pairwise parameter estimates correlations, while the other wasn?t even
able to produce a positive definite Hessian matrix (though it was able
to maximize the log-likelihood), I think I have good reasons to select
the properly converged model. I assume here that the lack of
convergence is a symptom of model inadequacy to the data, that the AIC was not able to detect.
---
Ruben: For some reasons I don't find model averaging appealing. I
guess deep in my heart I expect more from my model than just the best
predictive ability.
-- This is a religious, not a scientific statement, and has no place
in any scientific discussion.
--
Ruben: Seriously, there is a wide and open place in scientific
discussion for mechanistic model-building. I expect the models I built
to be more than able predictors, I want them to capture some aspect of
nature, to teach me something about nature, so I refrain from model
averaging, which is an open admission that you don't care too much
about what's really going on.
-- The belief that certain data analysis practices -- standard or not
-- somehow can be trusted to yield reliable scientific results in the
face of clear theoretical (mathematical )and practical results to the
contrary, while widespread, impedes and often thwarts the progress of
science, Evidence-based medicine and clinical trials came about for a
reason. I would encourage you to reexamine the basis of your
scientific practice and the role that "magical thinking" plays in it.
Best,
Bert
--
Ruben: All right Bert. I often re-examine the basis of my scientific
praxis but less often than I did before, I have to confess. I like to
think it is because I am converging on the right praxis so there are
less issues to re-examine. But this problem of model selection is a tough one.
Being a likelihoodist in inference naturally leads you to AIC-based
model selection, Jim Lindsey being influent too. Wanting that your
models say some something about nature is another strong conditioning
factor. Put this two together and it becomes hard to do
model-averaging. And it has nothing to do with religion (yuck!).