model selection in lme4
In that case I might bring up B&A's "tapering effects" argument again -- selecting the correct model with a fixed number of parameters with non-tapering effects is what BIC is for, not what AIC is for.
I think this may be the case. The data that I have used is real data not simulated. However, I can tell that the parameters were unnecessary as they didn't explain any variation above and beyond the more simpler models. I think it may also be a case of the type of research questions that I ask in psychology vs. ecology. I'm always interested in knowing more about BIC/AIC and I'll check out your references. Thanks! Chris
On Sunday 15 February 2009 21:02:42 Ben Bolker wrote:
Took a (very) quick look at Raftery, which all seems sensible and well-argued. However ... the paper contrasts Bayes/BIC with classical hypothesis testing. Many of the points listed on p. 155 (better assessment of evidence, applicability to non-nested models, take model uncertainty into account, allow model averaging, easy to implement) apply to AIC as well as BIC. BIC does have many good qualities (approximation to Bayes factor, sensible "flat prior" interpretation, statistical consistency, ...). But the crux of the argument between BIC and AIC is the difference in their objective. BIC aims to identify the "true model", which essentially assumes that there is a sharp cutoff between parameters/processes that are in the model and those that are out. Burnham and Anderson have a lot to say about tapering effect sizes; they are zealots about AIC, and I often discount their enthusiasm, but after much percolation I've decided that AIC really does make sense for the kinds of questions I (and many ecologists) tend to ask. When you say that AIC selects an overly complex model, how do you know what the correct model is and which parameters are unnecessary? Is this a case of fitting to simulation output?
I have tried to say this more coherently at http://emdbolker.wikidot.com/blog:aic-vs-bic As an aside, I don't have a vested interest in this, and I don't claim that AIC is better for everything ... just that it seems most ecologists are working with "true models" that are of arbitrarily large dimension with tapering effects, which is where AIC should select the model with the best predictive capability ... Ben Bolker Christopher David Desjardins wrote:
For a discussion of BIC, please see Raftery (1995) in Sociological Methodology. Before you commit yourself on the AIC, I do encourage you to look at your BIC. In the models I've run when there is disagreement between the BIC and the AIC, it's usually that the AIC selects the overly complex model and includes unnecessary parameters. Cheers, Chris On Sunday 15 February 2009 19:50:30 Ben Bolker wrote:
It would be better to use AICc, but I'm not sure what I would use for "number of parameters" for a random effect with n levels: any number between 0.5 and n seems plausible! Someone should send Shane Richards (who has done some very nice work testing (Q)AIC(c) in ecological settings) and see if he's willing to tackle this one, although I can imagine he's getting sick of this kind of exercise ... Ben Bolker Renwick, A. R. wrote:
Just a quickie Ben, Are you saying that you would use AIC rather than AICc even with small sample size - due to difficulty in counting residual degrees of
freedom?
Thanks Anna p.s. this forum really is fantastic
________________________________________ From: r-sig-mixed-models-bounces at r-project.org [r-sig-mixed-models-bounces at r-project.org] On Behalf Of Ben Bolker [bolker at ufl.edu] Sent: 15 February 2009 23:07 To: Christopher David Desjardins Cc: r-sig-mixed-models at r-project.org; tahirajamil at yahoo.com Subject: Re: [R-sig-ME] model selection in lme4 Some caution on this advice: you seem to be quoting the general advice on AIC/BIC/AICc 1. The AIC/BIC distinction is between "best prediction" and "consistent estimation of true model" dimension, e.g. Yang, Yuhong. 2005. Can the strengths of AIC and BIC be shared? A conflict between model identification and regression estimation. Biometrika 92, no. 4 (December 1): 937-950. doi:10.1093/biomet/92.4.937. I favor AIC on these grounds, but you can decide for yourself. 2. For models with different random effects, AIC and BIC share a "degrees of freedom counting" problem with all model selection approaches -- there are two aspects here, (1) whether you are focused on individual-level prediction or population-level prediction (Vaida and Blanchard 2005, Spiegelhalter et al 2002) and (2) whether AIC/BIC share the boundary problems that also apply to likelihood ratio tests (Greven, Sonja. 2008. Non-Standard Problems in Inference for Additive and Linear Mixed Models. G?ttingen, Germany: Cuvillier Verlag. http://www.cuvillier.de/flycms/en/html/30/-UickI3zKPS,3cEY=/Buchdetails .h tml?SID=wVZnpL8f0fbc. ) 3. AIC and BIC are asymptotic tests (which can be especially problematic with random effects problems, when there are not large number of random blocks -- this makes likelihood ratio tests NOT OK for fixed-effect comparisons with small numbers of blocks (Pinheiro and Bates 2000)). If you want to use AICc then you are back to counting residual degrees of freedom ... as far as I know there isn't much guidance available on this issue. My bottom line: I would go ahead and use (Q)AIC with caution for data sets with large (?) numbers of blocks. With smaller numbers of blocks I would probably try to find some kind of randomization/permutation approach to get a sense of the relevant size of delta-AIC values ... ... or damn the torpedoes and see if you can get away with straight AIC. Ben Bolker Christopher David Desjardins wrote: You could use either the BIC or the AIC. My understanding is that the AIC tends to favor overly complex models whereas the BIC tends to favor parsimonious models. I am generally inclined to always use the BIC. If you have a small sample size you might also consider using the AICC which is a correction of the AIC for small sample sizes. That said, in my experience the AICC still selects more complex models than the BIC. Also if you have nested models you could use the chi-square tests. Cheers, Chris On Feb 15, 2009, at 4:44 PM, Tahira Jamil wrote: Hi I have run GLMM models in lme4 with different fixed effects and random effects . But now the problem is model selction Is AIC or BIC results are definitive specially for Gernalized linear mixed models or what critera should I use for model selction. So I can decide which explantory variable should be in the model because I have more than 10 explantory variables and some are entering in the model as random effect. In some cases If AIC has lower value but BIC is comparatively high. some suggestion for model selection would be highly appricated. WIth best wishes T Jamil Ph.D student Biometris Wageningen University and Research centre Netherlands. _______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models ----------------- Christopher David Desjardins Ph.D. Student Quantitative Methods in Education Department of Educational Psychology University of Minnesota http://blog.lib.umn.edu/desja004/educationalpsychology/ _______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models -- Ben Bolker Associate professor, Biology Dep't, Univ. of Florida bolker at ufl.edu / www.zoology.ufl.edu/bolker GPG key: www.zoology.ufl.edu/bolker/benbolker-publickey.asc _______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models The University of Aberdeen is a charity registered in Scotland, No SC013683.