Hello! I am fitting multilevel log-binomial models for a very large dataset (>53,000 observations), using glmer. I am getting this-- Error: (maxstephalfit) PIRLS step-halvings failed to reduce deviance in pwrssUpdate. And the model is not converging. I think that increasing the number of iterations may remediate the convergence issue. But I can?t find the right syntax for that online. Any advice re these two issues? Many thanks, Jennifer
new R user struggling with error and convergence issues
5 messages · Jennifer Yourkavitch, Ben Bolker
Follow-up: glmmPQL works like a charm. I don?t really understand the difference between the algorithms used by glmmPQL and glmer. Should I be concerned that the models ran with glmmPQL but not glmer? Thanks, Jennifer
On Feb 15, 2016, at 2:22 PM, Jennifer Yourkavitch <jenyourkavitch at yahoo.com> wrote:
Hello! I am fitting multilevel log-binomial models for a very large dataset (>53,000 observations), using glmer. I am getting this-- Error: (maxstephalfit) PIRLS step-halvings failed to reduce deviance in pwrssUpdate. And the model is not converging. I think that increasing the number of iterations may remediate the convergence issue. But I can?t find the right syntax for that online. Any advice re these two issues? Many thanks, Jennifer
It is admittedly hard to increase the "maxstephalfit" parameter, but
part of the reason we haven't put much effort into making it easier is
that in our experience it rarely helps. Log-binomial models in
particular are tricky because the log link (i.e. the exponential
inverse-link) doesn't naturally constrain the response to the allowed
range of (0,1), so these models are hard to fit (the same applies to
inverse-link Gamma models). The bigger your data set, the more likely
you are to run into cases where the predicted value is >1. We could
admittedly do a much better job giving useful warnings about when and
where this problem occurred, and (more controversially) allow users to
clamp the output of the inverse-link function to the allowable domain
...
library("lme4")
set.seed(101)
dd <- data.frame(x=seq(-4,4,length=500),
f=sample(letters[1:10],size=500,replace=TRUE))
dd$y <- simulate(~x+(1|f),family=binomial,
newdata=dd,newparams=list(beta=c(0,1),theta=1))[[1]]
library("ggplot2")
ggplot(dd,aes(x,y,colour=f))+geom_point()+
geom_smooth(method="glm",method.args=list(family=binomial))
m1 <- glmer(y~x+(1|f), dd, family=binomial)
m2 <- glmer(y~x+(1|f), dd, family=binomial(link="log")) ## PIRLS
step-halving problem
On Mon, Feb 15, 2016 at 2:22 PM, Jennifer Yourkavitch via
R-sig-mixed-models <r-sig-mixed-models at r-project.org> wrote:
Hello! I am fitting multilevel log-binomial models for a very large dataset (>53,000 observations), using glmer. I am getting this-- Error: (maxstephalfit) PIRLS step-halvings failed to reduce deviance in pwrssUpdate. And the model is not converging. I think that increasing the number of iterations may remediate the convergence issue. But I can?t find the right syntax for that online. Any advice re these two issues? Many thanks, Jennifer
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
1 day later
Thank you! What do you think about using glmmPQL? The model converged. Jennifer
On Feb 15, 2016, at 3:21 PM, Ben Bolker <bbolker at gmail.com> wrote:
It is admittedly hard to increase the "maxstephalfit" parameter, but
part of the reason we haven't put much effort into making it easier is
that in our experience it rarely helps. Log-binomial models in
particular are tricky because the log link (i.e. the exponential
inverse-link) doesn't naturally constrain the response to the allowed
range of (0,1), so these models are hard to fit (the same applies to
inverse-link Gamma models). The bigger your data set, the more likely
you are to run into cases where the predicted value is >1. We could
admittedly do a much better job giving useful warnings about when and
where this problem occurred, and (more controversially) allow users to
clamp the output of the inverse-link function to the allowable domain
...
library("lme4")
set.seed(101)
dd <- data.frame(x=seq(-4,4,length=500),
f=sample(letters[1:10],size=500,replace=TRUE))
dd$y <- simulate(~x+(1|f),family=binomial,
newdata=dd,newparams=list(beta=c(0,1),theta=1))[[1]]
library("ggplot2")
ggplot(dd,aes(x,y,colour=f))+geom_point()+
geom_smooth(method="glm",method.args=list(family=binomial))
m1 <- glmer(y~x+(1|f), dd, family=binomial)
m2 <- glmer(y~x+(1|f), dd, family=binomial(link="log")) ## PIRLS
step-halving problem
On Mon, Feb 15, 2016 at 2:22 PM, Jennifer Yourkavitch via
R-sig-mixed-models <r-sig-mixed-models at r-project.org> wrote:
Hello! I am fitting multilevel log-binomial models for a very large dataset (>53,000 observations), using glmer. I am getting this-- Error: (maxstephalfit) PIRLS step-halvings failed to reduce deviance in pwrssUpdate. And the model is not converging. I think that increasing the number of iterations may remediate the convergence issue. But I can?t find the right syntax for that online. Any advice re these two issues? Many thanks, Jennifer
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
3 days later
Jennifer Yourkavitch via R-sig-mixed-models <r-sig-mixed-models at ...> writes:
Follow-up: glmmPQL works like a charm. I don?t really understand the difference between the algorithms used by glmmPQL and glmer. Should I be concerned that the models ran with glmmPQL but not glmer? Thanks, Jennifer
The main case where Laplace approximation (the default glmer algorithm) is significantly better than PQL is when there are a small number of values per group; PQL depends more strongly than Laplace approximation on the approximate Normality of the sampling distributions of the conditional modes (predictions of the random effects for particular groups). Do the predicted values make sense? Ben Bolker