Skip to content

logistic regression - glm.fit: fitted probabilities numerically 0 or 1 occurred

10 messages · Ben quant, Peter Dalgaard, Patrick Breheny

Original
Peter Dalgaard
#
On Dec 1, 2011, at 18:54 , Ben quant wrote:

            

      
      
      
    

        
Still not there. Sometimes it's because your mailer doesn't label them with the appropriate mime-type (e.g. as application/octet-stream, which is "arbitrary binary"). Anyways, see below

[snip]

            

      
      
      
    

        
I bet that you do... You can get the warning without that effect (one of my own examples is  the probability of menarche in a data set that includes infants and old age pensioners), but not with a huge odds ratio as well. Take a look at 

d <- as.data.frame(l_yx) 
with(d, y[order(x)])

if it comes out as all zeros followed by all ones or vice versa, then you have the problem.

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        
      

    

    

      
      

        


  

        

      Peter Dalgaard
    

    

      
      #
    

  


  

      
On Dec 1, 2011, at 21:32 , Ben quant wrote:

        

            

      
      
      
    

        
Hum, then maybe it really is a case of a transition region being short relative to the range of your data. Notice that the warning is just that: a warning. I do notice that the distribution of your x values is rather extreme -- you stated a range of 0--14 and a mean of 0.01. And after all, an odds ratio of 3000 per unit is only a tad over a doubling per 0.1 units.

Have a look at  range(x[y==0]) and range(x[y==1]).

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        
      

    

    

      
      

        
      

    

    

      
      

        
      

    

    

      
      

        


  

        

      Peter Dalgaard
    

    

      
      #
    

  


  

      
On Dec 1, 2011, at 23:43 , Ben quant wrote:

        

            

      
      
      
    

        
It's easier to explain why you got the warning before. If the OR for a one unit change is 3000, the OR for a 14 unit change is on the order of 10^48 and that causes over/underflow in the conversion to probabilities.

I'm still baffled at how you can get that model fitted to your data, though. One thing is that you can have situations where there are fitted probabilities of one corresponding to data that are all one and/or fitted zeros where data are zero, but you seem to have cases where you have both zeros and ones at both ends of the range of x. Fitting a zero to a one or vice versa would make the likelihood zero, so you'd expect that the algorithm would find a better set of parameters rather quickly. Perhaps the extremely large number of observations that you have has something to do with it? 

You'll get the warning if the fitted zeros or ones occur at any point of the iterative procedure. Maybe it isn't actually true for the final model, but that wouldn't seem consistent with the OR that you cited.

Anyways, your real problem lies with the distribution of the x values. I'd want to try transforming it to something more sane. Taking logarithms is the obvious idea, but you'd need to find out what to do about the zeros -- perhaps log(x + 1e-4) ? Or maybe just cut the outliers down to size with pmin(x,1).

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        
      

    

    

      
      

        


  

        

      Patrick Breheny
    

    

      
      #
    

  


  

      
On 12/01/2011 08:00 PM, Ben quant wrote:

            

      
      
      
    

        
The logistic regression model you are fitting assumes a linear 
relationship between x and the log odds of y; that does not seem to be 
the case for your data.  To illustrate:

x <- l_yx[,"x"]
y <- l_yx[,"y"]
ind1 <- x <= .002
ind2 <- (x > .002 & x <= .0065)
ind3 <- (x > .0065 & x <= .13)
ind4 <- (x > .0065 & x <= .13)

 > summary(glm(y[ind1]~x[ind1],family=binomial))
...
Coefficients:
              Estimate Std. Error z value Pr(>|z|)
(Intercept)  -2.79174    0.02633 -106.03   <2e-16 ***
x[ind1]     354.98852   22.78190   15.58   <2e-16 ***

 > summary(glm(y[ind2]~x[ind2],family=binomial))
Coefficients:
              Estimate Std. Error z value Pr(>|z|)
(Intercept)  -2.15805    0.02966 -72.766   <2e-16 ***
x[ind2]     -59.92934    6.51650  -9.197   <2e-16 ***

 > summary(glm(y[ind3]~x[ind3],family=binomial))
...
Coefficients:
              Estimate Std. Error z value Pr(>|z|)
(Intercept) -2.367206   0.007781 -304.22   <2e-16 ***
x[ind3]     18.104314   0.346562   52.24   <2e-16 ***

 > summary(glm(y[ind4]~x[ind4],family=binomial))
...
Coefficients:
             Estimate Std. Error z value Pr(>|z|)
(Intercept) -1.31511    0.08549 -15.383   <2e-16 ***
x[ind4]      0.06261    0.08784   0.713    0.476

To summarize, the relationship between x and the log odds of y appears 
to vary dramatically in both magnitude and direction depending on which 
interval of x's range we're looking at.  Trying to summarize this 
complicated pattern with a single line is leading to the fitted 
probabilities near 0 and 1 you are observing (note that only 0.1% of the 
data is in region 4 above, although region 4 accounts for 99.1% of the 
range of x).