Skip to content

GLM output for deviance and loglikelihood

2 messages · Jeffrey Pollock, Peter Dalgaard

Original
Jeffrey Pollock
# 
First of all thank you both for the replies, my understanding is a lot clearer after thinking about the example you showed.

One further question though, looking at the code for 'polr' in MASS suggests that ordinal (and I would guess nominal too) data with levels >2 (ie not binary) also has a saturated model log-likelihood of 0;
So am I right in saying that Binary data isnt the only case where this is true? It would make sense to me that for a multinomial model you could have a unique factor for each data point and thus be able to create a likelihood of 1.

I have an example of this similar to the poisson one;
# weights:  33 (20 variable)
initial  value 10.986123 
iter  10 value 0.073035
final  value 0.000086 
converged
Call:
multinom(formula = y ~ factor)

Coefficients:
  (Intercept)   factor2   factor3   factor4  factor5  factor6  factor7  factor8   factor9 factor10
2   -14.33414 -21.50626  28.45961  28.45961 -9.18573 -9.18573 -9.18573 -9.18573  28.45961 -9.18573
3   -12.71240 -27.36982 -14.22512 -14.22512 23.75682 23.75682 23.75682 23.75682 -14.22512 23.75682

Residual Deviance: 0.0001713779 
AIC: 40.00017
'log Lik.' -8.568897e-05 (df=20)
y factor
 [1,] 1      1
 [2,] 1      2
 [3,] 2      3
 [4,] 2      4
 [5,] 3      5
 [6,] 3      6
 [7,] 3      7
 [8,] 3      8
 [9,] 2      9
[10,] 3     10

My understanding of this is that; if I observe a factor of '1' then the model will say with probability 1 that the outcome will be 1 and so on for the other rows in the dataset, and this shows in the estimated coefficients. I think the reason the log-likelihood doesn?t return exactly 0 is that the fitting algorithm used gets suitably close and then stops. Wouldn't make sense to continue the algorithm until the coefficients where either 'Inf' or '-Inf'.

Please let me know your thoughts on this.

Thanks again,

Jeff

-----Original Message-----
From: peter dalgaard [mailto:pdalgd at gmail.com] 
Sent: 21 April 2011 09:32
To: Juliet Hannah
Cc: Jeffrey Pollock; r-help at r-project.org
Subject: Re: [R] GLM output for deviance and loglikelihood
On Apr 21, 2011, at 05:14 , Juliet Hannah wrote:

            

      
      
      
    

        
To be precise, it only happens when the log likelihood of the saturated model is 0, which for discrete models implies that the probability of the observed data under the saturated model is 1.  Binary data is pretty much the _only_ case where this is true (because individual fitted probabilities become either zero or one).

  
    

      
      
    

  


  


    

    

      
      

        


  

        

      Peter Dalgaard
    

    

      
      #
    

  


  

      
On Apr 21, 2011, at 11:30 , Jeffrey Pollock wrote:

        

            

      
      
      
    

        
Yes. (I did say "pretty much"...). There are also some synthetic cases like when you enter a 2x2 table as 4 separate records:

            

      
      
      
    

        
Call:
glm(formula = outcome ~ g, family = binomial, data = d, weights = n)

Deviance Residuals: 
     1       2       3       4  
-1.482   1.274  -2.255   2.116  

Coefficients:
            Estimate Std. Error z value Pr(>|z|)
(Intercept)   1.0986     2.5658   0.428    0.669
g            -0.4055     1.4434  -0.281    0.779

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 13.460  on 3  degrees of freedom
Residual deviance: 13.380  on 2  degrees of freedom
AIC: 17.380

Number of Fisher Scoring iterations: 3

(The results are fine as long as you don't actually use the "residual deviance" for anything!)