Skip to content

negetative AIC values: How to compare models with negative AIC's

8 messages · Jan Verbesselt, Brian Ripley, Douglas Bates +1 more

Original
Jan Verbesselt
# 
Dear,

When fitting the following model
knots <- 5
lrm.NDWI <- lrm(m.arson ~ rcs(NDWI,knots) 

I obtain the following result:

Logistic Regression Model

lrm(formula = m.arson ~ rcs(NDWI, knots))


Frequencies of Responses
  0   1 
666  35 

       Obs  Max Deriv Model L.R.       d.f.          P          C        Dxy
Gamma      Tau-a         R2      Brier 
       701      5e-07      34.49          4          0      0.777      0.553
0.563      0.053      0.147      0.045 

          Coef     S.E.    Wald Z P     
Intercept   -4.627   3.188 -1.45  0.1467
NDWI         5.333  20.724  0.26  0.7969
NDWI'        6.832  74.201  0.09  0.9266
NDWI''      10.469 183.915  0.06  0.9546
NDWI'''   -190.566 254.590 -0.75  0.4541

When analysing the glm fit of the same model

Call:  glm(formula = m.arson ~ rcs(NDWI, knots), x = T, y = T) 

Coefficients:
            (Intercept)     rcs(NDWI, knots)NDWI    rcs(NDWI, knots)NDWI'
rcs(NDWI, knots)NDWI''  rcs(NDWI, knots)NDWI'''  
                0.02067                  0.08441                 -0.54307
3.99550                -17.38573  

Degrees of Freedom: 700 Total (i.e. Null);  696 Residual
Null Deviance:      33.25 
Residual Deviance: 31.76        AIC: -167.7 

A negative AIC occurs!

How can the negative AIC from different models be compared with each other?
Is this result logical? Is the lowest AIC still correct?


Thanks,
Jan

_______________________________________________________________________
ir. Jan Verbesselt 
Research Associate 
Lab of Geomatics Engineering K.U. Leuven
Vital Decosterstraat 102. B-3000 Leuven Belgium 
Tel: +32-16-329750   Fax: +32-16-329760
http://gloveg.kuleuven.ac.be/
Brian Ripley
# 
AICs (like log-likelihoods) can be positive or negative.
However, you fitted a Gaussian and not a binomial glm (as lrm does if 
m.arson is binary).

For a discrete response with the usual dominating measure (counting 
measure) the log-likelihood is negative and hence the AIC is positive,
but not in general (and it is matter of convention even there).

In any case, Akaike only suggested comparing AIC for nested models, no one
suggests comparing continuous and discrete models.
On Fri, 15 Apr 2005, Jan Verbesselt wrote:

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Douglas Bates
    

    

      
      #
    

  


  

      
Jan Verbesselt wrote:

            

      
      
      
    

        
I'm not sure about this particular example but in general there is no 
problem with a negative AIC or a negative deviance just as there is no 
problem with a positive log-likelihood.  It is a common misconception 
that the log-likelihood must be negative.  If the likelihood is derived 
from a probability density it can quite reasonably exceed 1 which means 
that log-likelihood is positive, hence the deviance and the AIC are 
negative.

If you believe that comparing AICs is a good way to choose a model then 
it would still be the case that the (algebraically) lower AIC is preferred.

  
  


  


      

    

    

      
      

        


  

        

      Jan Verbesselt
    

    

      
      #
    

  


  

      
Thanks a lot for the input!

I forgot to add family=binomial, for a binomial glm. Now the AIC's are
positive!

I was planning to use AIC (from the binomial glm) and c-index (lrm) to
compare and rank different uni-variate (one continue explanatory variable)
logistic models to evaluate the 'performance' of the different explanatory
variables in the different models.

What is the best technique to compare these lrm.models, which are not
nested? I found in literature that ranking based on different parameters
(goodness of fit and predictability) that these can be used to compare
uni-variate models.

Thanks in advance,
Regards,
Jan-


_______________________________________________________________________
ir. Jan Verbesselt 
Research Associate 
Lab of Geomatics Engineering K.U. Leuven
Vital Decosterstraat 102. B-3000 Leuven Belgium 
Tel: +32-16-329750   Fax: +32-16-329760
http://gloveg.kuleuven.ac.be/
_______________________________________________________________________

-----Original Message-----
From: Prof Brian Ripley [mailto:ripley at stats.ox.ac.uk] 
Sent: Friday, April 15, 2005 5:06 PM
To: Jan Verbesselt
Cc: r-help at stat.math.ethz.ch
Subject: Re: [R] negetative AIC values: How to compare models with negative
AIC's

AICs (like log-likelihoods) can be positive or negative.
However, you fitted a Gaussian and not a binomial glm (as lrm does if 
m.arson is binary).

For a discrete response with the usual dominating measure (counting 
measure) the log-likelihood is negative and hence the AIC is positive,
but not in general (and it is matter of convention even there).

In any case, Akaike only suggested comparing AIC for nested models, no one
suggests comparing continuous and discrete models.

        
On Fri, 15 Apr 2005, Jan Verbesselt wrote:

        

            

      
      
      
    

        
Dxy

            

      
      
      
    

        
0.553

            

      
      
      
    

        
other?

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Brian Ripley
    

    

      
      #
    

  


  

      
Compare them by `goodness for purpose': you have not told us the purpose.
Please do read some of the extensive literature on model comparison.

        
On Sat, 16 Apr 2005, Jan Verbesselt wrote:

        

            

      
      
      
    

        

      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Frank E Harrell Jr
    

    

      
      #
    

  


  

      
Prof Brian Ripley wrote:

            

      
      
      
    

        
In addition to Brian's comment, AIC may be of use.  You can't really use 
c-index (ROC area) as it is not sensitive enough for comparing two 
models.  But whatever you use, the bad news is that you can't use the 
results to compare more than 2 or 3 completely pre-chosen models or you 
will invalidate inference and estimates if you use these comparisons to 
build a final model.

Frank

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Jan Verbesselt
    

    

      
      #
    

  


  

      
Dear all,

Thanks a lot for the input. I will take the considerations into account. 

Referring to;
"2 or 3 completely pre-chosen models or you will invalidate inference and
estimates if you use these comparisons to build a final model"

The aim is not use the comparisons to build a final model but to select the
explanatory variable which explains most of the variance or has the best
predictive ability (p247 10.8 Harrell, 2001).

I'm comparing variables, which are all related to the remotely sensed water
content of vegetation, with binary fire occurrence data (1: fire / 0: no
fire). The aim is to select the water related variable which has the best
'performance' (Referring to literature about logistic regression used for
evaluation of fire danger indices).

e.g. a lrm model is  lrm(firedata~waterrelated.variable)

Thanks a lot and best regards,
Jan

***
"
In addition to Brian's comment, AIC may be of use.  You can't really use 
c-index (ROC area) as it is not sensitive enough for comparing two 
models.  But whatever you use, the bad news is that you can't use the 
results to compare more than 2 or 3 completely pre-chosen models or you 
will invalidate inference and estimates if you use these comparisons to 
build a final model.

Frank
"
***

  
  


  


      

    

    

      
      

        


  

        

      Frank E Harrell Jr
    

    

      
      #
    

  


  

      
Jan Verbesselt wrote:

            

      
      
      
    

        
One procedure that will shed light on this is to bootstrap the ranks of 
the chi-square statistics for competing variables.  I think you will be 
surprised how wide the confidence intervals for the ranks are.  There is 
an example in the Alzola & Harrell document although it is for partial 
chi-squares for competing variables in a single model.

-FH