Skip to content

testing fixed effects in binomial lmer...again?

7 messages · Achaz von Hardenberg, Douglas Bates, Dimitris Rizopoulos +4 more

Original
Douglas Bates
#
On Jan 8, 2008 5:38 AM, Achaz von Hardenberg <fauna at pngp.it> wrote:
Yes, that is the best choice in lmer.  (In the development version it
is, in fact, the only choice.)
The change in the log-likelihood between two nested models is, in my
opinion, the most sensible test statistic for comparing the models.
However, it is not clear how one should convert this test statistic to
a p-value.  The use of the chi-squared distribution is based on
asymptotic results and can give an "anti-conservative" (i.e. lower
than would be obtained through a randomization test or via simulation)
p-value for small samples.  As far as I can see, the justification for
the use of AIC as a comparison criterion is even more vague.

For linear fixed-effects models one can compensate for small samples
by changing from z-tests to t-tests and from chi-squared tests to F
tests.  The exact theory breaks down for mixed-effects models or for
generalized linear models and is even more questionable for
generalized linear mixed models.  As Ben Bolker mentioned, I think
that one way to deal with the hypothesis testing question while
preserving the integrity of the model is to base inferences on a
Markov-chain Monte Carlo sample from the (Bayesian) posterior
distribution of the parameters.

Code for MCMC samples for parameters in GLMMs is not yet fully
developed (or documented).  In the meantime I would use the likelihood
ratio tests but exercise caution in reporting p-values for
small-sample cases.
Dimitris Rizopoulos
# 
----- Original Message ----- 
From: "Douglas Bates" <bates at stat.wisc.edu>
To: "Achaz von Hardenberg" <fauna at pngp.it>
Cc: <r-sig-mixed-models at r-project.org>
Sent: Tuesday, January 08, 2008 3:10 PM
Subject: Re: [R-sig-ME] testing fixed effects in binomial 
lmer...again?
What about Bootstrap (parametric or not)? Would it be useful in this 
case?

(For instance, something along the following lines:

library(lme4)

form.null <- # formula under null
form.altr <- # formula under alternative
fm1 <- lmer(form.null, family = binomial, data = data)
fm2 <- lmer(form.altr, family = binomial, data = data)

# observed value of the LRT
Tobs <- anova(fm1, fm2)$Chisq[2]

B <- 199
Tvals <- numeric(B)
# 'id' is the grouping variable
unq.ids <- unique(data$id)
for (b in 1:B) {
    dat.new <- # a sample with replacement from the original subjects
    fm1 <- lmer(form.null, family = binomial, data = data.new)
    fm2 <- lmer(form.altr, family = binomial, data = data.new)
    Tvals[b] <- anova(fm1, fm2)$Chisq[2]
}
# estimated p-value
(1 + sum(Tvals >= Tobs)) / (B + 1)


if the estimated p-value is near the significance level, 'B' can be 
increased accordingly.)

Best,
Dimitris
Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm
David Duffy
#
On Tue, 8 Jan 2008, Dimitris Rizopoulos wrote:
The only problem is specifying a bootstrap mechanism that respects the
structure of the random effects.  So for time series data, your bootstrap
samples have to remain AR1 or whatever (ie you don't want gaps appearing that
aren't in the observed data), and for genetic type data (the kind I have),
that pseudosample people are appropriately related to one another.  Resampling
clusters works for that kind of data, though I think you need many clusters.
There are several papers in the area of genetic linkage analysis that have 
validated bootstrapping for a test that a variance component is zero.

But for testing simple hypotheses about particular fixed effects, 
a permutation/randomization test should work, I think.

David Duffy.
Ben Bolker
# 
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
David Duffy wrote:
| On Tue, 8 Jan 2008, Dimitris Rizopoulos wrote:
|>> On Jan 8, 2008 5:38 AM, Achaz von Hardenberg <fauna at pngp.it> wrote:
|>>
|>>> However, I am not sure about what I should do to test for the
|>>> significance of fixed effects in the binomial case
|> What about Bootstrap (parametric or not)? Would it be useful in this
|> case?
|>
|
| The only problem is specifying a bootstrap mechanism that respects the
| structure of the random effects.  So for time series data, your bootstrap
| samples have to remain AR1 or whatever (ie you don't want gaps
appearing that
| aren't in the observed data), and for genetic type data (the kind I have),
| that pseudosample people are appropriately related to one another.
Resampling
| clusters works for that kind of data, though I think you need many
clusters.
| There are several papers in the area of genetic linkage analysis that
have
| validated bootstrapping for a test that a variance component is zero.
|
| But for testing simple hypotheses about particular fixed effects,
| a permutation/randomization test should work, I think.
|
| David Duffy.

~  My favorite solution (which worked in nlme, I think, but might
take some time to get for lme4 ...) would to be able to generate
posterior simulations from the reduced model, then use these to
generate a null distribution for F statistics (or whatever) for
the model comparison.  This seems as though it would actually be
a relatively straightforward extension of mcmcsamp, once it exists --
although arguably once you have mcmcsamp you wouldn't need it
any more ...

~   Ben Bolker

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHg/drc5UpGjwzenMRAgtdAJ9iill9KFLLGOAUnLyXvCRVEWthEQCdFLb9
s+A2eFe1JfDrAy/0zW9MFhA=
=Aitb
-----END PGP SIGNATURE-----
5 days later
Ken Beath
#
On 09/01/2008, at 10:45 AM, Reinhold Kliegl wrote:

            

      
      
      
    

        
A solution to this problem is

Carpenter, JR; Goldstein, H; Rasbash, J "A novel bootstrap procedure  
for assessing the relationship between class size and achievement",
JOURNAL OF THE ROYAL STATISTICAL SOCIETY SERIES C-APPLIED STATISTICS,  
52: 431-443 Part 4 2003

but I expect that parametric bootstrapping is enough for most problems.

Ken