Wald F tests

Does anyone out there have opinions on this subject?
How should one test hypotheses about fixed effects in
(G)LMMs, especially for small to moderate sample sizes?
(Please ignore issues of _estimation_ (PQL
vs Laplace vs AGQ vs ...)

Should it amuse you to do so, you can vote at:

http://www.surveymonkey.com/s.aspx?sm=yLyfrV_2ftw6WGx2dEFLWnIw_3d_3d

(since we all know that scientific questions are settled
by a democratic process)

a hypothesis testing is soooo 20th century, don't bother
b likelihood ratio tests [ignore known anticonservatism]
c F tests (LMM) or Wald tests (GLMM) [ignore mismatch with  
hypothesized
null distributions]
d bootstrapped confidence intervals
e [mcmcsamp confidence intervals -- if available]
f randomization/simulation tests of nested null hypotheses
g AIC comparisons [ignore that prediction != hypothesis testing]

Note that Wald Z tests [option c] are more or less what you're
doing, implicitly, if you just eyeball the estimated
parameter values and their standard errors.

cheers
  Ben Bolker

-------- Original Message --------
Subject: [Fwd: Re: [R-sig-ME] Wald F tests]
Date: Tue, 07 Oct 2008 17:51:01 -0400
From: Ben Bolker <bolker at ufl.edu>
To: R Mixed Models <r-sig-mixed-models at r-project.org>


But ... LRTs are non-recommended (anticonservative) for
comparing fixed effects of LMMs hence (presumably) for
GLMMs, unless sample size (# blocks/"residual" total sample
size) is large, no?

I just got through telling readers of
a forthcoming TREE (Trends in Ecology and Evolution) article
that they should use Wald Z, chi^2, t, or F (depending on
whether testing a single or multiple parameters, and whether
there is overdispersion or not), in preference to LRTs,
for testing fixed effects ... ?  Or do you consider LRT
better than Wald in this case (in which case as far as
we know _nothing_ works very well for GLMMs, and I might
just start to cry ...)  Or perhaps I have to get busy
running some simulations ...

Where would _you_ go to find advice on inference
(as opposed to estimation) on estimated GLMM parameters?

cheers
 Ben Bolker

Douglas Bates wrote:

If I were using glmer to fit a generalized linear mixed model I would
use likelihood ratio tests rather than Wald tests.  That is, I would
fit a model including a particular term then fit it again without  
that
term and calculate the difference in the deviance values, comparing
that to a chi-square.

I'm not sure how one would do this using the results from glmmPQL.

On Fri, Oct 3, 2008 at 3:37 PM, Ben Bolker <bolker at ufl.edu> wrote:

[forwarding to R-sig-mixed, where it is likely to get more
responses]

Mark Fowler wrote:

Hello,
     Might anyone know how to conduct Wald-type F-tests of the fixed
effects estimated by glmmPQL? I see this implemented in SAS  
(GLIMMIX),
and have seen it recommended in user group discussions, but  
haven't come
across any code to accomplish it. I understand the anova function  
treats
a glmmPQL fit as an lme fit, with the test assumptions based on  
maximum
likelihood, which is inappropriate for PQL. I'm using S-Plus 7. I  
also
have R 2.7 and S-Plus 8 if necessary.

c F tests (LMM) or Wald tests (GLMM) [ignore mismatch with hypothesized
null distributions]
d bootstrapped confidence intervals
e [mcmcsamp confidence intervals -- if available]
f randomization/simulation tests of nested null hypotheses
g AIC comparisons [ignore that prediction != hypothesis testing]

I think it is going too far to say that one should not be
testing hypotheses (the implication of that "is soooo 20th C"?).
But the place of that activity is much more limited than is
commonly recognized.

Basically, I do not like the range of options that this (half-serious?)
survey has on offer, and I'd need to write half a page or more
to explain why.  Democracy maybe, but (as I suppose is
always the case in the political democracies that are on offer)
the choices are severely constrained.

Where such a hypothesis testing perspective may be
appropriate, the preferred starting point is almost always
a confidence interval.  Why not ask the comparable questions
arise for estimation?

There's an editorial in Volume 72(5) (pp.1057-1058) of the
Journal of Wildlife Management with which I pretty much agree:
"... understand that the average reader of the Journal is
interested in the biological questions addressed with your
work.  The analytical framework and resulting results should
support those questions and flow from them, not overwhelm
them."

But I guess that Ben would like us to assume that the proper
support framework is in place!

Note also, on pages 1272-1278 of the same issue:
"Suggestions for Basic Graph Use When Reporting Wildlife
Research Results", by Brett Collier.

John Maindonald             email: john.maindonald at anu.edu.au
phone : +61 2 (6125)3473    fax  : +61 2(6125)5549
Centre for Mathematics & Its Applications, Room 1194,
John Dedman Mathematical Sciences Building (Building 27)
Australian National University, Canberra ACT 0200.


On 11/10/2008, at 6:47 AM, Ben Bolker wrote:

Does anyone out there have opinions on this subject?
How should one test hypotheses about fixed effects in
(G)LMMs, especially for small to moderate sample sizes?
(Please ignore issues of _estimation_ (PQL
vs Laplace vs AGQ vs ...)

Should it amuse you to do so, you can vote at:

http://www.surveymonkey.com/s.aspx?sm=yLyfrV_2ftw6WGx2dEFLWnIw_3d_3d

(since we all know that scientific questions are settled
by a democratic process)

a hypothesis testing is soooo 20th century, don't bother
b likelihood ratio tests [ignore known anticonservatism]
c F tests (LMM) or Wald tests (GLMM) [ignore mismatch with hypothesized
null distributions]
d bootstrapped confidence intervals
e [mcmcsamp confidence intervals -- if available]
f randomization/simulation tests of nested null hypotheses
g AIC comparisons [ignore that prediction != hypothesis testing]

Note that Wald Z tests [option c] are more or less what you're
doing, implicitly, if you just eyeball the estimated
parameter values and their standard errors.

cheers
  Ben Bolker

-------- Original Message --------
Subject: [Fwd: Re: [R-sig-ME] Wald F tests]
Date: Tue, 07 Oct 2008 17:51:01 -0400
From: Ben Bolker <bolker at ufl.edu>
To: R Mixed Models <r-sig-mixed-models at r-project.org>


But ... LRTs are non-recommended (anticonservative) for
comparing fixed effects of LMMs hence (presumably) for
GLMMs, unless sample size (# blocks/"residual" total sample
size) is large, no?

I just got through telling readers of
a forthcoming TREE (Trends in Ecology and Evolution) article
that they should use Wald Z, chi^2, t, or F (depending on
whether testing a single or multiple parameters, and whether
there is overdispersion or not), in preference to LRTs,
for testing fixed effects ... ?  Or do you consider LRT
better than Wald in this case (in which case as far as
we know _nothing_ works very well for GLMMs, and I might
just start to cry ...)  Or perhaps I have to get busy
running some simulations ...

Where would _you_ go to find advice on inference
(as opposed to estimation) on estimated GLMM parameters?

cheers
 Ben Bolker

Douglas Bates wrote:

If I were using glmer to fit a generalized linear mixed model I would
use likelihood ratio tests rather than Wald tests.  That is, I would
fit a model including a particular term then fit it again without that
term and calculate the difference in the deviance values, comparing
that to a chi-square.

I'm not sure how one would do this using the results from glmmPQL.

On Fri, Oct 3, 2008 at 3:37 PM, Ben Bolker <bolker at ufl.edu> wrote:

[forwarding to R-sig-mixed, where it is likely to get more
responses]

Mark Fowler wrote:

Hello,
     Might anyone know how to conduct Wald-type F-tests of the fixed
effects estimated by glmmPQL? I see this implemented in SAS (GLIMMIX),
and have seen it recommended in user group discussions, but haven't
come
across any code to accomplish it. I understand the anova function
treats
a glmmPQL fit as an lme fit, with the test assumptions based on maximum
likelihood, which is inappropriate for PQL. I'm using S-Plus 7. I also
have R 2.7 and S-Plus 8 if necessary.