same old question - lme4 and p-values

I agree with Lorenz Gygax.  I'll come back to p-values below.

Confidence intervals (CIs) make, for me, a lot more sense than
p-values. The reality, though, is that users will interpret CIs as
some kind of probability statement.  For all practical purposes,
a CI is just the Bayesian credible interval that one gets with
some suitable "non-informative prior".  Why not then be specific
about the prior, and go with the Bayesian credible interval?
(There is an issue whether such a prior can always be found.
Am right in judging this no practical consequence?)

There are cases where the prior is informative in a sense that
breaks the nexus between the CI and a realistic Bayesian
credible interval.  A similar issue arises for a p-value; the
probability of the evidence given innocence (or freedom from
some rare disease)  (this is H0) is dramatically different from
the probability of innocence given the evidence, and it may
be a difference between 1/100000 and 1/2.  Where the
Bayesian credible interval and the CI are dramatically different,
a p-value or CI can only mislead.  In the way that p-values are
commonly taught, it may take considerable strength of will to
avoid confusion between P(A | H0) and (P(H0 | A)!

For intervals for variances, the prior can matter a lot, if a
smallish number of independent pieces of information is used
to  estimate the variance and/or those pieces of information
have widely varying weights.  I guess that emphasizes how
insecure inference about variances can be.  It is much worse
than the common forms of CI indicate.

If one is to take abs(t) > 2 as indicating significance, this is
under iid Normal assumptions a p-value of 0.1 for 5df, and 0.18
for 2 degrees of freedom.  One has to ask members of the
relevant scientific community whether they are comfortable
with that, given also that those p-values are likely to be more
than otherwise suspect because of the small number of
degrees of freedom.  Or are we discussing experiments where
we always have at least 10 degrees of freedom?  If not, and
there is an insistence on making claims of "significance",
maybe we want abs(t) > 2.5 or abs(t) > 3.

I do not see any cogent reason to be concerned that the
distribution of the Bayesian p-value may, under H0, be far
from uniform on (0,1).  This, if it is an issue, is an especial issue
for intervals for variances.

Why not then, for models fitted using lmer, a Bayesian HPD
interval, given that Douglas has made it so easy to calculate
these? This seems to me more than otherwise pertinent if the
emphasis is on effect size.

None of these various measures is more than a very crude
summary of what has been achieved.  Maybe plots of posterior
density estimates might be given for key parameters, ideally
with some indication of sensitivity to the prior (this would need
more than mcmcsamp()).  In any case, publish the data, so that
the sceptical reader can make his/her own checks, and/or use
it in the design of future experiments, and/or so that it can be
used as a teaching resource.

John Maindonald             email: john.maindonald at anu.edu.au
phone : +61 2 (6125)3473    fax  : +61 2(6125)5549
Centre for Mathematics & Its Applications, Room 1194,
John Dedman Mathematical Sciences Building (Building 27)
Australian National University, Canberra ACT 0200.

same old question - lme4 and p-values

Thread (29 messages)