same old question - lme4 and p-values
For a one-sided situation, the posterior probability that the parameter is on the wrong side of zero makes a lot of sense. (Ironically, the classic mistake of interpreting a p-value as a probability statement about the parameter is correct here!) The two-sided case seems more challenging. The posterior probability content of an "indifference zone" around zero can be computed, but it's easy to play games by carefully picking the width of the zone. A graph of the probability against the width of the zone could be a useful exploratory tool, but would take up a lot of space if used in reporting results. Regards, Rob Kushler
Martin Maechler wrote:
"Jon" == Jonathan Baron <baron at psych.upenn.edu>
on Sat, 5 Apr 2008 07:21:19 -0400 writes:
Jon> On 04/05/08 12:10, Reinhold Kliegl wrote:
[...]
>> In perspective, I think the p-value problem will simply
>> go away.
Jon> I'm not sure what you mean here. If you mean to
Jon> replace them with confidence intervals, I have no
Jon> problem with that. But, as a journal editor, I am
Jon> afraid that I will continue to insist on some sort of
Jon> evidence that effects are real. This can be done in
Jon> many ways. But too many authors submit articles in
Jon> which the claimed effects can result from random
Jon> variation, either in subjects ("participants*") or
Jon> items, and they don't correctly reject such alternative
Jon> explanations of a difference in means.
Jon> I have noticed a kind of split among those who comment
Jon> on this issue. On the one side are those who are
Jon> familiar with fields such as epidemiology or economics
Jon> (excluding experimental economics), where the claim is
Jon> often made that "the null hypothesis is always false
Jon> anyway, so why bother rejecting it?" These are the
Jon> ones interested in effect sizes, variance accounted
Jon> for, etc. They are correct for this kind of research,
Jon> but there are other kinds of research.
Jon> On the other side, are those from (e.g.) experimental
Jon> psychology, where the name of the game is to design
Jon> experiments that are so well controlled that the null
Jon> hypothesis will be true if the effect of interest is
Jon> absent. As a member of this group, when I read people
Jon> from the first group, I find it very discouraging. It
Jon> is almost as if they are saying that what I work so
Jon> hard to try to do is impossible.
Jon> To get a little specific, although I found Gelman and
Jon> Hill's book very helpful on many points (and it does
Jon> not deny the existence of people like me), it is
Jon> written largely for members of the first group. By
Jon> contrast, Baayen's book is written for people like me,
Jon> as is the Baayen, Davidson, and Bates article, "Mixed
Jon> effects modeling with crossed random effects for
Jon> subjects and items."
Jon> I'm afraid we do need significance tests, or confidence
Jon> intervals, or something.
I agree even though I'm very deeply inside the camp of statisticians
who know that all models are wrong but some are useful, and
hence I do not "believe" any P-values (or exact confidence /
credibility intervals).
For those who need ``something like a P-value'' I've heard
yesterday Lorenz Gygax (also subscriber here) proposing
to report the "credibility of 0", possibly "2-sided", as a
pseudo-P value;, i.e. basically that would be
2 * k/n, for an MCMC sample b_1,b_2, ..., b_n
k := {min k'; b_k' > 0}.
The reasoning would be the following:
Use the 1-to-1 correspondence between confidence intervals and
testing pretending that the credibility intervals are confidence
intervals, and consequently you just need to look at which
confidence level 0 will be at the exact border of the
credibility interval.
Yesterday after the talk, I found that a good idea.
Just now, it seems a bit doubtful, since under the null
hypothesis, I don't think such a pseudo P-value would be uniform
in [0,1].
Martin
Jon> * On "participants" vs. "subjects" see:
Jon> http://www.psychologicalscience.org/observer/getArticle.cfm?id=1549
Jon> -- Jonathan Baron, Professor of Psychology, University
Jon> of Pennsylvania Home page:
Jon> http://www.sas.upenn.edu/~baron Editor: Judgment and
Jon> Decision Making (http://journal.sjdm.org)
Jon> _______________________________________________
Jon> R-sig-mixed-models at r-project.org mailing list
Jon> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models