Message-ID: <701552A0-ED8A-434E-9FD9-6CDA2F11EF4B@uos.de>
Date: 2008-09-10T14:11:07Z
From: Stefan Evert
Subject: A question about the hypergeometric distribution and phyper()
In-Reply-To: <8975119BCD0AC5419D61A9CF1A923E9506D87BB9@iahce2ksrv1.iah.bbsrc.ac.uk>
On 10 Sep 2008, at 15:19, michael watson (IAH-C) wrote:
> Example 1: I have a universe of 6187 objects, and 164 have a
> particular
> attribute, therefore 6187-164 do not have that attribute. I sample
> 249
> of those objects, and find that 19 have that attribute. I get a p-
> value
> here (looking at just over-representation):
>
> phyper(19, 164, 6187-164, 249, lower.tail=FALSE)
> [1] 7.816235e-06
Actually, if you look at ?phyper, you'll see that this should be
phyper(18, 164, 6187-164, 249, lower.tail=FALSE)
[1] 2.775819e-05
if you want to calculate Pr(X >= 19) = Pr(X > 18). Similarly:
> phyper(4, 12, 6187-12, 249, lower.tail=FALSE)
> [1] 6.368919e-05
phyper(3, 12, 6187-12, 249, lower.tail=FALSE)
[1] 0.0009816739
Which you'll still find counterintuitive, of course.
> It seems to me that the probability of seeing 19 out of 164 in a
> sample
> of 249 being less than the probability of seeing 4 out of 12 in a
> sample
> of the same size is counter-intuitive.
>
> Secondly, can someone point me to some documentation explaining why
> these seemingly counter-intuitive p-values occur?
I think it's just because the hypergeometric distribution becomes very
skewed and non-normal for expected values < 1 (expectations should be
roughly 6.6 in the first case and 0.5 in the second case). Perhaps it
helps to visualize the two distributions?
M <- rbind(dhyper(0:20, 164, 6187-164, 249), dhyper(0:20, 12, 6187-12,
249))
rownames(M) <- c("164 out of 6187", "12 out of 6187")
colnames(M) <- 0:20
barplot(M, beside=TRUE, legend = TRUE)
Best regards,
Stefan Evert
[ stefan.evert at uos.de | http://purl.org/stefan.evert ]