On Corrections for Chi-Sq Goodness of Fit Test

When the p-value is calculated via simulation the test is not, strictly
speaking a ``chi-squared test''.    That is, it is *not* assumed that the
distribution of the test statistic, under the null hypothesis, is a chi-squared
distribution.  Instead an empirical distribution derived from simulating
samples under the null hypothesis is used.  The *same statistic* is calculated,
but the chi-squared distribution is never invoked.  The simulated p-value
facility is provided so that tests can be conducted even when the requirements
for the validity of the chi-squared (null) distribution are not met.

The test, whether you want to call it a chi-squared test or not, is a perfectly
valid goodness of fit test.  (Modulo the caveat that I previously mentioned,
i.e. that it is required that there be no ties amongst the values of the test
statistics.  Which is a mild worry, but has nothing to do with continuity corrections
which are meaningless in this context.)

   cheers,

       Rolf Turner

On 25/12/11 08:26, Michael Fuller wrote:

On Dec 22, 2011, at 8:56 PM, Rolf Turner wrote:

On 20/12/11 10:24, Michael Fuller wrote:

TOPIC
My question regards the philosophy behind how R implements corrections to chi-square statistical tests. At least in recent versions (I'm using 2.13.1 (2011-07-08) on OSX 10.6.8.), the chisq.test function applies the Yates continuity correction for 2 by 2 contingency tables. But when used as a goodness of fit test (GoF, aka likelihood ratio test), chisq.test does not appear to implement any corrections for widely recognized problems, such as small sample size, non-uniform expected frequencies, and one D.F.

From the help page:

"In the goodness-of-fit case simulation is done by random sampling from the discrete distribution specified by p, each sample being of size n = sum(x)."

Is the thinking that random sampling completely obviates the need for corrections?

   Yes.

Wouldn't the same statistical issues still apply

   No.

(e.g. poor continuity approximation with one D.F.,

   There are no degrees of freedom involved.  There is no continuity involved.
   The observed test statistics (say "Stat") is compared with a number of
   test statistics, Stat_1, ..., Stat_N, calculated from data sets simulated under
   the null hypothesis.  If the null is true, then Stat and Stat_1, ...., Stat_N are
   all of ``equal status''.  If there are m values of the Stat_i which are greater
   than Stat, then the ``probability of observing, under the null hypothesis,
   data as extreme as, or more extreme than, what you actually observed''
   is the probability of randomly selecting one of a specified set of m+1 ``slots''
   out of a total of N+1 slots (where each slot has probability 1/(N+1)).

But if the test is truly a chi-square Goodness of Fit (GoF) test, then:

(1) the test compares the Stat to a chi-square distribution with (k-1) degrees of
freedom, for k frequency categories.

(2) the shape of the distribution depends on the degrees of freedom

(3) continuity is an issue, because the values for Stat are discrete, whereas
the chi-square distribution is continuous. Therefore the p-value is not exact.

If I understand your description, the function generates a probability
distribution by sampling a set of values with equal probability. The probability of
any value in the final distribution depends on its frequency in the sample set. This
is a standard Monte Carlo method, but the PDF it generates is not chi-square (the
resulting distribution depends upon the distribution of the sample set). Clearly,
using a permutation method such as this to determine the statistical significance of
a given test statistic is non-parametric and in particular, not chi-square.

If what you say is true, then the chisq.test function in R does not implement a true GoF
test. I don't understand why the GoF test is not computed the standard way. It should
compute the chi-square statistic, using the values of p to generate expected
frequencies. And it should use a continuity correction when degrees of freedom = 1.

Thank you for taking the time to respond to my message.

Cheers,
Mike

   Thus the p-value is (exactly) equal to (m+1)/(N+1).

   The only restriction is that there be no ties amongst the values of Stat
   and Stat_1, ..., Stat_N.  There being ties is of fairly low probability, but is
   not of zero probability --- since there is a finite number of possible samples
   and hence of statistic values.  So this restriction is a mild worry.

   However a ``continuity correction'' would be of no help whatsoever.

problems with non-uniform expected frequencies, etc) with random sampling?

   Don't understand what you mean by this.

       cheers,

           Rolf Turner

====================
Michael M. Fuller, Ph.D.
Department of Biology
University of New Mexico
Albuquerque, NM
EMAIL: mmfuller at unm.edu
WEB: biology.unm.edu/mmfuller