Skip to content

Assumptions for ANOVA: the right way to check the normality

13 messages · Frodo Jedi, Robert Baer, Greg Snow +2 more

Original
Robert Baer
# 
Try the following:
# using your scrd data and your proposed models
fit1<- lm(response ~ stimulus + condition + stimulus:condition, data=scrd)
fit2<- lm(response ~ stimulus + condition, data=scrd)
fit3<- lm(response ~ condition, data=scrd)

# Set up for 6 plots on 1 panel
op = par(mfrow=c(2,3))

# residuals function extracts residuals
# Visual inspection is a good start for checking normality
# You get a much better feel than from some "magic number" statistic
hist(residuals(fit1))
hist(residuals(fit2))
hist(residuals(fit3))

# especially qqnorm() plots which are linear for normal data
qqnorm(residuals(fit1))
qqnorm(residuals(fit2))
qqnorm(residuals(fit3))

# Restore plot parameters
par(op)
Indeed - Kruskal-Wallis is a good test to use for one factor data that is 
ordinal so it is a good alternative to your fit3.
Your "response" seems to be a discrete variable rather than a continuous 
variable.
You must decide if it is reasonable to approximate it with a normal 
distribution which is by definition continuous.

        

      
      
    

  
  


  


      

    

    

      
      

        
      

    

    

      
      

        


  

        

      Greg Snow
    

    

      
      #
    

  


  

      
Remember that an non-significant result (especially one that is still near alpha like yours) does not give evidence that the null is true.  The reason that the 1st 2 tests below don't show significance is more due to lack of power than some of the residuals being normal.  The only test that I would trust for this is SnowsPenultimateNormalityTest (TeachingDemos package, the help page is more useful than the function itself).

But I think that you are mixing up 2 different concepts (a very common misunderstanding).  What is important if we want to do normal theory inference is that the coefficients/effects/estimates are normally distributed.  Now since these coefficients can be shown to be linear combinations of the error terms, if the errors are iid normal then the coefficients are also normally distributed.  So many people want to show that the residuals come from a perfectly normal distribution.  But it is the theoretical errors, not the observed residuals that are important (the observed residuals are not iid).  You need to think about the source of your data to see if this is a reasonable assumption.  Now I cannot fathom any universe (theoretical or real) in which normally distributed errors added to means that they are independent of will result in a finite set of integers, so an assumption of exact normality is not reasonable (some may want to argue this, but convincing me will be very difficult).  But looking for exact normality is a bit of a red herring because, we also have the Central Limit Theorem that says that if the errors are not normal (but still iid) then the distribution of the coefficients will approach normality as the sample size increases.  This is what make statistics doable (because no real dataset entered into the computer is exactly normal).  The more important question is are the residuals "normal enough"?  for which there is not a definitive test (experience and plots help).

But this all depends on another assumption that I don't think that you have even considered.  Yes we can use normal theory even when the random part of the data is not normally distributed, but this still assumes that the data is at least interval data, i.e. that we firmly believe that the difference between a response of 1 and a response of 2 is exactly the same as a difference between a 6 and a 7 and that the difference from 4 to 6 is exactly twice that of 1 vs. 2.  From your data and other descriptions, I don't think that that is a reasonable assumption.  If you are not willing to make that assumption (like me) then means and normal theory tests are meaningless and you should use other approaches.  One possibility is to use non-parametric methods (which I believe Frank has already suggested you use), another is to use proportional odds logistic regression.



--
Gregory (Greg) L. Snow Ph.D.
Statistical Data Center
Intermountain Healthcare
greg.snow at imail.org
801.408.8111

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        
      

    

    

      
      

        
      

    

    

      
      

        
      

    

    

      
      

        
      

    

    

      
      

        


  

        

      Greg Snow
    

    

      
      #
    

  


  

      
A lot of this depends on what question you are really trying to answer.  For one way anova replacing y-values with their ranks essentially transforms the distribution to uniform (under the null) and the Central Limit Theorem kicks in for the uniform with samples larger than about 5, so the normal approximations are pretty good and the theory works, but what are you actually testing?  The most meaningful null that is being tested is that all data come from the exact same distribution.  So what does it mean when you reject that null?  It means that all the groups are not representing the same distribution, but is that because the means differ? Or the variances? Or the shapes? It can be any of those.  Some point out that if you make certain assumptions such as symmetry or shifts of the same distributions, then you can talk about differences in means or medians, but usually if I am using non-parametrics it is because I don't believe that things are symmetric and the shift idea doesn't fit in my mind.

Some alternatives include bootstrapping or permutation tests, or just transforming the data to get something closer to normal.

Now what does replacing by ranks do in 2-way anova where we want to test the difference in one factor without making assumptions about whether the other factor has an effect or not?  I'm not sure on this one.

I have seen regression on ranks, it basically tests for some level of relationship, but regression is usually used for some type of prediction and predicting from a rank-rank regression does not seem meaningful to me.

Fitting the regression model does not require normality, it is the tests on the coefficients and confidence and prediction intervals that assume normality (again the CLT helps for large samples (but not for prediction intervals)).  Bootstrapping is an option for regression without assuming normality, transformations can also help.

--
Gregory (Greg) L. Snow Ph.D.
Statistical Data Center
Intermountain Healthcare
greg.snow at imail.org
801.408.8111

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Ben Ward
    

    

      
      #
    

  


  

      
I believe what I'm doing, is an ancova, because I have two categorical 
and a numerical explanatory variables, and a numerical response variable 
(this is the same experiment as before, the bacteria), and I'm just, at 
the minute (because I'm only half way through), doing some modelling and 
seeing what I get with what I currently have. And I'm paying attention 
to 95% CI for the different terms of a model, as well as the 
coefficient, and the explanatory power of the term and likelyhood that 
the same result could be obtained at random through the P values, 
derived from F. To be honest I havent checked much what my data 
distributions are like and such becasue I'm not finished collecting it 
yet. I mainly mentioned the ranking because it was given considerable 
mention in one of my texts sections on hypothesis testing on models.

        
On 07/01/2011 18:34, Greg Snow wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Frodo Jedi
    

    

      
      #
    

  


  

      
Dear Greg,
many thanks for your answer. Now I have a problem then in understanding how to 
check 

normality in case of ANOVA with repeated measures.
I would need an help with a numeric example, as I haven?tu fully understood how 
it works with the
proj() command as it as suggested by another R user in this mailing list.


For example, in attachment you find a .csv table resulting from an experiment, 
you can access it by means of this command:

            

      
      
      
    

        
The data are from an experiment where participants had to evaluate on a seven 
point likert scale
the realism of some stimuli, which are presented both in condition "A" and in 
condition "AH".

I need to perform the ANOVA by means of this command:

            

      
      
      
    

        
but the problem is that I cannot plot as usually do the qqnorm on the residuals 
of the fit because
lm does not support the Error term present in aov.
I normally check normality through a plot (or a shapiro.test function). Now 
could you please 

illustrate me how will you be able to undestand from my data if they are 
normally distributed?


Please enlighten me

Best regards

        

      
      
    

  
  


  


      

    

    
2 days later

    

      
      

        
      

    

    

      
      

        


  

        

      Mike Marchywka
    

    

      
      #
    

  


  

      
I can't get hotmail to indicate the original text so I'm going to top
post. There seems to be a lot of back and forth here, let me see if these
comment help guide discussion a bit. 

I tried to run some histograms of your experiment (prior to a bunch of other things )
and IIRC in many cases
you have counts under 10. At minimum, anything you do or any test you
run you want to do some senistivity analyses and perturb your data a bit.
Your objective of course is important- say you want to calibrate your
response data and try to validate your assumption that your survey 
question relfect some continuous variable ( but a respondent can only
round his response to an int as in teh case of taking a temperature for example, otherwise
all you can really say is that these things are like ranks, 7>6>5 etc ). 
Personally I always avoid non-parametrics
( just personal bias) but with small samples and a response that is closer
to a rank than a continuous variable with some meaning, it may make sense.

If you plot hitograms of responses versus A and AH, visually they look
different, you could try fitting the histos to various pdf's and see what you
get etc. This is all retro/post-hoc so you may as well explore away.


From: Greg.Snow at imail.org
To: frodo.jedi at yahoo.com
Date: Mon, 10 Jan 2011 11:26:05 -0700
CC: r-help at r-project.org
Subject: Re: [R] Assumptions for ANOVA: the right way to check the normality


What is the question you are really trying to find the answer for?  Knowing that may help us give more meaningful answers.

You keep wanting to test the residuals for normality, but it looks like you are doing it because some outdate recipe suggests it rather than that you understand why.

It is fairly easy to create a distribution that is definitely not normal, that gives the wrong answer most of the time if normality is assumed, yet will pass most normality tests most of the time (well except for SnowsPenualtimateNormalityTest, but that one has an unfair advantage in this situation).  So just because the residuals look normal (or close enough) does not mean that the theory holds.

R. A. Fisher is said to have said that the quality of a statistician can be judged by the amount of rat droppings under his finger nails.  Now if we take that literally, then I must not be very good.  But more what he meant is that a statistician must understand the source of the data, not just get a file and put it through some canned routines.  So these questions are really for you or the source of your data.

Also remember that the normality of the data/residuals/etc. is not as important as the CLT for your sample size.  The main things that make the CLT not work (for samples that are not large enough) are outliers and strong skewness, since your outcome is limited to the numbers 1-7, I don't see outliers or skewness being a real problem.  So you are probably fine for fixed effects style models (though checking with experts in your area or doing simulations can support/counter this).  But when you add in random effects then there is a lot of uncertainty about if the normal theory still holds, the latest lme code uses mcmc sampling rather than depending on normal theory and is still being developed.

This now comes back to my first question: what are you trying to find out?

You may not need to do anova or that type of model.  Some simple hypotheses may be answered using McNemars test on your data.  If you want to do predictions then linear models will be meaningless (what would a prediction of -3.2, 4.493, or 8.1 mean on a 7 point likert scale?) and something like proportional odds logistic regression will be much more meaningful.  Between those are bootstrap and permutation methods that may answer you question without any normality assumptions.

--
Gregory (Greg) L. Snow Ph.D.
Statistical Data Center
Intermountain Healthcare
greg.snow at imail.org
801.408.8111

From: Frodo Jedi [mailto:frodo.jedi at yahoo.com]
Sent: Saturday, January 08, 2011 3:20 AM
To: Greg Snow
Cc: r-help at r-project.org
Subject: Re: [R] Assumptions for ANOVA: the right way to check the normality

Dear Greg,
many thanks for your answer. Now I have a problem then in understanding how to check
normality in case of ANOVA with repeated measures.
I would need an help with a numeric example, as I haven?tu fully understood how it works with the
proj() command as it as suggested by another R user in this mailing list.


For example, in attachment you find a .csv table resulting from an experiment, you can access it by means of this command:

            

      
      
      
    

        
The data are from an experiment where participants had to evaluate on a seven point likert scale
the realism of some stimuli, which are presented both in condition "A" and in condition "AH".

I need to perform the ANOVA by means of this command:

            

      
      
      
    

        
but the problem is that I cannot plot as usually do the qqnorm on the residuals of the fit because
lm does not support the Error term present in aov.
I normally check normality through a plot (or a shapiro.test function). Now could you please
illustrate me how will you be able to undestand from my data if they are normally distributed?


Please enlighten me

Best regards