Brian has some really sound advice here. Using a Monte Carlo approximation
rather than the exact result kind of misses the entire point. 0.05 is
arbitrary and using an approximation to an exact distribution that you can
easily calculate is misguided. Getting a p-value of 0.1 versus 0.05 really
shouldn't result in any groundbreaking scientific differences in
interpretation, despite what a journal might say about your p-values.
You should read Greenland et al (http://link.springer.com/
article/10.1007/s10654-016-0149-3) which has some choice quotes from
Neyman and Pearson and a lot of wisdom regarding the insanity surrounding
our use of p-values.
Jason Law
Statistician, City of Portland
Water Pollution Control Laboratory
6543 N Burlington Ave,
Portland, OR
503-823-1038
jason.law at portlandoregon.gov
-----Original Message-----
From: R-sig-ecology [mailto:r-sig-ecology-bounces at r-project.org] On
Behalf Of Cade, Brian
Sent: Friday, November 18, 2016 8:51 AM
To: Ellen Pape <ellen.pape at gmail.com>
Cc: <r-sig-ecology at r-project.org> <r-sig-ecology at r-project.org>
Subject: Re: [R-sig-eco] How to obtain P value from Monte Carlo sampling
for adonis (permanova)?
Ellen: If you are running permutation procedures with data that have very
small sample sizes in each group (your two groups of n = 3 each yields only
6!/(3!3!) = M = 20 permutations under Ho), then you just have to live with
the fact that the smallest possible P-value is 1/M (= 0.05 for your two
group example). There is nothing magical about P < 0.05 anyways. But as
the Monte Carlo resampling approach to obtaining permutation P-values
really is just a method to attempt to approximate the exact permutation
P-value (and usually used when M is so large that you can not enumerate it
exactly in reasonable computation time), you should not rely on it when the
number or permutations M is so small, and especially not just because you
might obtain a P < 1/M. If you obtain a P-value <0.05 in your example
using the Monte Carlo resampling procedure, all that indicates is that the
Monte Carlo resampling approach is a poor approximation in this small
sample situation. I think it is always preferable to obtain and interpret
the exact permutation distribution if it is easily calculable. Using a
crummy approximation just because you want P < 0.05 seems unreasonable to
me.
Brian
Brian S. Cade, PhD
U. S. Geological Survey
Fort Collins Science Center
2150 Centre Ave., Bldg. C
Fort Collins, CO 80526-8818
email: cadeb at usgs.gov <brian_cade at usgs.gov>
tel: 970 226-9326
On Fri, Nov 18, 2016 at 1:39 AM, Ellen Pape <ellen.pape at gmail.com> wrote:
Dear all,
I have recently decided to switch from Permanova/Primer to R, because
the latter is freeware (and I don't know for how long I will still
have a license). However, if I cannot seem to resolve my problem (see
below), I might have to go back to using Primer/Permanova.
If I run pairwise permanova/adonis tests on my data, the number of
unique permutations is small (I have two groups, each group has 3
observations) and the minimum P value I can get is larger than the
alpha value I (and most people) that I use to determine statistical
significance (i.e. 0.05).
In the manual of the PERMANOVA+ add-on in Primer by Anderson et al.
(2008) it is mentioned (page 28 and onwards) that when the number of
unique permutations is small (<100) than one should preferably
interpret the Monte-Carlo p value.
Does anyone know how to do this in R?
In my internet search I stumbled upon this page:
http://r.789695.n4.nabble.com/monte-carlo-simulations-in-
permanova-in-vegan-package-td4714034.html.
"JAri Oksanen answered here: 2. If you want to do so, you can generate
your resampling matrices by hand and use that matrix as the argument
of permutations=. See the documentations (?adonis) which tells how to do
", but I don't understand how to this..
Thank you very much!
Ellen
[[alternative HTML version deleted]]