Randomization Test: SOLVED

Ogbos,

To share data (in particularly lengthy one as yours), check out ?dput

To replicate sampling, look at ?replicate (will output to a data frame to use further) - that should answer your Q1.

Apart from that (and regarding Q2), the task you are after is getting more and more obscure to me. I don't see what you really want to test - between levels of n (or A in oodf)? If so, you would need to permute levels within each set (are they sets of -5:10, or are all observations completely independent? In the latter case, across all sets). What is the null hypothesis you want to test? And what is the data exactly? I don't understand what the two columns indicate. Then you need to decide on what the test statistic is you want to use. The average? The difference between each pair of averages? Anything like that? Do you want to test all levels simultaneously, or by pairs?

Clearly, your way of sampling is inappropriate for a randomization test. You are rather simulating data that can take any value between min and max with equal probability. That does not seem to be the right null model to me (it might be, though, depending on your null hypothesis). It would not make a randomization test either way, but rather a Monte Carlo simulation under the null hypothesis. Then you would determine your test statistic(s) (whichever they are) and subsequently repeat that often enough and check whether your observed value is higher than 90 or 99% of the simulated ones. Again, that would be a simulation, not a randomization test.

For the latter, you'd need to permute the data in an appropriate way (again depending on the null hypothesis, and on the structure, i.e. are they are coming in blocks of any kind, or all independent), and then recalculate the test statistic every time, and then proceed as before. I'm not sure whether the data was the result of a randomized experiment - if not, you can still use the idea, but fall into something that some refer to as "permutation testing" - the difference being that you need to make much stronger assumptions on independence etc of the data. Many use the terms equivalently, but just so you are aware.

I really think you need to look into a textbook (eg. Edgington & Onghena) or some papers to understand the concept of randomization tests, or consult with a statistician with good background in that field. What you are suggesting is not near it, and unless you have a clear hypothesis and a good understanding of how the data was generated, it is impossible for you (and anyone else) to say how such a test might be designed.

Michael

-----Original Message-----
From: Ogbos Okike <giftedlife2014 at gmail.com>
Sent: Mittwoch, 27. Februar 2019 22:53
To: Meyners, Michael <meyners.m at pg.com>
Cc: r-help <r-help at r-project.org>
Subject: Re: [R] Randomization Test

Dear Kind List,

I am still battling with this. I have, however, made some progress with the
suggestions of Micheal and others. At least, I have a better picture of what I
want to do now as I will attempt a detailed description here.

I am aware I should show you just a small part of my code and data.
But when I copied out a small portion and run to see what you get when I
send that,  I was not satisfied with the signal displayed. The epoch analysis
averages data and is quite sensitive to leveraging, especially if a small sample
is used.

So please permit/exercise patience  me to display the series of epoch that
give the averaged valued used. You can just run the code and see the signal
of interest. Here is the code and the data:

dta <- read.table( text ="n CR
 -5 8969

.
SNIP...
.

10 9566
",header=TRUE)

 data<-matrix(c(dta$CR),ncol=71)
A<-matrix(rep(-5:10,71))
B<-matrix(data)

 oodf<-data.frame(A,B)
 a<--5:10
oodf<-data.frame(A,B)
library(plotrix)
std.error<-function(x) return(sd(x)/(sum(!is.na(x))))
oomean<-as.vector(by(oodf$B,oodf$A,mean))
oose<-as.vector(by(oodf$B,oodf$A,std.error))
plot(-5:10,oomean,type="l",ylim=c(8890,9100),
 )
A<-oomean-1.96*oose
 B<-oomean+1.96*oose
lines(a,A,col="red")
 lines(a,B,col="red")

 My Question:
I wish to conduct a randomization test of significance (90 and 99
percentile) of the reductions/decreases as displayed by the signal.

I am attempting using:
x<-sample(8890:9500,1136,replace=T )

to generate the random numbers, where 8890, 9500 and 1136 are the
minimum  and maximum of the signal and 1136 the length of sample data.
Q1: Please how do I generate many samples as x above, say up to 5000 or
10,000? I manually generated and stored as x1,x2, x3 up to x100.

Q2: Please how do I use this randomly generated numbers to test the
statistical significance level of the signal generated by plot(-
5:10,oomean,type="l",ylim=c(8890,9100),  )?

I wish to test for 90% and 99% percentile.

I am sorry that this is too long.

Many thanks for your kind contributions

Best
Ogbos







On Sun, Feb 10, 2019 at 3:55 PM Ogbos Okike <giftedlife2014 at gmail.com>
wrote:

Dear Michael,
This is great! Thank you.

I have not really got any response other than yours.

I have long before now included what I have in a paper submitted to a

journal.

I am awaiting the feedback of the reviewer. I will compare the
comments with your input here and determine the corrections to make
and probably return to the list for additional help.

Best wishes
Ogbos

On Fri, Feb 8, 2019 at 4:31 PM Meyners, Michael <meyners.m at pg.com>

wrote:

Ogbos,

You do not seem to have received a reply over the list yet, which might

be due to the fact that this seems rather a stats than an R question. Neither
got your attachment (Figure) through - see posting guide.

I'm not familiar with epoch analysis, so not sure what exactly you are

doing / trying to achieve, but some general thoughts:

* You do NOT want to restrict your re-randomizations in a way that "none

of the dates corresponds with the ones in the real event" - actually, as a
general principle, the true data must be an admissible re-randomization as
well. You seem to have excluded that (and a lot of other randomizations at
the same time which might have occurred, i.e. dates 1 and 2 reversed but all
others the same), thereby rendering the test invalid. Any restrictions you
have on your re-randomizations must've applied to the original
randomization as well.

* If you have rather observational data (which I suspect, but not sure),

Edgington & Onghena (2007) would rather refer to this as a permutation test
- the difference being that you have to make strong assumptions (similar to
parametric tests) on the nature of the data, which are designed-in to be true
for randomization tests. It might be a merely linguistic discrimination, but it is
important to note which assumptions have to be (implicitly) made.

* I'm not sure what you mean by "mean differences" of the events - is

that two groups you are comparing? If so, that seems reasonable, but just
make sure the test statistic you use is reasonable and sensitive against the
alternatives you are mostly interested in. The randomization/permutation
test will never proof that, e.g., means are significantly different, but only that
there is SOME difference. By selecting the appropriate test statistic, you can
influence what will pop up more easily and what not, but you can never be
sure (unless you make strong assumptions about everything else, like in
many parametric tests).

* For any test statistic, you would then determine the proportion of its

values among the 5000 samples where it is as large or larger than the one
observed (or as small or smaller, or either, depending on the nature of the
test statistic and whether you aim for a one- or a two-sided test). That is your
p value. If small enough, conclude significance. At least conceptually
important: The observed test statistic is always part of the re-randomization
(i.e. your 5000) - so you truly only do 4999 plus the one you observed.
Otherwise the test may be more or less liberal. Your p value is hence no
smaller than 1/n, where n is the total number of samples you looked at
(including the observed one), a p value of 0 is not possible in randomization
tests (nor in other tests, of course).

I hope this is helpful, but you will need to go through these and refer to

your own setup to check whether you adhered to the principles or not,
which is impossible for me to judge based on the information provided (and I
won't be able to look at excessive code to check either).

Michael

-----Original Message-----
From: R-help <r-help-bounces at r-project.org> On Behalf Of Ogbos
Okike
Sent: Montag, 28. Januar 2019 19:42
To: r-help <r-help at r-project.org>
Subject: [R] Randomization Test

Dear Contributors,

I conducting epoch analysis. I tried to test the significance of
my result using randomization test.

Since I have 71 events, I randomly selected another 71 events,
making sure that none of the dates in the random events
corresponds with the ones in the real event.

Following the code I found here

(https://www.uvm.edu/~dhowell/StatPages/R/RandomizationTestsWithR/

R andom2Sample/TwoIndependentSamplesR.html),
I combined these two data set and used them to generate another
5000 events. I then plotted the graph of the mean differences for
the 5000 randomly generated events. On the graph, I indicated the
region of the mean difference between the real 71 epoch and the
randomly selected 71 epoch.

Since the two tail test shows that the mean difference falls at
the extreme of the randomly selected events, I concluded that my
result is statistically significant.



I am attaching the graph to assistance you in you suggestions.

I can attach both my code and the real and randomly generated
events if you ask for it.

My request is that you help me to understand if I am on the right track

or no.

This is the first time I am doing this and except the experts
decide, I am not quite sure whether I am right or not.

Many thanks for your kind concern.

Best
Ogbos