Hi, Imagine the two problems: 1) You have an event that occurs repeatedly over time. You want to identify periods when the event occurs more frequently than the base rate of occurrence. Ideally, you don't want to have to specify the period (e.g., break into months), so the analysis can be sensitive to scenarios such as many events happening only between, e.g., June 10 and June 15 - even though the overall number of events for the month may not be much greater than usual. Similarly, there may be a cluster of events that occur from March 28 to April 3. Ideally, you want to pull out the base rate of occurrence and highlight only the periods when the frequency is less or greater than the base rate. 2) Events again occur repeatedly over time in an inconsistent way. However, this time, the event has positive or negative outcomes - such as a spot check of conformity to regulations. You again want to know whether there is a group of negative outcomes close together in time. This analysis should take into account the negative outcomes as well though. E.g., if from June 10 to June 15 you get 5 negative outcomes and no positive outcomes it should be flagged. On the other hand, if from June 10 to June 15 you get 5 negative outcomes interspersed between many positive outcomes it should be ignored. I'm guessing that there is some statistical approach designed to look at these types of issues. What is it called? What package in R implements it? I basically just need to know where to start. Thanks, Mark
How to identify runs or clusters of events in time
3 messages · Mark Shanks, Clint Bowman, Charles C. Berry
Mark,
I did something similar a couple of year ago by coding non-events as 0,
positive events as +1 and negative events as -1 then summing the value
through time. In my case the patterns showed up quite clearly and I used
other criteria to define the actual periods.
Clint
Clint Bowman INTERNET: clint at ecy.wa.gov
Air Quality Modeler INTERNET: clint at math.utah.edu
Department of Ecology VOICE: (360) 407-6815
PO Box 47600 FAX: (360) 407-7534
Olympia, WA 98504-7600
USPS: PO Box 47600, Olympia, WA 98504-7600
Parcels: 300 Desmond Drive, Lacey, WA 98503-1274
On Fri, 1 Jul 2016, Mark Shanks wrote:
Hi, Imagine the two problems: 1) You have an event that occurs repeatedly over time. You want to identify periods when the event occurs more frequently than the base rate of occurrence. Ideally, you don't want to have to specify the period (e.g., break into months), so the analysis can be sensitive to scenarios such as many events happening only between, e.g., June 10 and June 15 - even though the overall number of events for the month may not be much greater than usual. Similarly, there may be a cluster of events that occur from March 28 to April 3. Ideally, you want to pull out the base rate of occurrence and highlight only the periods when the frequency is less or greater than the base rate. 2) Events again occur repeatedly over time in an inconsistent way. However, this time, the event has positive or negative outcomes - such as a spot check of conformity to regulations. You again want to know whether there is a group of negative outcomes close together in time. This analysis should take into account the negative outcomes as well though. E.g., if from June 10 to June 15 you get 5 negative outcomes and no positive outcomes it should be flagged. On the other hand, if from June 10 to June 15 you get 5 negative outcomes interspersed between many positive outcomes it should be ignored. I'm guessing that there is some statistical approach designed to look at these types of issues. What is it called? What package in R implements it? I basically just need to know where to start. Thanks, Mark [[alternative HTML version deleted]]
______________________________________________ R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
See below
On Fri, 1 Jul 2016, Mark Shanks wrote:
Hi, Imagine the two problems: 1) You have an event that occurs repeatedly over time. You want to identify periods when the event occurs more frequently than the base rate of occurrence. Ideally, you don't want to have to specify the period (e.g., break into months), so the analysis can be sensitive to scenarios such as many events happening only between, e.g., June 10 and June 15 - even though the overall number of events for the month may not be much greater than usual. Similarly, there may be a cluster of events that occur from March 28 to April 3. Ideally, you want to pull out the base rate of occurrence and highlight only the periods when the frequency is less or greater than the base rate.
A good place to start is: Siegmund, D. O., N. R. Zhang, and B. Yakir. "False discovery rate for scanning statistics." Biometrika 98.4 (2011): 979-985. and Aldous, David. Probability approximations via the Poisson clumping heuristic. Vol. 77. Springer Science & Business Media, 2013. --- A nice illustration of how scan statistcis can be used is: Aberdein, Jody, and David Spiegelhalter. "Have London's roads become more dangerous for cyclists?." Significance 10.6 (2013): 46-48.
2) Events again occur repeatedly over time in an inconsistent way. However, this time, the event has positive or negative outcomes - such as a spot check of conformity to regulations. You again want to know whether there is a group of negative outcomes close together in time. This analysis should take into account the negative outcomes as well though. E.g., if from June 10 to June 15 you get 5 negative outcomes and no positive outcomes it should be flagged. On the other hand, if from June 10 to June 15 you get 5 negative outcomes interspersed between many positive outcomes it should be ignored. I'm guessing that there is some statistical approach designed to look at these types of issues. What is it called?
`Scan statistic' is a good search term. `Poisson clumping', too.
What package in R implements it? I basically just need to know where to start.
There are some R packages. CRAN has packages SNscan and graphscan, which sound like they might interest you. My BioConductor package geneRxCluster: http://bioconductor.org/packages/release/bioc/html/geneRxCluster.html seeks clusters in a binary sequence as described in detail at http://bioinformatics.oxfordjournals.org/content/30/11/1493 HTH, Chuck