Maximum number of patterns and speed in grep
On Fri, Jul 13, 2012 at 1:41 PM, mdvaan <mathijsdevaan at gmail.com> wrote:
Here's some data (which should give you the error messages):
# read in data
data <- read.csv("https://dl.dropbox.com/u/13631687/data.csv", header =
T, sep = ",")
# first paste all data
data1 <- paste(data[,1], collapse = "|")
# second paste subsets of the data
data2a <- paste(data[1:750,1], collapse = "|")
data2b <- paste(data[751:1500,1], collapse = "|")
# define the object to be searched
text <- c("the first is Santa Fe Gold Corp", "the second is Starpharma
Holdings")
# match
strapplyc(text, data1)
strapplyc(text, data2a)
strapplyc(text, data2b)
Thanks in advance!
Although it seems that strapplyc can handle larger regular expressions
than grep in R it seems neither can handle as many as in your example
so process it in chunks:
k <- 3000 # chunk size
f <- function(from, text) {
to <- min(from + k - 1, nrow(data))
r <- paste(data[seq(from, to), 1], collapse = "|")
r <- gsub("[().*?+{}]", "", r)
strapply(text, r)
}
ix <- seq(1, nrow(data), k)
out <- lapply(text, function(text) unlist(lapply(ix, f, text)))
Statistics & Software Consulting GKX Group, GKX Associates Inc. tel: 1-877-GKX-GROUP email: ggrothendieck at gmail.com