what is the best way to process the following data?

You can make a step-number variable with cumsum(grepl("^Step ", ...)) and
use it as the splitting variable in split.  E.g.,
dat <- read.table(yourFile, stringsAsFactors=FALSE, sep="|",
colClasses=c("NULL", "character", "character", "character"),
col.names=c("Junk","Date","Time","Type"))
dat <- with(dat, data.frame(DateTime=as.POSIXct(paste(Date, Time),
format="%m/%d/%Y %H:%M:%S"), Type=Type, stringsAsFactors=FALSE))
head(dat)
DateTime           Type
1 2016-06-16 03:44:16       Step 001
2 2016-06-16 03:44:16 Initialization
3 2016-06-16 03:44:16        Filters
4 2016-06-16 03:45:03    Split Items
5 2016-06-16 03:46:20           Sort
6 2016-06-16 03:46:43          Check
split(dat, cumsum(grepl("^Step ", dat$Type)))
$`1`
              DateTime                                        Type
1  2016-06-16 03:44:16                                    Step 001
2  2016-06-16 03:44:16                              Initialization
...
13 2016-06-16 04:06:33 BOP processing for 7,960 items has finished

$`2`
              DateTime                                        Type
14 2016-06-16 04:06:34                                    Step 002
15 2016-06-16 04:06:35                              Initialization
...

Bill Dunlap
TIBCO Software
wdunlap tibco.com

On Thu, Jun 16, 2016 at 8:42 PM, Satish Vadlamani <

Hello,
I have multiple text files with the format shown below (see the two files
that I pasted below). Each file is a log of multiple steps that the system
has processed and for each step, it has shown the start time of the process
step. For example, in the data below, the filter started at
|06/16/2016|03:44:16

How to read this data so that Step 001 is one data frame, Step 002 is
another, and so on. After I do this, I will then compare the Step 001 times
with and without parallel process.

For example, the files pasted below "no_parallel_process_SLS_4.txt" and
"parallel_process_SLS_4.txt" will make it clear what I am trying to do. I
want to compare the parallel process times taken for each step with the non
parallel process times.

If there are better ways of performing this task that what I am thinking,
could you let me know? Thanks in advance.

Satish Vadlamani

parallel_process_file.txt
|06/16/2016|03:44:16|Step 001
|06/16/2016|03:44:16|Initialization
|06/16/2016|03:44:16|Filters
|06/16/2016|03:45:03|Split Items
|06/16/2016|03:46:20|Sort
|06/16/2016|03:46:43|Check
|06/16/2016|04:01:13|Save
|06/16/2016|04:04:35|Update preparation
|06/16/2016|04:04:36|Update comparison
|06/16/2016|04:04:38|Update
|06/16/2016|04:04:38|Update
|06/16/2016|04:06:01|Close
|06/16/2016|04:06:33|BOP processing for 7,960 items has finished
|06/16/2016|04:06:34|Step 002
|06/16/2016|04:06:35|Initialization
|06/16/2016|04:06:35|Filters
|06/16/2016|04:07:14|Split Items
|06/16/2016|04:08:57|Sort
|06/16/2016|04:09:06|Check
|06/16/2016|04:26:36|Save
|06/16/2016|04:39:29|Update preparation
|06/16/2016|04:39:31|Update comparison
|06/16/2016|04:39:43|Update
|06/16/2016|04:39:45|Update
|06/16/2016|04:44:28|Close
|06/16/2016|04:45:26|BOP processing for 8,420 items has finished
|06/16/2016|04:45:27|Step 003
|06/16/2016|04:45:27|Initialization
|06/16/2016|04:45:27|Filters
|06/16/2016|04:48:50|Split Items
|06/16/2016|04:55:15|Sort
|06/16/2016|04:55:40|Check
|06/16/2016|05:13:35|Save
|06/16/2016|05:17:34|Update preparation
|06/16/2016|05:17:34|Update comparison
|06/16/2016|05:17:36|Update
|06/16/2016|05:17:36|Update
|06/16/2016|05:19:29|Close
|06/16/2016|05:19:49|BOP processing for 8,876 items has finished
|06/16/2016|05:19:50|Step 004
|06/16/2016|05:19:50|Initialization
|06/16/2016|05:19:50|Filters
|06/16/2016|05:20:43|Split Items
|06/16/2016|05:22:14|Sort
|06/16/2016|05:22:29|Check
|06/16/2016|05:37:27|Save
|06/16/2016|05:38:43|Update preparation
|06/16/2016|05:38:44|Update comparison
|06/16/2016|05:38:45|Update
|06/16/2016|05:38:45|Update
|06/16/2016|05:39:09|Close
|06/16/2016|05:39:19|BOP processing for 5,391 items has finished
|06/16/2016|05:39:20|Step 005
|06/16/2016|05:39:20|Initialization
|06/16/2016|05:39:20|Filters
|06/16/2016|05:39:57|Split Items
|06/16/2016|05:40:21|Sort
|06/16/2016|05:40:24|Check
|06/16/2016|05:46:01|Save
|06/16/2016|05:46:54|Update preparation
|06/16/2016|05:46:54|Update comparison
|06/16/2016|05:46:54|Update
|06/16/2016|05:46:55|Update
|06/16/2016|05:47:24|Close
|06/16/2016|05:47:31|BOP processing for 3,016 items has finished
|06/16/2016|05:47:32|Step 006
|06/16/2016|05:47:32|Initialization
|06/16/2016|05:47:32|Filters
|06/16/2016|05:47:32|Update preparation
|06/16/2016|05:47:32|Update comparison
|06/16/2016|05:47:32|Update
|06/16/2016|05:47:32|Close
|06/16/2016|05:47:33|BOP processing for 0 items has finished
|06/16/2016|05:47:33|Step 007
|06/16/2016|05:47:33|Initialization
|06/16/2016|05:47:33|Filters
|06/16/2016|05:47:34|Split Items
|06/16/2016|05:47:34|Sort
|06/16/2016|05:47:34|Check
|06/16/2016|05:47:37|Save
|06/16/2016|05:47:37|Update preparation
|06/16/2016|05:47:37|Update comparison
|06/16/2016|05:47:37|Update
|06/16/2016|05:47:37|Update
|06/16/2016|05:47:37|Close
|06/16/2016|05:47:37|BOP processing for 9 items has finished
|06/16/2016|05:47:37|Step 008
|06/16/2016|05:47:37|Initialization
|06/16/2016|05:47:37|Filters
|06/16/2016|05:47:38|Update preparation
|06/16/2016|05:47:38|Update comparison
|06/16/2016|05:47:38|Update
|06/16/2016|05:47:38|Close
|06/16/2016|05:47:38|BOP processing for 0 items has finished

no_parallel_process_file.txt
|06/15/2016|22:52:46|Step 001
|06/15/2016|22:52:46|Initialization

|06/15/2016|22:52:46|Filters

|06/15/2016|22:54:21|Split Items

|06/15/2016|22:55:10|Sort

|06/15/2016|22:55:15|Check

|06/15/2016|23:04:43|Save

|06/15/2016|23:06:38|Update preparation

|06/15/2016|23:06:38|Update comparison

|06/15/2016|23:06:39|Update

|06/15/2016|23:06:39|Update

|06/15/2016|23:12:04|Close

|06/15/2016|23:13:16|BOP processing for 7,942 items has finished

|06/15/2016|23:13:17|Step 002
|06/15/2016|23:13:17|Initialization

|06/15/2016|23:13:17|Filters

|06/15/2016|23:16:27|Split Items

|06/15/2016|23:20:18|Sort

|06/15/2016|23:20:34|Check

|06/16/2016|00:08:08|Save

|06/16/2016|00:26:19|Update preparation

|06/16/2016|00:26:20|Update comparison

|06/16/2016|00:26:30|Update

|06/16/2016|00:26:31|Update

|06/16/2016|00:42:31|Close

|06/16/2016|00:45:09|BOP processing for 8,400 items has finished

|06/16/2016|00:45:11|Step 003
|06/16/2016|00:45:12|Initialization

|06/16/2016|00:45:12|Filters

|06/16/2016|00:53:01|Split Items

|06/16/2016|01:01:44|Sort

|06/16/2016|01:02:55|Check

|06/16/2016|01:41:40|Save

|06/16/2016|01:44:37|Update preparation

|06/16/2016|01:44:37|Update comparison

|06/16/2016|01:44:39|Update

|06/16/2016|01:44:39|Update

|06/16/2016|01:47:37|Close

|06/16/2016|01:48:07|BOP processing for 8,867 items has finished

|06/16/2016|01:48:08|Step 004
|06/16/2016|01:48:08|Initialization

|06/16/2016|01:48:08|Filters

|06/16/2016|01:49:51|Split Items

|06/16/2016|01:50:35|Sort

|06/16/2016|01:50:39|Check

|06/16/2016|01:59:12|Save

|06/16/2016|02:00:47|Update preparation

|06/16/2016|02:00:47|Update comparison

|06/16/2016|02:00:48|Update

|06/16/2016|02:00:48|Update

|06/16/2016|02:02:40|Close

|06/16/2016|02:02:55|BOP processing for 5,383 items has finished

|06/16/2016|02:02:56|Step 005
|06/16/2016|02:02:56|Initialization

|06/16/2016|02:02:56|Filters

|06/16/2016|02:03:47|Split Items

|06/16/2016|02:04:19|Sort

|06/16/2016|02:04:21|Check

|06/16/2016|02:08:08|Save

|06/16/2016|02:09:22|Update preparation

|06/16/2016|02:09:22|Update comparison

|06/16/2016|02:09:22|Update

|06/16/2016|02:09:22|Update

|06/16/2016|02:11:03|Close

|06/16/2016|02:11:14|BOP processing for 3,016 items has finished

|06/16/2016|02:11:14|Step 006
|06/16/2016|02:11:14|Initialization

|06/16/2016|02:11:14|Filters

|06/16/2016|02:11:15|Update preparation

|06/16/2016|02:11:15|Update comparison

|06/16/2016|02:11:15|Update

|06/16/2016|02:11:15|Close

|06/16/2016|02:11:15|BOP processing for 0 items has finished

|06/16/2016|02:11:15|Step 007
|06/16/2016|02:11:15|Initialization

|06/16/2016|02:11:15|Filters

|06/16/2016|02:11:17|Split Items

|06/16/2016|02:11:17|Sort

|06/16/2016|02:11:17|Check

|06/16/2016|02:11:20|Save

|06/16/2016|02:11:20|Update preparation

|06/16/2016|02:11:20|Update comparison

|06/16/2016|02:11:20|Update

|06/16/2016|02:11:20|Update

|06/16/2016|02:11:20|Close

|06/16/2016|02:11:20|BOP processing for 9 items has finished

|06/16/2016|02:11:20|Step 008
|06/16/2016|02:11:20|Initialization

|06/16/2016|02:11:21|Filters

|06/16/2016|02:11:21|Update preparation

|06/16/2016|02:11:21|Update comparison

|06/16/2016|02:11:21|Update

|06/16/2016|02:11:21|Close

|06/16/2016|02:11:21|BOP processing for 0 items has finished

--

Satish Vadlamani

        [[alternative HTML version deleted]]

______________________________________________
R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

what is the best way to process the following data?

Thread (2 messages)