Variance of multiple non-contiguous time periods?

On 04/11/14 17:02, David Winsemius wrote:

On Nov 4, 2014, at 8:35 AM, CJ Davies wrote:

On 04/11/14 16:13, PIKAL Petr wrote:

Hi

-----Original Message-----
From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-
project.org] On Behalf Of CJ Davies
Sent: Tuesday, November 04, 2014 2:50 PM
To: Jim Lemon; r-help at r-project.org
Subject: Re: [R] Variance of multiple non-contiguous time periods?

On 04/11/14 09:11, Jim Lemon wrote:

On Mon, 3 Nov 2014 12:45:03 PM CJ Davies wrote:

...
On 30/10/14 21:33, Jim Lemon wrote:
If I understand, you mean to calculate deviations for each

individual

'chunk' of each transition & then aggregate the results? This is

what

I'd been thinking about, but is there a sensible manner within R to
achieve this, or is it something for which it would be easier to
preprocess the data in an external tool? Is there some way to subset

the

data such that I can work over just contiguous 'chunks'?

Exactly. If there is some combination of existing variables that can
be combined to make a set of unique values for each "chunk", you can
calculate the deviations within each "chunk", then average the

squared

deviations for each type of "chunk", weighting by the duration of the
"chunks" so that you don't bias the pooled variance toward the longer
"chunks".

Jim

I am stumped for a way of automating this process though. Each line of
log data looks like this;

2406  55.4    (-11.2, 1.0, -0.9)      (-4.1, 1.0, 0.0)        7.077912
      0.9203392       (0.0,
0.7, -0.1, 0.7)       8.129684        89.41537        -8.212769       (0.0, 0.7, -0.1,
0.7)
8.129684      89.41537        351.7872        1       0       0       False   0.15    3
      37.76761        True    False   0
transition 1

First you need to import it to R which could be tricky based on above line.
Some values will probably need to process through regular expression.

If I understand correctly number after transition is a signal which estimets continuous chunks. If it is true then

?rle is a function which can estimate length of chunks.

Cheers
Petr

Where the last variable defines which transition is currently active.
However to separate these data into 'chunks' would involve making a
comparison between each line of data & the preceding line of data to
determine whether it is part of the same contiguous 'chunk'. Is this
something that would be better achieved using external preprocessing
written in a language I am more familiar with, as I haven't the
foggiest how I would approach this within R?

Regards,
CJ Davies

snipped

Importing into R wasn't an issue; some of the fields contain spaces & symbols, but all the fields are tab separated so I can simply use;

foo <- read.csv("bar",header=T,sep="\t")

I've just written a hacky bit of Java that gives me the lines of each 'chunk' as a separate list & I think I'll then calculate these particular values using Java's Math class rather than trying to come up with a sensible way to import these 'chunks' back into R. When it comes to string/list manipulation like this I think my knowledge in Java & lack of knowledge in R makes the former the better option!

If you had offered the output of dput(head(foo, 20) ) and explained what defined a "chunk-defining transition", it would have been fairly easy to show you how to use cumsum in an ave() call to construct a grouping variable.

Regards,
CJ Davies

David Winsemius
Alameda, CA, USA

Here is an example 100 lines of the input --> http://paste2.org/2LZVGP5K

The final value on each line, under the header "environment", is always one of ["real", "transition 1", "transition 2", "transition 3", "transition 4"]. A 'chunk-defining transition' is when this value changes.

If there is a way to do this in R in a more elegant fashion than my hacky Java, then I would be glad to learn.

download.file("http://paste2.org/2LZVGP5K", "bar.txt")

foo <- read.csv("bar.txt",header=T,sep="\t")
str(foo)

Regards,
CJ Davies