apply and cousins

Thanks Jim and others (and sorry Jim - an early version of this slipped
into your inbox :))

Apologies for not giving some concrete code - I was trying to explain in
words.

What I need to do is to fit a simple linear model to successive sections
of a long matrix.

So far, the best solution I have come up with uses apply twice:

Generate some data in a 100000*3 matrix:

N = 100000
Z = cbind(1:N,cumsum(rnorm(N,1,0.01)),rnorm(N,1.2,0.1)) #

where the first column is an index, the second a monotonic increasing
value representing time and the third just the measurements I want to
process.

Then write a function dVals1:

dVals1 = function(Y,DD,dT){which.min((Y[2] - dT) > DD[,2])))

which will identify the first row where the time is greater than current
time - dT.

So to identify the start of the data (say) 10 units before for each row,
we use apply and prepended this as a column to the array for later use:

ZZ = cbind(apply(Z,1,dVals1,Z,10),Z)

There may be some cases, particularly at the start, where later values are
extracted because the minimum returned by which.min is 1.

I now have start and finish pointers for each position so can proceed to
fit a simple linear model with the following function:

dVals2=function(D2,DD){
  if((D2[2]-D2[1])<10){return(rep(0,2))} # reject short examples
  DX=DD[D2[1]:D2[2],]
  Res=as.vector(lm(DX[,3]~DX[,2])$coefficients)
  return(Res)
}

which returns 2 0's either if there are fewer than 10 values, otherwise it
returns the intercept and slope calculated over the specified range.

Applying this to the whole data by:

t(apply(ZZ,1,dVals2,DD=ZZ))

does the job I think returning the results as an N * 2 matrix.