Skip to content

how to smooth timeseries without the lagging?

7 messages · Michael, Josuah Rechtsteiner, Sean Carmody +3 more

Original
Michael
# 
Hi all,

If I use a moving average, it will smooth the choppy time series, but
it will lead to lagging...

How do I smooth timeseries without the lagging effect?

Thanks!
Josuah Rechtsteiner
# 
maybe kalman filter is what you are looking for.



Am 25.07.2009 um 05:44 schrieb Michael:
michael.sankowski at gmail.com
# 
Also try MESA. I was suppposed to do this for josh but have been on other projects. 
Sent via BlackBerry from T-Mobile

-----Original Message-----
From: Sean Carmody <seancarmody at gmail.com>

Date: Sat, 25 Jul 2009 19:56:59 
To: Josuah Rechtsteiner<rechtsteiner at bgki.net>
Cc: <r-sig-finance at stat.math.ethz.ch>
Subject: Re: [R-SIG-Finance] how to smooth timeseries without the lagging?


Or a Henderson filter. If you have a filter that looks forward as well as
back, it will not have the lag effect. Then it'll need special treatment at
the end of the series.

(Sorry for the double email Josuah).

Sean.

On Sat, Jul 25, 2009 at 7:12 PM, Josuah Rechtsteiner
<rechtsteiner at bgki.net>wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Eric Zivot
    

    

      
      #
    

  


  

      
"smoothing" is not a well defined term. In the Kalman filter literature, smoothing refers to state extraction at time t using all available information (before and after t) and typically results in two-sided moving average type algorithms. The whole point of smoothing is to reduce the noise in the data and extract a "smooth" component. This invariably involves some kind of averaging of the observation both before and after the observation. So it doesn't make sense to have a "smoother" that does not involve some lagging effect. Otherwise, you couldn't smooth. See


Harvey, A. C. and Koopman, S. J. (2000). Signal extraction and the formulation of unobserved components
models. Econometrics Journal, Vol. 3, pp. 84-107.

for a nice discussion.

****************************************************************
*  Eric Zivot                  			               *
*  Professor and Gary Waterman Distinguished Scholar           *
*  Department of Economics                                     *
*  Adjunct Professor of Finance                                *
*  Adjunct Professor of Statistics
*  Box 353330                  email:  ezivot at u.washington.edu *
*  University of Washington    phone:  206-543-6715            *
*  Seattle, WA 98195-3330                                      *                                                           *
*  www:  http://faculty.washington.edu/ezivot                  *
****************************************************************

        
On Sat, 25 Jul 2009 michael.sankowski at gmail.com wrote:

        

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    
1 day later

    

      
      

        


  

        

      michael.sankowski at gmail.com
    

    

      
      #
    

  


  

      
What is the application?  Optimal smoothing is related to extracting high signal to noise. If your application is not dependent on constant or perfectly stable parameter then you can look at optimizing signal to noise rather than choosing unchanging smoothing. 

I typically use MESA to extract cycle information when dealing with a changing cyclical component.   I can post some code in a bit. 
Sent via BlackBerry from T-Mobile

-----Original Message-----
From: Eric Zivot <ezivot at u.washington.edu>

Date: Sat, 25 Jul 2009 08:31:04 
To: <michael.sankowski at gmail.com>
Cc: Sean Carmody<seancarmody at gmail.com>; <r-sig-finance-bounces at stat.math.ethz.ch>; Josuah Rechtsteiner<rechtsteiner at bgki.net>; r-sig-finance at stat.math.ethz.ch<r-sig-finance at stat.math.ethz.ch>
Subject: Re: [R-SIG-Finance] how to smooth timeseries without the lagging?


"smoothing" is not a well defined term. In the Kalman filter literature, smoothing refers to state extraction at time t using all available information (before and after t) and typically results in two-sided moving average type algorithms. The whole point of smoothing is to reduce the noise in the data and extract a "smooth" component. This invariably involves some kind of averaging of the observation both before and after the observation. So it doesn't make sense to have a "smoother" that does not involve some lagging effect. Otherwise, you couldn't smooth. See


Harvey, A. C. and Koopman, S. J. (2000). Signal extraction and the formulation of unobserved components
models. Econometrics Journal, Vol. 3, pp. 84-107.

for a nice discussion.

****************************************************************
*  Eric Zivot                  			               *
*  Professor and Gary Waterman Distinguished Scholar           *
*  Department of Economics                                     *
*  Adjunct Professor of Finance                                *
*  Adjunct Professor of Statistics
*  Box 353330                  email:  ezivot at u.washington.edu *
*  University of Washington    phone:  206-543-6715            *
*  Seattle, WA 98195-3330                                      *                                                           *
*  www:  http://faculty.washington.edu/ezivot                  *
****************************************************************

        
On Sat, 25 Jul 2009 michael.sankowski at gmail.com wrote:

        

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Gerard M. Keogh
    

    

      
      #
    

  


  

      
Sorry guys!

This is not a well posed question.
If you intend to "smooth" a time series all methods mentioned (and that I'm
aware of anyway) use some kind of TIME averaging - the best simple method
is probably kernel smoothing (nadaraya-watson smoother). Lowess (available
in R) locally weighted regression is another alternative.
One way or another you'll be doing some kind of "time averaging" - afterall
the smoothed solution is generally a projection of x_t onto the subspace
spanned by the history (e.g. x_(t-1) etc).

As I see it the only way to avoid "time averaging" is to use regression of
y_t on x1, x2, .. - this once again is the projection of y_t onto the
subspace spanned by x1, x2 etc. - but this is still an "average"!
Here of course you need independent predictors and once again these may
have time effects that are correlated with y_t - e.g. y_t may be correlated
with x_1(t-3), x_1 lagged 3 periods. In this situation you'll need dynamic
regression models.
But, nevertheless using regression is the most straightforward option to
avoid time averaging.

Gerard



                                                                           
             Eric Zivot                                                    
             <ezivot at u.washing                                             
             ton.edu>                                                   To 
             Sent by:                  michael.sankowski at gmail.com         
             r-sig-finance-bou                                          cc 
             nces at stat.math.et         "r-sig-finance at stat.math.ethz.ch"   
             hz.ch                     <r-sig-finance at stat.math.ethz.ch>,  
                                       r-sig-finance-bounces at stat.math.eth 
                                       z.ch                                
             25/07/2009 16:31                                      Subject 
                                       Re: [R-SIG-Finance] how to smooth   
                                       timeseries without the lagging?     
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           




"smoothing" is not a well defined term. In the Kalman filter literature,
smoothing refers to state extraction at time t using all available
information (before and after t) and typically results in two-sided moving
average type algorithms. The whole point of smoothing is to reduce the
noise in the data and extract a "smooth" component. This invariably
involves some kind of averaging of the observation both before and after
the observation. So it doesn't make sense to have a "smoother" that does
not involve some lagging effect. Otherwise, you couldn't smooth. See


Harvey, A. C. and Koopman, S. J. (2000). Signal extraction and the
formulation of unobserved components
models. Econometrics Journal, Vol. 3, pp. 84-107.

for a nice discussion.

****************************************************************
*  Eric Zivot
*
*  Professor and Gary Waterman Distinguished Scholar           *
*  Department of Economics                                     *
*  Adjunct Professor of Finance                                *
*  Adjunct Professor of Statistics
*  Box 353330                  email:  ezivot at u.washington.edu *
*  University of Washington    phone:  206-543-6715            *
*  Seattle, WA 98195-3330                                      *
*
*  www:  http://faculty.washington.edu/ezivot                  *
****************************************************************

        
On Sat, 25 Jul 2009 michael.sankowski at gmail.com wrote:

        

            

      
      
      
    

        
projects.

            

      
      
      
    

        
lagging?

            

      
      
      
    

        
at

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        
_______________________________________________
R-SIG-Finance at stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only.
-- If you want to post, subscribe first.



**********************************************************************************
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.  It is the policy of the Department of Justice, Equality and Law Reform and the Agencies and Offices using its IT services to disallow the sending of offensive material.
Should you consider that the material contained in this message is offensive you should contact the sender immediately and also mailminder[at]justice.ie.

Is le haghaidh an duine n? an eintitis ar a bhfuil s? d?rithe, agus le haghaidh an duine n? an eintitis sin amh?in, a bhearta?tear an fhaisn?is a tarchuireadh agus f?adfaidh s? go bhfuil ?bhar faoi r?n agus/n? faoi phribhl?id inti. Toirmisctear aon athbhreithni?, atarchur n? leathadh a dh?anamh ar an bhfaisn?is seo, aon ?s?id eile a bhaint aisti n? aon ghn?omh a dh?anamh ar a hiontaoibh, ag daoine n? ag eintitis seachas an faighteoir beartaithe. M? fuair t? ? seo tr? dhearmad, t?igh i dteagmh?il leis an seolt?ir, le do thoil, agus scrios an t-?bhar as aon r?omhaire. Is ? beartas na Roinne Dl? agus Cirt, Comhionannais agus Athch?irithe Dl?, agus na nOif?g? agus na nGn?omhaireachta? a ?s?ideann seirbh?s? TF na Roinne, seoladh ?bhair chol?il a dh?chead?.
M?s rud ? go measann t? gur ?bhar col?il at? san ?bhar at? sa teachtaireacht seo is ceart duit dul i dteagmh?il leis an seolt?ir l?ithreach agus le mailminder[ag]justice.ie chomh maith. 
***********************************************************************************