Prof. Sornette has spent years forcasting bubble burst with "log-periodic power law". The latest paper gives "a self-consistent model for explosive financial bubbles, which combines a mean-reverting volatility process and a stochastic conditional return which reflects nonlinear positive feedbacks and continuous updates of the investors' beliefs and sentiments." And his latest predicting is the burst of Chinese equity bubble at the end of July. http://arxiv.org/abs/0907.1827 While waiting to see the result, I wonder whether it is possible to replicate the forcast with R. The model is in the page 10 of the "A Consistent Model of `Explosive' Financial Bubbles With Mean-Reversing Residuals", http://arxiv.org/abs/0905.0128 . The output chart is in the page 3 of "The Chinese Equity Bubble: Ready to Burst", http://arxiv.org/abs/0907.1827 . I guess the authors of the latter paper use the same model as described in the first paper. Because statistics is still challenging for me though I could use R for basic data manipulations, I wonder which package or function would be necessary to implement the model in the paper. The model seems more complicated than the models in the R tutorials for me. By the way, the author of the paper used Python and the codes are private. Any suggestion would be highly appreciated. Wind
LPPL model for bubble burst forcasting
10 messages · Brian G. Peterson, Wind, Gabor Grothendieck
So first, using your real name and ideally your professional identity, ask for the python code. Better yet, get an academic buddy to do it. Usually getting access to the code isn't too tough. Mention things like "repeatable research" and "collaboration" in your email. Two of the authors publish their email addresses in one of the papers you reference, so contacting them should be easy. Next port the python code to R. If you can't do that, then replicate the model in R "from scratch". A trivial scan of the paper in question lends several techniques that are well covered in R: AR, GARCH, power laws, linear regression, stochastic discount factor, Ornstein-Uhlenbeck, etc. There are volumes of information available on these topics from within R, in numerous books, and in the archives of this mailing list and r-help. You're going to have to do your replication in pieces, probably starting with their implementation of the log periodic power law (LPPL), for which I do not believe there is an existing direct analogue in R though all the component parts necessary to replicate it should be readily available. As you work on each step of the replication, share your code with this list and the problems you are having with a particular step. Ask specific, directed questions with code to back them up. Someone will likely help you solve the specific problem. In R generally, it is not necessary that you be able to *do* the math (think pencil and paper), but if you plan to replicate published work, it will be necessary to *understand* at least some of how the math works, and to be able to pick out the names of techniques that you can search for an utilize. Basically, I'm recommending that you (specifically) and others (more generally) should share the process of replicating a technique like this, as well as the final product, to give all the rest of us who are likely to be helping "you" get all this done. quid pro quo. Cheers, - Brian
Brian G. Peterson http://braverock.com/brian/ Ph: 773-459-4973 IM: bgpbraverock Wind wrote: > Prof. Sornette has spent years forcasting bubble burst with > "log-periodic power law". The latest paper gives "a > self-consistent model for explosive financial bubbles, which combines > a mean-reverting volatility process and a stochastic conditional > return which reflects nonlinear positive feedbacks and continuous > updates of the investors' beliefs and sentiments." > > And his latest predicting is the burst of Chinese equity bubble at > the end of July. http://arxiv.org/abs/0907.1827 > > While waiting to see the result, I wonder whether it is possible to > replicate the forcast with R. The model is in the page 10 of the "A > Consistent Model of `Explosive' Financial Bubbles With Mean-Reversing > Residuals", http://arxiv.org/abs/0905.0128 . The output chart is > in the page 3 of "The Chinese Equity Bubble: Ready to Burst", > http://arxiv.org/abs/0907.1827 . I guess the authors of the latter > paper use the same model as described in the first paper. > > Because statistics is still challenging for me though I could use R > for basic data manipulations, I wonder which package or function > would be necessary to implement the model in the paper. The model > seems more complicated than the models in the R tutorials for me. > By the way, the author of the paper used Python and the codes are > private. > > Any suggestion would be highly appreciated. >
Thanks Brian. You're always warm-hearted and very professional. I will try my best following your detailed instructions. It would be a great improvement for myself if I could work out the final solution with the help of the list. Since I am just an independent individual investor with only master degree in finance, I guess it would take some time. Thanks again. I will post the progress to the list if there is some progress :) wind
On Thu, Jul 16, 2009 at 9:25 PM, Brian G. Peterson<brian at braverock.com> wrote:
So first, using your real name and ideally your professional identity, ask for the python code. ?Better yet, get an academic buddy to do it. Usually getting access to the code isn't too tough. ?Mention things like "repeatable research" and "collaboration" in your email. ?Two of the authors publish their email addresses in one of the papers you reference, so contacting them should be easy. Next port the python code to R. If you can't do that, then replicate the model in R "from scratch". ?A trivial scan of the paper in question lends several techniques that are well covered in R: AR, GARCH, power laws, linear regression, stochastic discount factor, Ornstein-Uhlenbeck, etc. There are volumes of information available on these topics from within R, in numerous books, and in the archives of this mailing list and r-help. You're going to have to do your replication in pieces, probably starting with their implementation of the log periodic power law (LPPL), for which I do not believe there is an existing direct analogue in R though all the component parts necessary to replicate it should be readily available. As you work on each step of the replication, share your code with this list and the problems you are having with a particular step. ?Ask specific, directed questions with code to back them up. ?Someone will likely help you solve the specific problem. In R generally, it is not necessary that you be able to *do* the math (think pencil and paper), but if you plan to replicate published work, it will be necessary to *understand* at least some of how the math works, and to be able to pick out the names of techniques that you can search for an utilize. Basically, I'm recommending that you (specifically) and others (more generally) should share the process of replicating a technique like this, as well as the final product, to give all the rest of us who are likely to be helping "you" get all this done. quid pro quo. Cheers, ?- Brian -- Brian G. Peterson http://braverock.com/brian/ Ph: 773-459-4973 IM: bgpbraverock Wind wrote:
Prof. Sornette has spent years forcasting bubble burst with "log-periodic power law". ? ?The latest paper ?gives "a self-consistent model for explosive financial bubbles, which combines a mean-reverting volatility process and a stochastic conditional return which reflects nonlinear positive feedbacks and continuous updates of the investors' beliefs and sentiments." And his ?latest ?predicting is the burst of Chinese equity bubble at the end of July. ? ? http://arxiv.org/abs/0907.1827 While waiting to see the result, I wonder whether it is possible to replicate the forcast with R. ?The model is in the page 10 of the "A Consistent Model of `Explosive' Financial Bubbles With Mean-Reversing Residuals", ?http://arxiv.org/abs/0905.0128 ?. ? The output chart is in the page 3 of "The Chinese Equity Bubble: Ready to Burst", http://arxiv.org/abs/0907.1827 . ? I guess the authors of the latter paper use the same model as described in the first paper. Because statistics is still challenging for me though I could use R for ?basic data manipulations, ?I wonder which package or function would be necessary to implement the model in the paper. ?The model seems more complicated than the models in the R tutorials for me. By the way, the author of the paper used Python and the codes are private. Any suggestion would be highly appreciated.
There is a chart of Heng Seng index in page 24 of Prof. Sornette's paper: ## Financial Bubbles, Real Estate bubbles, Derivative Bubbles, and the Financial and Economic Crisis ## http://arxiv.org/abs/0905.0220 The picture has also been attached as .hong kong.jpg The y axis of the chart is log-axis. And there is a straight line in the chart. "This is indeed the long-term behavior of this market, as shown by the best linear fit represented by the solid straight line, corresponding to an average constant growth rate of 13.8% per year." The following codes could not plot the same straight line. I wonder how could plot the straight best fit line in the log plot. ## Financial Bubbles, Real Estate bubbles, Derivative Bubbles, and the Financial and Economic Crisis ## http://arxiv.org/abs/0905.0220 ## chart in Page 24 hsi<-read.csv("http://32xiang.appspot.com/static/hsi-1970.csv",header=TRUE,stringsAsFactors=FALSE) pr<-hsi$close plot(pr,type="l",log="y") grid() ti<-index(pr) ti2<-index(pr)^2 lines(lm(pr~ti+ti2)$fit,col="red") lines(lm(pr~ti)$fit,col="blue") lines(lm(pr~ti2)$fit,col="pink") Thanks in advance. wind -------------- next part -------------- A non-text attachment was scrubbed... Name: hong kong .jpg Type: image/jpeg Size: 48355 bytes Desc: not available URL: <https://stat.ethz.ch/pipermail/r-sig-finance/attachments/20090717/bbd27fcd/attachment.jpg>
Sorry that I forgot including the first line of the code:
library(quantmod)
So the replicable codes as following:
library(quantmod)
hsi<-read.csv("http://32xiang.appspot.com/static/hsi-1970.csv",header=TRUE,stringsAsFactors=FALSE)
pr<-hsi$close
plot(pr,type="l",log="y")
grid()
ti<-index(pr)
ti2<-index(pr)^2
lines(lm(pr~ti+ti2)$fit,col="red")
lines(lm(pr~ti)$fit,col="blue")
lines(lm(pr~ti2)$fit,col="pink")
On Fri, Jul 17, 2009 at 3:12 PM, Wind<windspeedo99 at gmail.com> wrote:
There is a chart of Heng Seng index ?in page 24 of Prof. Sornette's paper: ## Financial Bubbles, Real Estate bubbles, Derivative Bubbles, and the Financial and Economic Crisis ## http://arxiv.org/abs/0905.0220 The picture has also been attached as .hong kong.jpg The y axis of the chart is log-axis. ?And there is a straight line in the chart. ?"This is indeed the long-term behavior of this market, as shown by the best linear fit represented by the solid straight line, corresponding to an average constant growth rate of 13.8% per year." The following codes could not plot the same straight line. ?I wonder how could plot the straight best fit line ?in the log plot. ## Financial Bubbles, Real Estate bubbles, Derivative Bubbles, and the Financial and Economic Crisis ## http://arxiv.org/abs/0905.0220 ## chart in Page 24 hsi<-read.csv("http://32xiang.appspot.com/static/hsi-1970.csv",header=TRUE,stringsAsFactors=FALSE) pr<-hsi$close plot(pr,type="l",log="y") grid() ti<-index(pr) ti2<-index(pr)^2 lines(lm(pr~ti+ti2)$fit,col="red") lines(lm(pr~ti)$fit,col="blue") lines(lm(pr~ti2)$fit,col="pink") Thanks in advance. wind
Try this: lines(exp(fitted(lm(log(pr) ~ ti))), col = "purple")
On Fri, Jul 17, 2009 at 4:11 AM, Wind<windspeedo99 at gmail.com> wrote:
Sorry that I forgot including the first line of the code:
library(quantmod)
So the replicable codes as following:
library(quantmod)
hsi<-read.csv("http://32xiang.appspot.com/static/hsi-1970.csv",header=TRUE,stringsAsFactors=FALSE)
pr<-hsi$close
plot(pr,type="l",log="y")
grid()
ti<-index(pr)
ti2<-index(pr)^2
lines(lm(pr~ti+ti2)$fit,col="red")
lines(lm(pr~ti)$fit,col="blue")
lines(lm(pr~ti2)$fit,col="pink")
On Fri, Jul 17, 2009 at 3:12 PM, Wind<windspeedo99 at gmail.com> wrote:
There is a chart of Heng Seng index ?in page 24 of Prof. Sornette's paper: ## Financial Bubbles, Real Estate bubbles, Derivative Bubbles, and the Financial and Economic Crisis ## http://arxiv.org/abs/0905.0220 The picture has also been attached as .hong kong.jpg The y axis of the chart is log-axis. ?And there is a straight line in the chart. ?"This is indeed the long-term behavior of this market, as shown by the best linear fit represented by the solid straight line, corresponding to an average constant growth rate of 13.8% per year." The following codes could not plot the same straight line. ?I wonder how could plot the straight best fit line ?in the log plot. ## Financial Bubbles, Real Estate bubbles, Derivative Bubbles, and the Financial and Economic Crisis ## http://arxiv.org/abs/0905.0220 ## chart in Page 24 hsi<-read.csv("http://32xiang.appspot.com/static/hsi-1970.csv",header=TRUE,stringsAsFactors=FALSE) pr<-hsi$close plot(pr,type="l",log="y") grid() ti<-index(pr) ti2<-index(pr)^2 lines(lm(pr~ti+ti2)$fit,col="red") lines(lm(pr~ti)$fit,col="blue") lines(lm(pr~ti2)$fit,col="pink") Thanks in advance. wind
_______________________________________________ R-SIG-Finance at stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. -- If you want to post, subscribe first.
It works exactly as the chart. Thanks Gabor On Fri, Jul 17, 2009 at 6:58 PM, Gabor
Grothendieck<ggrothendieck at gmail.com> wrote:
Try this: lines(exp(fitted(lm(log(pr) ~ ti))), col = "purple") On Fri, Jul 17, 2009 at 4:11 AM, Wind<windspeedo99 at gmail.com> wrote:
Sorry that I forgot including the first line of the code:
library(quantmod)
So the replicable codes as following:
library(quantmod)
hsi<-read.csv("http://32xiang.appspot.com/static/hsi-1970.csv",header=TRUE,stringsAsFactors=FALSE)
pr<-hsi$close
plot(pr,type="l",log="y")
grid()
ti<-index(pr)
ti2<-index(pr)^2
lines(lm(pr~ti+ti2)$fit,col="red")
lines(lm(pr~ti)$fit,col="blue")
lines(lm(pr~ti2)$fit,col="pink")
On Fri, Jul 17, 2009 at 3:12 PM, Wind<windspeedo99 at gmail.com> wrote:
There is a chart of Heng Seng index ?in page 24 of Prof. Sornette's paper: ## Financial Bubbles, Real Estate bubbles, Derivative Bubbles, and the Financial and Economic Crisis ## http://arxiv.org/abs/0905.0220 The picture has also been attached as .hong kong.jpg The y axis of the chart is log-axis. ?And there is a straight line in the chart. ?"This is indeed the long-term behavior of this market, as shown by the best linear fit represented by the solid straight line, corresponding to an average constant growth rate of 13.8% per year." The following codes could not plot the same straight line. ?I wonder how could plot the straight best fit line ?in the log plot. ## Financial Bubbles, Real Estate bubbles, Derivative Bubbles, and the Financial and Economic Crisis ## http://arxiv.org/abs/0905.0220 ## chart in Page 24 hsi<-read.csv("http://32xiang.appspot.com/static/hsi-1970.csv",header=TRUE,stringsAsFactors=FALSE) pr<-hsi$close plot(pr,type="l",log="y") grid() ti<-index(pr) ti2<-index(pr)^2 lines(lm(pr~ti+ti2)$fit,col="red") lines(lm(pr~ti)$fit,col="blue") lines(lm(pr~ti2)$fit,col="pink") Thanks in advance. wind
_______________________________________________ R-SIG-Finance at stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. -- If you want to post, subscribe first.
2 days later
Some progress. The LPPL curve could be plotted with the following codes. The problem now is how to get the best fit parameters. Some researchers use python or matlab for LPPL calibrating. It seems that some of them prefer tabu search for optimums locating. It seems that there's still no general function for tabu search in R. At the end of codes, I give the possible parameter combinations to be searched in, maybe there are other functions for optimum searching in R. Any suggestion would be appreciated. ## Financial Bubbles, Real Estate bubbles, Derivative Bubbles, and the Financial and Economic Crisis ## http://arxiv.org/abs/0905.0220 ## Fig. 23 S&P500 index (in logarithmic scale)in Page 39 library(quantmod) LPPL1<-function(p,dtc=20,alpha=0.35,omega=0.1,phi=1) { #function in page 26 of http://arxiv.org/abs/0905.0220 #the basic form of LPPL dtc=abs(floor(dtc)) tc<-length(p)+dtc dt<-abs(tc-(1:length(p))) x1<-dt^alpha x2<-(dt^alpha)*cos(omega*log(dt)+phi) f<-lm(log(p) ~ x1+x2) f$para<-list(recno=dim(f$model)[1],dtc=dtc,alpha=alpha,omega=omega,phi=phi,sigma=summary(f)$sigma) return(f) } opt.lppl<-function(x) { #derived function for optim return(LPPL1(p,dtc=x[1],alpha=x[2],omega=x[3],phi=x[4])$para$sigma) } LPPL1.x<-function(lpplf,pt=100) { #x axis for predicting dt<-abs((lpplf$recno+lpplf$dtc)-(1:(lpplf$recno+lpplf$dtc+pt))) dt[dt==0]<-0.5 x1<-dt^lpplf$alpha x2<-(dt^lpplf$alpha)*cos(lpplf$omega*log(dt)+lpplf$phi) return(list(x1=x1,x2=x2)) } #get the SP500 index pr<-getSymbols("^GSPC",auto.assign=FALSE,from="2003-10-1",to="2007-05-16")[,4] p<-as.numeric(pr) plot(p,type="l",log="y",xlim=c(0,length(p)+100),ylim=c(min(p),max(p)*1.2)) abline(v=length(p),col="green") #something like the Fig. 23 in Page 39 of http://arxiv.org/abs/0905.0220 #but obviously the result is not calibrated well #using optim like this can not calibrate the LPPL model opts<-sapply(seq(1,50,10),function(x){ o<-optim(c(x,0.6,20,1),opt.lppl) f3<-LPPL1(p,dtc=o$par[1],alpha=o$par[2],omega=o$par[3],phi=o$par[4]) xp<-LPPL1.x(f3$para,200) f3p<-predict(f3,data.frame(x1=xp$x1,x2=xp$x2)) lines(exp(f3p),col="blue") lines(exp(fitted(f3)),col="red") f3$para }) ##crash point after dtc days dtc<-seq(1,100,1) ##appropraite range of the parametes of LPPL ##according to Dr. W.X. Zhou's new book which is in Chinese ##the increments of the sequences are added according to my own judement alpha<-seq(0.01,1.2,0.1) omega<-seq(0,40,1) phi<-seq(0,7,0.1) ##millions possible combinations #complete test would be difficult para<-expand.grid(dtc=dtc,alpha=alpha,omega=omega,phi=phi) dim(para) system.time(sigs<-apply(para[1:100,],1,function(x){LPPL1(p,dtc=x[1],alpha=x[2],omega=x[3],phi=x[4])$para$sigma})) ##methods for minimum sigma searching within the parameter combinations ##not implemented yet wind
On Thu, Jul 16, 2009 at 9:25 PM, Brian G. Peterson<brian at braverock.com> wrote:
So first, using your real name and ideally your professional identity, ask for the python code. ?Better yet, get an academic buddy to do it. Usually getting access to the code isn't too tough. ?Mention things like "repeatable research" and "collaboration" in your email. ?Two of the authors publish their email addresses in one of the papers you reference, so contacting them should be easy. Next port the python code to R. If you can't do that, then replicate the model in R "from scratch". ?A trivial scan of the paper in question lends several techniques that are well covered in R: AR, GARCH, power laws, linear regression, stochastic discount factor, Ornstein-Uhlenbeck, etc. There are volumes of information available on these topics from within R, in numerous books, and in the archives of this mailing list and r-help. You're going to have to do your replication in pieces, probably starting with their implementation of the log periodic power law (LPPL), for which I do not believe there is an existing direct analogue in R though all the component parts necessary to replicate it should be readily available. As you work on each step of the replication, share your code with this list and the problems you are having with a particular step. ?Ask specific, directed questions with code to back them up. ?Someone will likely help you solve the specific problem. In R generally, it is not necessary that you be able to *do* the math (think pencil and paper), but if you plan to replicate published work, it will be necessary to *understand* at least some of how the math works, and to be able to pick out the names of techniques that you can search for an utilize. Basically, I'm recommending that you (specifically) and others (more generally) should share the process of replicating a technique like this, as well as the final product, to give all the rest of us who are likely to be helping "you" get all this done. quid pro quo. Cheers, ?- Brian -- Brian G. Peterson http://braverock.com/brian/ Ph: 773-459-4973 IM: bgpbraverock Wind wrote:
Prof. Sornette has spent years forcasting bubble burst with "log-periodic power law". ? ?The latest paper ?gives "a self-consistent model for explosive financial bubbles, which combines a mean-reverting volatility process and a stochastic conditional return which reflects nonlinear positive feedbacks and continuous updates of the investors' beliefs and sentiments." And his ?latest ?predicting is the burst of Chinese equity bubble at the end of July. ? ? http://arxiv.org/abs/0907.1827 While waiting to see the result, I wonder whether it is possible to replicate the forcast with R. ?The model is in the page 10 of the "A Consistent Model of `Explosive' Financial Bubbles With Mean-Reversing Residuals", ?http://arxiv.org/abs/0905.0128 ?. ? The output chart is in the page 3 of "The Chinese Equity Bubble: Ready to Burst", http://arxiv.org/abs/0907.1827 . ? I guess the authors of the latter paper use the same model as described in the first paper. Because statistics is still challenging for me though I could use R for ?basic data manipulations, ?I wonder which package or function would be necessary to implement the model in the paper. ?The model seems more complicated than the models in the R tutorials for me. By the way, the author of the paper used Python and the codes are private. Any suggestion would be highly appreciated.
Glad to see you're making progress on this problem. This paper http://www.diegm.uniud.it/satt/papers/DiSc06b.pdf implements a tabu search in R, though they didn't publish their code. You might want to contact them for the implementation and permission to share their tabu search algorithm/code with the R community. They also reference an R package called RACE by this gentleman: http://iridia.ulb.ac.be/~mbiro/ that they use for evaluating the solution. More generally, from my limited understanding, tabu search is an extension and refinement of simulated annealing approaches easily implemented in R. In brief the approach is to take your best 'n' solutions from a random space search, and then search "near" those solutions. Simulated annealing and its close cousins have a lot of benefits in finance, where a single true optima from a closed form problem is not likely to be available. I personally have rarely found 'optim' to be usable for my problem space, and have had to use other solvers for practical problems in finance. Regards, - Brian
Wind wrote:
Some progress. The LPPL curve could be plotted with the following codes. The problem now is how to get the best fit parameters. Some researchers use python or matlab for LPPL calibrating. It seems that some of them prefer tabu search for optimums locating. It seems that there's still no general function for tabu search in R. At the end of codes, I give the possible parameter combinations to be searched in, maybe there are other functions for optimum searching in R. Any suggestion would be appreciated. ## Financial Bubbles, Real Estate bubbles, Derivative Bubbles, and the Financial and Economic Crisis ## http://arxiv.org/abs/0905.0220 ## Fig. 23 S&P500 index (in logarithmic scale)in Page 39 library(quantmod) LPPL1<-function(p,dtc=20,alpha=0.35,omega=0.1,phi=1) { #function in page 26 of http://arxiv.org/abs/0905.0220 #the basic form of LPPL dtc=abs(floor(dtc)) tc<-length(p)+dtc dt<-abs(tc-(1:length(p))) x1<-dt^alpha x2<-(dt^alpha)*cos(omega*log(dt)+phi) f<-lm(log(p) ~ x1+x2) f$para<-list(recno=dim(f$model)[1],dtc=dtc,alpha=alpha,omega=omega,phi=phi,sigma=summary(f)$sigma) return(f) } opt.lppl<-function(x) { #derived function for optim return(LPPL1(p,dtc=x[1],alpha=x[2],omega=x[3],phi=x[4])$para$sigma) } LPPL1.x<-function(lpplf,pt=100) { #x axis for predicting dt<-abs((lpplf$recno+lpplf$dtc)-(1:(lpplf$recno+lpplf$dtc+pt))) dt[dt==0]<-0.5 x1<-dt^lpplf$alpha x2<-(dt^lpplf$alpha)*cos(lpplf$omega*log(dt)+lpplf$phi) return(list(x1=x1,x2=x2)) } #get the SP500 index pr<-getSymbols("^GSPC",auto.assign=FALSE,from="2003-10-1",to="2007-05-16")[,4] p<-as.numeric(pr) plot(p,type="l",log="y",xlim=c(0,length(p)+100),ylim=c(min(p),max(p)*1.2)) abline(v=length(p),col="green") #something like the Fig. 23 in Page 39 of http://arxiv.org/abs/0905.0220 #but obviously the result is not calibrated well #using optim like this can not calibrate the LPPL model opts<-sapply(seq(1,50,10),function(x){ o<-optim(c(x,0.6,20,1),opt.lppl) f3<-LPPL1(p,dtc=o$par[1],alpha=o$par[2],omega=o$par[3],phi=o$par[4]) xp<-LPPL1.x(f3$para,200) f3p<-predict(f3,data.frame(x1=xp$x1,x2=xp$x2)) lines(exp(f3p),col="blue") lines(exp(fitted(f3)),col="red") f3$para }) ##crash point after dtc days dtc<-seq(1,100,1) ##appropraite range of the parametes of LPPL ##according to Dr. W.X. Zhou's new book which is in Chinese ##the increments of the sequences are added according to my own judement alpha<-seq(0.01,1.2,0.1) omega<-seq(0,40,1) phi<-seq(0,7,0.1) ##millions possible combinations #complete test would be difficult para<-expand.grid(dtc=dtc,alpha=alpha,omega=omega,phi=phi) dim(para) system.time(sigs<-apply(para[1:100,],1,function(x){LPPL1(p,dtc=x[1],alpha=x[2],omega=x[3],phi=x[4])$para$sigma})) ##methods for minimum sigma searching within the parameter combinations ##not implemented yet wind On Thu, Jul 16, 2009 at 9:25 PM, Brian G. Peterson<brian at braverock.com> wrote:
So first, using your real name and ideally your professional identity, ask for the python code. Better yet, get an academic buddy to do it. Usually getting access to the code isn't too tough. Mention things like "repeatable research" and "collaboration" in your email. Two of the authors publish their email addresses in one of the papers you reference, so contacting them should be easy. Next port the python code to R. If you can't do that, then replicate the model in R "from scratch". A trivial scan of the paper in question lends several techniques that are well covered in R: AR, GARCH, power laws, linear regression, stochastic discount factor, Ornstein-Uhlenbeck, etc. There are volumes of information available on these topics from within R, in numerous books, and in the archives of this mailing list and r-help. You're going to have to do your replication in pieces, probably starting with their implementation of the log periodic power law (LPPL), for which I do not believe there is an existing direct analogue in R though all the component parts necessary to replicate it should be readily available. As you work on each step of the replication, share your code with this list and the problems you are having with a particular step. Ask specific, directed questions with code to back them up. Someone will likely help you solve the specific problem. In R generally, it is not necessary that you be able to *do* the math (think pencil and paper), but if you plan to replicate published work, it will be necessary to *understand* at least some of how the math works, and to be able to pick out the names of techniques that you can search for an utilize. Basically, I'm recommending that you (specifically) and others (more generally) should share the process of replicating a technique like this, as well as the final product, to give all the rest of us who are likely to be helping "you" get all this done. quid pro quo. Cheers, - Brian -- Brian G. Peterson http://braverock.com/brian/ Ph: 773-459-4973 IM: bgpbraverock Wind wrote:
Prof. Sornette has spent years forcasting bubble burst with "log-periodic power law". The latest paper gives "a self-consistent model for explosive financial bubbles, which combines a mean-reverting volatility process and a stochastic conditional return which reflects nonlinear positive feedbacks and continuous updates of the investors' beliefs and sentiments." And his latest predicting is the burst of Chinese equity bubble at the end of July. http://arxiv.org/abs/0907.1827 While waiting to see the result, I wonder whether it is possible to replicate the forcast with R. The model is in the page 10 of the "A Consistent Model of `Explosive' Financial Bubbles With Mean-Reversing Residuals", http://arxiv.org/abs/0905.0128 . The output chart is in the page 3 of "The Chinese Equity Bubble: Ready to Burst", http://arxiv.org/abs/0907.1827 . I guess the authors of the latter paper use the same model as described in the first paper. Because statistics is still challenging for me though I could use R for basic data manipulations, I wonder which package or function would be necessary to implement the model in the paper. The model seems more complicated than the models in the R tutorials for me. By the way, the author of the paper used Python and the codes are private. Any suggestion would be highly appreciated.
Brian G. Peterson http://braverock.com/brian/ Ph: 773-459-4973 IM: bgpbraverock
3 days later
Since I am not good at coding or statistics, genetic algo has been used instead of tabu search. Package rgenoud is great and idiot proof. Attached is the latest analysis on SSEC index, the equity market index of mainland China. genoud() has been run for 10 times, generating 10 fits. The residuals are all stationary according to ADF test from package urca. But the time window setting is subjective and maybe some minor problems on parameter conditions such as omega and C. So, just for fun. Thanks for all the encouragement and help from the list. And thanks for the detailed instructions on statistics issues by one of the author on LPPL, Dr. Lin. wind
On Mon, Jul 20, 2009 at 9:22 PM, Brian G. Peterson <brian at braverock.com>wrote:
Glad to see you're making progress on this problem. This paper http://www.diegm.uniud.it/satt/papers/DiSc06b.pdf implements a tabu search in R, though they didn't publish their code. You might want to contact them for the implementation and permission to share their tabu search algorithm/code with the R community. They also reference an R package called RACE by this gentleman: http://iridia.ulb.ac.be/~mbiro/ that they use for evaluating the solution. More generally, from my limited understanding, tabu search is an extension and refinement of simulated annealing approaches easily implemented in R. In brief the approach is to take your best 'n' solutions from a random space search, and then search "near" those solutions. Simulated annealing and its close cousins have a lot of benefits in finance, where a single true optima from a closed form problem is not likely to be available. I personally have rarely found 'optim' to be usable for my problem space, and have had to use other solvers for practical problems in finance. Regards, - Brian Wind wrote:
Some progress. The LPPL curve could be plotted with the following codes. The problem now is how to get the best fit parameters. Some researchers use python or matlab for LPPL calibrating. It seems that some of them prefer tabu search for optimums locating. It seems that there's still no general function for tabu search in R. At the end of codes, I give the possible parameter combinations to be searched in, maybe there are other functions for optimum searching in R. Any suggestion would be appreciated. ## Financial Bubbles, Real Estate bubbles, Derivative Bubbles, and the Financial and Economic Crisis ## http://arxiv.org/abs/0905.0220 ## Fig. 23 S&P500 index (in logarithmic scale)in Page 39 library(quantmod) LPPL1<-function(p,dtc=20,alpha=0.35,omega=0.1,phi=1) { #function in page 26 of http://arxiv.org/abs/0905.0220 #the basic form of LPPL dtc=abs(floor(dtc)) tc<-length(p)+dtc dt<-abs(tc-(1:length(p))) x1<-dt^alpha x2<-(dt^alpha)*cos(omega*log(dt)+phi) f<-lm(log(p) ~ x1+x2) f$para<-list(recno=dim(f$model)[1],dtc=dtc,alpha=alpha,omega=omega,phi=phi,sigma=summary(f)$sigma) return(f) } opt.lppl<-function(x) { #derived function for optim return(LPPL1(p,dtc=x[1],alpha=x[2],omega=x[3],phi=x[4])$para$sigma) } LPPL1.x<-function(lpplf,pt=100) { #x axis for predicting dt<-abs((lpplf$recno+lpplf$dtc)-(1:(lpplf$recno+lpplf$dtc+pt))) dt[dt==0]<-0.5 x1<-dt^lpplf$alpha x2<-(dt^lpplf$alpha)*cos(lpplf$omega*log(dt)+lpplf$phi) return(list(x1=x1,x2=x2)) } #get the SP500 index pr<-getSymbols("^GSPC",auto.assign=FALSE,from="2003-10-1",to="2007-05-16")[,4] p<-as.numeric(pr) plot(p,type="l",log="y",xlim=c(0,length(p)+100),ylim=c(min(p),max(p)*1.2)) abline(v=length(p),col="green") #something like the Fig. 23 in Page 39 of http://arxiv.org/abs/0905.0220 #but obviously the result is not calibrated well #using optim like this can not calibrate the LPPL model opts<-sapply(seq(1,50,10),function(x){ o<-optim(c(x,0.6,20,1),opt.lppl) f3<-LPPL1(p,dtc=o$par[1],alpha=o$par[2],omega=o$par[3],phi=o$par[4]) xp<-LPPL1.x(f3$para,200) f3p<-predict(f3,data.frame(x1=xp$x1,x2=xp$x2)) lines(exp(f3p),col="blue") lines(exp(fitted(f3)),col="red") f3$para }) ##crash point after dtc days dtc<-seq(1,100,1) ##appropraite range of the parametes of LPPL ##according to Dr. W.X. Zhou's new book which is in Chinese ##the increments of the sequences are added according to my own judement alpha<-seq(0.01,1.2,0.1) omega<-seq(0,40,1) phi<-seq(0,7,0.1) ##millions possible combinations #complete test would be difficult para<-expand.grid(dtc=dtc,alpha=alpha,omega=omega,phi=phi) dim(para) system.time(sigs<-apply(para[1:100,],1,function(x){LPPL1(p,dtc=x[1],alpha=x[2],omega=x[3],phi=x[4])$para$sigma})) ##methods for minimum sigma searching within the parameter combinations ##not implemented yet wind On Thu, Jul 16, 2009 at 9:25 PM, Brian G. Peterson<brian at braverock.com> wrote:
So first, using your real name and ideally your professional identity, ask for the python code. Better yet, get an academic buddy to do it. Usually getting access to the code isn't too tough. Mention things like "repeatable research" and "collaboration" in your email. Two of the authors publish their email addresses in one of the papers you reference, so contacting them should be easy. Next port the python code to R. If you can't do that, then replicate the model in R "from scratch". A trivial scan of the paper in question lends several techniques that are well covered in R: AR, GARCH, power laws, linear regression, stochastic discount factor, Ornstein-Uhlenbeck, etc. There are volumes of information available on these topics from within R, in numerous books, and in the archives of this mailing list and r-help. You're going to have to do your replication in pieces, probably starting with their implementation of the log periodic power law (LPPL), for which I do not believe there is an existing direct analogue in R though all the component parts necessary to replicate it should be readily available. As you work on each step of the replication, share your code with this list and the problems you are having with a particular step. Ask specific, directed questions with code to back them up. Someone will likely help you solve the specific problem. In R generally, it is not necessary that you be able to *do* the math (think pencil and paper), but if you plan to replicate published work, it will be necessary to *understand* at least some of how the math works, and to be able to pick out the names of techniques that you can search for an utilize. Basically, I'm recommending that you (specifically) and others (more generally) should share the process of replicating a technique like this, as well as the final product, to give all the rest of us who are likely to be helping "you" get all this done. quid pro quo. Cheers, - Brian -- Brian G. Peterson http://braverock.com/brian/ Ph: 773-459-4973 IM: bgpbraverock Wind wrote:
Prof. Sornette has spent years forcasting bubble burst with "log-periodic power law". The latest paper gives "a self-consistent model for explosive financial bubbles, which combines a mean-reverting volatility process and a stochastic conditional return which reflects nonlinear positive feedbacks and continuous updates of the investors' beliefs and sentiments." And his latest predicting is the burst of Chinese equity bubble at the end of July. http://arxiv.org/abs/0907.1827 While waiting to see the result, I wonder whether it is possible to replicate the forcast with R. The model is in the page 10 of the "A Consistent Model of `Explosive' Financial Bubbles With Mean-Reversing Residuals", http://arxiv.org/abs/0905.0128 . The output chart is in the page 3 of "The Chinese Equity Bubble: Ready to Burst", http://arxiv.org/abs/0907.1827 . I guess the authors of the latter paper use the same model as described in the first paper. Because statistics is still challenging for me though I could use R for basic data manipulations, I wonder which package or function would be necessary to implement the model in the paper. The model seems more complicated than the models in the R tutorials for me. By the way, the author of the paper used Python and the codes are private. Any suggestion would be highly appreciated.
-- Brian G. Peterson http://braverock.com/brian/ Ph: 773-459-4973 IM: bgpbraverock
-------------- next part -------------- An HTML attachment was scrubbed... URL: <https://stat.ethz.ch/pipermail/r-sig-finance/attachments/20090724/82f44d7f/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: ssec final 1.png Type: image/png Size: 10268 bytes Desc: not available URL: <https://stat.ethz.ch/pipermail/r-sig-finance/attachments/20090724/82f44d7f/attachment.png>