Dear Group, I have a large set of stocks and want to determine the efficient frontier. The data set covers approx. 1.5 years and S&P 500 companies (nothing weird). portfolio.optim from the PerformanceAnalytics package works very well and fast. However, whenever I decrease the number of stocks in the portfolio (to 10 or 400), I receive an error message: "solve.QP(Dmat, dvec, Amat, bvec = b0, meq = 2) : matrix D in quadratic function is not positive definite!" My command settings for portfolio.optim were: seed <- portfolio.optim(t(x), pm = current_er, riskless = FALSE, shorts = FALSE, rf = 0.0) Even when I tried it with shorts = TRUE the error would still remain. x is the set of stocks (stocks in columns, time in rows), current_er is the target return (lies between the minimal mean and the maximum mean of a long only portfolio). I can not post the stock data here - so maybe you have some general suggestions for me of what could have gone wrong... The covariance matrix is positive definite. What could cause the problem? It works fine with the large data set but does not work at all with the small one... Thanks a lot for your suggestions! Lui
portfolio.optim and error in solve.QP: matrix D not positive definite
7 messages · Arun Kumar Saha, Guy Yollin, Krishna Kumar +2 more
Hi Lui, I never worked with such kind of portfolio optimization problem but in Risk management practice it often comes as estimated VCV matrix is not a PD, hence it is not a truly VCV matrix. Root of this problem might be many, most importantly, it is incomplete and inconsistent return values. In such case, common practice is to disturb this estimated VCV matrix slightly, so that you would get **nearest** VCV matrix which is PD. Here you might be interested in: http://eprints.ma.man.ac.uk/232/01/covered/MIMS_ep2006_70.pdf Therefore I guess, what you need to do is perhaps debug the underlying codes and do some reverse-engineering to modify the underlying matrix to a nearest PD. HTH _____________________________________________________ Arun Kumar Saha, FRM QUANTITATIVE RISK AND HEDGE CONSULTING SPECIALIST Visit me at: http://in.linkedin.com/in/ArunFRM _____________________________________________________
View this message in context: http://r.789695.n4.nabble.com/portfolio-optim-and-error-in-solve-QP-matrix-D-not-positive-definite-tp3241430p3241453.html Sent from the Rmetrics mailing list archive at Nabble.com.
Hi Lui, Without seeing the data this is just speculation but... Are you sure you want t(x)? If you're mixing up your observations versus your assets this may explain the error. The first parameter of portfolio.optim (in the tseries package) is a returns matrix, one column for each asset and one row for each day (assuming daily returns). If you have this wrong then for your small datasets you'd have more columns than rows and this could produce that error. Also, you don't have to pass the entire returns matrix to portfolio.optim, you could pass just the covariance matrix you calculate yourself and a vector (1-row matrix) of mean returns as follows: library(tseries) set.seed(2) R <- matrix(rnorm(100*10),nrow=100,ncol=10) # 10 assets, 100 observations averet <- matrix(apply(R,2,mean),nrow=1) rcov <- cov(R) current_er <- 0.05 (op <- portfolio.optim(x=averet,pm=current_er,covmat=rcov,riskless = FALSE,shorts = FALSE, rf = 0.0)) Hope this helps. Best, Guy
On 1/26/2011 7:51 PM, Lui ## wrote:
Dear Group, I have a large set of stocks and want to determine the efficient frontier. The data set covers approx. 1.5 years and S&P 500 companies (nothing weird). portfolio.optim from the PerformanceAnalytics package works very well and fast. However, whenever I decrease the number of stocks in the portfolio (to 10 or 400), I receive an error message: "solve.QP(Dmat, dvec, Amat, bvec = b0, meq = 2) : matrix D in quadratic function is not positive definite!" My command settings for portfolio.optim were: seed<- portfolio.optim(t(x), pm = current_er, riskless = FALSE, shorts = FALSE, rf = 0.0) Even when I tried it with shorts = TRUE the error would still remain. x is the set of stocks (stocks in columns, time in rows), current_er is the target return (lies between the minimal mean and the maximum mean of a long only portfolio). I can not post the stock data here - so maybe you have some general suggestions for me of what could have gone wrong... The covariance matrix is positive definite. What could cause the problem? It works fine with the large data set but does not work at all with the small one... Thanks a lot for your suggestions! Lui
_______________________________________________ R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
1 day later
Also the link below might help with a large number of assets this is a common problem. https://stat.ethz.ch/pipermail/r-sig-finance/2008q3/002854.html Cheers Krishna On Jan 27, 2011, at 12:03 PM, Guy Yollin <gyollin at r-programming.org> wrote:
Hi Lui, Without seeing the data this is just speculation but... Are you sure you want t(x)? If you're mixing up your observations versus your assets this may explain the error. The first parameter of portfolio.optim (in the tseries package) is a returns matrix, one column for each asset and one row for each day (assuming daily returns). If you have this wrong then for your small datasets you'd have more columns than rows and this could produce that error. Also, you don't have to pass the entire returns matrix to portfolio.optim, you could pass just the covariance matrix you calculate yourself and a vector (1-row matrix) of mean returns as follows: library(tseries) set.seed(2) R <- matrix(rnorm(100*10),nrow=100,ncol=10) # 10 assets, 100 observations averet <- matrix(apply(R,2,mean),nrow=1) rcov <- cov(R) current_er <- 0.05 (op <- portfolio.optim(x=averet,pm=current_er,covmat=rcov,riskless = FALSE,shorts = FALSE, rf = 0.0)) Hope this helps. Best, Guy On 1/26/2011 7:51 PM, Lui ## wrote:
Dear Group, I have a large set of stocks and want to determine the efficient frontier. The data set covers approx. 1.5 years and S&P 500 companies (nothing weird). portfolio.optim from the PerformanceAnalytics package works very well and fast. However, whenever I decrease the number of stocks in the portfolio (to 10 or 400), I receive an error message: "solve.QP(Dmat, dvec, Amat, bvec = b0, meq = 2) : matrix D in quadratic function is not positive definite!" My command settings for portfolio.optim were: seed<- portfolio.optim(t(x), pm = current_er, riskless = FALSE, shorts = FALSE, rf = 0.0) Even when I tried it with shorts = TRUE the error would still remain. x is the set of stocks (stocks in columns, time in rows), current_er is the target return (lies between the minimal mean and the maximum mean of a long only portfolio). I can not post the stock data here - so maybe you have some general suggestions for me of what could have gone wrong... The covariance matrix is positive definite. What could cause the problem? It works fine with the large data set but does not work at all with the small one... Thanks a lot for your suggestions! Lui
_______________________________________________ R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
_______________________________________________ R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
1 day later
Hello everybody, sorry for my delayed "thanks" note - I was travelling. @Arun: Debugging the underlying code is a little bit difficult since the optimizer was written in FORTRAN. I think going for the nearest PD (as Krishna also suggested) might be the best way. However, I honestly don't understand why it is not PD... Does anybody have an explanation for that? @Guy: The weird thing is that I got the error code without the t(x) in the first place. t(x) solved the problem (for some assets) and the result indicated that it took a look at the assets and not the observations... I am going to give it a try with the covariance matrix again and let you know if it worked out... strange though. @Krishna: Thanks for your link! I think that really helps... I am going to try it out! Do you have an explanation why this is a common problem with a large number of assets? Thank you! Have a nice weekend! Lui
On Fri, Jan 28, 2011 at 3:51 PM, krishna <kriskumar at earthlink.net> wrote:
Also the link below might help with a large number of assets this is a common problem. https://stat.ethz.ch/pipermail/r-sig-finance/2008q3/002854.html Cheers Krishna On Jan 27, 2011, at 12:03 PM, Guy Yollin <gyollin at r-programming.org> wrote:
Hi Lui, Without seeing the data this is just speculation but... Are you sure you want t(x)? If you're mixing up your observations versus your assets this may explain the error. The first parameter of portfolio.optim (in the tseries package) is a returns matrix, one column for each asset and one row for each day (assuming daily returns). ?If you have this wrong then for your small datasets you'd have more columns than rows and this could produce that error. Also, you don't have to pass the entire returns matrix to portfolio.optim, you could pass just the covariance matrix you calculate yourself and a vector (1-row matrix) of mean returns as follows: library(tseries) set.seed(2) R <- matrix(rnorm(100*10),nrow=100,ncol=10) # 10 assets, 100 observations averet <- matrix(apply(R,2,mean),nrow=1) rcov <- cov(R) current_er <- 0.05 (op <- portfolio.optim(x=averet,pm=current_er,covmat=rcov,riskless = FALSE,shorts = FALSE, rf = 0.0)) Hope this helps. Best, Guy On 1/26/2011 7:51 PM, Lui ## wrote:
Dear Group, I have ?a large set of stocks and want to determine the efficient frontier. The data set covers approx. 1.5 years and S&P 500 companies (nothing weird). portfolio.optim from the PerformanceAnalytics package works very well and fast. However, whenever I decrease the number of stocks in the portfolio (to 10 or 400), I receive an error message: "solve.QP(Dmat, dvec, Amat, bvec = b0, meq = 2) : ?matrix D in quadratic function is not positive definite!" My command settings for portfolio.optim were: seed<- portfolio.optim(t(x), pm = current_er, riskless = FALSE, shorts = FALSE, rf = 0.0) Even when I tried it with shorts = TRUE the error would still remain. x is the set of stocks (stocks in columns, time in rows), current_er is the target return (lies between the minimal mean and the maximum mean of a long only portfolio). I can not post the stock data here - so maybe you have some general suggestions for me of what could have gone wrong... The covariance matrix is positive definite. What could cause the problem? It works fine with the large data set but does not work at all with the small one... Thanks a lot for your suggestions! Lui
_______________________________________________ R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
_______________________________________________ R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
_______________________________________________ R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
The problem could be due to few reasons the papers by Higham & Rebonato are a good read as to what the recipe does. The simplest recipe is to set the negative eigenvalues to a small positive number and rescale the matrix. There are a few causes for this if you create corr matrices from bivariate estimation and slap them together that may not be PD. In some cases in derivative pricing the matrix is hand glued together from implied correlations between two assets. Now a matrix of such implied corrs does not have to be PD and fails. Sample corr matrices are by definition PD however often times if you have a lot of missing data(illiquid names in your universe?) and if you do a na.locf (creating a const asset) this could do it as well. It might be useful to do multivariate imputation with some o the R packages rather than deletion or carrying fwd on your data. There are other reasons besides all of the above for your corr matrix being non PD. I think if you are working with such a large universe it might be easier to have factor corr matrices using PCA or ICA. This way if you manage to label the factors then you can see what bets your optimizer is taking. HTH Best Krishna On Jan 29, 2011, at 3:52 PM, "Lui ##" <lui.r.project at googlemail.com> wrote:
Hello everybody, sorry for my delayed "thanks" note - I was travelling. @Arun: Debugging the underlying code is a little bit difficult since the optimizer was written in FORTRAN. I think going for the nearest PD (as Krishna also suggested) might be the best way. However, I honestly don't understand why it is not PD... Does anybody have an explanation for that? @Guy: The weird thing is that I got the error code without the t(x) in the first place. t(x) solved the problem (for some assets) and the result indicated that it took a look at the assets and not the observations... I am going to give it a try with the covariance matrix again and let you know if it worked out... strange though. @Krishna: Thanks for your link! I think that really helps... I am going to try it out! Do you have an explanation why this is a common problem with a large number of assets? Thank you! Have a nice weekend! Lui On Fri, Jan 28, 2011 at 3:51 PM, krishna <kriskumar at earthlink.net> wrote:
Also the link below might help with a large number of assets this is a common problem. https://stat.ethz.ch/pipermail/r-sig-finance/2008q3/002854.html Cheers Krishna On Jan 27, 2011, at 12:03 PM, Guy Yollin <gyollin at r- programming.org> wrote:
Hi Lui, Without seeing the data this is just speculation but... Are you sure you want t(x)? If you're mixing up your observations versus your assets this may explain the error. The first parameter of portfolio.optim (in the tseries package) is a returns matrix, one column for each asset and one row for each day (assuming daily returns). If you have this wrong then for your small datasets you'd have more columns than rows and this could produce that error. Also, you don't have to pass the entire returns matrix to portfolio.optim, you could pass just the covariance matrix you calculate yourself and a vector (1-row matrix) of mean returns as follows: library(tseries) set.seed(2) R <- matrix(rnorm(100*10),nrow=100,ncol=10) # 10 assets, 100 observations averet <- matrix(apply(R,2,mean),nrow=1) rcov <- cov(R) current_er <- 0.05 (op <- portfolio.optim(x=averet,pm=current_er,covmat=rcov,riskless = FALSE,shorts = FALSE, rf = 0.0)) Hope this helps. Best, Guy On 1/26/2011 7:51 PM, Lui ## wrote:
Dear Group, I have a large set of stocks and want to determine the efficient frontier. The data set covers approx. 1.5 years and S&P 500 companies (nothing weird). portfolio.optim from the PerformanceAnalytics package works very well and fast. However, whenever I decrease the number of stocks in the portfolio (to 10 or 400), I receive an error message: "solve.QP(Dmat, dvec, Amat, bvec = b0, meq = 2) : matrix D in quadratic function is not positive definite!" My command settings for portfolio.optim were: seed<- portfolio.optim(t(x), pm = current_er, riskless = FALSE, shorts = FALSE, rf = 0.0) Even when I tried it with shorts = TRUE the error would still remain. x is the set of stocks (stocks in columns, time in rows), current_er is the target return (lies between the minimal mean and the maximum mean of a long only portfolio). I can not post the stock data here - so maybe you have some general suggestions for me of what could have gone wrong... The covariance matrix is positive definite. What could cause the problem? It works fine with the large data set but does not work at all with the small one... Thanks a lot for your suggestions! Lui
_______________________________________________ R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
_______________________________________________ R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
_______________________________________________ R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
_______________________________________________ R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
Krishna has a good point. Getting the
variance matrix to be barely positive
definite is still not a good thing for
optimization.
An eigenvalue that is essentially zero
says that there is a portfolio that is
essentially riskless. And we haven't
had any of those since subprime mortgages
fell out of favor.
As Krishna says a factor model will give
you a better result. A good (possibly
better) alternative is a Ledoit-Wolf
shrinkage estimate.
Functions to estimate either of those are
in the 'BurStFin' package which still hasn't
arrived in CRAN. But you can get it with:
install.packages('BurStFin', repos="http://www.burns-stat.com/R")
The 'tawny' package has a function for
Ledoit-Wolf.
On 29/01/2011 21:32, krishna wrote:
The problem could be due to few reasons the papers by Higham & Rebonato are a good read as to what the recipe does. The simplest recipe is to set the negative eigenvalues to a small positive number and rescale the matrix. There are a few causes for this if you create corr matrices from bivariate estimation and slap them together that may not be PD. In some cases in derivative pricing the matrix is hand glued together from implied correlations between two assets. Now a matrix of such implied corrs does not have to be PD and fails. Sample corr matrices are by definition PD however often times if you have a lot of missing data(illiquid names in your universe?) and if you do a na.locf (creating a const asset) this could do it as well. It might be useful to do multivariate imputation with some o the R packages rather than deletion or carrying fwd on your data. There are other reasons besides all of the above for your corr matrix being non PD. I think if you are working with such a large universe it might be easier to have factor corr matrices using PCA or ICA. This way if you manage to label the factors then you can see what bets your optimizer is taking. HTH Best Krishna On Jan 29, 2011, at 3:52 PM, "Lui ##" <lui.r.project at googlemail.com> wrote:
Hello everybody, sorry for my delayed "thanks" note - I was travelling. @Arun: Debugging the underlying code is a little bit difficult since the optimizer was written in FORTRAN. I think going for the nearest PD (as Krishna also suggested) might be the best way. However, I honestly don't understand why it is not PD... Does anybody have an explanation for that? @Guy: The weird thing is that I got the error code without the t(x) in the first place. t(x) solved the problem (for some assets) and the result indicated that it took a look at the assets and not the observations... I am going to give it a try with the covariance matrix again and let you know if it worked out... strange though. @Krishna: Thanks for your link! I think that really helps... I am going to try it out! Do you have an explanation why this is a common problem with a large number of assets? Thank you! Have a nice weekend! Lui On Fri, Jan 28, 2011 at 3:51 PM, krishna <kriskumar at earthlink.net> wrote:
Also the link below might help with a large number of assets this is a common problem. https://stat.ethz.ch/pipermail/r-sig-finance/2008q3/002854.html Cheers Krishna On Jan 27, 2011, at 12:03 PM, Guy Yollin <gyollin at r-programming.org> wrote:
Hi Lui, Without seeing the data this is just speculation but... Are you sure you want t(x)? If you're mixing up your observations versus your assets this may explain the error. The first parameter of portfolio.optim (in the tseries package) is a returns matrix, one column for each asset and one row for each day (assuming daily returns). If you have this wrong then for your small datasets you'd have more columns than rows and this could produce that error. Also, you don't have to pass the entire returns matrix to portfolio.optim, you could pass just the covariance matrix you calculate yourself and a vector (1-row matrix) of mean returns as follows: library(tseries) set.seed(2) R <- matrix(rnorm(100*10),nrow=100,ncol=10) # 10 assets, 100 observations averet <- matrix(apply(R,2,mean),nrow=1) rcov <- cov(R) current_er <- 0.05 (op <- portfolio.optim(x=averet,pm=current_er,covmat=rcov,riskless = FALSE,shorts = FALSE, rf = 0.0)) Hope this helps. Best, Guy On 1/26/2011 7:51 PM, Lui ## wrote:
Dear Group, I have a large set of stocks and want to determine the efficient frontier. The data set covers approx. 1.5 years and S&P 500 companies (nothing weird). portfolio.optim from the PerformanceAnalytics package works very well and fast. However, whenever I decrease the number of stocks in the portfolio (to 10 or 400), I receive an error message: "solve.QP(Dmat, dvec, Amat, bvec = b0, meq = 2) : matrix D in quadratic function is not positive definite!" My command settings for portfolio.optim were: seed<- portfolio.optim(t(x), pm = current_er, riskless = FALSE, shorts = FALSE, rf = 0.0) Even when I tried it with shorts = TRUE the error would still remain. x is the set of stocks (stocks in columns, time in rows), current_er is the target return (lies between the minimal mean and the maximum mean of a long only portfolio). I can not post the stock data here - so maybe you have some general suggestions for me of what could have gone wrong... The covariance matrix is positive definite. What could cause the problem? It works fine with the large data set but does not work at all with the small one... Thanks a lot for your suggestions! Lui
_______________________________________________ R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
_______________________________________________ R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
_______________________________________________ R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
_______________________________________________ R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
_______________________________________________ R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
Patrick Burns patrick at burns-stat.com http://www.burns-stat.com http://www.portfolioprobe.com/blog twitter: @portfolioprobe