Detail analysis of solvers' results
Alexios, I updated my script to use multiple solvers and compare results. It looks like nlminb is the most accurate solver for this generic dataset that I am using. I posted the results on stack exchange so its easier to view the results. If you want, I can post the results here. Here is the url with the updated script and results: https://stackoverflow.com/questions/51900177/should-the-positioning-of-the-external-regressors-change-the-output-of-arma-garc/ On Mon, Aug 20, 2018, 7:29 AM GALIB KHAN <ghk18 at scarletmail.rutgers.edu> wrote:
Alexios, I changed the solver to "hybrid" but kept the scaling that you provided and got different results when switching the positions for the external regressors. ugarchfit(spec = spec, data = as.matrix(temp$y),solver = "hybrid",fit.control=list(scale=1)) So it looks like you have to change the solver to "nlminb" and update fit.control with the scaling that you provided. Galib On Sun, Aug 19, 2018 at 11:41 PM, GALIB KHAN < ghk18 at scarletmail.rutgers.edu> wrote:
Alexios, Veryyyy interesting!!!!! No I cannot see any differences at all lol. I updated the code and indeed you are correct sir. Thank you for your time in investigating this. I will update my stack exchange post to reflect your answer in the morning. Again thank you for all your help!!!! Best, Galib Khan On Sun, Aug 19, 2018 at 11:19 PM, alexios galanos <alexios at 4dscape.com> wrote:
I did use the seed you provided.
Use the following code for estimation:
fit <- ugarchfit(spec = spec, data = as.matrix(temp$y),solver =
"nlminb", fit.control=list(scale=1))
model_maker(var1)
Estimate Std. Error t value Pr(>|t|)
mu -7.3998577 0.69086641 -10.7109821 0.0000000000
ar1 0.3387323 0.08280162 4.0908900 0.0000429721
ar2 -0.8834201 0.06569477 -13.4473414 0.0000000000
ma1 -0.2902069 0.08598589 -3.3750525 0.0007380161
ma2 0.8660807 0.06778418 12.7770320 0.0000000000
mxreg1 1.6782992 0.12769644 13.1428825 0.0000000000
mxreg2 2.5225382 0.04292728 58.7630625 0.0000000000
omega 12.0047145 0.82986864 14.4658010 0.0000000000
alpha1 0.0000000 0.07358520 0.0000000 1.0000000000
shape 63.0103309 98.49188643 0.6397515 0.5223341761
model_maker(var2)
Estimate Std. Error t value Pr(>|t|)
mu -7.3998549 0.69086651 -10.7109764 0.000000e+00
ar1 0.3387334 0.08280150 4.0909088 4.296861e-05
ar2 -0.8834206 0.06569433 -13.4474406 0.000000e+00
ma1 -0.2902081 0.08598562 -3.3750776 7.379487e-04
ma2 0.8660811 0.06778412 12.7770487 0.000000e+00
mxreg1 2.5225383 0.04292728 58.7630642 0.000000e+00
mxreg2 1.6782987 0.12769640 13.1428817 0.000000e+00
omega 12.0047142 0.82992363 14.4648419 0.000000e+00
alpha1 0.0000000 0.07359329 0.0000000 1.000000e+00
shape 63.0105962 98.49368444 0.6397425 5.223400e-01
I can?t see any ?significant? differences, can you?
It?s completely related to the optimization/starting parameters. The
?scale? is documented and not on by default (perhaps it should be).
Alexios
On Aug 19, 2018, at 9:02 PM, GALIB KHAN <ghk18 at scarletmail.rutgers.edu>
wrote:
Sorry for sending this again, I didn't include r-sig-finance in the
email address. I'm still adjusting in how to respond.
Alexios, Did you set the set the seed to 1, because I'm looking at your results
and the numbers do not match with the numbers that I have provided.
I understand why the coefficients' estimates are similar but it
doesn't explain why other columns such as the t-value and pr are off by a large margin. Also estimates for mu, ar*, ma*, omega, alpha1, and shape may have large differences.
Take mu as an example: -7.538187e+00 - (-7.877120e+00) = 0.338933, isn't that considered a
large difference to the point where it's safe to say that these two values are not similar?
Another example is the t-values for x1 and x2: x1 = 8.799994e+01 - 5.509361e+02 = -462.9362 x2 = 8.508606e+01 - 5.287634e+02 = -443.6773 An more alarming case that unfortunately I cannot share due to the
data being sensitive is that when the x variables' positions are switched, the p-values are not the same. The p-value for a particular external regressor went from 0 to 0.4385.
I will attempt to re-create a separate generic dataset that is similar
to the sensitive data that I am using.
Galib Khan On Sun, Aug 19, 2018 at 10:06 PM, alexios galanos <alexios at 4dscape.com>
wrote:
I run the code you provided and obtain the following results related
to the external parameters:
Case 1 (x1,x2)
# x2 is second
Estimate Std. Error t value Pr(>|t|)
mxreg1 1.6724148 1.203377e-01 1.389767e+01 0.0000000
mxreg2 2.5310286 1.878833e-02 1.347128e+02 0.0000000
Case 2 (x2,x1)
# i.e. x2 is now first
mxreg1 2.5225382 0.04292725 58.7631024 0.000000e+00
mxreg2 1.6782986 0.12769622 13.1428990 0.000000e+00
Small differences in the coefficients are the result of the optimizer.
There may be an issues in the
way starting parameters are being generated based on some recent input
from Josh Ulrich (still to investigate)
and related to arima0 (used to generate start parameters), but
otherwise don?t see a large problem at first glance.
Alexios
On Aug 19, 2018, at 5:46 PM, GALIB KHAN <
ghk18 at scarletmail.rutgers.edu> wrote:
Recently I have discovered a problem with a package called rugarch
that
creates arma-garch models. The issue is that if you literally change
the
positions of the x variables (external regressors) then you get two completely different results. In other words: - model1 = (arma(2,2) + garch(1,0) + x1 + x2) - model2 = (arma(2,2) + garch(1,0) + x2 + x1) - rugarch's output is essentially saying that model1 != model2 - When the correct result should be model1 == model2 I may not know a lot of statistics but I know for a fact that if you
move
the x variables around, the output should still be the same. Am I wrong on this? Here's my stack exchange post that shows a generic R script proving
my
point: Should the positioning of the external regressors change the
output
of arma-garch? (Possible rugarch bug/error) <
Any feedback is welcomed.
Thanks
[[alternative HTML version deleted]]
_______________________________________________ R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R
questions should go.
Galib Khan On Sun, Aug 19, 2018 at 11:41 PM, GALIB KHAN < ghk18 at scarletmail.rutgers.edu> wrote:
Alexios, Veryyyy interesting!!!!! No I cannot see any differences at all lol. I updated the code and indeed you are correct sir. Thank you for your time in investigating this. I will update my stack exchange post to reflect your answer in the morning. Again thank you for all your help!!!! Best, Galib Khan On Sun, Aug 19, 2018 at 11:19 PM, alexios galanos <alexios at 4dscape.com> wrote:
I did use the seed you provided.
Use the following code for estimation:
fit <- ugarchfit(spec = spec, data = as.matrix(temp$y),solver =
"nlminb", fit.control=list(scale=1))
model_maker(var1)
Estimate Std. Error t value Pr(>|t|)
mu -7.3998577 0.69086641 -10.7109821 0.0000000000
ar1 0.3387323 0.08280162 4.0908900 0.0000429721
ar2 -0.8834201 0.06569477 -13.4473414 0.0000000000
ma1 -0.2902069 0.08598589 -3.3750525 0.0007380161
ma2 0.8660807 0.06778418 12.7770320 0.0000000000
mxreg1 1.6782992 0.12769644 13.1428825 0.0000000000
mxreg2 2.5225382 0.04292728 58.7630625 0.0000000000
omega 12.0047145 0.82986864 14.4658010 0.0000000000
alpha1 0.0000000 0.07358520 0.0000000 1.0000000000
shape 63.0103309 98.49188643 0.6397515 0.5223341761
model_maker(var2)
Estimate Std. Error t value Pr(>|t|)
mu -7.3998549 0.69086651 -10.7109764 0.000000e+00
ar1 0.3387334 0.08280150 4.0909088 4.296861e-05
ar2 -0.8834206 0.06569433 -13.4474406 0.000000e+00
ma1 -0.2902081 0.08598562 -3.3750776 7.379487e-04
ma2 0.8660811 0.06778412 12.7770487 0.000000e+00
mxreg1 2.5225383 0.04292728 58.7630642 0.000000e+00
mxreg2 1.6782987 0.12769640 13.1428817 0.000000e+00
omega 12.0047142 0.82992363 14.4648419 0.000000e+00
alpha1 0.0000000 0.07359329 0.0000000 1.000000e+00
shape 63.0105962 98.49368444 0.6397425 5.223400e-01
I can?t see any ?significant? differences, can you?
It?s completely related to the optimization/starting parameters. The
?scale? is documented and not on by default (perhaps it should be).
Alexios
On Aug 19, 2018, at 9:02 PM, GALIB KHAN <ghk18 at scarletmail.rutgers.edu>
wrote:
Sorry for sending this again, I didn't include r-sig-finance in the
email address. I'm still adjusting in how to respond.
Alexios, Did you set the set the seed to 1, because I'm looking at your results
and the numbers do not match with the numbers that I have provided.
I understand why the coefficients' estimates are similar but it
doesn't explain why other columns such as the t-value and pr are off by a large margin. Also estimates for mu, ar*, ma*, omega, alpha1, and shape may have large differences.
Take mu as an example: -7.538187e+00 - (-7.877120e+00) = 0.338933, isn't that considered a
large difference to the point where it's safe to say that these two values are not similar?
Another example is the t-values for x1 and x2: x1 = 8.799994e+01 - 5.509361e+02 = -462.9362 x2 = 8.508606e+01 - 5.287634e+02 = -443.6773 An more alarming case that unfortunately I cannot share due to the
data being sensitive is that when the x variables' positions are switched, the p-values are not the same. The p-value for a particular external regressor went from 0 to 0.4385.
I will attempt to re-create a separate generic dataset that is similar
to the sensitive data that I am using.
Galib Khan On Sun, Aug 19, 2018 at 10:06 PM, alexios galanos <alexios at 4dscape.com>
wrote:
I run the code you provided and obtain the following results related
to the external parameters:
Case 1 (x1,x2)
# x2 is second
Estimate Std. Error t value Pr(>|t|)
mxreg1 1.6724148 1.203377e-01 1.389767e+01 0.0000000
mxreg2 2.5310286 1.878833e-02 1.347128e+02 0.0000000
Case 2 (x2,x1)
# i.e. x2 is now first
mxreg1 2.5225382 0.04292725 58.7631024 0.000000e+00
mxreg2 1.6782986 0.12769622 13.1428990 0.000000e+00
Small differences in the coefficients are the result of the optimizer.
There may be an issues in the
way starting parameters are being generated based on some recent input
from Josh Ulrich (still to investigate)
and related to arima0 (used to generate start parameters), but
otherwise don?t see a large problem at first glance.
Alexios
On Aug 19, 2018, at 5:46 PM, GALIB KHAN <
ghk18 at scarletmail.rutgers.edu> wrote:
Recently I have discovered a problem with a package called rugarch
that
creates arma-garch models. The issue is that if you literally change
the
positions of the x variables (external regressors) then you get two completely different results. In other words: - model1 = (arma(2,2) + garch(1,0) + x1 + x2) - model2 = (arma(2,2) + garch(1,0) + x2 + x1) - rugarch's output is essentially saying that model1 != model2 - When the correct result should be model1 == model2 I may not know a lot of statistics but I know for a fact that if you
move
the x variables around, the output should still be the same. Am I wrong on this? Here's my stack exchange post that shows a generic R script proving
my
point: Should the positioning of the external regressors change the
output
of arma-garch? (Possible rugarch bug/error) <
Any feedback is welcomed.
Thanks
[[alternative HTML version deleted]]
_______________________________________________ R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R
questions should go.