Dear Sirs, Since the start of April, I have gotten some errors on the Debian machines on the check on CRAN for my package dynamichazard. The places where I get the errors seems to come down to LAPACK's dtrtri routine. I have tried to reproduce the error on Debian 18.04.2 with clang 8.0.0 but I cannot reproduce them. The errors came suddenly and the tests passed before. I see similar issues (I think) in the following package: - RcppHMM - BNPmix - themetagenomics - tidytext - stm Any ideas what the error may be? Here is the relevant code parts for one of the failed tests: https://github.com/boennecd/dynamichazard/blob/c42105feea5e2d16a8d461b7c606245a743b3e0a/src/for_tests.cpp#L216 https://github.com/boennecd/dynamichazard/blob/c42105feea5e2d16a8d461b7c606245a743b3e0a/src/BLAS_LAPACK/arma_BLAS_LAPACK.cpp#L26 https://github.com/boennecd/dynamichazard/blob/c42105feea5e2d16a8d461b7c606245a743b3e0a/src/BLAS_LAPACK/R_BLAS_LAPACK.cpp#L67 Sincerely yours, Benjamin
[R-pkg-devel] dtrti2 error on CRAN checks
6 messages · Benjamin Christoffersen, Dirk Eddelbuettel, Iñaki Ucar +2 more
On 17 April 2019 at 11:06, Benjamin Christoffersen wrote:
| Since the start of April, I have gotten some errors on the Debian | machines on the check on CRAN for my package dynamichazard. The places | where I get the errors seems to come down to LAPACK's dtrtri routine. | I have tried to reproduce the error on Debian 18.04.2 with clang 8.0.0 | but I cannot reproduce them. | | The errors came suddenly and the tests passed before. I see similar | issues (I think) in the following package: | - RcppHMM | - BNPmix | - themetagenomics | - tidytext | - stm | | Any ideas what the error may be? Here is the relevant code parts for | one of the failed tests: | https://github.com/boennecd/dynamichazard/blob/c42105feea5e2d16a8d461b7c606245a743b3e0a/src/for_tests.cpp#L216 | https://github.com/boennecd/dynamichazard/blob/c42105feea5e2d16a8d461b7c606245a743b3e0a/src/BLAS_LAPACK/arma_BLAS_LAPACK.cpp#L26 | https://github.com/boennecd/dynamichazard/blob/c42105feea5e2d16a8d461b7c606245a743b3e0a/src/BLAS_LAPACK/R_BLAS_LAPACK.cpp#L67 There are four or more different sources of LAPACK and BLAS on Debian as we (since the late 1990s !!) utilize the nature of the _interface_ allowing different packages to fill in. So there could be i) the R internal source with a partial library -- Fedora/CentOS use this but the Debian distro builds do not ii) "reference BLAS", external, unoptimized iii) OpenBLAS, the successor to Goto, parallel via multithreading iv) Atlas, "tuned" but not mulitthreading and more as eg Intel MKL via a quick script (see my blog). As a historical aside, and way-back-when, ii) earned us year's long growling from the direction of Oxfordshire because some early/old versions of LAPACK had bugs. But it cuts both ways: the external versions tend to be complete whereas what R ships with internally contains a subset -- and at least one (early) user of RcppArmadillo was bitten when R built from sources using defaults would not have the complex operations he needed. To R Core's credit these missing functions got filled in over the years. In short, you need to look more closely. On the Ubuntu 18.10 machine that I type this on, sessionInfo()'s second paragraph has Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3 LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.3.3.so which IIRC is also the default (via some Debian/Ubuntu internal 'ordering' of the available alternatives). Hth, Dirk
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
Thanks for the quick reply. First a correction. I meant Ubuntu 18.04.2 of course, sorry.
In short, you need to look more closely.
Is there a way to tell which BLAS and LAPACK is used on the Debian machines on CRAN? I checked the "CRAN Package Check Flavors" page but there is no information there regarding BLAS or LAPACK. I have tried to test the package with Atlas, OpenBlas, and the reference implementation with gcc 7.3.0 and 8.2.0. It all worked without any errors. It seems that gcc has changed since January from version 8.2.0 and 7.3.0 to 8.3.0 on the tests on CRAN. Further, it is now Debian version 8.3.0-2 instead of Debian 8.2.0-7 and Debian 7.3.0-29. I do not know whether this can matter. Sincerely yours, Benjamin Christoffersen Den ons. 17. apr. 2019 kl. 11.38 skrev Dirk Eddelbuettel <edd at debian.org>:
On 17 April 2019 at 11:06, Benjamin Christoffersen wrote: | Since the start of April, I have gotten some errors on the Debian | machines on the check on CRAN for my package dynamichazard. The places | where I get the errors seems to come down to LAPACK's dtrtri routine. | I have tried to reproduce the error on Debian 18.04.2 with clang 8.0.0 | but I cannot reproduce them. | | The errors came suddenly and the tests passed before. I see similar | issues (I think) in the following package: | - RcppHMM | - BNPmix | - themetagenomics | - tidytext | - stm | | Any ideas what the error may be? Here is the relevant code parts for | one of the failed tests: | https://github.com/boennecd/dynamichazard/blob/c42105feea5e2d16a8d461b7c606245a743b3e0a/src/for_tests.cpp#L216 | https://github.com/boennecd/dynamichazard/blob/c42105feea5e2d16a8d461b7c606245a743b3e0a/src/BLAS_LAPACK/arma_BLAS_LAPACK.cpp#L26 | https://github.com/boennecd/dynamichazard/blob/c42105feea5e2d16a8d461b7c606245a743b3e0a/src/BLAS_LAPACK/R_BLAS_LAPACK.cpp#L67 There are four or more different sources of LAPACK and BLAS on Debian as we (since the late 1990s !!) utilize the nature of the _interface_ allowing different packages to fill in. So there could be i) the R internal source with a partial library -- Fedora/CentOS use this but the Debian distro builds do not ii) "reference BLAS", external, unoptimized iii) OpenBLAS, the successor to Goto, parallel via multithreading iv) Atlas, "tuned" but not mulitthreading and more as eg Intel MKL via a quick script (see my blog). As a historical aside, and way-back-when, ii) earned us year's long growling from the direction of Oxfordshire because some early/old versions of LAPACK had bugs. But it cuts both ways: the external versions tend to be complete whereas what R ships with internally contains a subset -- and at least one (early) user of RcppArmadillo was bitten when R built from sources using defaults would not have the complex operations he needed. To R Core's credit these missing functions got filled in over the years. In short, you need to look more closely. On the Ubuntu 18.10 machine that I type this on, sessionInfo()'s second paragraph has Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3 LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.3.3.so which IIRC is also the default (via some Debian/Ubuntu internal 'ordering' of the available alternatives). Hth, Dirk -- http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
Benjamin, did you try with rhub? G?bor makes a great effort there to keep it up to date with CRAN machines. And even if you still are not able to reproduce it with the current config, you could try to modify the docker file yourself. I?aki On Thu, 18 Apr 2019 at 07:53, Benjamin Christoffersen
<boennecd at gmail.com> wrote:
Thanks for the quick reply. First a correction. I meant Ubuntu 18.04.2 of course, sorry.
In short, you need to look more closely.
Is there a way to tell which BLAS and LAPACK is used on the Debian machines on CRAN? I checked the "CRAN Package Check Flavors" page but there is no information there regarding BLAS or LAPACK. I have tried to test the package with Atlas, OpenBlas, and the reference implementation with gcc 7.3.0 and 8.2.0. It all worked without any errors. It seems that gcc has changed since January from version 8.2.0 and 7.3.0 to 8.3.0 on the tests on CRAN. Further, it is now Debian version 8.3.0-2 instead of Debian 8.2.0-7 and Debian 7.3.0-29. I do not know whether this can matter. Sincerely yours, Benjamin Christoffersen Den ons. 17. apr. 2019 kl. 11.38 skrev Dirk Eddelbuettel <edd at debian.org>:
On 17 April 2019 at 11:06, Benjamin Christoffersen wrote: | Since the start of April, I have gotten some errors on the Debian | machines on the check on CRAN for my package dynamichazard. The places | where I get the errors seems to come down to LAPACK's dtrtri routine. | I have tried to reproduce the error on Debian 18.04.2 with clang 8.0.0 | but I cannot reproduce them. | | The errors came suddenly and the tests passed before. I see similar | issues (I think) in the following package: | - RcppHMM | - BNPmix | - themetagenomics | - tidytext | - stm | | Any ideas what the error may be? Here is the relevant code parts for | one of the failed tests: | https://github.com/boennecd/dynamichazard/blob/c42105feea5e2d16a8d461b7c606245a743b3e0a/src/for_tests.cpp#L216 | https://github.com/boennecd/dynamichazard/blob/c42105feea5e2d16a8d461b7c606245a743b3e0a/src/BLAS_LAPACK/arma_BLAS_LAPACK.cpp#L26 | https://github.com/boennecd/dynamichazard/blob/c42105feea5e2d16a8d461b7c606245a743b3e0a/src/BLAS_LAPACK/R_BLAS_LAPACK.cpp#L67 There are four or more different sources of LAPACK and BLAS on Debian as we (since the late 1990s !!) utilize the nature of the _interface_ allowing different packages to fill in. So there could be i) the R internal source with a partial library -- Fedora/CentOS use this but the Debian distro builds do not ii) "reference BLAS", external, unoptimized iii) OpenBLAS, the successor to Goto, parallel via multithreading iv) Atlas, "tuned" but not mulitthreading and more as eg Intel MKL via a quick script (see my blog). As a historical aside, and way-back-when, ii) earned us year's long growling from the direction of Oxfordshire because some early/old versions of LAPACK had bugs. But it cuts both ways: the external versions tend to be complete whereas what R ships with internally contains a subset -- and at least one (early) user of RcppArmadillo was bitten when R built from sources using defaults would not have the complex operations he needed. To R Core's credit these missing functions got filled in over the years. In short, you need to look more closely. On the Ubuntu 18.10 machine that I type this on, sessionInfo()'s second paragraph has Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3 LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.3.3.so which IIRC is also the default (via some Debian/Ubuntu internal 'ordering' of the available alternatives). Hth, Dirk -- http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
______________________________________________ R-package-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
I?aki ?car
On 18.04.19 07:53, Benjamin Christoffersen wrote:
In short, you need to look more closely.
Is there a way to tell which BLAS and LAPACK is used on the Debian machines on CRAN?
I am sure CRAN machines use BLAS and LAPACK as shipped with R, i.e. not *any* variant of system BLAS/LAPACK. cheerio ralf
Ralf Stubner Senior Software Engineer / Trainer daqana GmbH Dortustra?e 48 14467 Potsdam T: +49 331 23 61 93 11 F: +49 331 23 61 93 90 M: +49 162 20 91 196 Mail: ralf.stubner at daqana.com Sitz: Potsdam Register: AG Potsdam HRB 27966 Ust.-IdNr.: DE300072622 Gesch?ftsf?hrer: Dr.-Ing. Stefan Knirsch, Prof. Dr. Dr. Karl-Kuno Kunze -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <https://stat.ethz.ch/pipermail/r-package-devel/attachments/20190418/cd88919a/attachment.sig>
Kurt Hornik submitted a bug report to the gfortran maintainers yesterday, and yes, it seems to have a lot to do with this. He and Brian Ripley have been able to pinpoint it to certain gfortran revisions (specifically gfortran-9 at least r268992, or gfortran-8 at least r269349), but we're lacking a simple test case. -pd
On 18 Apr 2019, at 07:53 , Benjamin Christoffersen <boennecd at gmail.com> wrote: Thanks for the quick reply. First a correction. I meant Ubuntu 18.04.2 of course, sorry.
In short, you need to look more closely.
Is there a way to tell which BLAS and LAPACK is used on the Debian machines on CRAN? I checked the "CRAN Package Check Flavors" page but there is no information there regarding BLAS or LAPACK. I have tried to test the package with Atlas, OpenBlas, and the reference implementation with gcc 7.3.0 and 8.2.0. It all worked without any errors. It seems that gcc has changed since January from version 8.2.0 and 7.3.0 to 8.3.0 on the tests on CRAN. Further, it is now Debian version 8.3.0-2 instead of Debian 8.2.0-7 and Debian 7.3.0-29. I do not know whether this can matter. Sincerely yours, Benjamin Christoffersen Den ons. 17. apr. 2019 kl. 11.38 skrev Dirk Eddelbuettel <edd at debian.org>:
On 17 April 2019 at 11:06, Benjamin Christoffersen wrote: | Since the start of April, I have gotten some errors on the Debian | machines on the check on CRAN for my package dynamichazard. The places | where I get the errors seems to come down to LAPACK's dtrtri routine. | I have tried to reproduce the error on Debian 18.04.2 with clang 8.0.0 | but I cannot reproduce them. | | The errors came suddenly and the tests passed before. I see similar | issues (I think) in the following package: | - RcppHMM | - BNPmix | - themetagenomics | - tidytext | - stm | | Any ideas what the error may be? Here is the relevant code parts for | one of the failed tests: | https://github.com/boennecd/dynamichazard/blob/c42105feea5e2d16a8d461b7c606245a743b3e0a/src/for_tests.cpp#L216 | https://github.com/boennecd/dynamichazard/blob/c42105feea5e2d16a8d461b7c606245a743b3e0a/src/BLAS_LAPACK/arma_BLAS_LAPACK.cpp#L26 | https://github.com/boennecd/dynamichazard/blob/c42105feea5e2d16a8d461b7c606245a743b3e0a/src/BLAS_LAPACK/R_BLAS_LAPACK.cpp#L67 There are four or more different sources of LAPACK and BLAS on Debian as we (since the late 1990s !!) utilize the nature of the _interface_ allowing different packages to fill in. So there could be i) the R internal source with a partial library -- Fedora/CentOS use this but the Debian distro builds do not ii) "reference BLAS", external, unoptimized iii) OpenBLAS, the successor to Goto, parallel via multithreading iv) Atlas, "tuned" but not mulitthreading and more as eg Intel MKL via a quick script (see my blog). As a historical aside, and way-back-when, ii) earned us year's long growling from the direction of Oxfordshire because some early/old versions of LAPACK had bugs. But it cuts both ways: the external versions tend to be complete whereas what R ships with internally contains a subset -- and at least one (early) user of RcppArmadillo was bitten when R built from sources using defaults would not have the complex operations he needed. To R Core's credit these missing functions got filled in over the years. In short, you need to look more closely. On the Ubuntu 18.10 machine that I type this on, sessionInfo()'s second paragraph has Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3 LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.3.3.so which IIRC is also the default (via some Debian/Ubuntu internal 'ordering' of the available alternatives). Hth, Dirk -- http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
______________________________________________ R-package-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Peter Dalgaard, Professor, Center for Statistics, Copenhagen Business School Solbjerg Plads 3, 2000 Frederiksberg, Denmark Phone: (+45)38153501 Office: A 4.23 Email: pd.mes at cbs.dk Priv: PDalgd at gmail.com