Hello All, First of all, thanks to Martin, Hadley, and Dirk (and everybody else involved) for setting up this new mailing list. I am sure this will become a tremendous resource for package developers. My question is in essence just a repost of what I asked on R-devel a few weeks ago: https://stat.ethz.ch/pipermail/r-devel/2015-May/071147.html At the time, the 'metafor' package failed CRAN checks on Sparc Solaris. To clarify: Before I submit a new version of metafor to CRAN, I check that it passes 'R CMD check --as-cran' on R-devel (both 64- and 32-bit), R-current (whatever version this is at the time), and the oldest version of R specified under 'Depends' (which is currently 3.1.0 for the metafor package) -- so 6 checks in total. All of these checks are run on Windows (yes, shoot me). Admittedly, I screwed up on some package tests (using the 'testthat' package) -- using a few is_identical_to() instead of expect_equivalent() comparisons -- which I shouldn't have done in the first place. But all of these passed the 6 checks that I did. Not so on Sparc Solaris though (and apparently also not on x86_64 Linux with --disable-long-double, but as far as I can tell, http://cran.r-project.org/web/checks/check_results_metafor.html does not indicate such a check). At any rate, I try to check the package thoroughly before submitting -- but clearly not thorough enough. While my mistake was easy to remedy, the more general question is this: Is there any possibility of checking packages on various architectures, and especially Sparc Solaris, before submitting? Also, I don't know how often it happens that packages pass the various flavors of R on Windows/OSX/Linux but not on Sparc Solaris, so maybe this is really not much of an issue. But I plan on adding additional checks to the package so that code coverage is much higher -- so is CRAN itself then the only place where I can end up running all checks? Best, Wolfgang -- Wolfgang Viechtbauer, Ph.D., Statistician Department of Psychiatry and Neuropsychology School for Mental Health and Neuroscience Faculty of Health, Medicine, and Life Sciences Maastricht University, P.O. Box 616 (VIJV1) 6200 MD Maastricht, The Netherlands +31 (43) 388-4170 | http://www.wvbauer.com
[R-pkg-devel] CRAN check for package on Sparc Solaris (or other architectures)
5 messages · Viechtbauer Wolfgang (STAT), Ben Bolker, Dirk Eddelbuettel
For what it's worth I did install a Solaris VM on one of my machines so I could run tests. This does seem a little over the top as a requirement for all package developers though ... I have easy access to MacOS and Ubuntu so I usually test there and on win-builder as well. In my case, I eventually found that the warnings (related to some parallelized code) could be reproduced in batch mode but *not* in interactive mode on Solaris, so I just gave up and disabled the tests on Solaris via if (Sys.info()["sysname"] != "SunOS") (shame on me ...) A wish-list/discussion item for the CRAN maintainers would to be add a step to the submission process, in between pushing the button to confirm submission to CRAN and any CRAN maintainers having to look at the submission/interact with the submitter, would be for the CRAN results to get posted somewhere the submitter could see them. If there were errors or warnings the submission would be automatically rejected ... if there were notes the submitter would then have to add a comment/press another button justifying that it was OK to ignore them ... if the submission was completely clean it could automatically go on to the CRAN maintainers ... I don't know what the costs of this would be, beyond implementing it in the first place. It might encourage more load on the CRAN system by allowing developers to use it as a test rig (although the "automatically submit to CRAN if clean" behaviour would certainly discourage this), but it's hard for me to imagine that it wouldn't be worth it ... On Mon, May 25, 2015 at 6:13 PM, Viechtbauer Wolfgang (STAT)
<wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
Hello All, First of all, thanks to Martin, Hadley, and Dirk (and everybody else involved) for setting up this new mailing list. I am sure this will become a tremendous resource for package developers. My question is in essence just a repost of what I asked on R-devel a few weeks ago: https://stat.ethz.ch/pipermail/r-devel/2015-May/071147.html At the time, the 'metafor' package failed CRAN checks on Sparc Solaris. To clarify: Before I submit a new version of metafor to CRAN, I check that it passes 'R CMD check --as-cran' on R-devel (both 64- and 32-bit), R-current (whatever version this is at the time), and the oldest version of R specified under 'Depends' (which is currently 3.1.0 for the metafor package) -- so 6 checks in total. All of these checks are run on Windows (yes, shoot me). Admittedly, I screwed up on some package tests (using the 'testthat' package) -- using a few is_identical_to() instead of expect_equivalent() comparisons -- which I shouldn't have done in the first place. But all of these passed the 6 checks that I did. Not so on Sparc Solaris though (and apparently also not on x86_64 Linux with --disable-long-double, but as far as I can tell, http://cran.r-project.org/web/checks/check_results_metafor.html does not indicate such a check). At any rate, I try to check the package thoroughly before submitting -- but clearly not thorough enough. While my mistake was easy to remedy, the more general question is this: Is there any possibility of checking packages on various architectures, and especially Sparc Solaris, before submitting? Also, I don't know how often it happens that packages pass the various flavors of R on Windows/OSX/Linux but not on Sparc Solaris, so maybe this is really not much of an issue. But I plan on adding additional checks to the package so that code coverage is much higher -- so is CRAN itself then the only place where I can end up running all checks? Best, Wolfgang -- Wolfgang Viechtbauer, Ph.D., Statistician Department of Psychiatry and Neuropsychology School for Mental Health and Neuroscience Faculty of Health, Medicine, and Life Sciences Maastricht University, P.O. Box 616 (VIJV1) 6200 MD Maastricht, The Netherlands +31 (43) 388-4170 | http://www.wvbauer.com
______________________________________________ R-package-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Just went back and re-read the mailing list thread -- I think my VM is Solaris x86, so doesn't address the Sparc Solaris question anyway ...
On Mon, May 25, 2015 at 6:24 PM, Ben Bolker <bbolker at gmail.com> wrote:
For what it's worth I did install a Solaris VM on one of my machines so I could run tests. This does seem a little over the top as a requirement for all package developers though ... I have easy access to MacOS and Ubuntu so I usually test there and on win-builder as well. In my case, I eventually found that the warnings (related to some parallelized code) could be reproduced in batch mode but *not* in interactive mode on Solaris, so I just gave up and disabled the tests on Solaris via if (Sys.info()["sysname"] != "SunOS") (shame on me ...) A wish-list/discussion item for the CRAN maintainers would to be add a step to the submission process, in between pushing the button to confirm submission to CRAN and any CRAN maintainers having to look at the submission/interact with the submitter, would be for the CRAN results to get posted somewhere the submitter could see them. If there were errors or warnings the submission would be automatically rejected ... if there were notes the submitter would then have to add a comment/press another button justifying that it was OK to ignore them ... if the submission was completely clean it could automatically go on to the CRAN maintainers ... I don't know what the costs of this would be, beyond implementing it in the first place. It might encourage more load on the CRAN system by allowing developers to use it as a test rig (although the "automatically submit to CRAN if clean" behaviour would certainly discourage this), but it's hard for me to imagine that it wouldn't be worth it ... On Mon, May 25, 2015 at 6:13 PM, Viechtbauer Wolfgang (STAT) <wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
Hello All, First of all, thanks to Martin, Hadley, and Dirk (and everybody else involved) for setting up this new mailing list. I am sure this will become a tremendous resource for package developers. My question is in essence just a repost of what I asked on R-devel a few weeks ago: https://stat.ethz.ch/pipermail/r-devel/2015-May/071147.html At the time, the 'metafor' package failed CRAN checks on Sparc Solaris. To clarify: Before I submit a new version of metafor to CRAN, I check that it passes 'R CMD check --as-cran' on R-devel (both 64- and 32-bit), R-current (whatever version this is at the time), and the oldest version of R specified under 'Depends' (which is currently 3.1.0 for the metafor package) -- so 6 checks in total. All of these checks are run on Windows (yes, shoot me). Admittedly, I screwed up on some package tests (using the 'testthat' package) -- using a few is_identical_to() instead of expect_equivalent() comparisons -- which I shouldn't have done in the first place. But all of these passed the 6 checks that I did. Not so on Sparc Solaris though (and apparently also not on x86_64 Linux with --disable-long-double, but as far as I can tell, http://cran.r-project.org/web/checks/check_results_metafor.html does not indicate such a check). At any rate, I try to check the package thoroughly before submitting -- but clearly not thorough enough. While my mistake was easy to remedy, the more general question is this: Is there any possibility of checking packages on various architectures, and especially Sparc Solaris, before submitting? Also, I don't know how often it happens that packages pass the various flavors of R on Windows/OSX/Linux but not on Sparc Solaris, so maybe this is really not much of an issue. But I plan on adding additional checks to the package so that code coverage is much higher -- so is CRAN itself then the only place where I can end up running all checks? Best, Wolfgang -- Wolfgang Viechtbauer, Ph.D., Statistician Department of Psychiatry and Neuropsychology School for Mental Health and Neuroscience Faculty of Health, Medicine, and Life Sciences Maastricht University, P.O. Box 616 (VIJV1) 6200 MD Maastricht, The Netherlands +31 (43) 388-4170 | http://www.wvbauer.com
______________________________________________ R-package-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
On 26 May 2015 at 00:13, Viechtbauer Wolfgang (STAT) wrote:
| Is there any possibility of checking packages on various architectures, and
| especially Sparc Solaris, before submitting?
Not that I know of (apart of course from win-builder, and related services
such as Travis CI and its kin) and at this point in time. Especially not for
that one OS we are all haunted by.
There {was, is, probably always will be} talk of better services. But none
have been built so far..
Dirk
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
Thanks for the reply, Ben. Excluding Solaris from (some of the) tests is of course one (rather brute-force) way of circumventing the issue. And in fact, if I hadn't been able to resolve the errors, I probably would have taken that shortcut as well. On the other hand, if this would become common practice, Sparc Solaris would slowly be denigrated to a sort-of second-class architecture/OS for R, which isn't ideal either. I really like your suggestion for adding a step to the submission process as you describe. That would be fantastic! And while it may take some effort to set something like this up initially, it may ultimately save CRAN maintainers time. I (together with another person whose package also failed some tests on Sparc Solaris) was contacted about the errors by Brian Ripley and it seemed to me that this was not an automated message. Again, I have no idea how common such issues are, but if the CRAN maintainers are writing such messages on a regular basis (and with close to 7000 packages on CRAN, this could very well be the case), it seems to me that it would make a lot of sense to set up the system so that package authors are able to spot and hopefully resolve such issues before manual intervention becomes necessary. Then again, it's easy for me to cheer for this suggestion. And as you say, there are some potential dangers here, as such a system could be apt to abuse. But indeed, the way you describe it would probably prevent this from happening in the first place. Also, one could consider only allowing a resubmission (of a package that initially failed some checks) only once per day or every x-many hours. At any rate, I think this is a great suggestion. Best, Wolfgang
-----Original Message----- From: Ben Bolker [mailto:bbolker at gmail.com] Sent: Tuesday, May 26, 2015 00:25 To: Viechtbauer Wolfgang (STAT) Cc: r-package-devel at r-project.org Subject: Re: [R-pkg-devel] CRAN check for package on Sparc Solaris (or other architectures) For what it's worth I did install a Solaris VM on one of my machines so I could run tests. This does seem a little over the top as a requirement for all package developers though ... I have easy access to MacOS and Ubuntu so I usually test there and on win-builder as well. In my case, I eventually found that the warnings (related to some parallelized code) could be reproduced in batch mode but *not* in interactive mode on Solaris, so I just gave up and disabled the tests on Solaris via if (Sys.info()["sysname"] != "SunOS") (shame on me ...) A wish-list/discussion item for the CRAN maintainers would to be add a step to the submission process, in between pushing the button to confirm submission to CRAN and any CRAN maintainers having to look at the submission/interact with the submitter, would be for the CRAN results to get posted somewhere the submitter could see them. If there were errors or warnings the submission would be automatically rejected ... if there were notes the submitter would then have to add a comment/press another button justifying that it was OK to ignore them ... if the submission was completely clean it could automatically go on to the CRAN maintainers ... I don't know what the costs of this would be, beyond implementing it in the first place. It might encourage more load on the CRAN system by allowing developers to use it as a test rig (although the "automatically submit to CRAN if clean" behaviour would certainly discourage this), but it's hard for me to imagine that it wouldn't be worth it ... On Mon, May 25, 2015 at 6:13 PM, Viechtbauer Wolfgang (STAT) <wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
Hello All, First of all, thanks to Martin, Hadley, and Dirk (and everybody else
involved) for setting up this new mailing list. I am sure this will become a tremendous resource for package developers.
My question is in essence just a repost of what I asked on R-devel a
few weeks ago:
https://stat.ethz.ch/pipermail/r-devel/2015-May/071147.html At the time, the 'metafor' package failed CRAN checks on Sparc Solaris. To clarify: Before I submit a new version of metafor to CRAN, I check
that it passes 'R CMD check --as-cran' on R-devel (both 64- and 32-bit), R-current (whatever version this is at the time), and the oldest version of R specified under 'Depends' (which is currently 3.1.0 for the metafor package) -- so 6 checks in total. All of these checks are run on Windows (yes, shoot me).
Admittedly, I screwed up on some package tests (using the 'testthat'
package) -- using a few is_identical_to() instead of expect_equivalent() comparisons -- which I shouldn't have done in the first place. But all of these passed the 6 checks that I did. Not so on Sparc Solaris though (and apparently also not on x86_64 Linux with --disable-long-double, but as far as I can tell, http://cran.r- project.org/web/checks/check_results_metafor.html does not indicate such a check).
At any rate, I try to check the package thoroughly before submitting --
but clearly not thorough enough. While my mistake was easy to remedy, the more general question is this: Is there any possibility of checking packages on various architectures, and especially Sparc Solaris, before submitting? Also, I don't know how often it happens that packages pass the various flavors of R on Windows/OSX/Linux but not on Sparc Solaris, so maybe this is really not much of an issue. But I plan on adding additional checks to the package so that code coverage is much higher -- so is CRAN itself then the only place where I can end up running all checks?
Best, Wolfgang -- Wolfgang Viechtbauer, Ph.D., Statistician Department of Psychiatry and Neuropsychology School for Mental Health and Neuroscience Faculty of Health, Medicine, and Life Sciences Maastricht University, P.O. Box 616 (VIJV1) 6200 MD Maastricht, The Netherlands +31 (43) 388-4170 | http://www.wvbauer.com
______________________________________________ R-package-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel