Hi, Recently I learned thanks to Martin Morgan that there are some files with the Date/Publication fields for Bioconductor packages: https://bioconductor.org/packages/3.7/bioc/VIEWS. I'm trying to reconstruct which packages from CRAN and Biocondctor were available at any moment and it was very helpful. However, these files have the latest version published by a package on a given Bioconductor release. Is there a way to know if there were more updates after a release? I thought about searching the git log for each package. But that wouldn't be enough, as they might have increased their version but not passed Bioconductor checks, and thus not be released. Related to this, this field is present from Bioconductor version 3.7 or later but I couldn't find it on previous releases. Is there a way to know previous packages' releases and their dates? Packages' updates on the release branch should on contain bug fixes, but for reproducibility purposes it might be necessary to get the same bugs again. Many thanks in advance, Llu?s
[Bioc-devel] Bioconductor archive?
5 messages · Lluís Revilla, Shepherd, Lori, Sean Davis
2 days later
It looks like the Date/Publication field is only present when there was a change on the branch post release. (ie. any package that has a version x.y.(z+n) instead of x.y.0. After a release is frozen and a new release occurs, Bioconductor does not allow any changes or fixes even to bugs. A release is frozen so there is no changes after the new release occurs. I would have to dig in the history but my guess is 3.7 might be when we either switched to git or started having archived versions so likely not available before this date. Lori Shepherd - Kern Bioconductor Core Team Roswell Park Comprehensive Cancer Center Department of Biostatistics & Bioinformatics Elm & Carlton Streets Buffalo, New York 14263
From: Bioc-devel <bioc-devel-bounces at r-project.org> on behalf of Llu?s Revilla <lluis.revilla at gmail.com>
Sent: Saturday, March 15, 2025 5:20 AM
To: bioc-devel <bioc-devel at r-project.org>
Subject: [Bioc-devel] Bioconductor archive?
Sent: Saturday, March 15, 2025 5:20 AM
To: bioc-devel <bioc-devel at r-project.org>
Subject: [Bioc-devel] Bioconductor archive?
Hi, Recently I learned thanks to Martin Morgan that there are some files with the Date/Publication fields for Bioconductor packages: https://secure-web.cisco.com/1WmVHwH9-fASq-_cRqjzutLif_scf2tV0oia7j9wcAlmEkD6LTfPr4hpDabt4CAjYBdFcUrtqQXG2zbH0HakIsmTnqgnHUbghB0qC_b3FyGAhL5dnDBbz1Oh7HlpVwyPV79vgW7FMsg__zeInCyPb_jmFBXAvFRuq-HsBLTAC-Bf2EfgTjG3y38kBOIGnb59DWA6ILkuC-oYK0RJe8h3JvV5RoaeA9FxDk6QokHUT-YeC7hIEd_hURH1dV0dKbJN717qRcgwyT42SNb1evj91AQrxGnEyIR2XFpm28A-qOih3N2V_YsWsZd0wzGApXcZy/https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.7%2Fbioc%2FVIEWS I'm trying to reconstruct which packages from CRAN and Biocondctor were available at any moment and it was very helpful. However, these files have the latest version published by a package on a given Bioconductor release. Is there a way to know if there were more updates after a release? I thought about searching the git log for each package. But that wouldn't be enough, as they might have increased their version but not passed Bioconductor checks, and thus not be released. Related to this, this field is present from Bioconductor version 3.7 or later but I couldn't find it on previous releases. Is there a way to know previous packages' releases and their dates? Packages' updates on the release branch should on contain bug fixes, but for reproducibility purposes it might be necessary to get the same bugs again. Many thanks in advance, Llu?s [[alternative HTML version deleted]] _______________________________________________ Bioc-devel at r-project.org mailing list https://secure-web.cisco.com/13SnGNaaDyFbctEb1TdAguAxRDGWtUJvQINgKyoWwg8r1Kce77xQNycHZxQSYbLF7m6L2z5y7dVIwm3y-9U1nxiyuzrQxuIQZc5HoTMPvbokKA1qJHn3CCb-Zlx3gtXWIW2VtFh_7loh_SYeLpi5ak38PFBFkLutgGFEwFhXbr0EFIo2W8HRtaqFNH9_U-hcBauAVzEJOJV9rFuxZom3twTGLLjMzaXn7ZhRdcG56Z_sAM0lzgdFeTgepY4mN7XAUwqNMoSSwjIeL10YspawZ6fy_yXLfIysgSN1DpVVdzc9Pv7GHlPjj7-EVYr-ScNbg/https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fbioc-devel This email message may contain legally privileged and/or confidential information. If you are not the intended recipient(s), or the employee or agent responsible for the delivery of this message to the intended recipient(s), you are hereby notified that any disclosure, copying, distribution, or use of this email message is prohibited. If you have received this message in error, please notify the sender immediately by e-mail and delete this email message from your computer. Thank you.
2 days later
Hi Lori, Many thanks for your answer. I have a couple of follow-up questions.
It looks like the Date/Publication field is only present when there was a change on the branch post release. (ie. any package that has a version x.y.(z+n) instead of x.y.0. After a release is frozen and a new release occurs, Bioconductor does not allow any changes or fixes even to bugs. A release is frozen so there is no changes after the new release occurs.
Thanks for reminding me of this. I'm interested on the x.y.z+n packages that were released on each release, not just the last one or the initial one. Is this historical information available? The file at https://bioconductor.org/packages/3.20/bioc/VIEWS only includes the latest date of a given release, but there could be a release within a given Bioconductor version before that.
I would have to dig in the history but my guess is 3.7 might be when we either switched to git or started having archived versions so likely not available before this date.
I thought it would be difficult if not impossible to check this but even for the current release I can't find this data. Does Bioconductor have an internal archive with this information? On CRAN even if it removes a package internally the activities of the archive are stored: each date-time of publication, archive and removal. Does something similar happen in Bioconductor? Even if a given package is not available knowing that there was a release could be helpful for reproducibility (as it could be used to compare with the git log). With that information finding which package versions were used for a script with only a date could become easier. Best, Llu?s
Lori Shepherd - Kern Bioconductor Core Team Roswell Park Comprehensive Cancer Center Department of Biostatistics & Bioinformatics Elm & Carlton Streets Buffalo, New York 14263
________________________________ From: Bioc-devel <bioc-devel-bounces at r-project.org> on behalf of Llu?s Revilla <lluis.revilla at gmail.com> Sent: Saturday, March 15, 2025 5:20 AM To: bioc-devel <bioc-devel at r-project.org> Subject: [Bioc-devel] Bioconductor archive? Hi, Recently I learned thanks to Martin Morgan that there are some files with the Date/Publication fields for Bioconductor packages: https://secure-web.cisco.com/1WmVHwH9-fASq-_cRqjzutLif_scf2tV0oia7j9wcAlmEkD6LTfPr4hpDabt4CAjYBdFcUrtqQXG2zbH0HakIsmTnqgnHUbghB0qC_b3FyGAhL5dnDBbz1Oh7HlpVwyPV79vgW7FMsg__zeInCyPb_jmFBXAvFRuq-HsBLTAC-Bf2EfgTjG3y38kBOIGnb59DWA6ILkuC-oYK0RJe8h3JvV5RoaeA9FxDk6QokHUT-YeC7hIEd_hURH1dV0dKbJN717qRcgwyT42SNb1evj91AQrxGnEyIR2XFpm28A-qOih3N2V_YsWsZd0wzGApXcZy/https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.7%2Fbioc%2FVIEWS I'm trying to reconstruct which packages from CRAN and Biocondctor were available at any moment and it was very helpful. However, these files have the latest version published by a package on a given Bioconductor release. Is there a way to know if there were more updates after a release? I thought about searching the git log for each package. But that wouldn't be enough, as they might have increased their version but not passed Bioconductor checks, and thus not be released. Related to this, this field is present from Bioconductor version 3.7 or later but I couldn't find it on previous releases. Is there a way to know previous packages' releases and their dates? Packages' updates on the release branch should on contain bug fixes, but for reproducibility purposes it might be necessary to get the same bugs again. Many thanks in advance, Llu?s [[alternative HTML version deleted]] _______________________________________________ Bioc-devel at r-project.org mailing list https://secure-web.cisco.com/13SnGNaaDyFbctEb1TdAguAxRDGWtUJvQINgKyoWwg8r1Kce77xQNycHZxQSYbLF7m6L2z5y7dVIwm3y-9U1nxiyuzrQxuIQZc5HoTMPvbokKA1qJHn3CCb-Zlx3gtXWIW2VtFh_7loh_SYeLpi5ak38PFBFkLutgGFEwFhXbr0EFIo2W8HRtaqFNH9_U-hcBauAVzEJOJV9rFuxZom3twTGLLjMzaXn7ZhRdcG56Z_sAM0lzgdFeTgepY4mN7XAUwqNMoSSwjIeL10YspawZ6fy_yXLfIysgSN1DpVVdzc9Pv7GHlPjj7-EVYr-ScNbg/https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fbioc-devel This email message may contain legally privileged and/or confidential information. If you are not the intended recipient(s), or the employee or agent responsible for the delivery of this message to the intended recipient(s), you are hereby notified that any disclosure, copying, distribution, or use of this email message is prohibited. If you have received this message in error, please notify the sender immediately by e-mail and delete this email message from your computer. Thank you.
Hi, all. Perhaps a bit tangential, but I capture the results of all build reports for all packages daily (that is the intent, anyway) going back a year or so (a couple of years if we dig into archives). The reports are processed using code in this repo: https://github.com/seandavi/BiocBuildDB using a github action<https://github.com/seandavi/BiocBuildDB/actions/workflows/process_new_build_reports.yaml> that runs daily. This might not be exactly the format you are looking for, Lluis, but it does have a complete history of every build for every package for every day for all Bioc builds. The result is a set of three CSV files (one set for every build, about 3.5k CSV files right now) with rows for each package/machine/build step and the results of the build, including propagation status (whether the package gets pushed to release). Version numbers, git hashes, dates, Bioconductor versions, build commands, error logs, etc. are all captured. Thus, things like full text search over captured log output is possible over time, across branches, and across machines or packages. When a package enters the system is captured. The build_summary table currently checks in at about 6M rows (again, without going into archive data) and adds about 20k rows per day. I have pending issues<https://github.com/seandavi/BiocBuildDB/issues> to expose the data but just haven?t prioritized the work. I?m happy to discuss access and use cases either in a new thread here, on Slack, or via github issues. Sean From: Bioc-devel <bioc-devel-bounces at r-project.org> on behalf of Llu?s Revilla <lluis.revilla at gmail.com> Date: Wednesday, March 19, 2025 at 6:21?PM To: Kern, Lori <Lori.Shepherd at roswellpark.org> Cc: bioc-devel <bioc-devel at r-project.org> Subject: Re: [Bioc-devel] Bioconductor archive? Hi Lori, Many thanks for your answer. I have a couple of follow-up questions.
It looks like the Date/Publication field is only present when there was a change on the branch post release. (ie. any package that has a version x.y.(z+n) instead of x.y.0. After a release is frozen and a new release occurs, Bioconductor does not allow any changes or fixes even to bugs. A release is frozen so there is no changes after the new release occurs.
Thanks for reminding me of this. I'm interested on the x.y.z+n packages that were released on each release, not just the last one or the initial one. Is this historical information available? The file at https://bioconductor.org/packages/3.20/bioc/VIEWS only includes the latest date of a given release, but there could be a release within a given Bioconductor version before that.
I would have to dig in the history but my guess is 3.7 might be when we either switched to git or started having archived versions so likely not available before this date.
I thought it would be difficult if not impossible to check this but even for the current release I can't find this data. Does Bioconductor have an internal archive with this information? On CRAN even if it removes a package internally the activities of the archive are stored: each date-time of publication, archive and removal. Does something similar happen in Bioconductor? Even if a given package is not available knowing that there was a release could be helpful for reproducibility (as it could be used to compare with the git log). With that information finding which package versions were used for a script with only a date could become easier. Best, Llu?s
Lori Shepherd - Kern Bioconductor Core Team Roswell Park Comprehensive Cancer Center Department of Biostatistics & Bioinformatics Elm & Carlton Streets Buffalo, New York 14263
________________________________ From: Bioc-devel <bioc-devel-bounces at r-project.org> on behalf of Llu?s Revilla <lluis.revilla at gmail.com> Sent: Saturday, March 15, 2025 5:20 AM To: bioc-devel <bioc-devel at r-project.org> Subject: [Bioc-devel] Bioconductor archive? Hi, Recently I learned thanks to Martin Morgan that there are some files with the Date/Publication fields for Bioconductor packages: https://secure-web.cisco.com/1WmVHwH9-fASq-_cRqjzutLif_scf2tV0oia7j9wcAlmEkD6LTfPr4hpDabt4CAjYBdFcUrtqQXG2zbH0HakIsmTnqgnHUbghB0qC_b3FyGAhL5dnDBbz1Oh7HlpVwyPV79vgW7FMsg__zeInCyPb_jmFBXAvFRuq-HsBLTAC-Bf2EfgTjG3y38kBOIGnb59DWA6ILkuC-oYK0RJe8h3JvV5RoaeA9FxDk6QokHUT-YeC7hIEd_hURH1dV0dKbJN717qRcgwyT42SNb1evj91AQrxGnEyIR2XFpm28A-qOih3N2V_YsWsZd0wzGApXcZy/https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.7%2Fbioc%2FVIEWS I'm trying to reconstruct which packages from CRAN and Biocondctor were available at any moment and it was very helpful. However, these files have the latest version published by a package on a given Bioconductor release. Is there a way to know if there were more updates after a release? I thought about searching the git log for each package. But that wouldn't be enough, as they might have increased their version but not passed Bioconductor checks, and thus not be released. Related to this, this field is present from Bioconductor version 3.7 or later but I couldn't find it on previous releases. Is there a way to know previous packages' releases and their dates? Packages' updates on the release branch should on contain bug fixes, but for reproducibility purposes it might be necessary to get the same bugs again. Many thanks in advance, Llu?s [[alternative HTML version deleted]] _______________________________________________ Bioc-devel at r-project.org mailing list https://secure-web.cisco.com/13SnGNaaDyFbctEb1TdAguAxRDGWtUJvQINgKyoWwg8r1Kce77xQNycHZxQSYbLF7m6L2z5y7dVIwm3y-9U1nxiyuzrQxuIQZc5HoTMPvbokKA1qJHn3CCb-Zlx3gtXWIW2VtFh_7loh_SYeLpi5ak38PFBFkLutgGFEwFhXbr0EFIo2W8HRtaqFNH9_U-hcBauAVzEJOJV9rFuxZom3twTGLLjMzaXn7ZhRdcG56Z_sAM0lzgdFeTgepY4mN7XAUwqNMoSSwjIeL10YspawZ6fy_yXLfIysgSN1DpVVdzc9Pv7GHlPjj7-EVYr-ScNbg/https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fbioc-devel This email message may contain legally privileged and/or confidential information. If you are not the intended recipient(s), or the employee or agent responsible for the delivery of this message to the intended recipient(s), you are hereby notified that any disclosure, copying, distribution, or use of this email message is prohibited. If you have received this message in error, please notify the sender immediately by e-mail and delete this email message from your computer. Thank you.
_______________________________________________ Bioc-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
Thanks Sean, This looks awesome! Many thanks for storing this. I'll see how I could process the data and might contact you off-list or via the issues on the repo. Just by the numbers reported I'm a bit surprised by the daily increment of the summary table. Bioconductor software has around 2000 packages, checked on 5 different machines, per 5 outputs (Install, build, check, bin, propagate) (which results on that order of magnitudes), but not all builds and checks are run everyday (now I cannot find the page where the frequency is reported). At the moment I won't use build and check reports but I might be interested in that later (I too collect general checks results from CRAN without the log files). In any case, I'll get in touch. Ideally, I would like to export/use this from a package, as I have done for CRAN via the repo.data package I'm building. Best wishes and many thanks, Llu?s
On Thu, 20 Mar 2025 at 02:56, Sean Davis <seandavi at gmail.com> wrote:
Hi, all. Perhaps a bit tangential, but I capture the results of all build reports for all packages daily (that is the intent, anyway) going back a year or so (a couple of years if we dig into archives). The reports are processed using code in this repo: https://github.com/seandavi/BiocBuildDB using a github action that runs daily. This might not be exactly the format you are looking for, Lluis, but it does have a complete history of every build for every package for every day for all Bioc builds. The result is a set of three CSV files (one set for every build, about 3.5k CSV files right now) with rows for each package/machine/build step and the results of the build, including propagation status (whether the package gets pushed to release). Version numbers, git hashes, dates, Bioconductor versions, build commands, error logs, etc. are all captured. Thus, things like full text search over captured log output is possible over time, across branches, and across machines or packages. When a package enters the system is captured. The build_summary table currently checks in at about 6M rows (again, without going into archive data) and adds about 20k rows per day. I have pending issues to expose the data but just haven?t prioritized the work. I?m happy to discuss access and use cases either in a new thread here, on Slack, or via github issues. Sean From: Bioc-devel <bioc-devel-bounces at r-project.org> on behalf of Llu?s Revilla <lluis.revilla at gmail.com> Date: Wednesday, March 19, 2025 at 6:21?PM To: Kern, Lori <Lori.Shepherd at roswellpark.org> Cc: bioc-devel <bioc-devel at r-project.org> Subject: Re: [Bioc-devel] Bioconductor archive? Hi Lori, Many thanks for your answer. I have a couple of follow-up questions.
It looks like the Date/Publication field is only present when there was a change on the branch post release. (ie. any package that has a version x.y.(z+n) instead of x.y.0. After a release is frozen and a new release occurs, Bioconductor does not allow any changes or fixes even to bugs. A release is frozen so there is no changes after the new release occurs.
Thanks for reminding me of this. I'm interested on the x.y.z+n packages that were released on each release, not just the last one or the initial one. Is this historical information available? The file at https://bioconductor.org/packages/3.20/bioc/VIEWS only includes the latest date of a given release, but there could be a release within a given Bioconductor version before that.
I would have to dig in the history but my guess is 3.7 might be when we either switched to git or started having archived versions so likely not available before this date.
I thought it would be difficult if not impossible to check this but even for the current release I can't find this data. Does Bioconductor have an internal archive with this information? On CRAN even if it removes a package internally the activities of the archive are stored: each date-time of publication, archive and removal. Does something similar happen in Bioconductor? Even if a given package is not available knowing that there was a release could be helpful for reproducibility (as it could be used to compare with the git log). With that information finding which package versions were used for a script with only a date could become easier. Best, Llu?s
Lori Shepherd - Kern Bioconductor Core Team Roswell Park Comprehensive Cancer Center Department of Biostatistics & Bioinformatics Elm & Carlton Streets Buffalo, New York 14263
________________________________ From: Bioc-devel <bioc-devel-bounces at r-project.org> on behalf of Llu?s Revilla <lluis.revilla at gmail.com> Sent: Saturday, March 15, 2025 5:20 AM To: bioc-devel <bioc-devel at r-project.org> Subject: [Bioc-devel] Bioconductor archive? Hi, Recently I learned thanks to Martin Morgan that there are some files with the Date/Publication fields for Bioconductor packages: https://secure-web.cisco.com/1WmVHwH9-fASq-_cRqjzutLif_scf2tV0oia7j9wcAlmEkD6LTfPr4hpDabt4CAjYBdFcUrtqQXG2zbH0HakIsmTnqgnHUbghB0qC_b3FyGAhL5dnDBbz1Oh7HlpVwyPV79vgW7FMsg__zeInCyPb_jmFBXAvFRuq-HsBLTAC-Bf2EfgTjG3y38kBOIGnb59DWA6ILkuC-oYK0RJe8h3JvV5RoaeA9FxDk6QokHUT-YeC7hIEd_hURH1dV0dKbJN717qRcgwyT42SNb1evj91AQrxGnEyIR2XFpm28A-qOih3N2V_YsWsZd0wzGApXcZy/https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.7%2Fbioc%2FVIEWS I'm trying to reconstruct which packages from CRAN and Biocondctor were available at any moment and it was very helpful. However, these files have the latest version published by a package on a given Bioconductor release. Is there a way to know if there were more updates after a release? I thought about searching the git log for each package. But that wouldn't be enough, as they might have increased their version but not passed Bioconductor checks, and thus not be released. Related to this, this field is present from Bioconductor version 3.7 or later but I couldn't find it on previous releases. Is there a way to know previous packages' releases and their dates? Packages' updates on the release branch should on contain bug fixes, but for reproducibility purposes it might be necessary to get the same bugs again. Many thanks in advance, Llu?s [[alternative HTML version deleted]] _______________________________________________ Bioc-devel at r-project.org mailing list https://secure-web.cisco.com/13SnGNaaDyFbctEb1TdAguAxRDGWtUJvQINgKyoWwg8r1Kce77xQNycHZxQSYbLF7m6L2z5y7dVIwm3y-9U1nxiyuzrQxuIQZc5HoTMPvbokKA1qJHn3CCb-Zlx3gtXWIW2VtFh_7loh_SYeLpi5ak38PFBFkLutgGFEwFhXbr0EFIo2W8HRtaqFNH9_U-hcBauAVzEJOJV9rFuxZom3twTGLLjMzaXn7ZhRdcG56Z_sAM0lzgdFeTgepY4mN7XAUwqNMoSSwjIeL10YspawZ6fy_yXLfIysgSN1DpVVdzc9Pv7GHlPjj7-EVYr-ScNbg/https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fbioc-devel This email message may contain legally privileged and/or confidential information. If you are not the intended recipient(s), or the employee or agent responsible for the delivery of this message to the intended recipient(s), you are hereby notified that any disclosure, copying, distribution, or use of this email message is prohibited. If you have received this message in error, please notify the sender immediately by e-mail and delete this email message from your computer. Thank you.
_______________________________________________ Bioc-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel