Hi, I am preparing a data package and using the minfiData package as a reference. The .idat files in extdata and the .rda file in data are both present in both the compressed tarball source and the installed copy directory (in my case, under ~/R/x86-64.../3.0/minfiData). Isn't this redundant? Is there a way to have the prospective user only download the .rda files? Sorry if my question is misguided and thanks in advance for your help. --- Nicolas De Jay M.Sc. Student Department of Human Genetics Montreal Children's Hospital Research Institute, McGill University Health Centre 4060 Ste Catherine West, PT-239 Montreal, QC H3Z2Z3, Canada T: (514) 412-4440 | E: nicolas.dejay at mail.mcgill.ca
[Bioc-devel] Data Package Size Issues (.idat and .rda)
8 messages · Nicolas De Jay, Martin Morgan, Sean Davis +1 more
An embedded and charset-unspecified text was scrubbed... Name: not available URL: <https://stat.ethz.ch/pipermail/bioc-devel/attachments/20131107/029c5f99/attachment.pl>
Thanks for the prompt answer. The data set I am packaging closely resembles that of minfiData except that there are 52 samples; the IDAT files together are some 800MB whereas the Rda file is closer to 150MB. It is worth noting that my experiment data package will be submitted to Bioconductor along with a software package which makes use of these samples in the vignette. With this in mind, can I omit the IDAT files? If this goes against Bioconductor's underlying design, what would you say is the maximum size of an experiment data package? --- Nicolas De Jay On Thu, Nov 7, 2013 at 9:38 PM, Kasper Daniel Hansen
<kasperdanielhansen at gmail.com> wrote:
To give some background: it is true that the RGsetEx object (in data/RGsetEx.rda) is a 1-1 correspondence with the raw data files in inst/extdata, so one could consider it redundant. However, having the IDAT files are convenient for testing parsing, and also for other tools who want to have 450k example data and not want to depend on minfi. Those are the two main reasons for including the raw data as well. And then the fact that while the data size is "big" it is only 6 samples. Best, Kasper On Thu, Nov 7, 2013 at 3:58 PM, Nicolas De Jay <nicolas.dejay at mail.mcgill.ca> wrote:
Hi, I am preparing a data package and using the minfiData package as a reference. The .idat files in extdata and the .rda file in data are both present in both the compressed tarball source and the installed copy directory (in my case, under ~/R/x86-64.../3.0/minfiData). Isn't this redundant? Is there a way to have the prospective user only download the .rda files? Sorry if my question is misguided and thanks in advance for your help. --- Nicolas De Jay M.Sc. Student Department of Human Genetics Montreal Children's Hospital Research Institute, McGill University Health Centre 4060 Ste Catherine West, PT-239 Montreal, QC H3Z2Z3, Canada T: (514) 412-4440 | E: nicolas.dejay at mail.mcgill.ca
_______________________________________________ Bioc-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
On 11/07/2013 09:26 PM, Nicolas De Jay wrote:
Thanks for the prompt answer. The data set I am packaging closely resembles that of minfiData except that there are 52 samples; the IDAT files together are some 800MB whereas the Rda file is closer to 150MB. It is worth noting that my experiment data package will be submitted to Bioconductor along with a software package which makes use of these samples in the vignette. With this in mind, can I omit the IDAT files? If this goes against Bioconductor's underlying design, what would you say is the maximum size of an experiment data package?
Hi Nicolas -- Some things to bear in mind. Files are compressed in package tar balls, so your IDAT files may have a considerably smaller effective size. Generally, original text files are a much better way to store external data than Rda files. For instance, rda files require updating when / if the class definition changes, and the provenance and content of the data is unambiguous. Experiment data packages are meant to provide reusable examples for pedagogic purposes. One would hope that minfiData fulfills this requirement. If not, then it would be better to continue the current discussion with Kasper and others in the community to identify an appropriately comprehensive data set for use across many relevant packages. There is no formal statement about the maximum size of experiment data packages, but one would need to make a strong argument for why a Gb of experiment data is necessary (including why existing experiment data packages are fundamentally inadequate), especially if it is to support a single package. Martin
--- Nicolas De Jay On Thu, Nov 7, 2013 at 9:38 PM, Kasper Daniel Hansen <kasperdanielhansen at gmail.com> wrote:
To give some background: it is true that the RGsetEx object (in data/RGsetEx.rda) is a 1-1 correspondence with the raw data files in inst/extdata, so one could consider it redundant. However, having the IDAT files are convenient for testing parsing, and also for other tools who want to have 450k example data and not want to depend on minfi. Those are the two main reasons for including the raw data as well. And then the fact that while the data size is "big" it is only 6 samples. Best, Kasper On Thu, Nov 7, 2013 at 3:58 PM, Nicolas De Jay <nicolas.dejay at mail.mcgill.ca> wrote:
Hi, I am preparing a data package and using the minfiData package as a reference. The .idat files in extdata and the .rda file in data are both present in both the compressed tarball source and the installed copy directory (in my case, under ~/R/x86-64.../3.0/minfiData). Isn't this redundant? Is there a way to have the prospective user only download the .rda files? Sorry if my question is misguided and thanks in advance for your help. --- Nicolas De Jay M.Sc. Student Department of Human Genetics Montreal Children's Hospital Research Institute, McGill University Health Centre 4060 Ste Catherine West, PT-239 Montreal, QC H3Z2Z3, Canada T: (514) 412-4440 | E: nicolas.dejay at mail.mcgill.ca
_______________________________________________ Bioc-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
_______________________________________________ Bioc-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
Computational Biology / Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N. PO Box 19024 Seattle, WA 98109 Location: Arnold Building M1 B861 Phone: (206) 667-2793
An embedded and charset-unspecified text was scrubbed... Name: not available URL: <https://stat.ethz.ch/pipermail/bioc-devel/attachments/20131108/6f9f63a7/attachment.pl>
In that case, I will try to see if the public databases have the kind of data sets I am trying to package and run the idea by the team that is assigned to the project I am developing. Thank you Martin, Sean and Kasper for your valuable insight! --- Nicolas De Jay
On Fri, Nov 8, 2013 at 9:07 AM, Sean Davis <sdavis2 at mail.nih.gov> wrote:
On Fri, Nov 8, 2013 at 8:41 AM, Martin Morgan <mtmorgan at fhcrc.org> wrote:
On 11/07/2013 09:26 PM, Nicolas De Jay wrote:
Thanks for the prompt answer. The data set I am packaging closely resembles that of minfiData except that there are 52 samples; the IDAT files together are some 800MB whereas the Rda file is closer to 150MB. It is worth noting that my experiment data package will be submitted to Bioconductor along with a software package which makes use of these samples in the vignette. With this in mind, can I omit the IDAT files? If this goes against Bioconductor's underlying design, what would you say is the maximum size of an experiment data package?
Hi Nicolas -- Some things to bear in mind.
Hi, Nicolas. I just wanted to note that experiment data packages are meant as a convenient way to distribute data so that reproducible workflows and documentation can be created easily. There are other options such as accessing the data directly from public repositories using Bioconductor tools that serve the same purpose. While accessing such online resources does necessitate a one-time network connection (after which packages like GEOquery can use locally cached data), when appropriate datasets exist in public repositories, it may be a perfectly viable alternative to experiment data packages. In this particular case, as of today in NCBI GEO, there are 1711 Human 450k samples with IDAT files available. I am not arguing that this route should replace experiment data packages, just that stable public data resources are an alternative to them to consider. Sean
Files are compressed in package tar balls, so your IDAT files may have a considerably smaller effective size. Generally, original text files are a much better way to store external data than Rda files. For instance, rda files require updating when / if the class definition changes, and the provenance and content of the data is unambiguous. Experiment data packages are meant to provide reusable examples for pedagogic purposes. One would hope that minfiData fulfills this requirement. If not, then it would be better to continue the current discussion with Kasper and others in the community to identify an appropriately comprehensive data set for use across many relevant packages. There is no formal statement about the maximum size of experiment data packages, but one would need to make a strong argument for why a Gb of experiment data is necessary (including why existing experiment data packages are fundamentally inadequate), especially if it is to support a single package. Martin
--- Nicolas De Jay On Thu, Nov 7, 2013 at 9:38 PM, Kasper Daniel Hansen <kasperdanielhansen at gmail.com> wrote:
To give some background: it is true that the RGsetEx object (in data/RGsetEx.rda) is a 1-1 correspondence with the raw data files in inst/extdata, so one could consider it redundant. However, having the IDAT files are convenient for testing parsing, and also for other tools who want to have 450k example data and not want to depend on minfi. Those are the two main reasons for including the raw data as well. And then the fact that while the data size is "big" it is only 6 samples. Best, Kasper On Thu, Nov 7, 2013 at 3:58 PM, Nicolas De Jay <nicolas.dejay at mail.mcgill.ca> wrote:
Hi, I am preparing a data package and using the minfiData package as a reference. The .idat files in extdata and the .rda file in data are both present in both the compressed tarball source and the installed copy directory (in my case, under ~/R/x86-64.../3.0/minfiData). Isn't this redundant? Is there a way to have the prospective user only download the .rda files? Sorry if my question is misguided and thanks in advance for your help. --- Nicolas De Jay M.Sc. Student Department of Human Genetics Montreal Children's Hospital Research Institute, McGill University Health Centre 4060 Ste Catherine West, PT-239 Montreal, QC H3Z2Z3, Canada T: (514) 412-4440 | E: nicolas.dejay at mail.mcgill.ca
_______________________________________________ Bioc-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
_______________________________________________ Bioc-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
-- Computational Biology / Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N. PO Box 19024 Seattle, WA 98109 Location: Arnold Building M1 B861 Phone: (206) 667-2793
_______________________________________________ Bioc-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
An embedded and charset-unspecified text was scrubbed... Name: not available URL: <https://stat.ethz.ch/pipermail/bioc-devel/attachments/20131108/4c314595/attachment.pl>
An embedded and charset-unspecified text was scrubbed... Name: not available URL: <https://stat.ethz.ch/pipermail/bioc-devel/attachments/20131108/69eb6832/attachment.pl>