[Bioc-devel] Data Package Size Issues (.idat and .rda)

On Fri, Nov 8, 2013 at 8:41 AM, Martin Morgan <mtmorgan at fhcrc.org> wrote:

On 11/07/2013 09:26 PM, Nicolas De Jay wrote:

Thanks for the prompt answer.  The data set I am packaging closely
resembles that of minfiData except that there are 52 samples; the IDAT
files together are some 800MB whereas the Rda file is closer to 150MB.
  It is worth noting that my experiment data package will be submitted
to Bioconductor along with a software package which makes use of these
samples in the vignette.  With this in mind, can I omit the IDAT
files?  If this goes against Bioconductor's underlying design, what
would you say is the maximum size of an experiment data package?

Hi Nicolas -- Some things to bear in mind.

Hi, Nicolas.

I just wanted to note that experiment data packages are meant as a
convenient way to distribute data so that reproducible workflows and
documentation can be created easily.  There are other options such as
accessing the data directly from public repositories using Bioconductor
tools that serve the same purpose.  While accessing such online resources
does necessitate a one-time network connection (after which packages like
GEOquery can use locally cached data), when appropriate datasets exist in
public repositories, it may be a perfectly viable alternative to experiment
data packages.  In this particular case, as of today in NCBI GEO, there are
1711 Human 450k samples with IDAT files available.  I am not arguing that
this route should replace experiment data packages, just that stable public
data resources are an alternative to them to consider.

Sean

Files are compressed in package tar balls, so your IDAT files may have a
considerably smaller effective size.

Generally, original text files are a much better way to store external
data than Rda files. For instance, rda files require updating when / if the
class definition changes, and the provenance and content of the data is
unambiguous.

Experiment data packages are meant to provide reusable examples for
pedagogic purposes. One would hope that minfiData fulfills this requirement.
If not, then it would be better to continue the current discussion with
Kasper and others in the community to identify an appropriately
comprehensive data set for use across many relevant packages.

There is no formal statement about the maximum size of experiment data
packages, but one would need to make a strong argument for why a Gb of
experiment data is necessary (including why existing experiment data
packages are fundamentally inadequate), especially if it is to support a
single package.

Martin

---
Nicolas De Jay

On Thu, Nov 7, 2013 at 9:38 PM, Kasper Daniel Hansen
<kasperdanielhansen at gmail.com> wrote:

To give some background: it is true that the RGsetEx object (in
data/RGsetEx.rda) is a 1-1 correspondence with the raw data files in
inst/extdata, so one could consider it redundant.  However, having the
IDAT
files are convenient for testing parsing, and also for other tools who
want
to have 450k example data and not want to depend on minfi.  Those are
the
two main reasons for including the raw data as well.  And then the fact
that
while the data size is "big" it is only 6 samples.

Best,
Kasper


On Thu, Nov 7, 2013 at 3:58 PM, Nicolas De Jay
<nicolas.dejay at mail.mcgill.ca> wrote:

Hi,

I am preparing a data package and using the minfiData package as a
reference.  The .idat files in extdata and the .rda file in data are
both present in both the compressed tarball source and the installed
copy directory (in my case, under ~/R/x86-64.../3.0/minfiData).  Isn't
this redundant?  Is there a way to have the prospective user only
download the .rda files?

Sorry if my question is misguided and thanks in advance for your help.

---
Nicolas De Jay
M.Sc. Student
Department of Human Genetics
Montreal Children's Hospital Research Institute, McGill University
Health
Centre
4060 Ste Catherine West, PT-239
Montreal, QC H3Z2Z3, Canada
T: (514) 412-4440 | E: nicolas.dejay at mail.mcgill.ca

--
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793