[Bioc-devel] Reading and storing single cell ATAC data

Hi Caleb,
Hopefully Herve will chime in regarding SummarizedExperiment, but yes, I
think you can and should inherit from that. The `assays` slot must be an
object of type `Assays`, but that does appear to include a sparse Matrix.
See the comments at the top of Assays-class.R in the tarball for
SummarizedExperiment.  For example:

library(SummarizedExperiment)
library(Matrix)
library(GenomicRanges)
Nrow=1e6
Ncol=1e4
assay=Matrix::Matrix(0, nrow=Nrow, ncol=Ncol, sparse=TRUE)
gr <- GRanges(Rle("chr2", Nrow),
              IRanges(seq_len(Nrow), width=10))
se <- SummarizedExperiment(assays=assay, rowRanges=gr)

As far as out-of-core storage of sparse matrices, I do not know of any good
(portable) solutions.   If it makes more sense to chunk the matrix along
some dimension, you could always pickle the chunked, (sparse) Matrix
objects. In my experience, the decision to adopt sparse vs out-of-core
dense arrays has often required empirical testing to determine what is
fastest/most scalable, since you lose caching benefits from sequential
memory access once you go sparse.  I know there has been talk of extending
SummarizedExperiment to easily permit the Assays to be hdf5-based.

Is disk space really going to be a limiting factor? If so, then you will
probably be IO-bound, so you will need to distribute the data across
computing nodes for your analysis to scale anyways, which suggests some
sort of map-reduce formalism.  Which to my knowledge no one has considered
yet in Bioconductor.  But unless you are generating > 1 TB of
semi-processed data, maybe you don't need to go there?

-Andrew

Message: 3
Date: Fri, 23 Sep 2016 17:37:04 -0400
From: Caleb Lareau <caleblareau at g.harvard.edu>
To: bioc-devel at r-project.org
Subject: [Bioc-devel] Reading and storing single cell ATAC data
Message-ID: <83B8A778-4DAA-4A16-9DA8-22DEF9AD1252 at g.harvard.edu>
Content-Type: text/plain; charset="UTF-8"

Hi everyone?

I?m working with a team that?s generating single cell ATAC data in large
amounts and am designing the framework of an S4 object to facilitate
analyses in R. I have a couple of high-level questions that I wanted to
pose early to hopefully attain some community guidance in the
implementation of these data structures.


Question on S4 scATAC Structure--
It?s easy to imagine scATAC data as a matrix where the rows are particular
peaks and the columns are individual samples. We already have such an
impressive volume of data, such that if stored in an ordinary matrix, we
run into ~20 GB objects. As these data are very sparse, we store the peak
values in a sparse matrix (through the Matrix library). I wanted to collate
the peak information (probably in GRanges object) and sample information
(in a data frame) as well as some potential meta data in an S4 object.

Easy enough, sure, but after looking at the scRNA structure (e.g. scater <
https://bioconductor.org/packages/devel/bioc/vignettes/
scater/inst/doc/vignette.html>), I feel like I should be considering how
to inherit some of the nice properties from the canonical `ExpressionSet`
structure. However, since my constraints aren?t directly compatible (namely
the featureData slot really needs to be a GRanges and the exprs slot must
be an object from Matrix), it wasn?t clear to me how to maximize the
inheritance properties while adjusting to my unique constraints. Also, it
wasn?t clear to me whether or not I could inherent `SummarizedExperiment`
due to the different nature of the sparse matrix. Does anyone have any
advice on this structure?


Question on reading sparse matrices from disk--
I?m trying to work out the best to selectively read certain rows and
columns from a sparse matrix on disk into memory. I anticipate a time
fairly soon that loading our full scATAC data, even in sparse matrices, is
going to be untenable. Any matrix reading/slicing implementations that I?ve
seen don?t play friendly with sparse matrices. So, I hacked together two
solutions? 1) reads and subsets a gzipped matrix with 3 columns (row index;
column index; non-zero value) through a system call to awk. 2) converts
that same 3 column matrix into an SQLite object and send queries to read
values based on indices. The hiccups are that 1) doesn?t play friendly on
non-unix platforms and always scans the full file, and 2) is faster for
querying, but the binary object is ~7x larger than the gzipped object. I?ve
played around with hdf5 as well, but it didn?t seem to give me much back in
terms of speed or storage benefits comparatively. Has anyone found an
implementation that achieves a decent!
  lookup time and compression, or am I essentially needing to choose
between the two?



Thanks and have a great weekend!
-Caleb
        [[alternative HTML version deleted]]



------------------------------

Subject: Digest Footer

	[[alternative HTML version deleted]]