[Bioc-devel] Reading and storing single cell ATAC data

Message: 3
Date: Fri, 23 Sep 2016 17:37:04 -0400
From: Caleb Lareau <caleblareau at g.harvard.edu>
To: bioc-devel at r-project.org
Subject: [Bioc-devel] Reading and storing single cell ATAC data
Message-ID: <83B8A778-4DAA-4A16-9DA8-22DEF9AD1252 at g.harvard.edu>
Content-Type: text/plain; charset="UTF-8"

Hi everyone?

I?m working with a team that?s generating single cell ATAC data in large
amounts and am designing the framework of an S4 object to facilitate
analyses in R. I have a couple of high-level questions that I wanted to
pose early to hopefully attain some community guidance in the
implementation of these data structures.


Question on S4 scATAC Structure--
It?s easy to imagine scATAC data as a matrix where the rows are particular
peaks and the columns are individual samples. We already have such an
impressive volume of data, such that if stored in an ordinary matrix, we
run into ~20 GB objects. As these data are very sparse, we store the peak
values in a sparse matrix (through the Matrix library). I wanted to collate
the peak information (probably in GRanges object) and sample information
(in a data frame) as well as some potential meta data in an S4 object.

Easy enough, sure, but after looking at the scRNA structure (e.g. scater <
https://bioconductor.org/packages/devel/bioc/vignettes/
scater/inst/doc/vignette.html>), I feel like I should be considering how
to inherit some of the nice properties from the canonical `ExpressionSet`
structure. However, since my constraints aren?t directly compatible (namely
the featureData slot really needs to be a GRanges and the exprs slot must
be an object from Matrix), it wasn?t clear to me how to maximize the
inheritance properties while adjusting to my unique constraints. Also, it
wasn?t clear to me whether or not I could inherent `SummarizedExperiment`
due to the different nature of the sparse matrix. Does anyone have any
advice on this structure?


Question on reading sparse matrices from disk--
I?m trying to work out the best to selectively read certain rows and
columns from a sparse matrix on disk into memory. I anticipate a time
fairly soon that loading our full scATAC data, even in sparse matrices, is
going to be untenable. Any matrix reading/slicing implementations that I?ve
seen don?t play friendly with sparse matrices. So, I hacked together two
solutions? 1) reads and subsets a gzipped matrix with 3 columns (row index;
column index; non-zero value) through a system call to awk. 2) converts
that same 3 column matrix into an SQLite object and send queries to read
values based on indices. The hiccups are that 1) doesn?t play friendly on
non-unix platforms and always scans the full file, and 2) is faster for
querying, but the binary object is ~7x larger than the gzipped object. I?ve
played around with hdf5 as well, but it didn?t seem to give me much back in
terms of speed or storage benefits comparatively. Has anyone found an
implementation that achieves a decent!
  lookup time and compression, or am I essentially needing to choose
between the two?



Thanks and have a great weekend!
-Caleb
        [[alternative HTML version deleted]]



------------------------------

Subject: Digest Footer