Skip to content

[Bioc-devel] Adding a new annotation file (size ~ 9MB) into the Bioconductor Rsubread package

7 messages · Yang Liao, Shepherd, Lori, Hervé Pagès

Original
Yang Liao
# 
Hi,

We are the maintainers of the Bioconductor Rsubread package. We are trying to add a new gene annotation file (RefSeq GRCm39/mm39) into the Rsubread package for the users who have switched to the mm39 reference genome for their analyses.

We have built the annotation file, but we found that it was a little too large (~ 9 MBytes), larger than the 5MB limit. Hence the Git command refused to submit the file to the Rsubread (devel) repository, with an error message : "Error: file larger than 5 Mb".

Is it possible if we can have an exemption to add the mm39 annotation file into the Rsubread package?

All the best,
Yang
Shepherd, Lori
# 
Exceptions to file size are not permitted. We would prefer the data be downloaded and distributed through the AnnotationHub as we are moving away from traditional data packages.

Please see HubPub and the vignette on how to create a hub package https://bioconductor.org/packages/devel/bioc/vignettes/HubPub/inst/doc/CreateAHubPackage.html

It this case, since it is a single file and creating an entirely separate annotation package seems over kill and unnecessary overhead, we would advise using the annotationhub directory in Rsubread.
You may choose to host the data file yourself on a public accessible and reliable server (institutional level, AWS bucket, data lakes, zenodo); private servers and hosting data on github are not allowed by Bioconductor standards.  If you are not able to host the data yourself, you may upload the data file to the Bioconductor Azure Data Lake as described in the vignette link above.
Minimally, Rsubread would need to add the metadata.csv file that provides the necessary metadata information in inst/extdata.  And add the biocViews term AnnotationHubSoftware.

Please let us know when these files and changes are available and we can further assist adding the data officially to the AnnotationHub.

Cheers,



Lori Shepherd - Kern

Bioconductor Core Team

Roswell Park Comprehensive Cancer Center

Department of Biostatistics & Bioinformatics

Elm & Carlton Streets

Buffalo, New York 14263
Hervé Pagès
# 
Also just a reminder that RefSeq exons for mm39 are already available 
thru the TxDb.Mmusculus.UCSC.mm39.refGene package:

 ? library(TxDb.Mmusculus.UCSC.mm39.refGene)

 ? txdb <- TxDb.Mmusculus.UCSC.mm39.refGene

 ? mm39_exons <- sort(unlist(exonsBy(txdb, by="gene")))

 ? mcols(mm39_exons) <- DataFrame(GeneID=names(mm39_exons))

 ? names(mm39_exons) <- NULL

 ? mm39_exons
 ? # GRanges object with 243976 ranges and 1 metadata column:
 ? #? ????????????????? seqnames????????? ranges strand | GeneID
 ? # ? ??????????????????? <Rle>?????? <IRanges> <Rle> | <character>
 ? #? ????? [1]???????????? chr1 4878046-4878205????? + | 18777
 ? #? ????? [2]???????????? chr1 4878678-4878709????? + | 18777
 ? #? ????? [3]???????????? chr1 4898807-4898872????? + | 18777
 ? #? ????? [4]???????????? chr1 4900491-4900538????? + | 18777
 ? #? ????? [5]???????????? chr1 4902534-4902604????? + | 18777
 ? #? ????? ...????????????? ...???????????? ...??? ... . ...
 ? # ? [243972] chrUn_JH584304v1???? 55112-55248????? - | 66776
 ? # ? [243973] chrUn_JH584304v1???? 55465-55701????? - | 66776
 ? # ? [243974] chrUn_JH584304v1???? 56986-57151????? - | 66776
 ? # ? [243975] chrUn_JH584304v1???? 58564-58835????? - | 66776
 ? # ? [243976] chrUn_JH584304v1???? 59592-59689????? - | 66776
 ? # ? -------
 ? # ? seqinfo: 61 sequences (1 circular) from mm39 genome

so there should be no need to add anything to AnnotationHub or to 
Rsubread itself.

Dump the exons to a tab-delimited file similar to 
Rsubread/inst/annot/mm10_RefSeq_exon.txt with:

 ? df <- as.data.frame(mm39_exons)

 ? df <- cbind(df[ , "GeneID", drop=FALSE], df[ , setdiff(colnames(df), 
c("GeneID", "width"))])

 ? stopifnot(identical(colnames(df), c("GeneID", "seqnames", "start", 
"end", "strand")))

 ? colnames(df) <- c("GeneID", "Chr", "Start", "End", "Strand")

 ? write.table(df, file="mm39_RefSeq_exon.txt", quote=FALSE, sep="\t", 
row.names=FALSE)

The entire process of obtaining the exons and dumping them to the file 
takes about 2 seconds on my labtop ;-)

H.
On 08/04/2022 06:00, Kern, Lori wrote:

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Yang Liao
    

    

      
      #
    

  


  

      
Thank you, Herv? and Lori!

Indeed, the RefSeq mm39 annotation is available in TxDB, but in our case, we built a special version that were specifically treated and tested for RNA-seq analysis, so we still hope to use the inbuilt version in Rsubread. Maybe we will use a public server to host the annotation files, or further compress it to fit the 5MB file limit.

Thanks again for the very detailed and timely answers, and the example code!

All the best,
Yang

From: Herv? Pag?s<mailto:hpages.on.github at gmail.com>
Sent: Saturday, 9 April 2022 2:21 AM
To: Kern, Lori<mailto:Lori.Shepherd at RoswellPark.org>; Yang Liao<mailto:Yang.Liao at onjcri.org.au>; bioc-devel at r-project.org<mailto:bioc-devel at r-project.org>
Subject: Re: [Bioc-devel] Adding a new annotation file (size ~ 9MB) into the Bioconductor Rsubread package

This message originated from outside your organisation. Please be careful while clicking links, opening attachments, or replying to this email.

Also just a reminder that RefSeq exons for mm39 are already available
thru the TxDb.Mmusculus.UCSC.mm39.refGene package:

  library(TxDb.Mmusculus.UCSC.mm39.refGene)

  txdb <- TxDb.Mmusculus.UCSC.mm39.refGene

  mm39_exons <- sort(unlist(exonsBy(txdb, by="gene")))

  mcols(mm39_exons) <- DataFrame(GeneID=names(mm39_exons))

  names(mm39_exons) <- NULL

  mm39_exons
  # GRanges object with 243976 ranges and 1 metadata column:
  #                    seqnames          ranges strand | GeneID
  #                       <Rle>       <IRanges> <Rle> | <character>
  #        [1]             chr1 4878046-4878205      + | 18777
  #        [2]             chr1 4878678-4878709      + | 18777
  #        [3]             chr1 4898807-4898872      + | 18777
  #        [4]             chr1 4900491-4900538      + | 18777
  #        [5]             chr1 4902534-4902604      + | 18777
  #        ...              ...             ...    ... . ...
  #   [243972] chrUn_JH584304v1     55112-55248      - | 66776
  #   [243973] chrUn_JH584304v1     55465-55701      - | 66776
  #   [243974] chrUn_JH584304v1     56986-57151      - | 66776
  #   [243975] chrUn_JH584304v1     58564-58835      - | 66776
  #   [243976] chrUn_JH584304v1     59592-59689      - | 66776
  #   -------
  #   seqinfo: 61 sequences (1 circular) from mm39 genome

so there should be no need to add anything to AnnotationHub or to
Rsubread itself.

Dump the exons to a tab-delimited file similar to
Rsubread/inst/annot/mm10_RefSeq_exon.txt with:

  df <- as.data.frame(mm39_exons)

  df <- cbind(df[ , "GeneID", drop=FALSE], df[ , setdiff(colnames(df),
c("GeneID", "width"))])

  stopifnot(identical(colnames(df), c("GeneID", "seqnames", "start",
"end", "strand")))

  colnames(df) <- c("GeneID", "Chr", "Start", "End", "Strand")

  write.table(df, file="mm39_RefSeq_exon.txt", quote=FALSE, sep="\t",
row.names=FALSE)

The entire process of obtaining the exons and dumping them to the file
takes about 2 seconds on my labtop ;-)

H.

        
On 08/04/2022 06:00, Kern, Lori wrote:

            

      
      
      
    

        

      
      
    

        
--
Herv? Pag?s

Bioconductor Core Team
hpages.on.github at gmail.com

  
  


  


      

    

    

      
      

        


  

        

      Hervé Pagès
    

    

      
      #
    

  


  

      
On 08/04/2022 09:26, Yang Liao wrote:

        

            

      
      
      
    

        
Would be good to know what that means exactly. If Rsubread uses a subset 
of RefSeq exons, the curation process should be documented somewhere, 
for the sake of reproducibility.

Best,

H.

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Yang Liao
    

    

      
      #
    

  


  

      
Thanks for the reply! We used the flattenGTF function in Rsubread to merge the overlapping exons in each gene; this procedure is documented the manual page of featureCounts. We also checked/tested if there are "tricky" genes in the annotation that we need to take extra care/treatments (e.g. some genes can span multiple chromosomes and/or strands). It is hard to automate all the checks reliably.

Also, I think it can be helpful to the reproducibility of DGE analyses if we can have a version of gene annotations relatively stable, not changing when the RefSeq annotation changes between builds.

All the best,
Yang

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Hervé Pagès
    

    

      
      #
    

  


  

      
On 08/04/2022 10:02, Yang Liao wrote:

        

            

      
      
      
    

        
I see. thanks for clarifying.


Best,

H.