Skip to content

[Bioc-devel] Old package versions / Bioc archive of package's *.tar.gz releases?

10 messages · Henrik Bengtsson, Martin Morgan, Gabriel Becker

Original
Henrik Bengtsson
# 
Is there an easily accessible archive for Bioconductor packages
similar to what is provided on CRAN where you can find all released
versions of a package, e.g.
https://cran.r-project.org/src/contrib/Archive/PSCBS/?

Say I want to access the source code for affy 1.18.0.  Here are the
two approaches I'm aware of and none of them are particularly
appealing to me.  Does anyone know of a better approach?


# APPROACH 1: Download from http://bioconductor.org

The best approach I know now is to try to guess the date when this was
released in order to identify the Bioconductor release version.
Something like this:

1. Guess around 2010.

2. Go to http://bioconductor.org/about/release-announcements/ and see
what R versions were in use during 2010.  I find R 2.6.x and R 2.7.x.
The Bioc version for those R versions (same URL) are Bioc 2.1 and Bioc
2.2.  Let's focus on Bioc 2.2 (because I happen to know that is the
one)

3. Following the Bioc 2.2 link on above URL to get to
http://bioconductor.org/packages/2.2/BiocViews.html.

4. Click through, one eventually gets to
http://bioconductor.org/packages/2.2/bioc/html/affy.html

5. The "Source" link points to
http://bioconductor.org/packages/2.2/bioc/src/contrib/affy_1.18.2.tar.gz

Say I wanted affy 1.16.0 instead and I made the wrong guess in Step 2,
I can extrapolate from (Bioc 2.2, affy 1.18.x) finding that I should
go to Bioc 2.1 to find affy 1.16.x (because releases have even minor
version numbers).   It works, but is a bit tedious if you need to do
this more than once.

Also, I'm pretty sure I read somewhere that Bioc on archive the most
recent package version under each release, which means there is no
affy_1.18.0.tar.gz available for download.  Is that correct?


# APPROACH 2: Version control

$ git clone https://git.bioconductor.org/packages/affy
$ cd affy

# Package releases/versions are not tagged
$ git tag
[empty]

# Check Bioc release branches
$ git branch -a
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/RELEASE_1_0
  remotes/origin/RELEASE_1_0_branch
  remotes/origin/RELEASE_1_4
  remotes/origin/RELEASE_1_4_branch
  remotes/origin/RELEASE_1_5
[...]
  remotes/origin/RELEASE_3_5
  remotes/origin/master

That's back to above game of trying to narrow down which Bioc release
I should look at.  A similar approach is to look at the commit log:

$ git log DESCRIPTION

commit 35573048255b398f99ff1d3560906b2121912248
Author: Herve Pages <hpages at fhcrc.org>
Date:   Mon Apr 24 19:50:57 2017 +0000

    bump x.y.z versions to odd y after creation of 3_5 branch

    git-svn-id:
file:///home/git/hedgehog.fhcrc.org/bioconductor/trunk/madman/Rpacks/affy at 129129
bc3139a8-67e5-0310-9ffc-ced21a209358

commit aa4c2d648658e8c2cca2baf651aea92df55a4392
Author: Herve Pages <hpages at fhcrc.org>
Date:   Mon Apr 24 19:25:24 2017 +0000

    bump x.y.z versions to even y prior to creation of 3_5 branch

    git-svn-id:
file:///home/git/hedgehog.fhcrc.org/bioconductor/trunk/madman/Rpacks/affy at 129126
bc3139a8-67e5-0310-9ffc-ced21a209358

[...]

and try to locate affy 1.18.0 by peeking at the DESCRIPTION file history.

Does anyone know of a better/more automated approach?

Thanks,

Henrik
Martin Morgan
#
On 10/05/2017 01:50 PM, Henrik Bengtsson wrote:
The only option is to scrape, and that's approximate. One could build an 
archive

pkg,version,branch,from_svn_rev,to_svn_rev

and then consult that. Packages are supposed to increment the 'z' of 
x.y.z, but I'm sure there are many exceptions. I believe Jim Hester has 
an svn script for this, but I wasn't able to locate it; it would be fast 
in git.

For your future self, this

   https://bioconductor.org/packages/3.5/bioc/src/contrib/Archive/S4Vectors/

provides a hint of a change coming with the next release -- archives of 
all RELEASE package versions, starting in Bioc 3.6. (Kudos to Val for 
implementing this)

Martin
This email message may contain legally privileged and/or...{{dropped:2}}
Henrik Bengtsson
# 
On Thu, Oct 5, 2017 at 1:46 PM, Martin Morgan
<martin.morgan at roswellpark.org> wrote:
Thanks.  About 'z' not being increased.  Does the Bioc build servers
release (a) continuously or (b) only when it detects a version change
x.y.z -> x.y.z+1?  If it does it continuously, then what x.y.z is
installed does matter on when it was downloaded/installed, correct?
On the other hand, if it only builds in when a version bump is
detected, then one can at least narrow it down to a much narrow set of
x.y.z submits (if multiple exists).
This is great!  Thanks Val for this.

Thanks

Henrik
Martin Morgan
#
On 10/05/2017 05:14 PM, Henrik Bengtsson wrote:
The builder only pushes for upward increments, so a commit without a 
'positive' version bump would be built but not pushed to the public 
repository. I'm not sure how rigorously this policy was enforced before, 
e.g., 2005.

Of course there are exceptions, e.g., it is occasionally (at most one or 
two times a release cycle) necessary to flush the public repository 
entirely, and then whatever is built is pushed. And there is nothing 
stopping the user from doing a check-out from svn. Perhaps others will 
chime in with the gory / correct details.

Martin
This email message may contain legally privileged and/or...{{dropped:2}}
Gabriel Becker
# 
Henrik et al.,

My switchr package (on CRAN, github at: http://github.com/gmbecker/switchr,
preprint of the paper here: https://arxiv.org/abs/1501.02284
<https://arxiv.org/abs/1501.02284>) can do this.

In fact, installing (cohorts of) old versions of packages is one of it's
primary purposes. Specifically, it can install old source versions of
packages from CRAN, Bioconductor, and general Git and SVN repos you tell it
about.

With the caveat that it's a bad idea in the general case to specify an old
version of one package without specifying versions of its dependencies
(switchr allows you to do this via a manifest, which can be constructed
from sessionInfo output or guessed in the case of a CRAN package), you can
just do
And you will successfully completely break your Bioc installation by
installing IRanges 2.6.0 into it. ;-)

Switchr also gives you tools to more easilly maintain multiple libraries
which contain, for example, different bioc versions in them.

NB: switchr is subject to the caveat Martin pointed out and will fail to
retrieve a buildable version of the package if said buildable version is
not the first commit in SCM bearing that version in its DESCRIPTION file.

Hope that helps.

Best,
~G

On Thu, Oct 5, 2017 at 2:21 PM, Martin Morgan <martin.morgan at roswellpark.org

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Gabriel Becker
    

    

      
      #
    

  


  

      
In point of fact, it looks like IRanges 2.6.0 is an instance of that
weakness, so was probably a bad example. 2.6.1 installs correctly, or would
in it's native R/base bioc environment... (it fails for me in the library I
have...)

Also, the version on CRAN uses the bioc SVN, so may not work for recent
versions.

        
On Thu, Oct 5, 2017 at 3:58 PM, Gabe Becker <beckerg4 at gene.com> wrote:

        

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Henrik Bengtsson
    

    

      
      #
    

  


  

      
That's really nice; I didn't know it could do all that.  For my
clarification, when using PkgManifest(..., type = "bioc") it'll search
(i) the CRAN archives, (ii) the Bioconductor repos(es), and then (iii)
the Bioconductor Git repos - is that a correct observation? (I
installed from https://github.com/gmbecker/switchr)

/Henrik

        
On Thu, Oct 5, 2017 at 4:15 PM, Gabe Becker <becker.gabe at gene.com> wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Gabriel Becker
    

    

      
      #
    

  


  

      
Correct. The actual order of checks is:

   1. did switchr already retrieve that exact version for something else
   and keep it around,
   2. A GRANRepository object if one is specified (don't worry much about
   this one)
   3. the manifest itself (cran and bioc source types are ignored, but it
   would walk SCM if it had git/svn type manifest entry for the package)
   4. cran repo and cran archives,
   5. bioc repositories (all of them in descending order),
   6. bioc git (bioc SVN for the version on CRAN, which appeared to still
   work last time I checked it a week or two ago)

See the code in
https://github.com/gmbecker/switchr/blob/master/R/retrievePkgVersion.R for
details.

Best,
~G




On Oct 5, 2017 9:06 PM, "Henrik Bengtsson" <henrik.bengtsson at gmail.com>
wrote:

That's really nice; I didn't know it could do all that.  For my
clarification, when using PkgManifest(..., type = "bioc") it'll search
(i) the CRAN archives, (ii) the Bioconductor repos(es), and then (iii)
the Bioconductor Git repos - is that a correct observation? (I
installed from https://github.com/gmbecker/switchr)

/Henrik

        
On Thu, Oct 5, 2017 at 4:15 PM, Gabe Becker <becker.gabe at gene.com> wrote:

            

      
      
      
    

        
would

            

      
      
      
    

        
I

            

      
      
      
    

        
it

            

      
      
      
    

        
old

            

      
      
      
    

        
from

            

      
      
      
    

        
just

            

      
      
      
    

        
not

            

      
      
      
    

        
svn

            

      
      
      
    

        
chime

            

      
      
      
    

        
ve/S4Vectors/

            

      
      
      
    

        
of

            

      
      
      
    

        
was

            

      
      
      
    

        
Bioc

            

      
      
      
    

        
1.18.2.tar.gz

            

      
      
      
    

        
2,

            

      
      
      
    

        
n/Rpacks/affy at 129129

            

      
      
      
    

        
n/Rpacks/affy at 129126

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        
employee

            

      
      
      
    

        
by

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Gabriel Becker
    

    

      
      #
    

  


  

      
Oh, and the svn/git walking knows about release branches so it will find
hot-patch versions correctly, as well.

~G

        
On Thu, Oct 5, 2017 at 10:01 PM, Gabe Becker <beckerg4 at gene.com> wrote:

        

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Gabriel Becker
    

    

      
      #
    

  


  

      
On Thu, Oct 5, 2017 at 10:03 PM, Gabe Becker <beckerg4 at gene.com> wrote:

        

            

      
      
      
    

        
In the Bioc case, I mean, not generally.

And with that flurry of emails I'm off to bed.

Best,
~G