Skip to content

[Bioc-devel] Bioconductor Git/GitHub Mirrors

17 messages · Ryan, Leonardo Collado Torres, Dan Tenenbaum +3 more

#
Dear Bioconductors,

We're pleased to announce the availability of Bioconductor Git Mirrors. 
These are read-only GitHub repositories (available under https://github.com/Bioconductor-mirror) 
for every Bioconductor software package. These repositories are synchronized with our 
Subversion repository. Package maintainers (or anyone else) can fork these repositories
and do their development on the fork. Complete documentation of the mirrors is at

http://bioconductor.org/developers/how-to/git-mirror/

These mirrors supersede the Git-SVN bridge, which is now deprecated. 
Creation of new bridges is disabled and maintainers who are using the bridge
should migrate to the Git mirrors as soon as it's convenient, as the bridge
will eventually go away. Instructions for migrating can be found at the above link.

Some features of the new Git Mirrors, and why we feel they are a better solution than the Git-SVN bridge:

 - The mirrors contain complete commit history.
 - The mirrors contain release branches for Bioconductor 3.0 and 3.1, and 
   new releases will be added as they happen. You will no longer need 
   separate repositories for release and devel.
 - Setup is easy and you no longer have to grant any permissions on your
   repository to other users. You can commit directly to Subversion
   using git-svn (https://git-scm.com/docs/git-svn).
 - Each git commit appears in the SVN log as a distinct SVN commit; commits 
   are no longer grouped together as they were with the bridge. Each git
   commit can be mapped to a specific svn commit, and vice versa.
 - The Git mirrors are much more reliable.
 - Use is flexible. You can use git locally (without GitHub) or you 
   can use GitHub as well, to take advantage of all its social coding features.
 - Using GitHub's code search, you can search the entire Bioconductor codebase.
   Here's a sample search: https://goo.gl/jI92Ys
 - Subversion is fully supported and remains the cannonical repository; use
   of Git and GitHub is optional.

We are excited about these new mirrors (brought to you by the hard
work of Jim Hester) and we hope you are too. Questions and comments are
welcome on the bioc-devel mailing list.

Dan
#
Correct link is http://bioconductor.org/developers/how-to/git-mirrors/

On Tue, Jun 16, 2015 at 3:00 PM Dan Tenenbaum <dtenenba at fredhutch.org>
wrote:

  
  
#
This is great to hear. I sometimes want to delve into the source code of 
a package's internals, but doing so through the SVN web interface is 
clunky. Being able to use Github's repo browsing functionality for Bioc 
packages is great.
On 06/16/2015 12:00 PM, Dan Tenenbaum wrote:
#
Hi,


This sounds great! Not having to setup all those bridges everytime a
new BioC version is released should be a great feature, plus keeping
the commit history tidy. And we can still use Travis or other things
like Slack notifications on our own bridges. I guess that it'd be good
to remind users to pull new content from the bioc remote before trying
to commit to svn.

Anyhow, great work!




I'm having a few problems with this new setup which hopefully are easy
to iron out. Basically, I am stuck at the "git svn rebase && git svn
dcommit --add-author-from" step. See 2b) below.


1) My git was pointing to an older version even after updated with
brew. This was because I still had MacPorts around.
How to diagnose:
$ which git
Wrong version was pointing to /opt/local/bin/git instead of
/usr/local/bin/git Can also see by running
$ git --version ## if it doesn't match the version number you
installed from brew
Solution:
* Uninstall all of MacPorts following instructions from
https://guide.macports.org/chunked/installing.macports.uninstalling.html
* Follow part of
http://stackoverflow.com/questions/20648235/error-linking-git-to-homebrew
$ sudo brew prune
$ brew update
$ brew install git
$ brew unlink git && brew link git

You might want to update subversion.
http://blog.victorquinn.com/fix-git-svn-in-mountain-lion might be
relevant for you.

2) Test case: derfinderHelper.

Smallest and simplest package I have, which won't matter much if
things break because it's stable.

a) You might want to delete update_remotes.sh after using it.
* I deleted the git-svn bridges to both
https://github.com/leekgroup/derfinderHelper and
https://github.com/leekgroup/derfinderHelper-release
* On https://github.com/leekgroup/derfinderHelper I removed bioc as a
collaborator and deleted the push-hook since it seems like we don't
need this anymore
* Download and ran update_remotes.sh as instructed. It doesn't seem
like we need it anymore after that point. If you leave it there, it
seems to give problems.
Easy to solve, but might want to clarify in the instructions.

b) Following the section "Use Git Locally And Publicly" I get stuck on
the "git svn rebase && git svn dcommit --add-author-from" step as
shown at https://gist.github.com/lcolladotor/ac8838b25955caeebcf1. The
"CONFLICT (add/add): Merge conflict" message lead me to
http://stackoverflow.com/questions/19475387/how-to-handle-fix-git-add-add-conflicts
from where I could resolve the initial problem that I saw using "git
diff". But was basically the changes between the very first version of
README.md and the very latest version of it.
There must be a way to not have to go through every single commit
and/or file one by one. Using "git rebase --continue" doesn't help.

c) Given the issue with b, the devel branch already has a merge
commit. Using http://stackoverflow.com/questions/927358/how-to-undo-the-last-commit
I cleaned things up. I don't think it's needed, but I don't want to
run into more issues.
$ git rebase --abort
$ git reset HEAD~1


As you can tell, I'm no "git rebase" expert. From
http://git-scm.com/docs/git-rebase it seems like the "ours" option
might be useful. But right now I prefer to get some pointers on how to
proceed before playing around further :P

Thanks,
Leo


PS At the "Unable to determine upstream SVN information" section, I
believe that there is a typo. "git svn dcomit" should be "git svn
dcommit"
On Tue, Jun 16, 2015 at 3:20 PM, Ryan C. Thompson <rct at thompsonclan.org> wrote:
#
On Tue, Jun 16, 2015 at 8:55 PM, Leonardo Collado Torres
<lcollado at jhu.edu> wrote:
I meant repos here


I guess that it'd be good
#
----- Original Message -----
Good to know. Things seem to work on the Macs I have tested with the vanilla git-svn that ships with the OS.
I haven't needed to update svn.
Yes, you can delete the push hook. Leaving it does no harm as long as the bridge has been deleted (nobody will listen to the hook on the other end once the bridge is deleted).
Can you explain what problems?
I do this in a separate directory:

git clone https://github.com/Bioconductor/mirror.git

and then I have a repos containing the script where I can run 'git pull' if it ever changes. You could also symlink to it in /usr/local/bin. We'll see about changing the docs for this.
I didn't have this problem. Here is what I did, which should be the same as what you did:

forked derfinderHelper to my personal (dtenenba) account
cloned the fork
ran the update_remotes.sh script in the fork

git checkout devel
git svn rebase
git merge master --no-edit
git svn rebase
git svn dcommit --add-author-from 

None of these lines gave any errors and the last line said:

Committing to https://hedgehog.fhcrc.org/bioconductor/trunk/madman/Rpacks/derfinderHelper ...

Doesn't seem like it actually needed to commit anything.
Well, since I can't reproduce this situation I'm not exactly sure what to tell you. If you know you want all the changes from git and none from svn (or vice versa) then the --ours and --theirs options are appropriate. But you might just want to start over and see if the problem goes away.

Here are my relevant versions:

$ git --version
git version 1.9.5 (Apple Git-50.3)
$ git svn --version
git-svn version 1.9.5 (Apple Git-50.3) (svn 1.7.19)
$ svn --version
svn, version 1.8.11 (r1643975)
$ uname -a
Darwin viper 13.4.0 Darwin Kernel Version 13.4.0: Wed Mar 18 16:20:14 PDT 2015; root:xnu-2422.115.14~1/RELEASE_X86_64 x86_64

It does look like on this particular machine I have a non-standard svn:

$ which svn
/opt/subversion/bin/svn


Looks like it came from: https://www.wandisco.com/subversion/os/downloads
Will fix.

Thanks for the comments and for trying it out.

Dan
#
----- Original Message -----
PS., the docs do say this:

 2. git svn rebase to get the latest SVN changes.

However, you are reminding me that we should make sure we document that you should merge any changes from back into master
after you dcommit.

Dan
#
----- Original Message -----
Sorry, I wasn't following along correctly, I didn't start with your github repos (formerly bridged). What's the link to it?

Dan
#
----- Original Message -----
Answering my own question: https://github.com/leekgroup/derfinderHelper

I can reproduce the issue, this is the first time it has come up in all the formerly-bridged repos I have tested. Will get back to you.

Dan
#
On Tue, Jun 16, 2015 at 10:02 PM, Dan Tenenbaum <dtenenba at fredhutch.org> wrote:
Yeah, I've had mine for a while and forgot I still had MacPorts around.
Ok, I just did it to be safe.

Link is about those updating to Mountain Lion or Mavericks and having
to re-download Xcode then setting up the command line tools and them
some potential perl issues which I didn't have.
Ok!
Git complains that you have an untracked file in the repo when running
git svn rebase. There's no need to actually track it, or even exclude
it with a .gitignore file, so I simply deleted it. I did check that
the remotes had been added using "git remote -v".
Ok, maybe this is why you don't have the problem. I didn't fork
https://github.com/Bioconductor-mirror/derfinderHelper into my
personal (or organization, doesn't matter here) account. I did all of
this with my local clone of
https://github.com/leekgroup/derfinderHelper which already existed.
https://github.com/leekgroup/derfinderHelper has the full commit
history since I've been using git from the beginning, instead of the
aggregated svn commit history.

If I fork https://github.com/Bioconductor-mirror/derfinderHelper then
I'll lose the detailed commit history, right?

In terms of the deprecated git-svn bridge, I want git to win
unconditionally on the merge. I'm not sure how I would do that.


Another reason why I would like to avoid doing a fork is that several
users already "watch", starred the original github repo, or created
issues (some might be open still, others closed with info others might
want to read). Well, not really with the tiny derfinderHelper. But I
guess that this is just really a weak argument since I could just let
svn win while preserving the original github repo.
Ahh, ok. Sounds like what I want!

http://stackoverflow.com/questions/2959443/why-is-the-meaning-of-ours-and-theirs-reversed-with-git-svn
seems like a good starting point
#
----- Original Message -----
Laurent Gatto, helping me with a separate but (probably) related issue, may have helped with this one too.

Try starting over again. Remove your local repository and do a fresh clone:

git clone https://github.com/leekgroup/derfinderHelper.git
cd derfinderHelper
bash /path/to/update_remotes.sh
git checkout devel
git svn rebase
git merge master --no-edit
git svn dcommit --add-author-from

That all worked for me and did not end up committing anything.
The difference between this and our documentation is that here there is no rebase in between the merge and dcommit.

You can then review the commits in devel (with git log) and see if you want to merge them into master. If so, do

git checkout master
git merge devel

git status

will now show that master is 30 commits ahead of origin/master. 

Push if you want. Not sure if you want to as you said you wanted git to "win". But note that we are no longer in the land of one SCM unconditionally "winning" over another. That was a concept from the git-svn bridge (a somewhat dangerous one) which in any case only applied to the initial setup of a bridge. With git-svn you have complete control over what goes where.

So anyway, try the above and if it works we will discuss whether and how the documentation needs to be updated.

Thanks,
Dan
#
It would be cool if we could somehow get the equivalent of the
bioconductor package page to show up as the "readme" on the github
page. Or at least, if there could be a very obvious link from the
mirror repository to the maintainer repository, without having to
click through to the Bioconductor package page, which might confuse
new users a bit.
On Tue, Jun 16, 2015 at 12:00 PM, Dan Tenenbaum <dtenenba at fredhutch.org> wrote:
#
Why not write your own *readme.md* file for the package and put the link
there? There is nothing stopping the maintainer from having a README file
in the package. Although it does not show up on the Bioconductor page, it
shows up nicely on Github:

My package:

https://github.com/Bioconductor-mirror/categoryCompare
https://github.com/rmflight/categoryCompare

-Robert

On Wed, Jun 17, 2015 at 8:43 AM Michael Lawrence <lawrence.michael at gene.com>
wrote:

  
  
#
I have (in the past couple of minutes) added a more detailed description to
all of the GitHub pages.  If you have URL or BugReports fields in your
package DESCRIPTION those are used to provide links to the project page and
bug tracker if they are available, see Roberts
https://github.com/Bioconductor-mirror/categoryCompare for an example.

I would also recommend doing as Robert suggests and adding a link in a
README.md file if you would like to direct users elsewhere.  These files
are automatically rendered by GitHub and users are used to looking at them
for project documentation.

Thank you all for being patient as we roll out this functionality.

Jim

On Wed, Jun 17, 2015 at 9:06 AM, Robert M. Flight <rflight79 at gmail.com>
wrote:

  
  
2 days later
#
So this seems to work quite nicely. But in my own testing of committing
with my categoryCompare package, there is one interesting issue of Github
complaining about how many commits ahead / behind the fork is.

So I moved my old github repo, and then made a fresh fork from the
Bioconductor mirror, and made a new clone of that fork on my local machine,
and ran update_remotes.sh.

Made some changes to the readme (pointing to personal repo for issues) and
committed in *master

git push origin master (push back up to my personal copy)
git checkout devel
git svn rebase
git merge master --no-edit
git svn rebase && git svn dcommit --add-author-from

When I looked at my personal fork, it shows 1 commit ahead & behind the
original. So I pulled from *bioc* remote into master (and merge it), and
then pushed back up to master.

I still have this message: This branch is 2 commits ahead of
Bioconductor-mirror:master, but there are no file differences.

Does anyone know how to make this message go away? Should I just use my own
repo instead of a *fork* of the Bioconductor-mirror?

-Robert

On Wed, Jun 17, 2015 at 10:10 AM Jim Hester <james.f.hester at gmail.com>
wrote:

  
  
#
I think these are unavoidable artifacts with the current setup.  The
commits from the Bioconductor-mirror are coming from SVN after being
translated so they have extra metadata attached to the commit message,
while your original commits do not have this.  Therefore GitHub cannot
properly disambiguate the two commits, and you get this message.  You will
also notice you get the same commit twice in your log (
https://github.com/rmflight/categoryCompare/commits/master) due to the same
reason.

Using your own repo instead of forking Bioconductor-mirror will make the
message go away and you should also not have the issue with double commits
I believe.

On Fri, Jun 19, 2015 at 10:23 AM, Robert M. Flight <rflight79 at gmail.com>
wrote:

  
  
#
Hmmm, ok. I kind of liked having the message about being a fork of the
Bioc-mirror, but the difference message makes it a little messy. Oh well.
Was an interesting experiment.

On Fri, Jun 19, 2015 at 11:19 AM Jim Hester <james.f.hester at gmail.com>
wrote: