[Bioc-devel] Git pack file greater than 5MB
Hi all, Thank you all for taking the time to discuss this issue. While the package does have a fairly long history with multiple authors, I am currently the only active developer on the project. So, if it preferable from Bioconductor?s perspective, I will rewrite the repository's history. I do have one remaining question regarding the extent of the rewrite. Currently the package (not including the pack file) sits at ~1.6MB. After a test run of the rewrite I was able to reduce the pack file to ~4MB. So, the total package is still over 5MB, but each individual file is under the threshold. Is this acceptable? Or will I need to delete more from the history? I ask because I imagine it is preferable to limit the extent of the rewrite to a minimal acceptable standard. Thanks, Max
From: Bioc-devel <bioc-devel-bounces at r-project.org> on behalf of Martin Morgan <mtmorgan.bioc at gmail.com>
Sent: Thursday, October 1, 2020 10:32 AM
To: Henrik Bengtsson <henrik.bengtsson at gmail.com>; Nitesh Turaga <nturaga.bioc at gmail.com>
Cc: bioc-devel at r-project.org <bioc-devel at r-project.org>
Subject: Re: [Bioc-devel] Git pack file greater than 5MB
Sent: Thursday, October 1, 2020 10:32 AM
To: Henrik Bengtsson <henrik.bengtsson at gmail.com>; Nitesh Turaga <nturaga.bioc at gmail.com>
Cc: bioc-devel at r-project.org <bioc-devel at r-project.org>
Subject: Re: [Bioc-devel] Git pack file greater than 5MB
yes Herv? has made this point too -- mucking with the history of the package, potentially breaking historical checkouts (when large files are deleted from the history, too).
It's relevant because when a package is added to our repository we do a full clone of the master branch; an alternative would be to do a --depth 1 clone of the master branch, but to me this doesn't seem ideal at all -- from the Bioconductor perspective the git.bioconductor.org package is definitive, and all we would have would be 'and then a miracle occurred' for early package development. I'm also nervous about side-effects associated with maintaining the Bioconductor and non-Bioconductor repositories that have different historical starts.
My own feel is that most of these cases are packages that are still 'new' and seldom have clones / forks.
One could take a hybrid approach, where if a maintainer insists on the integrity of their git repository (or even automatically, if they do have large files in their history we automatically change strategy) then we do a --depth 1 clone.
Martin
?On 10/1/20, 12:17 PM, "Bioc-devel on behalf of Henrik Bengtsson" <bioc-devel-bounces at r-project.org on behalf of henrik.bengtsson at gmail.com> wrote:
I understood that it's a submission. Just wanted to make sure that it's
clear there might be side effects, e.g. people clone and collaborate also
before submitting to Bioc and a rewrite might surprise existing
collaborators etc.
/H
On Thu, Oct 1, 2020, 09:04 Nitesh Turaga <nturaga.bioc at gmail.com> wrote:
> This package isn?t yet a Bioconductor package Henrik. It will break other
> forks most likely. This package hasn?t been submitted to the Contributions
> either to be reviewed. So this is the time to break what needs to be broken
> before it?s submitted to Bioconductor and gets into the Bioconductor git
> repository.
>
> Nitesh
>
> On Oct 1, 2020, at 11:57 AM, Henrik Bengtsson <henrik.bengtsson at gmail.com>
> wrote:
>
> Doesn't a git rewrite break all existing clones, forks out there? I'm
> happy to be corrected, if this is not the case.
>
> /Henrik
>
> On Thu, Oct 1, 2020, 08:16 Nitesh Turaga <nturaga.bioc at gmail.com> wrote:
>
>> Hi,
>>
>> The BiocCheck will complain on the build system about the > 5MB package
>> size.
>>
>> The rewrite of the history with BFG cleaner (
>> https://rtyley.github.io/bfg-repo-cleaner/ <
>> https://rtyley.github.io/bfg-repo-cleaner/>) is not as severe as you
>> think it is to be honest. It is just removing these pack files which don?t
>> have a place in the tree structure. These are more often than not, orphan
>> files.
>>
>> If you are suspect of this solution, I would suggest you make a backup
>> clone of your repo and try it on that first before you touch the main repo.
>> Check the history (git log) to see if anything important is missing.
>>
>> But usually a software package has to be below 5MB. If you have some data
>> in there which is needed for the package, consider Experiment Hub.
>>
>> Best,
>>
>> Nitesh
>>
>> > On Sep 30, 2020, at 12:46 PM, McGrath, Max <MAX.MCGRATH at UCDENVER.EDU>
>> wrote:
>> >
>> > Hi all,
>> >
>> > We have a package that is ready for submission, but when running
>> BiocCheck a warning is generated noting that "The following files are over
>> 5MB in size: '.git/objects/pack/pack-xxx...". I've pruned, repacked, and
>> run git gc which reduced the file size from 5.2 to 5.1MB, but I have been
>> unable to reduce it further.
>> >
>> > I'm reaching out to determine if this is an issue, and if so to ask for
>> recommendations for solving it. Currently, the only solution I've come up
>> with is to rewrite the repository's history using a tool like
>> "git-filter-repo", but this is a more drastic action than I would prefer to
>> take. I would greatly appreciate any advice on the matter.
>> >
>> > Thank you,
>> > Max McGrath
>> >
>> > [[alternative HTML version deleted]]
>> >
>> > _______________________________________________
>> > Bioc-devel at r-project.org mailing list
>> > https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>
>>
>> [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> Bioc-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>
>
>
[[alternative HTML version deleted]]
_______________________________________________
Bioc-devel at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
_______________________________________________
Bioc-devel at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel