Skip to content

the case of building R snapshot without svn nor network connection.

11 messages · Hin-Tak Leung, Simon Urbanek, Peter Dalgaard +3 more

Original
Hin-Tak Leung
# 
The decision to actively discourage non-subsersion usage of snapshot build is already made (r62183). So I am just here to register a differing opinion.

- it is not about subversion vs other-version-control-tools. There are two parts of R's dev build process which requires an active network connection - tools/rsync-recommended and capturing `svn info` into R's headers. The former can be overridden with "./configure --with-recommended-packages=no". A few changes had been made in the last 6 months to make the latter mandatory. It used to be optional.

--- there are genuine needs for testing snapshots in a non-networked minimalist environment - e.g. banks or telecom industries - where one wants a "standardised host" build, and often it means an "outdated host"; or in a virtual machine environment - which are non-networked for security reasons, and also do not have tools beyond necessary for compiling and building. This is quite a common scenario.

--- AFAIK, 6 months ago, a snapshot copied to an non-networked host will build with "./configure --with-recommended-packages=no". Of course copying those recommended package tar balls across would be nicer. This is sadly no longer the case.

- dependent on `svn info` and sitting on top of a svn checkout possibly also affects cross-compiling. But then, it is not clear whether it is still possible to cross-compile R, since quite a few changes have been made to remove the capability of cross-compiling R for windows on unix hosts in the last few years. 

- testing dev snapshots is about trying things and fixing bugs before release. Making it difficult for non-core people to "try", seem to go against that ethos. If that's the case, maybe the repository could be closed off to anonymous check outs altogether, just to make it clear that testing snapshots before releases or even close to release, is not welcomed. Just a thought.

- although the official repository of the "main" linux kernel branch is git-based, there are mercurial mirrors; AFAIK the digital video broadcasting hardware support of the linux kernel is officially in mercurial repositories, but have git mirrors; likewise much of Xorg's is in mercurial but have git mirrors. I haven't heard of any much bigger public projects than R where some actively discourage others.

- To be honest r62183 itself is probably a good reason to move away from server-side repositories to distributed repositories (hg/git/arch/bzr). Local enhancements - i.e. an in-house fork - some of which are never going upstream, such as a local revert of r62183 (or a local revert of any other upstream commits) is one reason why some have distributed repositories.

Lastly, a minor grip. The current head of the HK government is probably sometimes called "HK Leung", just as some might call a certain old lady "UK Windsor" and a certain black person "US Obama".
Simon Urbanek
#
On Mar 15, 2013, at 7:29 PM, Hin-Tak Leung wrote:

            

      
      
      
    

        
That is a false statement - svn info doesn't require any network connection.

Cheers,
Simon

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Hin-Tak Leung
    

    

      
      #
    

  


  

      
I'll quantify the first part - R is perhaps the only public software project hosted on a subversion repository for which the result of 'svn export ...' does not build. Not only that it does not build, but make it a feature that it does not build.

Very few other projects actively try to go in that direction.

        
--- On Fri, 15/3/13, Hin-Tak Leung <hintak_leung at yahoo.co.uk> wrote:

        

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Peter Dalgaard
    

    

      
      #
    

  


  

      
On Mar 16, 2013, at 02:50 , Hin-Tak Leung wrote:

        

            

      
      
      
    

        
It is not a design goal in any project that I know of, that svn exports should be equivalent to distribution tarballs. In simple projects, that might be the case, but requiring a "make dist" step is quite common before the final shipment. 

An actual design goal has been to ensure that snapshot builds carry an SVN revision number so that we can detect whether issues are reported on versions prior to a known fix. This is done via an "svn info" command because that was the path of least resistance (you can't really e.g. maintain the SVN-REVISION file in svn because it would need to change on every commit). There's no particular intention to hardwire SVN as the source code management tool, but as it happens to be what we use, that tiny subsystem of the build system has to be specific to SVN.

The generic point is that you are given access to a working tool that is internal to the core R developers. We are not putting restrictions on what you do with that access, but if you want to play the game by other rules than we do, you need to take the consequences. If things don't work and you start complaining about them being "broken", steps may be taken to make it clearer who broke them.

As Jari Oksanen put it recently:

"I think the rule is that you can do anything as long as you don't complain. If you want to complain, you must follow the instructions."

Peter D.

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Hin-Tak Leung
    

    

      
      #
    

  


  

      
--- On Sat, 16/3/13, peter dalgaard <pdalgd at gmail.com> wrote:

        

            

      
      
      
    

        
Yes and no. Many projects much larger than R have a "make dist" target. However, I don't know of any where they make it a feature and a design goal and actively go forward that a 'make dist' tar ball differs substantially in functionality from a snapshot close to the release revision, and also actively make sure that a snapshot does not work.

Try name one.

Even if you can name one other such project, is that honestly good practice to emulate?

            

      
      
      
    

        
As is evident, dependence on svn is already hardwired. Look at the issue leading up to this discussion: there were (are, since there isn't a new release yet) code in Matrix, and also elsewhere from a cursory grep, where code paths are conditional on specific version commit numbers, and do different things before/after specific svn revision numbers.

            

      
      
      
    

        
There is a difference between "unsupported" ("I don't want to spend time hearing about issues arising from going off the beaten path"), versus 'actively discourage going off "beaten" path'. 

Where, of course, "beaten path" is defined by a small group of people, as is "design goal". (it is a feature to break).

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Dirk Eddelbuettel
    

    

      
      #
    

  


  

      
Can you please stop this?  You made your point, repeatedly.  Nobody has come
to your side.  

And you *do* have easy alternatives:

   i)  git repo,  see https://github.com/wch/r-source    

   ii) nighly tarballs,  see ftp://ftp.stat.math.ethz.ch/Software/R

Whatever your particular circumstances are (which you never detailed in a way
that made me understand what your issue is), one of these should work.

As I said before, 'svn up && make distclean && ~/bin/makeRdevel' works just
fine for me with the configure choice I prefer hardwired in the script.  I
never sync to r-recommended and *honestly* have no idea what your issue is.
I just rebuilt two days ago without an issue:

   edd at max:~$ ~/bin/R-devel.sh --version|head -1
   R Under development (unstable) (2013-03-15 r62262) -- "Unsuffered Consequences"


Dirk

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Hin-Tak Leung
    

    

      
      #
    

  


  

      
Network access is *not* a given, nor is the privilege of installing arbitrary "uncertified" and "non-essential" tools - whatever the meaning of "uncertified" and "non-essential" are, those being defined, as is "design goal", etc, by some small committee.

It is a very common scenario, e.g. banks & telecom, some part of public/government service and health care. This does not seem to sink in without repeating.

        
--- On Sat, 16/3/13, Hin-Tak Leung <htl10 at users.sourceforge.net> wrote:

        

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Simon Urbanek
    

    

      
      #
    

  


  

      
On Mar 16, 2013, at 10:13 AM, Hin-Tak Leung wrote:

        

            

      
      
      
    

        
Network access is not needed to build R - apparently that fact did not seem to sink in, either. This entire thread is based on false assumptions and as such the only place for it is /dev/null

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Yihui Xie
    

    

      
      #
    

  


  

      
"the only place for it is /dev/null" --> hi Achim, I bet this is a
nice fortune candidate :)

Regards,
Yihui
--
Yihui Xie <xieyihui at gmail.com>
Phone: 515-294-2465 Web: http://yihui.name
Department of Statistics, Iowa State University
2215 Snedecor Hall, Ames, IA


On Sat, Mar 16, 2013 at 9:39 AM, Simon Urbanek

        
<simon.urbanek at r-project.org> wrote:

            

      
      
      
    

  
  


  


      

    

    
2 days later

    

      
      

        


  

        

      Martin Maechler
    

    

      
      #
    

  


  

          

      
      
      
    

        

        
> On Mar 16, 2013, at 10:13 AM, Hin-Tak Leung wrote:

        
>> Network access is *not* a given, nor is the privilege of
    >> installing arbitrary "uncertified" and "non-essential"
    >> tools - whatever the meaning of "uncertified" and
    >> "non-essential" are, those being defined, as is "design
    >> goal", etc, by some small committee.
    >> 
    >> It is a very common scenario, e.g. banks & telecom, some
    >> part of public/government service and health care. This
    >> does not seem to sink in without repeating.
    >> 

    > Network access is not needed to build R - apparently that
    > fact did not seem to sink in, either. This entire thread
    > is based on false assumptions and as such the only place
    > for it is /dev/null

Indeed!

Hin-Tak, do you really think that we, R core, would cripple
ourselves in such a way ??
I bet that almost all of us build R (from svn) without internet
connection many times a year.

But we do follow the build presciptions (*) which you have taken
enormous lengths *not* to go along with.

Martin

---
(*) builddir != srcdir


    >> --- On Sat, 16/3/13, Hin-Tak Leung

        
>> <htl10 at users.sourceforge.net> wrote:

        
>> 
    >>> I'll quantify the first part - R is perhaps the only
    >>> public software project hosted on a subversion
    >>> repository for which the result of 'svn export ...' does
    >>> not build. Not only that it does not build, but make it
    >>> a feature that it does not build.
    >>> 
    >>> Very few other projects actively try to go in that
    >>> direction.
    >>> 
    >>> --- On Fri, 15/3/13, Hin-Tak Leung

        
>>> <hintak_leung at yahoo.co.uk> wrote:

        
>>> 
    >>>> The decision to actively discourage non-subsersion
    >>>> usage of snapshot build is already made (r62183). So I
    >>>> am just here to register a differing opinion.
    >>>> 
    >>>> - it is not about subversion vs
    >>> other-version-control-tools.
    >>>> There are two parts of R's dev build process which
    >>> requires
    >>>> an active network connection - tools/rsync-recommended
    >>> and
    >>>> capturing `svn info` into R's headers. The former can
    >>> be
    >>>> overridden with "./configure
    >>>> --with-recommended-packages=no". A few changes had been
    >>> made
    >>>> in the last 6 months to make the latter mandatory. It
    >>> used
    >>>> to be optional.
    >>>> 
    >>>> --- there are genuine needs for testing snapshots in a
    >>>> non-networked minimalist environment - e.g. banks or
    >>> telecom
    >>>> industries - where one wants a "standardised host"
    >>> build,
    >>>> and often it means an "outdated host"; or in a virtual
    >>>> machine environment - which are non-networked for
    >>> security
    >>>> reasons, and also do not have tools beyond necessary
    >>> for
    >>>> compiling and building. This is quite a common
    >>> scenario.
    >>>> 
    >>>> --- AFAIK, 6 months ago, a snapshot copied to an
    >>>> non-networked host will build with "./configure
    >>>> --with-recommended-packages=no". Of course copying
    >>> those
    >>>> recommended package tar balls across would be nicer.
    >>> This is
    >>>> sadly no longer the case.
    >>>> 
    >>>> - dependent on `svn info` and sitting on top of a svn
    >>>> checkout possibly also affects cross-compiling. But
    >>> then, it
    >>>> is not clear whether it is still possible to
    >>> cross-compile
    >>>> R, since quite a few changes have been made to remove
    >>> the
    >>>> capability of cross-compiling R for windows on unix
    >>> hosts in
    >>>> the last few years.
    >>>> 
    >>>> - testing dev snapshots is about trying things and
    >>> fixing
    >>>> bugs before release. Making it difficult for non-core
    >>> people
    >>>> to "try", seem to go against that ethos. If that's the
    >>> case,
    >>>> maybe the repository could be closed off to anonymous
    >>> check
    >>>> outs altogether, just to make it clear that testing
    >>>> snapshots before releases or even close to release, is
    >>> not
    >>>> welcomed. Just a thought.
    >>>> 
    >>>> - although the official repository of the "main" linux
    >>>> kernel branch is git-based, there are mercurial
    >>> mirrors;
    >>>> AFAIK the digital video broadcasting hardware support
    >>> of the
    >>>> linux kernel is officially in mercurial repositories,
    >>> but
    >>>> have git mirrors; likewise much of Xorg's is in
    >>> mercurial
    >>>> but have git mirrors. I haven't heard of any much
    >>> bigger
    >>>> public projects than R where some actively discourage
    >>>> others.
    >>>> 
    >>>> - To be honest r62183 itself is probably a good reason
    >>> to
    >>>> move away from server-side repositories to distributed
    >>>> repositories (hg/git/arch/bzr). Local enhancements -
    >>> i.e. an
    >>>> in-house fork - some of which are never going upstream,
    >>> such
    >>>> as a local revert of r62183 (or a local revert of any
    >>> other
    >>>> upstream commits) is one reason why some have
    >>> distributed
    >>>> repositories.
    >>>> 
    >>>> Lastly, a minor grip. The current head of the HK
    >>> government
    >>>> is probably sometimes called "HK Leung", just as some
    >>> might
    >>>> call a certain old lady "UK Windsor" and a certain
    >>> black
    >>>> person "US Obama".
    >>>> 
    >>> 
    >> 
    >> ______________________________________________
    >> R-devel at r-project.org mailing list
    >> https://stat.ethz.ch/mailman/listinfo/r-devel
    >> 
    >> 

    > ______________________________________________
    > R-devel at r-project.org mailing list
    > https://stat.ethz.ch/mailman/listinfo/r-devel

  
  


  


      

    

    

      
      

        


  

        

      Hin-Tak Leung
    

    

      
      #
    

  


  

      
--- On Sat, 16/3/13, Hin-Tak Leung <htl10 at users.sourceforge.net> wrote:

        

            

      
      
      
    

        
Oh, and some might call using a tool beyond its initial purpose and context an enhancement, rather than 'misuse'. Quite a lot of words, like 'certified', 'essential', have no meaning outside the small group of people who use and misuse them, as is 'misuse'.