Skip to content

download.file suggestion: if(OSX){ use curl}

4 messages · Timothy Bates, Simon Urbanek

Timothy Bates
# 
Dear R mac users,

As services that enforce secure connections like https sftp proliferate, it would be nice if R could handle accessing files over the secure services.

Currently download.file() doesn?t handle https (or anything secure)

The special interest for Mac users is that we have <curl> out of the box. 

I wonder if the download.file() function could be special cased for our platforms that have curl on board and use that?

best wishes,
tim


Wikipedia snippet of the day: 
In 1861, income tax was reduced to ninepence; in 1863 to sevenpence; in 1864 to fivepence; and in 1865 to fourpence. Gladstone believed that government was extravagant and wasteful with taxpayers' money and so sought to let money "fructify in the pockets of the people" by keeping taxation levels down through "peace and retrenchment".

http://en.wikipedia.org/wiki/William_Ewart_Gladstone
3 days later
Simon Urbanek
#
On Jul 4, 2011, at 7:38 AM, Timothy Bates wrote:

            

      
      
      
    

        
It already supports curl so why should it special-case anything? If you want to use curl by default (I don't for example) just set the download.file.method option.

Cheers,
Simon

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Timothy Bates
    

    

      
      #
    

  


  

      
Thanks Simon: that?s good news (if somewhat hidden?)

But?.
While this works

   secureURL = "https://dl.dropbox.com/s/pl60zb6cljootdj/test.txt?
   download.file(secureURL, "junk.txt", method=?curl?)

This doesn?t?
  
  read.table(secureURL)
  # Error in file(file, "rt") : cannot open the connection
  # In addition: Warning message:
  # In file(file, "rt") : unsupported URL scheme


The reason I thought that curl was explicitly not supported was the Rd help for <file> and <url> in ~/bin/R/trunk/src/library/base/man/connections.Rd

            

      
      
      
    

        
and

            

      
      
      
    

        
file() does not support this option, even if download.file() is told too.

If there is a way around this, or file and url can be changed, then I?m happy to submit Rd diffs or other help so people can find this valuable functionality at the  read.xxx level.

Something like 

url and file always support the http://, ftp:// and file:// URL schemes, and may support secure connections (https, sftp etc.).

#ifdef unix
  Secure connections (for instance https://) are supported via use of helpers such as curl which can be set via the <download.file.method> option of download.file.
#endif

With an example of how to do that. Thoughts welcome.

        
On 7 Jul 2011, at 2:33 PM, Simon Urbanek wrote:

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Simon Urbanek
    

    

      
      #
    

  


  

      
download.file() and file() are two entirely independent facilities. file()/url() etc. are connections - and none of them supports https - not even on the Mac. In contrast, download.file() is simply a utility to download a file - it has nothing to do with connections (except for the fact that it may choose to use connections to accomplish the task - or not). You could equally well just call curl yourself -- and in fact that is probably the closest to what you want:

f=pipe("curl -s -S https://rforge.net")
read.table(f)
close(f)

Finally, you may want to have a look at RCurl and getURLContent() and alike.

Cheers,
Simon

        
On Jul 7, 2011, at 10:33 AM, Timothy Bates wrote: