Skip to content

[R-pkg-devel] download.file and https

7 messages · Duncan Murdoch, Paul Gilbert, Dirk Eddelbuettel +1 more

Original
Paul Gilbert
#
On 07/02/2015 10:52 PM, Henrik Bengtsson wrote:
I do have TRUE for this. The default behaviour still does not work.

Paul
Duncan Murdoch
#
On 03/07/2015 5:24 AM, Paul Gilbert wrote:
Have you tried specifying the protocol explicitly, i.e.

 download.file("https://research.stlouisfed.org/fred2/series/M2/downloaddata/M2.csv", destfile = tmp, method="libcurl")

?

Duncan Murdoch
Paul Gilbert
#
On 07/02/2015 11:52 PM, Duncan Murdoch wrote:
Yes. The default does not accept a string with https.  It gives 
"unsupported URL scheme".

A few people have mentioned workarounds or setting options. But, as 
Joshua mentioned, the question is really about best practice to make a 
package robust on all platforms, and tests that pass on CRAN.

Paul
Dirk Eddelbuettel
# 
In random, I do the following, using R 3.2.0 or later with libcurl where
installed, and Jeroen's lightweight curl::curl otherwise.


getConnection <- function(urltxt, ...) {
    if (getRversion() >= "3.2.0" && capabilities()["libcurl"]) {
        url(urltxt, ..., method="libcurl")
    } else {
        curl(urltxt, ...)
    }
}

closeConnection <- function(con) {
    if (getRversion() >= "3.2.0" && capabilities()["libcurl"]) {
        close(con)
    } else {
        ## nothing to be done for curl()
    }
}


Hth,  Dirk
Paul Gilbert
# 
The difficulty with most workarounds suggested so far is the expanded 
list of dependencies. My package and quantmod, have had a simple 
dependency on download.file in utils. Picking on Dirks suggestion as an 
example, it would add a system dependency on libcurl and an R package 
dependency on curl, which in turn has system dependencies libcurl-devel 
(rpm) or libcurl4-openssl-dev (deb). Furthermore, it suggests testthat, 
knitr, jsonlite, rmarkdown, and magrittr, which means that these and 
their dependencies, imports, and suggests all need to be installed and 
working in order for me to test, and working with R-devel for me to 
submit to CRAN.

Let me mention again that this problem is not specific to my package or 
quantmod. It will be quickly affecting everyone that downloads from US 
government websites, not to mention that other governments and 
commercial web sites may start doing the same thing.

Paul
On 07/03/2015 06:39 AM, Dirk Eddelbuettel wrote:
Winston Chang
# 
My downloader package was created to deal with https on various platforms:
  http://cran.r-project.org/web/packages/downloader/index.html

The package dates back to before R had libcurl support. It's very
simple: if you're on Windows, it uses setInternet2(), which enable
https support, and if you're on non-Windows, it looks for wget, curl,
and then lynx.

You don't have to import downloader in order to do this - there's just
one function you'd need, which, as far as I'm concerned, you can put
in your code:
  https://github.com/wch/downloader/blob/master/R/download.r

If you need the code to officially relicensed, feel free to contact me by email.

-Winston
On Fri, Jul 3, 2015 at 9:24 AM, Paul Gilbert <pgilbert902 at gmail.com> wrote:
Dirk Eddelbuettel
#
On 3 July 2015 at 10:24, Paul Gilbert wrote:
| The difficulty with most workarounds suggested so far is the expanded 
| list of dependencies. My package and quantmod, have had a simple 
| dependency on download.file in utils. Picking on Dirks suggestion as an 
| example, it would add a system dependency on libcurl and an R package 
| dependency on curl, which in turn has system dependencies libcurl-devel 
| (rpm) or libcurl4-openssl-dev (deb). Furthermore, it suggests testthat, 

Well but you cannot have https without these.

Moreover, Depends != Suggests.  That really matters.

| knitr, jsonlite, rmarkdown, and magrittr, which means that these and 
| their dependencies, imports, and suggests all need to be installed and 
| working in order for me to test, and working with R-devel for me to 
| submit to CRAN.

Not my experience.

The .travis.yml I use for random (still) installs curl from source, yet is
done in under a minute.  I never remember whether others can browse my test
history but try

   https://travis-ci.org/eddelbuettel/random/builds   

As Michael Rutter has curl in his cran2deb4ubuntu PPA knows by the R-Travis
setup, I might as well install his r-cran-curl which would make this yet a
little faster.
 
| Let me mention again that this problem is not specific to my package or 
| quantmod. It will be quickly affecting everyone that downloads from US 
| government websites, not to mention that other governments and 
| commercial web sites may start doing the same thing.

Use of curl (to provide https) will become much more common. R now supports
it in R 3.2.0, but only as optional build-time choice.  With that I find
Jeroen's package a pretty good choice.

But that is of course just one suggestion, and if it doesn't work for you I
fully understand.  I was faced with the exact same problem -- Mads switched
random.org to https-only -- and would like to see what other suggestions
people may have.

Dirk
 
| Paul
|
| On 07/03/2015 06:39 AM, Dirk Eddelbuettel wrote:
| >
| > In random, I do the following, using R 3.2.0 or later with libcurl where
| > installed, and Jeroen's lightweight curl::curl otherwise.
| >
| >
| > getConnection <- function(urltxt, ...) {
| >      if (getRversion() >= "3.2.0" && capabilities()["libcurl"]) {
| >          url(urltxt, ..., method="libcurl")
| >      } else {
| >          curl(urltxt, ...)
| >      }
| > }
| >
| > closeConnection <- function(con) {
| >      if (getRversion() >= "3.2.0" && capabilities()["libcurl"]) {
| >          close(con)
| >      } else {
| >          ## nothing to be done for curl()
| >      }
| > }
| >
| >
| > Hth,  Dirk
| >