Does anyone know how to close an FTP session with RCurl?? I am trying to automate the process of downloading snow data from a government website, and their server is throttling the connection after a few files are downloaded.? I contacted their system administrator, who wrote: ?My suspicion at this point is that the getURL commands are opened and perform the function asked, then linger in wait for 15 minutes until or ftp server closes the idle sessions. Is there a way to tell R to close the sessions??
I?ve perused the RCurl manual but I don?t see a way to close sessions.? I tried copying the following example from the RCurl manual, but it didn?t solve the problem.? I?m a novice at this and I don?t understand the relationship between handles and sessions, so I am probably missing something.
#EXAMPLE from getCurl(), p. 39
if(url.exists("http://www.omegahat.net/RCurl/index.html")) {
??curl = getCurlHandle()
?? getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)
?? #getCurlInfo(curl) # I skipped this step
?? rm(curl) # release the curl! (does this end the session???)
}
Thanks!
John
Closing FTP sessions with RCurl
6 messages · Tom, J Payne, William Dunlap
No expert here, and this isn't tested. It seems you can set the
forbid.reuse option which will cause curl to shutdown the connection
after transfer is complete.
if(url.exists("http://www.omegahat.net/RCurl/index.html")) {
curl <- getCurlHandle()
curlSetOpt(.opts=list(forbid.reuse=1),curl=curl)
getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)
}
On Tue, Jun 14, 2016 at 2:41 PM, J Payne <jcpayne at uw.edu> wrote:
Does anyone know how to close an FTP session with RCurl? I am trying to automate the process of downloading snow data from a government website, and their server is throttling the connection after a few files are downloaded. I contacted their system administrator, who wrote: ?My suspicion at this point is that the getURL commands are opened and perform the function asked, then linger in wait for 15 minutes until or ftp server closes the idle sessions. Is there a way to tell R to close the sessions??
I?ve perused the RCurl manual but I don?t see a way to close sessions. I tried copying the following example from the RCurl manual, but it didn?t solve the problem. I?m a novice at this and I don?t understand the relationship between handles and sessions, so I am probably missing something.
#EXAMPLE from getCurl(), p. 39
if(url.exists("http://www.omegahat.net/RCurl/index.html")) {
curl = getCurlHandle()
getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)
#getCurlInfo(curl) # I skipped this step
rm(curl) # release the curl! (does this end the session???)
}
Thanks!
John
[[alternative HTML version deleted]]
______________________________________________ R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Thanks Tom! I tried that and it didn?t work, but perhaps there are other problems. The system administrator wrote ?I think closing sessions immediately upon completion of the transfer, or continuing to use the same session for additional requests would both have a similar and positive effect.? However, I?ve now tried your solution to close the session, and also tried using one curl handle throughout, but the throttling persists. John
On 6/14/16, 1:00 PM, "Tom Wright" <tom at maladmin.com> wrote:
No expert here, and this isn't tested. It seems you can set the
forbid.reuse option which will cause curl to shutdown the connection
after transfer is complete.
if(url.exists("http://www.omegahat.net/RCurl/index.html")) {
curl <- getCurlHandle()
curlSetOpt(.opts=list(forbid.reuse=1),curl=curl)
getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)
}
On Tue, Jun 14, 2016 at 2:41 PM, J Payne <jcpayne at uw.edu> wrote:
Does anyone know how to close an FTP session with RCurl? I am trying to automate the process of downloading snow data from a government website, and their server is throttling the connection after a few files are downloaded. I contacted their system administrator, who wrote: ?My suspicion at this point is that the getURL commands are opened and perform the function asked, then linger in wait for 15 minutes until or ftp server closes the idle sessions. Is there a way to tell R to close the sessions??
I?ve perused the RCurl manual but I don?t see a way to close sessions. I tried copying the following example from the RCurl manual, but it didn?t solve the problem. I?m a novice at this and I don?t understand the relationship between handles and sessions, so I am probably missing something.
#EXAMPLE from getCurl(), p. 39
if(url.exists("http://www.omegahat.net/RCurl/index.html")) {
curl = getCurlHandle()
getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)
#getCurlInfo(curl) # I skipped this step
rm(curl) # release the curl! (does this end the session???)
}
Thanks!
John
[[alternative HTML version deleted]]
______________________________________________ R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
rm(curl) # release the curl! (does this end the session???)
Try adding a call to gc() immediately after this removal. That will force an immediate run of any finalizer associated with the object just removed. With the call to gc(), the garbage collector will be called some time in the future and the finalizers will be run then. (I don't know if curl has a finalizer that closes the session.) Bill Dunlap TIBCO Software wdunlap tibco.com
On Wed, Jun 15, 2016 at 12:34 PM, J Payne <jcpayne at uw.edu> wrote:
Thanks Tom! I tried that and it didn?t work, but perhaps there are other problems. The system administrator wrote ?I think closing sessions immediately upon completion of the transfer, or continuing to use the same session for additional requests would both have a similar and positive effect.? However, I?ve now tried your solution to close the session, and also tried using one curl handle throughout, but the throttling persists. John On 6/14/16, 1:00 PM, "Tom Wright" <tom at maladmin.com> wrote:
No expert here, and this isn't tested. It seems you can set the
forbid.reuse option which will cause curl to shutdown the connection
after transfer is complete.
if(url.exists("http://www.omegahat.net/RCurl/index.html")) {
curl <- getCurlHandle()
curlSetOpt(.opts=list(forbid.reuse=1),curl=curl)
getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)
}
On Tue, Jun 14, 2016 at 2:41 PM, J Payne <jcpayne at uw.edu> wrote:
Does anyone know how to close an FTP session with RCurl? I am trying
to automate the process of downloading snow data from a government website, and their server is throttling the connection after a few files are downloaded. I contacted their system administrator, who wrote: ?My suspicion at this point is that the getURL commands are opened and perform the function asked, then linger in wait for 15 minutes until or ftp server closes the idle sessions. Is there a way to tell R to close the sessions??
I?ve perused the RCurl manual but I don?t see a way to close sessions.
I tried copying the following example from the RCurl manual, but it didn?t solve the problem. I?m a novice at this and I don?t understand the relationship between handles and sessions, so I am probably missing something.
#EXAMPLE from getCurl(), p. 39
if(url.exists("http://www.omegahat.net/RCurl/index.html")) {
curl = getCurlHandle()
getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)
#getCurlInfo(curl) # I skipped this step
rm(curl) # release the curl! (does this end the session???)
}
Thanks!
John
[[alternative HTML version deleted]]
______________________________________________ R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. ______________________________________________ R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Fantastic!? That did the trick.? I still suspect that there may be a command within RCurl that accomplishes the same thing, but in any case I?m very grateful to have a solution that works.? Best, John From: William Dunlap <wdunlap at tibco.com> Date: Wednesday, June 15, 2016 at 12:57 PM To: J Payne <jcpayne at uw.edu> Cc: Tom Wright <tom at maladmin.com>, R help list <r-help at r-project.org> Subject: Re: [R] Closing FTP sessions with RCurl
rm(curl) # release the curl! (does this end the session???)
Try adding a call to gc() immediately after this removal. That will force an immediate run of any finalizer associated with the object just removed. With the call to gc(), the garbage collector will be called some time in the future and the finalizers will be run then. (I don't know if curl has a finalizer that closes the session.) Bill Dunlap TIBCO Software wdunlap tibco.com
On Wed, Jun 15, 2016 at 12:34 PM, J Payne <jcpayne at uw.edu> wrote:
Thanks Tom! I tried that and it didn?t work, but perhaps there are other problems. The system administrator wrote ?I think closing sessions immediately upon completion of the transfer, or continuing to use the same session for additional requests would both have a similar and positive effect.? However, I?ve now tried your solution to close the session, and also tried using one curl handle throughout, but the throttling persists. John
On 6/14/16, 1:00 PM, "Tom Wright" <tom at maladmin.com> wrote:
No expert here, and this isn't tested. It seems you can set the
forbid.reuse option which will cause curl to shutdown the connection
after transfer is complete.
if(url.exists("http://www.omegahat.net/RCurl/index.html")) {
curl <- getCurlHandle()
curlSetOpt(.opts=list(forbid.reuse=1),curl=curl)
getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)
}
On Tue, Jun 14, 2016 at 2:41 PM, J Payne <jcpayne at uw.edu> wrote:
Does anyone know how to close an FTP session with RCurl? I am trying to automate the process of downloading snow data from a government website, and their server is throttling the connection after a few files are downloaded. I contacted their system administrator, who wrote: ?My suspicion at this point is that the getURL commands are opened and perform the function asked, then linger in wait for 15 minutes until or ftp server closes the idle sessions. Is there a way to tell R to close the sessions??
I?ve perused the RCurl manual but I don?t see a way to close sessions. I tried copying the following example from the RCurl manual, but it didn?t solve the problem. I?m a novice at this and I don?t understand the relationship between handles and sessions, so I am probably missing something.
#EXAMPLE from getCurl(), p. 39
if(url.exists("http://www.omegahat.net/RCurl/index.html")) {
curl = getCurlHandle()
getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)
#getCurlInfo(curl) # I skipped this step
rm(curl) # release the curl! (does this end the session???)
}
Thanks!
John
[[alternative HTML version deleted]]
______________________________________________ R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
______________________________________________ R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
With the call to gc(), the garbage collector will be called some time in
the
future and the finalizers will be run then.
Typo: that initial 'With' should be 'Without'. Bill Dunlap TIBCO Software wdunlap tibco.com
On Wed, Jun 15, 2016 at 12:57 PM, William Dunlap <wdunlap at tibco.com> wrote:
rm(curl) # release the curl! (does this end the session???)
Try adding a call to gc() immediately after this removal. That will force an immediate run of any finalizer associated with the object just removed. With the call to gc(), the garbage collector will be called some time in the future and the finalizers will be run then. (I don't know if curl has a finalizer that closes the session.) Bill Dunlap TIBCO Software wdunlap tibco.com On Wed, Jun 15, 2016 at 12:34 PM, J Payne <jcpayne at uw.edu> wrote:
Thanks Tom! I tried that and it didn?t work, but perhaps there are other problems. The system administrator wrote ?I think closing sessions immediately upon completion of the transfer, or continuing to use the same session for additional requests would both have a similar and positive effect.? However, I?ve now tried your solution to close the session, and also tried using one curl handle throughout, but the throttling persists. John On 6/14/16, 1:00 PM, "Tom Wright" <tom at maladmin.com> wrote:
No expert here, and this isn't tested. It seems you can set the
forbid.reuse option which will cause curl to shutdown the connection
after transfer is complete.
if(url.exists("http://www.omegahat.net/RCurl/index.html")) {
curl <- getCurlHandle()
curlSetOpt(.opts=list(forbid.reuse=1),curl=curl)
getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)
}
On Tue, Jun 14, 2016 at 2:41 PM, J Payne <jcpayne at uw.edu> wrote:
Does anyone know how to close an FTP session with RCurl? I am trying
to automate the process of downloading snow data from a government website, and their server is throttling the connection after a few files are downloaded. I contacted their system administrator, who wrote: ?My suspicion at this point is that the getURL commands are opened and perform the function asked, then linger in wait for 15 minutes until or ftp server closes the idle sessions. Is there a way to tell R to close the sessions??
I?ve perused the RCurl manual but I don?t see a way to close
sessions. I tried copying the following example from the RCurl manual, but it didn?t solve the problem. I?m a novice at this and I don?t understand the relationship between handles and sessions, so I am probably missing something.
#EXAMPLE from getCurl(), p. 39
if(url.exists("http://www.omegahat.net/RCurl/index.html")) {
curl = getCurlHandle()
getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)
#getCurlInfo(curl) # I skipped this step
rm(curl) # release the curl! (does this end the session???)
}
Thanks!
John
[[alternative HTML version deleted]]
______________________________________________ R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. ______________________________________________ R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.