Skip to content
Prev 16831 / 63424 Next
Original Raw

download.file() yields incomplete files with method="internal" (PR#7991)

mbrmbr@acm.org 
Summary:
When I use method="wget" with download.file(), I consistently get 
a download of the entire file. When I use method="internal", I 
infrequently get the entire file, but usually get only part of the file. This
behavior occurs with .cdf (a weather file format - basically binary)
from a UCAR site.

I am not sure this is a bug, since it could be some internet
inconsistency, or I may be doing something wrong, but given that wget
works and internal doesn't, it might be a bug.

Appended to this message is an example of the output from a download.file
using method="wget" and method="internal", including warnings. After that
are the warnings from looping 10 times using method="internal".

Thank you for your time.

Mark.
=================================================================
******* method="wget" ************
--14:10:57--  ftp://ftp.atd.ucar.edu/pub/archive/weather/foothills/flab.20050619.cdf
           => `/export/home/mbr/boulder/foothills-weather-data/flab.20050619.cdf'
Resolving ftp.atd.ucar.edu... 128.117.80.209
Connecting to ftp.atd.ucar.edu[128.117.80.209]:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /pub/archive/weather/foothills ... done.
==> PASV ... done.    ==> RETR flab.20050619.cdf ... done.
Length: 16,940 (unauthoritative)

100%[====================================>] 16,940        48.51K/s             

14:10:58 (48.43 KB/s) - `/export/home/mbr/boulder/foothills-weather-data/flab.20050619.cdf' saved [16,940]

************** method="internal" ************************
trying URL 'ftp://ftp.atd.ucar.edu/pub/archive/weather/foothills/flab.20050619.cdf'
ftp data connection made, file length 16940 bytes
opened URL
==========================================
downloaded 14Kb

There were 33 warnings (use warnings() to see them)
Warning messages:
1: RxmlNanoFTPGetMore : read 82 [0 - 82]
2: 
<<<
220-----------------------------------------------------------------------------

--
3: RxmlNanoFTPGetMore : read 77 [82 - 159]
4: 
<<<
220-malbec.atd.ucar.edu: NCAR/ATD SE host, for official NCAR business only.
---

--
5: RxmlNanoFTPGetMore : read 512 [0 - 512]
6: 
<<<
220-
220-        ***        National Center for Atmospheric Research       ***
220-        ***           Atmospheric Technology Division             ***
220-
220-        ***   Use of this system for unauthorized purposes is a   ***
220-        ***   violation of federal law.  See Public Law 99-474.   ***
220-        ***   Any activity may be logged and monitored.  By       ***
220-        ***   using this system, you consent to such logging and  ***
220-        ***   monitoring.                     
--
7: RxmlNanoFTPGetMore : read 112 [0 - 112]
8: 
<<<
                    ***
220----------------------------------------------------------------------------
220 

--
9: 
---

--
10: Got 220
11: USER anonymous
12: RxmlNanoFTPGetMore : read 23 [0 - 23]
13: 
<<<
230 Login successful.

--
14: 
---

--
15: Got 230
16: RxmlNanoFTPGetMore : read 51 [0 - 51]
17: 
<<<
227 Entering Passive Mode (128,117,80,209,192,27)

--
18: 
---

--
19: Got 227
20: RxmlNanoFTPGetMore : read 31 [0 - 31]
21: 
<<<
200 Switching to Binary mode.

--
22: 
---

--
23: Got 200
24: RxmlNanoFTPGetMore : read 109 [0 - 109]
25: 
<<<
150 Opening BINARY mode data connection for /pub/archive/weather/foothills/flab.20050619.cdf (16940 bytes).

--
26: 
---

--
27: Got 150
28: RxmlNanoFTPGetMore : read 19 [0 - 19]
29: 
<<<
226 File send OK.

--
30: 
---

--
31: Got 226
32: QUIT
33: downloaded length 14480 != reported length 16940

************** method="internal" 10 times ************************
s<-download.file("ftp://ftp.atd.ucar.edu/pub/archive/weather/foothills/flab.20050619.cdf",destfile="~/boulder/foothills-weather-data/flab.20050619.cdf",mode="wb",method="internal")
print(s)
}

[verbose output removed]
Warning messages:
1: downloaded length 13032 != reported length 16940 
2: downloaded length 13032 != reported length 16940 
3: downloaded length 11584 != reported length 16940 
4: downloaded length 13032 != reported length 16940 
5: downloaded length 11584 != reported length 16940 
6: downloaded length 8688 != reported length 16940 
7: downloaded length 14480 != reported length 16940 
8: downloaded length 13032 != reported length 16940 
9: downloaded length 13032 != reported length 16940 
10: downloaded length 13032 != reported length 16940
_                
platform i686-pc-linux-gnu
arch     i686             
os       linux-gnu        
system   i686, linux-gnu  
status                    
major    2                
minor    1.1              
year     2005             
month    06               
day      20               
language R