Possible bug in gzcon() (6161:src/main/connections.c)
? Fri, 25 Apr 2025 13:41:35 +0000 Andr? Wildberg <andre.wildberg at outlook.com> ?????:
Reproducible example: addr <- "https://www1.ncdc.noaa.gov/pub/data/ghcn/daily/by_station/USW00014839.csv.gz" # online/stream nrow(read.csv(gzcon(url(addr), text=T), header=F)) # [1] 1798 # local download.file(addr, destfile=basename(addr)) nrow(read.csv(gzcon(file(basename(addr), "r"), text=T), header=F)) # [1] 429498
I can reproduce the problem (with slightly different numbers):
length(readLines(gzcon(file("USW00014839.csv.gz", "rb"), text = TRUE)))
# [1] 28002
length(readLines("USW00014839.csv.gz"))
# [1] 429535
The underlying reason for the problem you're having with gzcon() is
most likely that the gzip archive has been concatenated from multiple
separate archives:
perl -MIO::Uncompress::Gunzip -E'
my $z = IO::Uncompress::Gunzip::->new(shift);
say "end of stream" while $z->nextStream() == 1;
' -- USW00014839.csv.gz
# end of stream
# end of stream
# end of stream
# end of stream
# end of stream
readLines("USW00014839.csv.gz") calls file(), which can transparently
switch to a gzfile() connection, which supports concatenated archives,
but gzcon() currently doesn't. Feature request submitted at
<https://bugs.r-project.org/show_bug.cgi?id=18887>.
Best regards, Ivan