Dear R Users,
I am encountering a problem when reading nc files into R using the ncdf and ncdf4 libraries. The nc files are too large to attach an example (but if someone is interested in helping out I could send a file privately via an online drive), but the code is basic:
for(i in 1:length(thesenames[,1])){
data <- nc_open(paste(INDIR, thesenames[i,c("wholename")], sep=""), write=F)
d.vars <- names(data$var)
d.size <- (data$var[[length(d.vars)]])$size
# Obtaining longitude and latitude values
d.lon <- as.vector(ncvar_get(data, varid="lon", start=c(1,1), count=c(d.size[1],d.size[2])))
d.lat <- as.vector(ncvar_get(data, varid="lat", start=c(1,1), count=c(d.size[1],d.size[2])))
# Obtaining climate data values
df.clim <- data.frame(rn=seq(1:length(d.lon)))
for(y in 1:d.size[3]){
df.clim[,1+y] <- as.vector(ncvar_get(data, varid=d.vars[length(d.vars)], start=c(1,1,y), count=c(d.size[1],d.size[2],1)))
names(df.clim)[1+y] <- paste("y",y,sep="") }
tosummarise[,,i] <- as.matrix(df.clim[,-1])
}
The data are temperature or precipitation, across space and time.
For most of the >250 files I have, there are no problems, but for around 8 of these files, I get strange values. The data should be within a relatively narrow range, yet I get values such as -8.246508e+07 or 7.659506e+11. The particularly strange part is that these kind of values occur at regularly spaced intervals across the data, usually within a single time step.
I have the same problem (including the exact same strange values) when using ArcMap, yet the data provider assures me that the data look normal when using CDO (climate data operators) to view them, and that there are no missing values.
I realise this is very difficult to diagnose without the nc files themselves, so my questions are (1) Has anyone encountered something like this before?, (2) Is there something I am failing to specify in the code when reading in?, and (3) Is anyone interested in digging into this and willing to play around with the nc files if I make them available privately?
Thanks very much in advance!
Louise
Unexpected values obtained when reading in data using ncdf and ncdf4
3 messages · Louise Mair, David W. Pierce, Roy Mendelssohn - NOAA Federal
On Fri, Apr 22, 2016 at 1:32 AM, Louise Mair <louise.mair at slu.se> wrote:
Dear R Users, I am encountering a problem when reading nc files into R using the ncdf and ncdf4 libraries. The nc files are too large to attach an example (but if someone is interested in helping out I could send a file privately via an online drive), but the code is basic:
?[...]? ?Hi Louise, I'm the author of the ncdf and ncdf4 libraries. What are the details -- what operating system are you running on, what version of R and the netcdf library are you using? If you make the files available to me I can take a look. Regards, --Dave Pierce ?
for(i in 1:length(thesenames[,1])){
data <- nc_open(paste(INDIR, thesenames[i,c("wholename")], sep=""),
write=F)
d.vars <- names(data$var)
d.size <- (data$var[[length(d.vars)]])$size
# Obtaining longitude and latitude values
d.lon <- as.vector(ncvar_get(data, varid="lon", start=c(1,1),
count=c(d.size[1],d.size[2])))
d.lat <- as.vector(ncvar_get(data, varid="lat", start=c(1,1),
count=c(d.size[1],d.size[2])))
# Obtaining climate data values
df.clim <- data.frame(rn=seq(1:length(d.lon)))
for(y in 1:d.size[3]){
df.clim[,1+y] <- as.vector(ncvar_get(data,
varid=d.vars[length(d.vars)], start=c(1,1,y),
count=c(d.size[1],d.size[2],1)))
names(df.clim)[1+y] <- paste("y",y,sep="") }
tosummarise[,,i] <- as.matrix(df.clim[,-1])
}
The data are temperature or precipitation, across space and time.
For most of the >250 files I have, there are no problems, but for around 8
of these files, I get strange values. The data should be within a
relatively narrow range, yet I get values such as -8.246508e+07 or
7.659506e+11. The particularly strange part is that these kind of values
occur at regularly spaced intervals across the data, usually within a
single time step.
I have the same problem (including the exact same strange values) when
using ArcMap, yet the data provider assures me that the data look normal
when using CDO (climate data operators) to view them, and that there are no
missing values.
I realise this is very difficult to diagnose without the nc files
themselves, so my questions are (1) Has anyone encountered something like
this before?, (2) Is there something I am failing to specify in the code
when reading in?, and (3) Is anyone interested in digging into this and
willing to play around with the nc files if I make them available privately?
Thanks very much in advance!
Louise
[[alternative HTML version deleted]]
______________________________________________ R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
David W. Pierce Division of Climate, Atmospheric Science, and Physical Oceanography Scripps Institution of Oceanography, La Jolla, California, USA (858) 534-8276 (voice) / (858) 534-8561 (fax) dpierce at ucsd.edu [[alternative HTML version deleted]]
Hi Louise: If Dave can?t figure it out, I can give a look also. A couple of things I would suggest: 1. Don?t use the name ?data? in the nc_open command, that is a reserved command in R and you never know what problems that can cause. 2. You are doing calculations to get set the start and count values in the ncvar_get commands, print those values out before you make the calls to make certain they are valid. HTH, -Roy
On Apr 22, 2016, at 8:08 AM, David W. Pierce <dpierce at ucsd.edu> wrote: On Fri, Apr 22, 2016 at 1:32 AM, Louise Mair <louise.mair at slu.se> wrote:
Dear R Users, I am encountering a problem when reading nc files into R using the ncdf and ncdf4 libraries. The nc files are too large to attach an example (but if someone is interested in helping out I could send a file privately via an online drive), but the code is basic:
?[...]? ?Hi Louise, I'm the author of the ncdf and ncdf4 libraries. What are the details -- what operating system are you running on, what version of R and the netcdf library are you using? If you make the files available to me I can take a look. Regards, --Dave Pierce ?
for(i in 1:length(thesenames[,1])){
data <- nc_open(paste(INDIR, thesenames[i,c("wholename")], sep=""),
write=F)
d.vars <- names(data$var)
d.size <- (data$var[[length(d.vars)]])$size
# Obtaining longitude and latitude values
d.lon <- as.vector(ncvar_get(data, varid="lon", start=c(1,1),
count=c(d.size[1],d.size[2])))
d.lat <- as.vector(ncvar_get(data, varid="lat", start=c(1,1),
count=c(d.size[1],d.size[2])))
# Obtaining climate data values
df.clim <- data.frame(rn=seq(1:length(d.lon)))
for(y in 1:d.size[3]){
df.clim[,1+y] <- as.vector(ncvar_get(data,
varid=d.vars[length(d.vars)], start=c(1,1,y),
count=c(d.size[1],d.size[2],1)))
names(df.clim)[1+y] <- paste("y",y,sep="") }
tosummarise[,,i] <- as.matrix(df.clim[,-1])
}
The data are temperature or precipitation, across space and time.
For most of the >250 files I have, there are no problems, but for around 8
of these files, I get strange values. The data should be within a
relatively narrow range, yet I get values such as -8.246508e+07 or
7.659506e+11. The particularly strange part is that these kind of values
occur at regularly spaced intervals across the data, usually within a
single time step.
I have the same problem (including the exact same strange values) when
using ArcMap, yet the data provider assures me that the data look normal
when using CDO (climate data operators) to view them, and that there are no
missing values.
I realise this is very difficult to diagnose without the nc files
themselves, so my questions are (1) Has anyone encountered something like
this before?, (2) Is there something I am failing to specify in the code
when reading in?, and (3) Is anyone interested in digging into this and
willing to play around with the nc files if I make them available privately?
Thanks very much in advance!
Louise
[[alternative HTML version deleted]]
______________________________________________ R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
-- David W. Pierce Division of Climate, Atmospheric Science, and Physical Oceanography Scripps Institution of Oceanography, La Jolla, California, USA (858) 534-8276 (voice) / (858) 534-8561 (fax) dpierce at ucsd.edu [[alternative HTML version deleted]]
______________________________________________ R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
********************** "The contents of this message do not reflect any position of the U.S. Government or NOAA." ********************** Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center ***Note new address and phone*** 110 Shaffer Road Santa Cruz, CA 95060 Phone: (831)-420-3666 Fax: (831) 420-3980 e-mail: Roy.Mendelssohn at noaa.gov www: http://www.pfeg.noaa.gov/ "Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected" "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.