write/readOGR issue with date / time strings
Hi Roger, Thanks for your patience. This one is confusing to me as I've done exactly (well, I think exactly) the same steps in the past and it has worked fine. The only difference is (a) I'm using Rstudio which made me (b) update my rgdal library version. I've always converted date/time fields using as.character or format(). Answers to your questions embedded below...
On 11/12/11 3:17 PM, "Roger Bivand" <Roger.Bivand at nhh.no> wrote:
On Fri, 11 Nov 2011, Corrie Curtice wrote:
sessionInfo()
R version 2.13.1 (2011-07-08) Platform: x86_64-apple-darwin9.8.0/x86_64 (64-bit) locale: [1] C/en_US.UTF-8/C/C/C/C attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] rgdal_0.7-1 sp_0.9-84
OK, you have the most recent release.
loaded via a namespace (and not attached):
[1] grid_2.13.1 lattice_0.19-30 tools_2.13.1
I installed rgdal and PROJ4 as follows:
setRepositories(ind=1:2)
install.packages("rgdal")
install.packages("PROJ.4")
Quoting the loading messages would help, to confirm the version of the underlying GDAL binary. The PROJ.4 package is not required or used by rgdal. My rgdal version on Fedora reports:
library(rgdal)
Loading required package: sp Geospatial Data Abstraction Library extensions to R successfully loaded Loaded GDAL runtime: GDAL 1.8.1, released 2011/06/04 Path to GDAL shared files: /usr/local/share/gdal Loaded PROJ.4 runtime: Rel. 4.7.1, 23 September 2009, [PJ_VERSION: 470] Path to PROJ.4 shared files: (autodetected)
Here's the output from loading rgdal:
library(rgdal)
Geospatial Data Abstraction Library extensions to R successfully loaded Loaded GDAL runtime: GDAL 1.8.0, released 2011/01/12 Path to GDAL shared files: /Users/corriecurtice/Library/R/2.13/library/rgdal/gdal Loaded PROJ.4 runtime: Rel. 4.7.1, 23 September 2009, [PJ_VERSION: 470] Path to PROJ.4 shared files: /Users/corriecurtice/Library/R/2.13/library/rgdal/proj
So there's a slightly newer GDAL release (1.8.1). I'm in the process of trying to get that installed but not having a lot of luck. I downloaded and installed gdal v1.8.1-1 leopard (and dependencies). I assume I need to re-compile the rgdal package so it sees this new version but I get an error when trying to do that: corrie-curtices-macbook-pro:Downloads corriecurtice$ !R R CMD INSTALL --configure-args='--with-gdal-config=/Library/Frameworks/GDAL.framework/Pro grams/gdal-config --with-proj-include=/Library/Frameworks/PROJ.framework/Headers --with-proj-lib=/Library/Frameworks/PROJ.framework/unix/lib' rgdal_0.7-4.tar.gz * installing to library ?/Users/corriecurtice/Library/R/2.13/library? * installing *source* package ?rgdal? ... gdal-config: /Library/Frameworks/GDAL.framework/Programs/gdal-config checking for gcc... gcc-4.2 -arch i386 -std=gnu99 checking whether the C compiler works... no configure: error: in `/private/var/folders/jD/jDD8BZCHHLC-VOQmq7vTGk+++TI/-Tmp-/RtmpTlKhSZ/R.INS TALL253fc47f/rgdal': configure: error: C compiler cannot create executables See `config.log' for more details ERROR: configuration failed for package ?rgdal? * removing ?/Users/corriecurtice/Library/R/2.13/library/rgdal? * restoring previous ?/Users/corriecurtice/Library/R/2.13/library/rgdal? I can't find the config.log file, the tmp directory referenced is empty and it's not in my cwd.
I do not see the problem in the R-forge development version, and cannot
reproduce the problem. This now works in (unreleased) 0.7-4:
library(rgdal)
dsn <- system.file("vectors", package = "rgdal")[1]
kiritimati_primary_roads <- readOGR(dsn=dsn,
layer="kiritimati_primary_roads")
summary(kiritimati_primary_roads)
kiritimati_primary_roads$dt <- as.POSIXct("2011-11-11 11:11:11")+(60*0:34)
summary(kiritimati_primary_roads)
td <- tempdir()
writeOGR(kiritimati_primary_roads, td, "kiri1", driver="ESRI Shapefile")
kiri1 <- readOGR(td, "kiri1")
summary(kiri1)
kiri1$dt <- as.POSIXct(as.character(kiri1$dt))
summary(kiri1)
This failed in 0.7-1 because the POSIXct column is not recognized as also
POSIXt; this is corrected in 0.7-4. However, this was not the original
problem. Had you reported the output of:
sapply(spdfUTM at data, class)
sapply(spdfUTM at data, storage.mode)
we would have known how krillGMTtime was stored - it does not seem to
have
been detected as a column requiring conversion to character string, or as
a column that should trigger an error. In general, as Alex wrote, all
date
time columns should be converted to character and converted back on
reading back from the shapefile. OGR drivers treat field types
differently, so writeOGR has to take a conservative view of possibilities.
Here's the output from the sapply:
sapply(spdfUTM at data, class)
krillGMTtime Lat Lon
"factor" "factor" "factor"
sapply(spdfUTM at data, storage.mode)
krillGMTtime Lat Lon "integer" "integer" "integer" To answer Don's question, yes I have bad LAT/LON rows when I first read them in but they are all 999. I discard them before making my spatial points data frame, which does in fact use those two columns for the coordinates. I have another similar dataset where the date/time field storage.mode is reported as "character" and not "integer", and has the same issue. Thanks for the help. Corrie
Hope this clarifies, Roger
Thanks, Corrie On 11/11/11 5:31 PM, "Roger Bivand" <Roger.Bivand at nhh.no> wrote:
On Fri, 11 Nov 2011, Corrie Curtice wrote:
Hello, Apologies if this has already been addressed somewhere, I did a brief search of archives but didn't find quite this issue.
Which versions of what are you using? You should provide the output of sessionInfo(), the messages given when rgdal loads, and details of how you installed rgdal. In addition, a use case would be helpful, but only when we know what version you are using. In current rgdal, data frame columns of class POSIXt are converted to character strings before writing with writeOGR(). Roger
I'm writing out an ESRI shapefile. My spdf has a date/time field. Looks like this:
head(spdfUTM at data)
krillGMTtime Lat Lon 1 2010-05-12 12:34:21 -64.67655969 -62.15040195 21 2010-05-12 12:35:12 -64.6771229 -62.1519511 38 2010-05-12 12:36:02 -64.67775863 -62.15340614 57 2010-05-12 12:36:53 -64.67838269 -62.15494829 78 2010-05-12 12:37:43 -64.67901497 -62.15647203 100 2010-05-12 12:38:37 -64.67973667 -62.15773224 All three fields are reported to be "factors" by str. writeOGR appears happy:
writeOGR(spdfUTM,dd,layer="krillPoints-UTM",driver="ESRI
Shapefile",verbose=TRUE,overwrite=TRUE) $object_type [1] "SpatialPointsDataFrame" $output_dsn [1] "/users/corriecurtice/documents/Data_2010/Shapefiles/" $output_layer [1] "krillPoints-UTM" $output_diver [1] "ESRI Shapefile" $output_n [1] 11179 $output_nfields [1] 3 $output_fields [1] "krillGMTtime" "Lat" "Lon" $output_fclasses [1] 4 4 4 $dataset_options NULL $layer_options NULL Warning message: In writeOGR(spdfUTM, dd, layer = "krillPoints-UTM", driver = "ESRI Shapefile", : existing layer removed When I read it right back in again, the date/time field is NAs. This is also true if I load the shapefile into ArcMap.
foo <- readOGR(dd,layer="krillPoints-UTM",verbose=TRUE)
OGR data source with driver: ESRI Shapefile Source: "/users/corriecurtice/documents/Data_2010/Shapefiles/", layer: "krillPoints-UTM" with 11179 features and 3 fields Feature type: wkbPoint with 2 dimensions
head(foo at data)
krillGMTti Lat Lon 1 <NA> -64.67655969 -62.15040195 2 <NA> -64.6771229 -62.1519511 3 <NA> -64.67775863 -62.15340614 4 <NA> -64.67838269 -62.15494829 5 <NA> -64.67901497 -62.15647203 6 <NA> -64.67973667 -62.15773224 Thoughts? Am I missing something obvious? I can break it up into separate date and time fields, but since the field is simply a factor anyways I'm not sure how that would help. Cheers, Corrie --- Corrie Curtice Research Analyst Marine Geospatial Ecology Lab Nicholas School of the Environment, Duke University http://mgel.env.duke.edu em: corrie.curtice at duke.edu ph: 252-504-7538 [[alternative HTML version deleted]]
_______________________________________________ R-sig-Geo mailing list R-sig-Geo at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo
-- Roger Bivand Department of Economics, NHH Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43 e-mail: Roger.Bivand at nhh.no
_______________________________________________ R-sig-Geo mailing list R-sig-Geo at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo
-- Roger Bivand Department of Economics, NHH Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43 e-mail: Roger.Bivand at nhh.no