write/readOGR issue with date / time strings

Hi Roger,

Thanks for your patience. This one is confusing to me as I've done exactly
(well, I think exactly) the same steps in the past and it has worked fine.
The only difference is (a) I'm using Rstudio which made me (b) update my
rgdal library version.  I've always converted date/time fields using
as.character or format().  Answers to your questions embedded below...

On 11/12/11 3:17 PM, "Roger Bivand" <Roger.Bivand at nhh.no> wrote:

On Fri, 11 Nov 2011, Corrie Curtice wrote:

sessionInfo()

R version 2.13.1 (2011-07-08)
Platform: x86_64-apple-darwin9.8.0/x86_64 (64-bit)

locale:
[1] C/en_US.UTF-8/C/C/C/C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] rgdal_0.7-1 sp_0.9-84

OK, you have the most recent release.

loaded via a namespace (and not attached):
[1] grid_2.13.1     lattice_0.19-30 tools_2.13.1

I installed rgdal and PROJ4 as follows:

setRepositories(ind=1:2)
install.packages("rgdal")
install.packages("PROJ.4")

Quoting the loading messages would help, to confirm the version of the
underlying GDAL binary. The PROJ.4 package is not required or used by
rgdal. My rgdal version on Fedora reports:

library(rgdal)

Loading required package: sp
Geospatial Data Abstraction Library extensions to R successfully loaded
Loaded GDAL runtime: GDAL 1.8.1, released 2011/06/04
Path to GDAL shared files: /usr/local/share/gdal
Loaded PROJ.4 runtime: Rel. 4.7.1, 23 September 2009, [PJ_VERSION: 470]
Path to PROJ.4 shared files: (autodetected)

Here's the output from loading rgdal:

library(rgdal)

Geospatial Data Abstraction Library extensions to R successfully loaded
Loaded GDAL runtime: GDAL 1.8.0, released 2011/01/12
Path to GDAL shared files:
/Users/corriecurtice/Library/R/2.13/library/rgdal/gdal
Loaded PROJ.4 runtime: Rel. 4.7.1, 23 September 2009, [PJ_VERSION: 470]
Path to PROJ.4 shared files:
/Users/corriecurtice/Library/R/2.13/library/rgdal/proj

So there's a slightly newer GDAL release (1.8.1).  I'm in the process of
trying to get that installed but not having a lot of luck.  I downloaded
and installed gdal v1.8.1-1 leopard (and dependencies). I assume I need to
re-compile the rgdal package so it sees this new version but I get an
error when trying to do that:

corrie-curtices-macbook-pro:Downloads corriecurtice$ !R
R CMD INSTALL
--configure-args='--with-gdal-config=/Library/Frameworks/GDAL.framework/Pro
grams/gdal-config
--with-proj-include=/Library/Frameworks/PROJ.framework/Headers
--with-proj-lib=/Library/Frameworks/PROJ.framework/unix/lib'
rgdal_0.7-4.tar.gz
* installing to library ?/Users/corriecurtice/Library/R/2.13/library?
* installing *source* package ?rgdal? ...
gdal-config: /Library/Frameworks/GDAL.framework/Programs/gdal-config
checking for gcc... gcc-4.2 -arch i386 -std=gnu99
checking whether the C compiler works... no
configure: error: in
`/private/var/folders/jD/jDD8BZCHHLC-VOQmq7vTGk+++TI/-Tmp-/RtmpTlKhSZ/R.INS
TALL253fc47f/rgdal':
configure: error: C compiler cannot create executables
See `config.log' for more details
ERROR: configuration failed for package ?rgdal?
* removing ?/Users/corriecurtice/Library/R/2.13/library/rgdal?
* restoring previous ?/Users/corriecurtice/Library/R/2.13/library/rgdal?


I can't find the config.log file, the tmp directory referenced is empty
and it's not in my cwd.

I do not see the problem in the R-forge development version, and cannot
reproduce the problem. This now works in (unreleased) 0.7-4:

library(rgdal)
dsn <- system.file("vectors", package = "rgdal")[1]
  kiritimati_primary_roads <- readOGR(dsn=dsn,
layer="kiritimati_primary_roads")
summary(kiritimati_primary_roads)
kiritimati_primary_roads$dt <- as.POSIXct("2011-11-11 11:11:11")+(60*0:34)
summary(kiritimati_primary_roads)
td <- tempdir()
writeOGR(kiritimati_primary_roads, td, "kiri1", driver="ESRI Shapefile")
kiri1 <- readOGR(td, "kiri1")
summary(kiri1)
kiri1$dt <- as.POSIXct(as.character(kiri1$dt))
summary(kiri1)

This failed in 0.7-1 because the POSIXct column is not recognized as also
POSIXt; this is corrected in 0.7-4. However, this was not the original
problem. Had you reported the output of:

sapply(spdfUTM at data, class)
sapply(spdfUTM at data, storage.mode)

we would have known how krillGMTtime was stored - it does not seem to
have
been detected as a column requiring conversion to character string, or as
a column that should trigger an error. In general, as Alex wrote, all
date
time columns should be converted to character and converted back on
reading back from the shapefile. OGR drivers treat field types
differently, so writeOGR has to take a conservative view of possibilities.

Here's the output from the sapply:

sapply(spdfUTM at data, class)

krillGMTtime          Lat          Lon
   "factor"     "factor"     "factor"

sapply(spdfUTM at data, storage.mode)

krillGMTtime          Lat          Lon
  "integer"    "integer"    "integer"


To answer Don's question, yes I have bad LAT/LON rows when I first read
them in but they are all 999. I discard them before making my spatial
points data frame, which does in fact use those two columns for the
coordinates. I have another similar dataset where the date/time field
storage.mode is reported as "character" and not "integer", and has the
same issue.


Thanks for the help.

Corrie

Hope this clarifies,

Roger

Thanks,

Corrie



On 11/11/11 5:31 PM, "Roger Bivand" <Roger.Bivand at nhh.no> wrote:

On Fri, 11 Nov 2011, Corrie Curtice wrote:

Hello,

Apologies if this has already been addressed somewhere, I did a brief
search
of archives but didn't find quite this issue.

Which versions of what are you using? You should provide the output of
sessionInfo(), the messages given when rgdal loads, and details of how
you
installed rgdal. In addition, a use case would be helpful, but only
when
we know what version you are using. In current rgdal, data frame
columns
of class POSIXt are converted to character strings before writing with
writeOGR().

Roger

I'm writing out an ESRI shapefile.  My spdf has a date/time field.
Looks
like this:

head(spdfUTM at data)

          krillGMTtime          Lat          Lon
1   2010-05-12 12:34:21 -64.67655969 -62.15040195
21  2010-05-12 12:35:12  -64.6771229  -62.1519511
38  2010-05-12 12:36:02 -64.67775863 -62.15340614
57  2010-05-12 12:36:53 -64.67838269 -62.15494829
78  2010-05-12 12:37:43 -64.67901497 -62.15647203
100 2010-05-12 12:38:37 -64.67973667 -62.15773224

All three fields are reported to be "factors" by str.  writeOGR
appears
happy:

writeOGR(spdfUTM,dd,layer="krillPoints-UTM",driver="ESRI

Shapefile",verbose=TRUE,overwrite=TRUE)
$object_type
[1] "SpatialPointsDataFrame"
$output_dsn
[1] "/users/corriecurtice/documents/Data_2010/Shapefiles/"
$output_layer
[1] "krillPoints-UTM"
$output_diver
[1] "ESRI Shapefile"
$output_n
[1] 11179
$output_nfields
[1] 3
$output_fields
[1] "krillGMTtime" "Lat"          "Lon"
$output_fclasses
[1] 4 4 4
$dataset_options
NULL
$layer_options
NULL
Warning message:
In writeOGR(spdfUTM, dd, layer = "krillPoints-UTM", driver = "ESRI
Shapefile",  :
 existing layer removed

When I read it right back in again, the date/time field is NAs.  This
is
also true if I load the shapefile into ArcMap.

foo <- readOGR(dd,layer="krillPoints-UTM",verbose=TRUE)

OGR data source with driver: ESRI Shapefile
Source: "/users/corriecurtice/documents/Data_2010/Shapefiles/", layer:
"krillPoints-UTM"
with 11179 features and 3 fields
Feature type: wkbPoint with 2 dimensions

head(foo at data)

 krillGMTti          Lat          Lon
1       <NA> -64.67655969 -62.15040195
2       <NA>  -64.6771229  -62.1519511
3       <NA> -64.67775863 -62.15340614
4       <NA> -64.67838269 -62.15494829
5       <NA> -64.67901497 -62.15647203
6       <NA> -64.67973667 -62.15773224

Thoughts? Am I missing something obvious?

I can break it up into separate date and time fields, but since the
field is
simply a factor anyways I'm not sure how that would help.

Cheers,

Corrie

---
Corrie Curtice
Research Analyst
Marine Geospatial Ecology Lab
Nicholas School of the Environment, Duke University
http://mgel.env.duke.edu
em: corrie.curtice at duke.edu
ph: 252-504-7538




    [[alternative HTML version deleted]]

--
Roger Bivand
Department of Economics, NHH Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no

--
Roger Bivand
Department of Economics, NHH Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no