Issues importing BIG shapefile - Limitations of R or something else?

An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-geo/attachments/20090328/2a99c30d/attachment.pl>

Here are the sessionInfo() and traceback() results following the failed
import.  It looks like I am not using the most recent version of sp so I am
trying to figure out the terms in the call to update.package() and with the
most recent version I will try again.
Thanks! The indications are that coordinates is being passed a numeric 
vector, not a matrix, at:

Line(coords = shape$verts[from[i]:to[i], ])

so if from[i] == to[i], the code as shown converts a single coordinate 
(one row) matrix into a numeric vector (omitted drop=FALSE in "[" method, 
perhaps the most common coding error in S/R).

That is fixable, but a better workaround may be to use readOGR() in rgdal 
rather than readShapeLines() in maptools.

Roger
The .shp file is ~30 megabytes and
the associated .dbf is ~15 megabytes, though I don't think the latter is
included in the import.  I have 2G of memory on this machine....

Tyler

**
*> roads<-as(as(readShapeLines(fn="D:\\GIS\\road2004.shp"), "SpatialLines"),
"psp")
*Error in function (classes, fdef, mtable)  :
 unable to find an inherited method for function "coordinates", for
signature "numeric"

*> sessionInfo()*
R version 2.8.1 (2008-12-22)
i386-pc-mingw32

locale:
LC_COLLATE=English_Canada.1252;LC_CTYPE=English_Canada.1252;LC_MONETARY=English_Canada.1252;LC_NUMERIC=C;LC_TIME=English_Canada.1252

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base

other attached packages:
[1] spatstat_1.15-0 deldir_0.0-7    gpclib_1.4-3    mgcv_1.4-1.1
maptools_0.7-20 foreign_0.8-29  sp_0.9-29
loaded via a namespace (and not attached):
[1] grid_2.8.1      lattice_0.17-17 tools_2.8.1

*> traceback()*
13: stop("unable to find an inherited method for function \"", fdef at generic,

       "\", for signature ", cnames)
12: function (classes, fdef, mtable)
   {
       methods <- .findInheritedMethods(classes, fdef, mtable)
       if (length(methods) == 1)
           return(methods[[1]])
       else if (length(methods) == 0) {
           cnames <- paste("\"", sapply(classes, as.character),
               "\"", sep = "", collapse = ", ")
           stop("unable to find an inherited method for function \"",
               fdef at generic, "\", for signature ", cnames)
       }
       else stop("Internal error in finding inherited methods; didn't
return a unique method")
   }(list("numeric"), function (obj)
   standardGeneric("coordinates"), <environment>)
11: coordinates(coords)
10: Line(coords = shape$verts[from[i]:to[i], ])
9: .shapes2LinesList(shapes[[i]], ID = IDs[i])
8: .shp2LinesDF(Map, proj4string = proj4string)
7: withCallingHandlers(expr, warning = function(w)
invokeRestart("muffleWarning"))
6: suppressWarnings(.shp2LinesDF(Map, proj4string = proj4string))
5: readShapeLines(fn = "D:\\GIS\\road2004.shp")
4: .class1(object)
3: as(readShapeLines(fn = "D:\\GIS\\road2004.shp"), "SpatialLines")
2: .class1(object)
1: as(as(readShapeLines(fn = "D:\\GIS\\road2004.shp"), "SpatialLines"),
      "psp")

	[[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
R-sig-Geo at stat.math.ethz.ch
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no
An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-geo/attachments/20090329/8c94947f/attachment.pl>

As you've indicated Roger there does appear to be an error in
readShapeLines() under maptools.  However I can't help but wonder why I do
not have any problems using the same command sequence for a subset of
the same data (even after updating my version of sp)...?  With said subset I
am able to import and convert the .shp file to a SpatialLinesDataFrame
object and then (end result) a psp object.  This latter transformation is
critical in order to produce the type of output I am trying to model using
distmap().

Now via the alternate path, readOGR() successfully imported the spatial
lines polygon (full coverage) as a SpatialLinesDataFrame object;
**
*> roads<-readOGR("D:/GIS", layer="road2004")*
OGR data source with driver: ESRI Shapefile
Source: "D:/GIS", layer: "road2004"
with  51218  rows and  4  columns
Feature type: wkbLineString with 2 dimensions
However from this point when I try to convert it to a psp object it fails:
Tyler:

Thanks again for a full report. The failure in readShapeLines() was caused 
by a line with only a single coordinate (two, in fact). This isn't a 
problem in the readOGR() code, and will not be in the latest release of 
maptools.

In spatstat, if spatstat.options("checkpolygons") is TRUE, (xrange[2] < 
xrange[1]) and (yrange[2] < yrange[1]) are both expected to hold, and 
cannot, because for single coordinate lines, the min and max are equal.

You have two choices - based on the files you made available, either drop 
the two single coordinate lines:

ncrds <- sapply(slot(roads, "lines"), function(x) sapply(slot(x, "Lines"),
   function(y) nrow(slot(y, "coords"))))
summary(unlist(wh_singleton))
keep <- sapply(ncrds, function(x) all(x > 1))
roads_gt_1 <- roads[keep]
length(slot(roads, "lines"))
length(slot(roads_gt_1, "lines"))
test <- as(roads_gt_1, "psp")

or set checkpolygons to FALSE in spatstat.options(). I'm running the 
former at the moment - deleting the SpatialLinesDataFrame object after 
coercing to SpatialLines would save some memory.

Hope this helps,

Roger
** *> test<-as(roads, "psp")*
Error in owin(xr, yr, poly = list(x = xr[c(1, 2, 2, 1)], y = yr[c(1, 1,  :
 xrange should be a vector of length 2 giving (xmin, xmax)

Here I will provide similar information as before, in hopes that we might be
able to shed light on the issue, which appears to involve the window
being used to spatially frame the data:

*> sessionInfo()*
R version 2.8.1 (2008-12-22)
i386-pc-mingw32

locale:
LC_COLLATE=English_Canada.1252;LC_CTYPE=English_Canada.1252;LC_MONETARY=English_Canada.1252;LC_NUMERIC=C;LC_TIME=English_Canada.1252

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base

other attached packages:
[1] rgdal_0.6-7     spatstat_1.15-0 deldir_0.0-7    gpclib_1.4-3
mgcv_1.4-1.1    maptools_0.7-20 sp_0.9-32       foreign_0.8-29
loaded via a namespace (and not attached):
[1] grid_2.8.1      lattice_0.17-17 nlme_3.1-89     tools_2.8.1

*> traceback()
*14: stop("xrange should be a vector of length 2 giving (xmin, xmax)")
13: owin(xr, yr, poly = list(x = xr[c(1, 2, 2, 1)], y = yr[c(1, 1,
       2, 2)]), unitname = unitname(W))
12: switch(W$type, rectangle = {
       xr <- W$xrange
       yr <- W$yrange
       return(owin(xr, yr, poly = list(x = xr[c(1, 2, 2, 1)], y = yr[c(1,
           1, 2, 2)]), unitname = unitname(W)))
   }, polygonal = {
       return(W)
   }, mask = {
       stop("A mask cannot be converted to a polygon")
   })
11: as.polygonal(x)
10: owin2gpc(B)
9: union.owin(W, Wlist[[i]])
8: superimposePSP(y, window = window)
7: FUN(X[[24798L]], ...)
6: lapply(lin, as.psp.Lines)
5: as.psp.SpatialLines(y, window = window, marks = marks)
4: as.psp(y, window = window, marks = marks)
3: as.psp.SpatialLinesDataFrame(from)
2: asMethod(object)
1: as(roads, "psp")

Why does this error occur now when it works fine for a subset of the same
data?  Is there a fix that could help with this problem?
Tyler

On Sun, Mar 29, 2009 at 11:21 AM, Roger Bivand <Roger.Bivand at nhh.no> wrote:

On Sat, 28 Mar 2009, Tyler Dean Rudolph wrote:

Here are the sessionInfo() and traceback() results following the failed
import.  It looks like I am not using the most recent version of sp so I
am
trying to figure out the terms in the call to update.package() and with
the
most recent version I will try again.

Thanks! The indications are that coordinates is being passed a numeric
vector, not a matrix, at:

Line(coords = shape$verts[from[i]:to[i], ])

so if from[i] == to[i], the code as shown converts a single coordinate (one
row) matrix into a numeric vector (omitted drop=FALSE in "[" method, perhaps
the most common coding error in S/R).

That is fixable, but a better workaround may be to use readOGR() in rgdal
rather than readShapeLines() in maptools.

Roger

  The .shp file is ~30 megabytes and
the associated .dbf is ~15 megabytes, though I don't think the latter is
included in the import.  I have 2G of memory on this machine....

Tyler

**
*> roads<-as(as(readShapeLines(fn="D:\\GIS\\road2004.shp"),
"SpatialLines"),
"psp")
*Error in function (classes, fdef, mtable)  :
 unable to find an inherited method for function "coordinates", for
signature "numeric"

*> sessionInfo()*
R version 2.8.1 (2008-12-22)
i386-pc-mingw32

locale:

LC_COLLATE=English_Canada.1252;LC_CTYPE=English_Canada.1252;LC_MONETARY=English_Canada.1252;LC_NUMERIC=C;LC_TIME=English_Canada.1252

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base

other attached packages:
[1] spatstat_1.15-0 deldir_0.0-7    gpclib_1.4-3    mgcv_1.4-1.1
maptools_0.7-20 foreign_0.8-29  sp_0.9-29
loaded via a namespace (and not attached):
[1] grid_2.8.1      lattice_0.17-17 tools_2.8.1

*> traceback()*
13: stop("unable to find an inherited method for function \"",
fdef at generic,

      "\", for signature ", cnames)
12: function (classes, fdef, mtable)
  {
      methods <- .findInheritedMethods(classes, fdef, mtable)
      if (length(methods) == 1)
          return(methods[[1]])
      else if (length(methods) == 0) {
          cnames <- paste("\"", sapply(classes, as.character),
              "\"", sep = "", collapse = ", ")
          stop("unable to find an inherited method for function \"",
              fdef at generic, "\", for signature ", cnames)
      }
      else stop("Internal error in finding inherited methods; didn't
return a unique method")
  }(list("numeric"), function (obj)
  standardGeneric("coordinates"), <environment>)
11: coordinates(coords)
10: Line(coords = shape$verts[from[i]:to[i], ])
9: .shapes2LinesList(shapes[[i]], ID = IDs[i])
8: .shp2LinesDF(Map, proj4string = proj4string)
7: withCallingHandlers(expr, warning = function(w)
invokeRestart("muffleWarning"))
6: suppressWarnings(.shp2LinesDF(Map, proj4string = proj4string))
5: readShapeLines(fn = "D:\\GIS\\road2004.shp")
4: .class1(object)
3: as(readShapeLines(fn = "D:\\GIS\\road2004.shp"), "SpatialLines")
2: .class1(object)
1: as(as(readShapeLines(fn = "D:\\GIS\\road2004.shp"), "SpatialLines"),
     "psp")

       [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
R-sig-Geo at stat.math.ethz.ch
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no

Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no