sp/rgdal workflows with PROJ >= 6 and GDAL >= 3

Because PROJ >= 6 and GDAL >= 3 change the way that PROJ strings 
(representations of coordinate reference systems) are handled, steps are 
being taken to find ways to adapt sp/rgdal workflows. A current proposal is 
to store the WKT2_2018 string as a comment to CRS objects as defined in the 
sp package.

A draft development-in-progress version of rgdal is available at 
https://r-forge.r-project.org/R/?group_id=884, and for sp at 
https://github.com/rsbivand/sp (this version of sp requires rgdal >= 1.5-1). 
This adds the WKT comments to CRS objects on reading vector and raster data 
sources, and uses WKT comments if found when writing vector and raster 
objects (or at least does as far as I've checked, possibly fragile).

An RFC with tersely worked cases for using CRS object comments to carry WKT 
strings but maintaining full backward compatibility is online at 
http://rgdal.r-forge.r-project.org/articles/PROJ6_GDAL3.html.

If you have other ideas or concerns about trying to use this mechanism for sp 
CRS objects, please contribute at your earliest convenience.

http://rgdal.r-forge.r-project.org/reference/list_coordOps.html shows the 
beginning of the next step, to query transformation operations to find viable 
coordinate operation pipelines.

I'm assuming that the previous behaviour (transform without considering 
accuracy with whatever is to hand) is not viable going forward, and that we 
will need two steps: list coordinate operations between source and target CRS 
(using the WKT comments as better specifications than the PROJ strings), 
possibly intervene manually to install missing grids, then undertake the 
coordinate operation.

The fallback may be simply to choose the least inaccurate available 
coordinate operation, but this should be a fallback. This means that all uses 
of spTransform() will require intervention.

Is this OK (it is tiresome but modernises workflows once), or is it not OK 
(no user intervention is crucial)?

These behaviours may be set in an option, so that package maintainers and 
users may delay modernisation, but all are undoubtedly served by rapid 
adaptation (GRASS 7.8.1 released yesterday, libspatialite, pyproj, QGIS 
development versions all state that they list candidate coordinate 
operations).

We cannot ship all the grids, they are very bulky, and probably nobody needs 
sub-metre accuracy world-wide. Work in PROJ is starting to create a content 
delivery network for trusted download and mechanisms for registering 
downloaded grids on user platforms. We would for example not want Windows 
users of rgdal and sf to have to download the same grid twice.

Comments welcome here and at https://github.com/r-spatial/discuss/issues/28 
or https://github.com/r-spatial/sf/issues/1187

Roger

And this link explains the CDN proposal for grid distribution:

https://www.spatialys.com/en/crowdfunding/

Roger

On Wed, 13 Nov 2019, Roger Bivand wrote:

 Because PROJ >= 6 and GDAL >= 3 change the way that PROJ strings
 (representations of coordinate reference systems) are handled, steps are
 being taken to find ways to adapt sp/rgdal workflows. A current proposal
 is to store the WKT2_2018 string as a comment to CRS objects as defined in
 the sp package.

 A draft development-in-progress version of rgdal is available at
 https://r-forge.r-project.org/R/?group_id=884, and for sp at
 https://github.com/rsbivand/sp (this version of sp requires rgdal >=
 1.5-1). This adds the WKT comments to CRS objects on reading vector and
 raster data sources, and uses WKT comments if found when writing vector
 and raster objects (or at least does as far as I've checked, possibly
 fragile).

 An RFC with tersely worked cases for using CRS object comments to carry
 WKT strings but maintaining full backward compatibility is online at
 http://rgdal.r-forge.r-project.org/articles/PROJ6_GDAL3.html.

 If you have other ideas or concerns about trying to use this mechanism for
 sp CRS objects, please contribute at your earliest convenience.

 http://rgdal.r-forge.r-project.org/reference/list_coordOps.html shows the
 beginning of the next step, to query transformation operations to find
 viable coordinate operation pipelines.

 I'm assuming that the previous behaviour (transform without considering
 accuracy with whatever is to hand) is not viable going forward, and that
 we will need two steps: list coordinate operations between source and
 target CRS (using the WKT comments as better specifications than the PROJ
 strings), possibly intervene manually to install missing grids, then
 undertake the coordinate operation.

 The fallback may be simply to choose the least inaccurate available
 coordinate operation, but this should be a fallback. This means that all
 uses of spTransform() will require intervention.

 Is this OK (it is tiresome but modernises workflows once), or is it not OK
 (no user intervention is crucial)?

 These behaviours may be set in an option, so that package maintainers and
 users may delay modernisation, but all are undoubtedly served by rapid
 adaptation (GRASS 7.8.1 released yesterday, libspatialite, pyproj, QGIS
 development versions all state that they list candidate coordinate
 operations).

 We cannot ship all the grids, they are very bulky, and probably nobody
 needs sub-metre accuracy world-wide. Work in PROJ is starting to create a
 content delivery network for trusted download and mechanisms for
 registering downloaded grids on user platforms. We would for example not
 want Windows users of rgdal and sf to have to download the same grid
 twice.

 Comments welcome here and at
 https://github.com/r-spatial/discuss/issues/28 or
 https://github.com/r-spatial/sf/issues/1187

 Roger

The development version of rgdal on R-Forge is now at rev 894, and is now 
ready for trying out with PROJ6/GDAL3 workflows, and workflows that may 
migrate within 6 months to modern CRS representations. The motivating RFC is 
also updated to cover coordinate operations, the use of prepared 
(pre-searched) coordinate operations, and should be read carefully by anyone 
using rgdal::spTransform(). Note further that rgdal::project() will not be 
adapted for PROJ6, and is effectively deprecated.

I'll be running reverse dependency checks, and may be bugging package 
maintainers. I would really prefer that mainainers of packages using 
spTransform() checked themselves and joined this thread or the associated 
twitter thread: https://twitter.com/RogerBivand/status/1194586193108914177

Be ready for modern PROJ and GDAL, they are already being deployed across 
open source geospatial software, like GRASS, QGIS, pyproj, spatialite etc.

Waiting, hopefully not in vain, for contributions.

Roger

On Wed, 13 Nov 2019, Roger Bivand wrote:

 And this link explains the CDN proposal for grid distribution:

 https://www.spatialys.com/en/crowdfunding/

 Roger

 On Wed, 13 Nov 2019, Roger Bivand wrote:

  Because PROJ >= 6 and GDAL >= 3 change the way that PROJ strings
  (representations of coordinate reference systems) are handled, steps are
  being taken to find ways to adapt sp/rgdal workflows. A current proposal
  is to store the WKT2_2018 string as a comment to CRS objects as defined
  in
  the sp package.

  A draft development-in-progress version of rgdal is available at
  https://r-forge.r-project.org/R/?group_id=884, and for sp at
  https://github.com/rsbivand/sp (this version of sp requires rgdal >=
  1.5-1). This adds the WKT comments to CRS objects on reading vector and
  raster data sources, and uses WKT comments if found when writing vector
  and raster objects (or at least does as far as I've checked, possibly
  fragile).

  An RFC with tersely worked cases for using CRS object comments to carry
  WKT strings but maintaining full backward compatibility is online at
  http://rgdal.r-forge.r-project.org/articles/PROJ6_GDAL3.html.

  If you have other ideas or concerns about trying to use this mechanism
  for
  sp CRS objects, please contribute at your earliest convenience.

  http://rgdal.r-forge.r-project.org/reference/list_coordOps.html shows
  the
  beginning of the next step, to query transformation operations to find
  viable coordinate operation pipelines.

  I'm assuming that the previous behaviour (transform without considering
  accuracy with whatever is to hand) is not viable going forward, and that
  we will need two steps: list coordinate operations between source and
  target CRS (using the WKT comments as better specifications than the
  PROJ
  strings), possibly intervene manually to install missing grids, then
  undertake the coordinate operation.

  The fallback may be simply to choose the least inaccurate available
  coordinate operation, but this should be a fallback. This means that all
  uses of spTransform() will require intervention.

  Is this OK (it is tiresome but modernises workflows once), or is it not
  OK
  (no user intervention is crucial)?

  These behaviours may be set in an option, so that package maintainers
  and
  users may delay modernisation, but all are undoubtedly served by rapid
  adaptation (GRASS 7.8.1 released yesterday, libspatialite, pyproj, QGIS
  development versions all state that they list candidate coordinate
  operations).

  We cannot ship all the grids, they are very bulky, and probably nobody
  needs sub-metre accuracy world-wide. Work in PROJ is starting to create
  a
  content delivery network for trusted download and mechanisms for
  registering downloaded grids on user platforms. We would for example not
  want Windows users of rgdal and sf to have to download the same grid
  twice.

  Comments welcome here and at
  https://github.com/r-spatial/discuss/issues/28 or
  https://github.com/r-spatial/sf/issues/1187

  Roger

A description of the status now with regard to a prototype resolution is
online at:

https://rsbivand.github.io/ECS530_h19/ECS530_III.html

I'm planning to stream a talk about this at 09:15-11:00 CET on Tuesday 3
December. I need a volunteer to test the streaming link in advance during
next week. I'm unsure which technology to use for remote participants to
provide feedback.

Contributions/comments welcome!

Roger


On Fri, 15 Nov 2019, Roger Bivand wrote:

The development version of rgdal on R-Forge is now at rev 894, and is

now

ready for trying out with PROJ6/GDAL3 workflows, and workflows that may
migrate within 6 months to modern CRS representations. The motivating

RFC is

also updated to cover coordinate operations, the use of prepared
(pre-searched) coordinate operations, and should be read carefully by

anyone

using rgdal::spTransform(). Note further that rgdal::project() will not

be

adapted for PROJ6, and is effectively deprecated.

I'll be running reverse dependency checks, and may be bugging package
maintainers. I would really prefer that mainainers of packages using
spTransform() checked themselves and joined this thread or the

associated

twitter thread:

https://twitter.com/RogerBivand/status/1194586193108914177

Be ready for modern PROJ and GDAL, they are already being deployed

across

open source geospatial software, like GRASS, QGIS, pyproj, spatialite

etc.

Waiting, hopefully not in vain, for contributions.

Roger

On Wed, 13 Nov 2019, Roger Bivand wrote:

 And this link explains the CDN proposal for grid distribution:

 https://www.spatialys.com/en/crowdfunding/

 Roger

 On Wed, 13 Nov 2019, Roger Bivand wrote:

  Because PROJ >= 6 and GDAL >= 3 change the way that PROJ strings
  (representations of coordinate reference systems) are handled, steps

are

  being taken to find ways to adapt sp/rgdal workflows. A current

proposal

  is to store the WKT2_2018 string as a comment to CRS objects as

defined

  in
  the sp package.

  A draft development-in-progress version of rgdal is available at
  https://r-forge.r-project.org/R/?group_id=884, and for sp at
  https://github.com/rsbivand/sp (this version of sp requires rgdal >=
  1.5-1). This adds the WKT comments to CRS objects on reading vector

and

  raster data sources, and uses WKT comments if found when writing

vector

  and raster objects (or at least does as far as I've checked, possibly
  fragile).

  An RFC with tersely worked cases for using CRS object comments to

carry

  WKT strings but maintaining full backward compatibility is online at
  http://rgdal.r-forge.r-project.org/articles/PROJ6_GDAL3.html.

  If you have other ideas or concerns about trying to use this

mechanism

  for
  sp CRS objects, please contribute at your earliest convenience.

  http://rgdal.r-forge.r-project.org/reference/list_coordOps.html

shows

  the
  beginning of the next step, to query transformation operations to

find

  viable coordinate operation pipelines.

  I'm assuming that the previous behaviour (transform without

considering

  accuracy with whatever is to hand) is not viable going forward, and

that

  we will need two steps: list coordinate operations between source and
  target CRS (using the WKT comments as better specifications than the
  PROJ
  strings), possibly intervene manually to install missing grids, then
  undertake the coordinate operation.

  The fallback may be simply to choose the least inaccurate available
  coordinate operation, but this should be a fallback. This means that

all

  uses of spTransform() will require intervention.

  Is this OK (it is tiresome but modernises workflows once), or is it

not

  OK
  (no user intervention is crucial)?

  These behaviours may be set in an option, so that package maintainers
  and
  users may delay modernisation, but all are undoubtedly served by

rapid

  adaptation (GRASS 7.8.1 released yesterday, libspatialite, pyproj,

QGIS

  development versions all state that they list candidate coordinate
  operations).

  We cannot ship all the grids, they are very bulky, and probably

nobody

  needs sub-metre accuracy world-wide. Work in PROJ is starting to

create

  a
  content delivery network for trusted download and mechanisms for
  registering downloaded grids on user platforms. We would for example

not

  want Windows users of rgdal and sf to have to download the same grid
  twice.

  Comments welcome here and at
  https://github.com/r-spatial/discuss/issues/28 or
  https://github.com/r-spatial/sf/issues/1187

  Roger

--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en

The development version of rgdal on R-Forge is now at rev 894, and is now
ready for trying out with PROJ6/GDAL3 workflows, and workflows that may
migrate within 6 months to modern CRS representations. The motivating RFC is
also updated to cover coordinate operations, the use of prepared
(pre-searched) coordinate operations, and should be read carefully by anyone
using rgdal::spTransform(). Note further that rgdal::project() will not be
adapted for PROJ6, and is effectively deprecated.

I'll be running reverse dependency checks, and may be bugging package
maintainers. I would really prefer that mainainers of packages using
spTransform() checked themselves and joined this thread or the associated
twitter thread: https://twitter.com/RogerBivand/status/1194586193108914177

Be ready for modern PROJ and GDAL, they are already being deployed across
open source geospatial software, like GRASS, QGIS, pyproj, spatialite etc.

Waiting, hopefully not in vain, for contributions.

Roger

On Wed, 13 Nov 2019, Roger Bivand wrote:

 And this link explains the CDN proposal for grid distribution:

 https://www.spatialys.com/en/crowdfunding/

 Roger

 On Wed, 13 Nov 2019, Roger Bivand wrote:

  Because PROJ >= 6 and GDAL >= 3 change the way that PROJ strings
  (representations of coordinate reference systems) are handled, steps are
  being taken to find ways to adapt sp/rgdal workflows. A current proposal
  is to store the WKT2_2018 string as a comment to CRS objects as defined
  in
  the sp package.

  A draft development-in-progress version of rgdal is available at
  https://r-forge.r-project.org/R/?group_id=884, and for sp at
  https://github.com/rsbivand/sp (this version of sp requires rgdal >=
  1.5-1). This adds the WKT comments to CRS objects on reading vector and
  raster data sources, and uses WKT comments if found when writing vector
  and raster objects (or at least does as far as I've checked, possibly
  fragile).

  An RFC with tersely worked cases for using CRS object comments to carry
  WKT strings but maintaining full backward compatibility is online at
  http://rgdal.r-forge.r-project.org/articles/PROJ6_GDAL3.html.

  If you have other ideas or concerns about trying to use this mechanism
  for
  sp CRS objects, please contribute at your earliest convenience.

  http://rgdal.r-forge.r-project.org/reference/list_coordOps.html shows
  the
  beginning of the next step, to query transformation operations to find
  viable coordinate operation pipelines.

  I'm assuming that the previous behaviour (transform without considering
  accuracy with whatever is to hand) is not viable going forward, and that
  we will need two steps: list coordinate operations between source and
  target CRS (using the WKT comments as better specifications than the
  PROJ
  strings), possibly intervene manually to install missing grids, then
  undertake the coordinate operation.

  The fallback may be simply to choose the least inaccurate available
  coordinate operation, but this should be a fallback. This means that all
  uses of spTransform() will require intervention.

  Is this OK (it is tiresome but modernises workflows once), or is it not
  OK
  (no user intervention is crucial)?

  These behaviours may be set in an option, so that package maintainers
  and
  users may delay modernisation, but all are undoubtedly served by rapid
  adaptation (GRASS 7.8.1 released yesterday, libspatialite, pyproj, QGIS
  development versions all state that they list candidate coordinate
  operations).

  We cannot ship all the grids, they are very bulky, and probably nobody
  needs sub-metre accuracy world-wide. Work in PROJ is starting to create
  a
  content delivery network for trusted download and mechanisms for
  registering downloaded grids on user platforms. We would for example not
  want Windows users of rgdal and sf to have to download the same grid
  twice.

  Comments welcome here and at
  https://github.com/r-spatial/discuss/issues/28 or
  https://github.com/r-spatial/sf/issues/1187

  Roger

Thanks, we use the local, institutional interface which provides the 
quality we delivered at the 2014 Geostat summer school. The link will be 
posted on 
https://rsbivand.github.io/ECS530_h19/streaming_ecs530_h19.html when 
available (delayed because I've been off work). We'll just have to use 
Monday's classes to check that things are working, feedback info in the 
link above.

Roger

--
Roger Bivand
Norwegian School of Economics
Helleveien 30, 5045 Bergen, Norway
Roger.Bivand at nhh.no

Streaming link now published (thanks to Arild Schanke), only live when
transmission active.

Roger

On Fri, 29 Nov 2019, Roger Bivand wrote:

Thanks, we use the local, institutional interface which provides the
quality we delivered at the 2014 Geostat summer school. The link will be
posted on
https://rsbivand.github.io/ECS530_h19/streaming_ecs530_h19.html when
available (delayed because I've been off work). We'll just have to use
Monday's classes to check that things are working, feedback info in the
link above.

Roger

--
Roger Bivand
Norwegian School of Economics
Helleveien 30, 5045 Bergen, Norway
Roger.Bivand at nhh.no

confirmed?

Concerning testing the connection, I'd be happy to do it but I do not

know how much I will be available online in the coming days. You can drop
me a mail/DM on twitter, and if I am available I'll gladly help.

Concerning technology, Zoom usually works quite well (https://zoom.us/<

https://zoom.us/ent?zcid=3172>)

  regards,

Lorenzo

On Sat, 23 Nov 2019 at 13:04, Roger Bivand <Roger.Bivand at nhh.no<mailto:

Roger.Bivand at nhh.no>> wrote:

A description of the status now with regard to a prototype resolution is
online at:

https://rsbivand.github.io/ECS530_h19/ECS530_III.html

I'm planning to stream a talk about this at 09:15-11:00 CET on Tuesday 3
December. I need a volunteer to test the streaming link in advance during
next week. I'm unsure which technology to use for remote participants to
provide feedback.

Contributions/comments welcome!

Roger


On Fri, 15 Nov 2019, Roger Bivand wrote:

The development version of rgdal on R-Forge is now at rev 894, and is

now

ready for trying out with PROJ6/GDAL3 workflows, and workflows that may
migrate within 6 months to modern CRS representations. The motivating

RFC is

also updated to cover coordinate operations, the use of prepared
(pre-searched) coordinate operations, and should be read carefully by

anyone

using rgdal::spTransform(). Note further that rgdal::project() will not

be

adapted for PROJ6, and is effectively deprecated.

I'll be running reverse dependency checks, and may be bugging package
maintainers. I would really prefer that mainainers of packages using
spTransform() checked themselves and joined this thread or the

associated

twitter thread:

https://twitter.com/RogerBivand/status/1194586193108914177

Be ready for modern PROJ and GDAL, they are already being deployed

across

open source geospatial software, like GRASS, QGIS, pyproj, spatialite

etc.

Waiting, hopefully not in vain, for contributions.

Roger

On Wed, 13 Nov 2019, Roger Bivand wrote:

 And this link explains the CDN proposal for grid distribution:

 https://www.spatialys.com/en/crowdfunding/

 Roger

 On Wed, 13 Nov 2019, Roger Bivand wrote:

  Because PROJ >= 6 and GDAL >= 3 change the way that PROJ strings
  (representations of coordinate reference systems) are handled,

steps are

  being taken to find ways to adapt sp/rgdal workflows. A current

proposal

  is to store the WKT2_2018 string as a comment to CRS objects as

defined

  in
  the sp package.

  A draft development-in-progress version of rgdal is available at
  https://r-forge.r-project.org/R/?group_id=884, and for sp at
  https://github.com/rsbivand/sp (this version of sp requires rgdal

=

  1.5-1). This adds the WKT comments to CRS objects on reading vector

and

  raster data sources, and uses WKT comments if found when writing

vector

  and raster objects (or at least does as far as I've checked,

possibly

  fragile).

  An RFC with tersely worked cases for using CRS object comments to

carry

  WKT strings but maintaining full backward compatibility is online at
  http://rgdal.r-forge.r-project.org/articles/PROJ6_GDAL3.html.

  If you have other ideas or concerns about trying to use this

mechanism

  for
  sp CRS objects, please contribute at your earliest convenience.

  http://rgdal.r-forge.r-project.org/reference/list_coordOps.html

shows

  the
  beginning of the next step, to query transformation operations to

find

  viable coordinate operation pipelines.

  I'm assuming that the previous behaviour (transform without

considering

  accuracy with whatever is to hand) is not viable going forward, and

that

  we will need two steps: list coordinate operations between source

and

  target CRS (using the WKT comments as better specifications than the
  PROJ
  strings), possibly intervene manually to install missing grids, then
  undertake the coordinate operation.

  The fallback may be simply to choose the least inaccurate available
  coordinate operation, but this should be a fallback. This means

that all

  uses of spTransform() will require intervention.

  Is this OK (it is tiresome but modernises workflows once), or is it

not

  OK
  (no user intervention is crucial)?

  These behaviours may be set in an option, so that package

maintainers

  and
  users may delay modernisation, but all are undoubtedly served by

rapid

  adaptation (GRASS 7.8.1 released yesterday, libspatialite, pyproj,

QGIS

  development versions all state that they list candidate coordinate
  operations).

  We cannot ship all the grids, they are very bulky, and probably

nobody

  needs sub-metre accuracy world-wide. Work in PROJ is starting to

create

  a
  content delivery network for trusted download and mechanisms for
  registering downloaded grids on user platforms. We would for

example not

  want Windows users of rgdal and sf to have to download the same grid
  twice.

  Comments welcome here and at
  https://github.com/r-spatial/discuss/issues/28 or
  https://github.com/r-spatial/sf/issues/1187

  Roger

--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no<mailto:

Roger.Bivand at nhh.no>

https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en

--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en