rgdal release candidate 1.5-9 rev. 1000 ready for testing

Roger,

thanks for your extensive answer, much appreciated.

I won't commenting on every detail, this would make the discssion somewhat 
bloated.

It's always the "cat and mouse game" between the libraries, packages 
managers and client packages. However, the chain is library > package 
manager > client package in my opinion, meaning the client package has 
the burden to at least check the first two and provide **at least** 
instructions for every OS how to get the package installable.

The dev toolchain on macOS in general and how things are linked/build is a 
different topic.
For this, I am somewhat missing (until recently) public accessibility of the 
build process/toolchain, even though things are moving forward with the 
creation of https://github.com/R-macos?type=source lately.

Regarding GDAL2 and PROJ>=6: If there is such an offset in spatial 
projections using this combination, I welcome the blocker. However, this 
was not clear in any way before your last post a few hours ago. However, 
I would also welcome more information on this then, either during 
installation attempts or at some other place. Maybe even opening an 
issue at the respective package manager informing the people about the 
dangerous situation. The more attraction an issue gets, the earlier it 
gets fixed.

Apologies if some of my recent messages were offensive to you or anyone else.
However, sometimes discussions need "clear" statements with emotions 
excluded.

In the end, we all aim for a smooth experience and aim to have everybody on 
board.

Best, Patrick

On 6 Jun 2020, at 18:09, Roger Bivand wrote:

 On Sat, 6 Jun 2020, Patrick Schratz wrote:

 Roger,

 I am sorry for arguing differently so often recently, but if I think that
 unfair arguing is going on, I have the feeling to correct this.

 First, again I think that stating "I do not have access to this system"
 is a weak reason in 2020.

 Your view, not mine. I cannot ask my employer to provide such resources.

 I've said this previously but again: As a developer/maintainer, there is
 the implicit burden to set up a dev environment across different
 platforms. CI helps here.

 No, this is not the main burden. For forward-facing packages interfacing
 external software, it is vastly more important to track developments in
 that software, by reading development mailing lists, and checking against
 alpha, beta and RC, as well as tracking master/trunk in periods between
 releases or when things happening in the external software seems to have
 the potential to impact existing R packages. This obviously absorbs a lot
 of time and energy. https://github.com/r-spatial/sf/issues/545 documents
 where we were in November 2017 with this; there are lots of other examples
 in posts  on the PROJ and GDAL lists, and in email exchanges with others.

 For more detailed testing, local emulation is possible via virtual
 machines which also applies to macOS systems (setting up is harder but
 not impossible).

 Before switching back to macOS a few months ago, I had a virtual machine
 of macOS running and used it successfully for dev purposes.

 The problem with macOS compared to Windows has been that while providing
 static builds for Windows for three iterations of the Windows RTools build
 train, with help from CRAN and others, most recently Jeroen, has gone
 fairly smoothly for more than ten years, macOS moves too quickly. For
 Simon to keep R itself running has been an uphill struggle, with problems
 with Fortran, versions of system software under clang (most recently the
 archaic ssl shipped by Apple), so providing fresh builds of PROJ, GDAL and
 GEOS has understandably not been a priority. We do not build Windows or
 macOS PROJ, GDAL or GEOS libraries, we need others, probably close to
 CRAN, to do this, following CRAN static linking policy.

 Note that Brian Ripley continues to monitor and check possibilities for
 static builds of OSGeo software for static linking to CRAN binary packages
 for macOS, but does not have breakthroughs to report for recent PROJ or
 GDAL releases, unfortunately. We are very fortunate that he continues to
 help us with this. rgdal 1.5-9 is partly to resolve your problem, but just
 as much to resolve problems on Solaris and other corner-case CRAN check
 platforms.

 Static linking isn't the only policy, but only Linux platforms provide
 (fairly) mature dynamic build package managers. OSGeo4W - built on cygwin
 - has been tried by some over the years, because CRAN static linked binary
 packages may have different versions than say QGIS, GRASS or other Windows
 binaries. We have touched on the idea of proposing R-spatial as an OSGeo
 project, among other things to try to match the versions of key software
 components better. But having many alternatives for source install on
 macOS runs counter to this, we need to find a path that works - the
 previous go-to was Kyngchaos, but that is no longer a viable solution.

 Second: All package managers seek to provide stability and homebrew is
 one of the most sophisticated ones out there. I honestly do not
 understand the general bashing against package managers here.

 Package managers were debated on R-sig-mac (yes, I follow that too). If
 you use a package manager, that is your choice, but having CI for macOS
 (past and current releases) then gets multiplied  by the number of package
 managers and environments (conda), etc. For central resources, say curl,
 package managers may be OK, but get stuck when some downstream packages
 either get dropped because they are not up to speed with say PROJ/GDAL, or
 PROJ/GDAL get held up because the packages using them are seen as
 important. This happens in all package managers, and is frequently
 discussed on PROJ and GDAL lists, amonng others by those responsible for
 package manager releases. The decisions are not easy, but we cannot be
 held back by package managers' policy choices. My guess is that some
 package in homebrew depends on GDAL 2.* and has not yet upgraded to be
 able to use GDAL 3.*; since GDAL 2.* can be built (although using the
 deprecated API) with PROJ >= 6, they let things slide. They should have
 chosen to stop PROJ at 5.2.*, avoiding the used of the deprecated API.

 Third: What role does Apple play in all of this? I am not arguing that
 they made some decisions that did not necessarily enhance the dev
 experience on macOS.
 However, I do not see any of these having an effect on the spatial
 library stack, especially GDAL and PROJ.

 The current situation is that the **main** package manager on macOS,
 namely homebrew, has a temporary version situation of GDAL and PROJ that
 is (for no clear reason yet) blocked by the client package rgdal.

 Users on macOS can however use the formulas of the osgeo4mac
 (https://github.com/OSGeo/homebrew-osgeo4mac) tap which comes with PROJ7
 and GDAL3 to solve these issues.

 ```r
 brew tap osgeo/osgeo4mac
 brew unlink gdal
 brew unlink proj

 brew install osgeo-gdal
 brew install osgeo-proj
 brew link osgeo-gdal
 brew link osgeo-proj
 ```

 I am well aware that many users of spatial software are not developers in
 their every day life and should stick to binaries.
 However, there is a group that does source installs and there are CI
 checks that rely on proper source installations with the current stack of
 the main package managers on a platform.
 And exactly this group is blocked right now by blocking the rgdal
 installation at all, with a somewhat weak reasoning for this action.

 Block is your expression, what is being required is that users actively
 opt in to using a very-short-life deprecated API. If I relax the
 requirement (I have already opened up for using the deprecated API, which
 I regard as potentially leading to positional accuracy loss, and certainly
 encouraging business as usual rather than active migration of workflows to
 WKT2 from Proj4 strings), I would be demonstrating crass irresponsibility
 with regard to users of rgdal, who need to know that these changes (in
 GDAL/PROJ) may impact their work.

 In addition, arguing/ranting about specific platforms with points
 unrelated to the current issues is a thing that I absolutely dislike,
 getting me started arguing against it.
 I also do not like certain platforms and have personal favorites.
 However, I always check on all and make sure that everyone gets a
 pleasant experience on their platform, even if that's painful for me and
 costs a lot of time.

 Pleasant experience is not something that carries much weight. Things need
 to work, and if user choices may lead to degradation through use of a
 deprecated API, I feel that they should be warned, and that that is more
 important than any pleaasant experience.

 For many years, OSX has been checked by CRAN, and this will continue. I do
 not accept that package maintainers have to resolve these problems (source
 installs of packages using external software), if they follow up CRAN
 requirements and make sure that the source packages continue to install
 and check cleanly with successive versions of the external software.

 I am well aware that I might not be invited to the next imaginary party
 with my arguing here but maybe the discussion can make use of more real
 facts again, lower subjective views, and focus on providing support for
 everyone, on all platforms, without introducing something like a
 "platform racism" with semi-fake news.

 I'm uncertain what you are referring to. I do not have a problem with
 platforms, I just see the role of package developers differently, as I
 explained above. At some point this year or next, the CRAN Windows and
 macOS binaries will have caught up (Windows already at PROJ 6.3.1 aand
 GDAL 3.0.4), so that for all users and developers but those insisting on
 source installs, the temporary difficulties now being experienced will
 pass. For those installing from source, when package managers catch up
 with the speed of change in PROJ and GDAL, things will calm down.

 Roger

 Best, Patrick

 On 6 Jun 2020, at 15:45, Roger Bivand wrote:

  On Sat, 6 Jun 2020, Ista Zahn wrote:

 On Fri, Jun 5, 2020 at 7:47 PM Manuel Sp?nola <mspinola10 at gmail.com>
  wrote:

  Dear list members,

 Sorry for the confusion, but with all these suggestions, what is the
 way
  to
  have the updated versions of the external
  software GEOS, PROJ, and GDAL for macOS users.

  I think the current recommendation is "if you have to ask don't do
  it". Just wait for these to be updated in the OSX binary packages
 on
  CRAN.

  Thanks, a much better way of saying this!

 We really would like to be able to help macOS users who see "Install
 from
 source?" and are tempted to choose "yes", but not only do we not have
 the
 resources or access to running systems, but also, at the moment, things
 seem very unpredictable. We do not think that environments are helpful,
  and many package managers do not seem to have sufficient focussed
  attention, which is understandable given Apple's gift for moving
 the
  goalposts.

 If users can install external software from source (macOS, Linux),
 they/we
 have a good deal of freedom. But this takes time, insight, and for many
 is
 problematic because their production system is blocked until the new
 versions are ready (PROJ and GDAL are C++11 or more, and take an order
 of
  magnitude longer to compile than just a few years ago).

 So for Windows and macOS, waiting for the CRAN binaries is a reasonable
  choice.

 Beyond this, we need to find ways of providing share/proj and share/gdal
  metadata files for all of the packages now using the PROJ and GDAL
  libraries, and of navigating the content download network for
 geodetic
 transformation grids available from PROJ 7. But that is another story
 ...

  Roger

  Best,
  Ista

  Manuel

  El vie., 5 jun. 2020 a las 14:31, Patrick Schratz (<
  patrick.schratz at gmail.com>) escribi?:

  I am not sure if the part with

  use --with-proj_api="proj_api.h" for deprecated API

 Is of much help since c/p won?t work but the text let?s people
  assume that c/p could/should work.
  In fact, a full path to ?proj_api.h? is required?

  I still do not like this blocker and I still do not know if this
  combination causes serious issues in production or just limits
 new
  features.

 For the time being, using and linking osgeo-gdal (3.0.1) and
 osgeo-proj
 (7.0.1) works and can be used as a workaround until homebrew-core
  formulas catch up.

  checks OK on PROJ 7.0.1 and GDAL 2.2.4

  Again, since it was maybe caused by my typo a few mails ago: The
  homebred-core gdal version is at 2.4.4 and not 2.2.4.

  On 5 Jun 2020, at 13:29, Roger Bivand wrote:

 The release candidate of rgdal_1.5-9 is ready for testing on
 R-forge:

  https://r-forge.r-project.org/R/?group_id=884

 Those insisting on installing on PROJ >= 6 and GDAL < 3 must use
 configure argument --with-proj_api="proj_api.h"; with this used,
 this
 version builds with --no-build-vignettes and installs and checks OK
 on
  PROJ 7.0.1 and GDAL 2.2.4 with --with-proj_api="proj_api.h".

 Otherwise checked OK with PROJ 4.8.0, 4.9.2, 4.9.3 and 5.2.0 with
 GDAL
 1.11.4; with PROJ 5.2.0 and GDAL 2.2.4, 2.3.2 and 2.4.2; with PROJ
  6.3.2 and GDAL 3.0.4; with PROJ 7.0.1 and GDAL 3.0.4 and 3.1.0.

 All who have indicated issues with source installs are asked to try
 the release candidate and to report back here by midnight CEST
 Monday
 8 June. If no indications are forthcoming, I'll assume that problems
  with 1.5-8 are resolved.

  Roger

  --
  Roger Bivand
  Department of Economics, Norwegian School of Economics,
  Helleveien 30, N-5045 Bergen, Norway.
  voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no
  https://orcid.org/0000-0003-2392-6140
  https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en

  --
  *Manuel Sp?nola, Ph.D.*
  Instituto Internacional en Conservaci?n y Manejo de Vida
 Silvestre
  Universidad Nacional
  Apartado 1350-3000
  Heredia
  COSTA RICA
  mspinola at una.cr <mspinola at una.ac.cr>
  mspinola10 at gmail.com
  Tel?fono: (506) 8706 - 4662
  Personal website: Lobito de r?o
  <https://sites.google.com/site/lobitoderio/>
  Institutional website: ICOMVIS <http://www.icomvis.una.ac.cr/>

          [[alternative HTML version deleted]]

  --
  Roger Bivand
  Department of Economics, Norwegian School of Economics,
  Helleveien 30, N-5045 Bergen, Norway.
  voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no
  https://orcid.org/0000-0003-2392-6140
  https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________
  R-sig-Geo mailing list
  R-sig-Geo at r-project.org

 --
 Roger Bivand
 Department of Economics, Norwegian School of Economics,
 Helleveien 30, N-5045 Bergen, Norway.
 voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no
 https://orcid.org/0000-0003-2392-6140
 https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en