Reporting an issue with rgdal in macOS BigSur

Dear Roger

Many thanks for your support. I built GDAL from scratch and now it 
supports the JPEG-2000 driver using the ?openjpeg? library. The reported 
problems are now solved.

The whole exercise gave me some lessons on RGDAL and GDAL for users of 
MacOS BigSur:

1. The usual ways of installing GDAL in macOS (using Homebrew or the
   Frameworks provided by KingChaos) do not provide support for
   JPEG-2000. So, one has to install GDAL and RGDAL from source.

2. One has to use Apple?s C/C++ clang compiler version 12.0, following
   the advice given in Stack Overflow:

https://stackoverflow.com/questions/63972113/big-sur-clang-invalid-version-error-due-to-macosx-deployment-target 
<https://stackoverflow.com/questions/63972113/big-sur-clang-invalid-version-error-due-to-macosx-deployment-target>

3. For GDAL to compile properly, it is necessary to install first
   ?pkg-config?, and then the libraries ?proj?, ?geo?, ?tiff?, ?jpeg?,
   ?openjpeg?, and ?zstd?.

Thus, if other users of ?gdal? find problems with BigSur, I can 
volunteer to support them.

Best regards
Gilberto

===========================
Prof Dr Gilberto Camara
Secretariat Director
GEO - Group on Earth Observations
7 bis, Avenue de La Paix
CH-1211 Geneva - Switzerland
Tel: +41227308480
Web: www.earthobservations.org

On 26 Nov 2020, at 15:40, Roger Bivand <Roger.Bivand at nhh.no> wrote:

Dear Gilberto,

On Thu, 26 Nov 2020, GilbertoCamara wrote:

Dear R-SIG-GEO and Roger

I am developing an R package (https://github.com/e-sensing/sits <https://github.com/e-sensing/sits>) to analyse satellite image time series (SITS) which is heavily dependent on ?rgdal?.

I would like to report an issue with rgdal in the new macOS BigSur. While I agree with Roger that it is not wise to install a new OS until it is stable, I hope the information below will be useful.

For starters, let me praise Roger, Edzer and all those involved in ?rgdal?. I did some debugging on the source code and was humbled by the effort involved in making the code an operational software many of us rely on. Chapeau bas!

That said, please allow me to report on three issues:

1. Installation

I have installed GDAL-3.2.0 using Homebrew on BigSur; it works fine. When installing the binary version of rgdal-1.5-18 (SVN revision 1082) from CRAN, recently released by Simon Urbanek, ?rgdal" reports that is is using GDAL-3.1.1, which is not installed. The actual message is: "Loaded GDAL runtime: GDAL 3.1.1, released 2020/06/22?.

When installing from source, rgdal recognises GDAL-3.2.0, if the Homebrew libraries are on the path. The message changes to: "Loaded GDAL runtime: GDAL 3.2.0, released 2020/10/26."

It is odd that the binary version would report using a runtime that is not there.

On MacOS and Windows, CRAN binary packages are built using static linking, for a number of reasons.  One is to limit installation difficulties, another to make support easier, as all users of a updated package/R version will be using the same set of external software components (of course, locales will vary, but most variables are fixed).

2. Errors in /vsis3 file access

In earlier versions of macOS, rgdal had no problems with accessing files in AWS with commands such as:

s3_file <- "/vsis3/sentinel-s2-l2a/tiles/20/L/KP/2018/8/17/0/R20m/B02.jp2?
rgdal::GDALinfo(s3_file)

In BigSur, it produces an error. Debugging the rgdal code, I found the error to arise from line 723 of the ?gdal-bindings.cpp? file. This is one of the places where rgdal calls the GDAL function ?GDALOpenEx?, as follows:

Please first check on running systems with pre-Big Sur and Big Sur that the returned lists of drivers match. Crucially, jp2 (Jpeg 2000) needs GDAL built against openjpeg. If one of the systems has GDAL built with the driver, and the other without, this outcome would be expected. I suspect that the Big Sur GDAL (or the CRAN MacOS rgdal binary) has missing/defective drivers.

For my Fedora 33 system with GDAL installed from source, I have:

drvs <- rgdal::gdalDrivers()$name; drvs[grep("JP2", drvs)]
## [1] "JP2OpenJPEG"

There are other JP2* drivers, but they need closed source libraries when GDAL itself is built, and cannot be redistributed from CRAN.

GDALDataset *pDataset = (GDALDataset *) GDALOpenEx(fn, RWFlag,
  papszAllowedDrivers, papszOpenOptions, NULL);

The function call returns a NULL pointer. Since calling gdalinfo externally or using gdalUtils::gdalinfo works, it seems there is a problem with the return parameters which are provided to rgdal by GDAL.

To reproduce the issue, one needs AWS credentials. If Roger and/or Edzer wish to have a go at this issue, I can provide a temporary credential for doing so.

First, we need to be sure that the driver is present, then present and working.

3. Errors in TIFF Decoding

Running devtools::check() produces a strange error on TIFF decoding on a very small file. The error is odd:

"TIFFReadDirectory: Warning, Unknown field with tag 42112 (0xa480) encountered."

The call stack is as follows:

 1: .Call("RGDAL_OpenDataset", .normalize_if_path(filename, mustWork = NA),     TRUE, silent, allowedDrivers, options, PACKAGE = "rgdal")
 2: .local(.Object, ...)
 3: initialize(value, ...)
 4: initialize(value, ...)
 5: new("GDALReadOnlyDataset", filename, silent = silent, allowedDrivers = allowedDrivers,     options = options)
 6: GDAL.open(fname, silent = silent, allowedDrivers = allowedDrivers,     options = options)
 7: rgdal::GDALinfo(file, silent = FALSE)

The error appears to be in the same place as the one reported above. However, it is hard to reproduce, because when running on the console, rgdal works.

This feels again like a driver mismatch. Please try with the CRAN MacOS binary package, for which https://cran.r-project.org/web/checks/check_results_rgdal.html looks clean. The examples for rgdal::readGDAL() include reading TIFF files, so are run in CRAN daily checks.

I have no access to any MacOS platform, so cannot do more than arm's length debug. My clear suspicion is that the GDAL you are using has drivers missing/defective. A small JP2 raster could be added to the examples to trap driver malfunction.

Best wishes,

Roger

ndvi_file <- c(system.file("extdata/raster/mod13q1/sinop-ndvi-2014.tif", package = "sits?))
rgdal::GDALinfo(ndvi_file, silent = FALSE)

However, given that the place on the ?rgdal? code where the error occurs is the same, it might be worthwhile to look into the RGDAL-GDAL interface in more detail.

I know it?s my responsibility to have moved to macOS BigSur too soon. Thus, I do not expect that you address the issue. That said, any help would be much appreciated.

Best regards
Gilberto
===========================
Prof Dr Gilberto Camara
Secretariat Director
GEO - Group on Earth Observations
7 bis, Avenue de La Paix
CH-1211 Geneva - Switzerland
Tel: +41227308480
Web: www.earthobservations.org <http://www.earthobservations.org/>

--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en