rgdal: About the new gdal3 and proj >=6 condition
On Fri, 29 May 2020, Patrick Schratz wrote:
The URL returns a 404.
Do you mean http://spatial.nhh.no/R/rgdal/? Please do say what you mean. If your browser blocks http: rather than https:, that is your choice. I imagine that it silently inserts an "s".
FWIW I?ve tried it locally using the latest svn revision but have no clue where I should find `proj_api.h` (I?ve looked in the complete proj installation directory).
If PROJ 7.0.1 is properly installed, the header will be in $prefix/include. If your packager put all the headers somewhere else, use the command argument: $ ls /usr/local/include/proj* /usr/local/include/proj_api.h /usr/local/include/proj.h /usr/local/include/proj_constants.h /usr/local/include/proj_symbol_rename.h /usr/local/include/proj_experimental.h /usr/local/include/proj: common.hpp crs.hpp metadata.hpp coordinateoperation.hpp datum.hpp nn.hpp coordinatesystem.hpp io.hpp util.hpp You should not need to know. What does pkg-config proj --cflags say (once you've set PKG_CONFIG_PATH, of course.
I see ``` ls /usr/local/Cellar/proj/7.0.1/lib/ libproj.19.dylib libproj.a libproj.dylib@ libproj.la* pkgconfig/ ```
In your earlier post, you negected to say that you used the necessary configure argument. The error message says that you did not use it. Please report back when you have done so. Futher, I quote from your message: "It further causes complete breakage for people relying on the homebrew package manager on macOS since current versions are gdal v2.2.4 and proj 7.0.1." Please accept that I pay attention to what you write.
Even if this custom installation might work at some point, making such adjustments until gdal3 arrives in homebrew will be a major pain, also for CI builds. It would still be highly appreciated if this internal assertion would be removed again.
Getting things right may be painful; GDAL 2 with PROJ 7 is wrong, because it mixes two incompatible metadata storage systems. Migrating all users to GDAL >= 3 & PROJ >= 6 is a matter of urgency to avoid silent positional error of about 125m, or about 4 satellite image cells. See the vignette here http://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html and this r-spatial blog https://www.r-spatial.org/r/2020/03/17/wkt.html. I am not willing to let silent error creep in because some people find it inconvenient to avoid the problem. Both sf and sp/rgdal are covering your backs even if it feels rough. We've already helped dozens of maintainers of reverse dependencies future-proof their users from the consequences of changes made in PROJ and GDAL, where we have also contributed to making the transition easier by exposing helper functions in their APIs, which we use. I expect you to be able to CMD check a tarball from the commmand line, no more than that, to report back, and to try to grasp that these external software components are really important and completely non-trivial. Roger
On 29 May 2020, at 18:37, Roger Bivand wrote:
I have put a tarball, built without vignettes with GDAL 2.2.4 and PROJ 7.0.1 that passes CMD check (with warnings for missing vignettes) on: http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz It involves code copying to provide duplicated functions in three settings: PROJ < 6 & GDAL < 3 PROJ > = 6 & GDAL >= 3 PROJ > = 6 & GDAL < 3 (your homebrew case) You need the configure argument: --configure-args=--with-proj_api=proj_api.h. Without the argument, you now get directed to use it. For now I've dropped the configure tests for sqlite3, curl and tiff; works for me but may not work for you. Please make the output of: R CMD check --install-args="--configure-args=--with-proj_api=proj_api.h" rgdal_1.5-9.tar.gz available ASAP. I count on an immediate response. Roger On Fri, 29 May 2020, Roger Bivand wrote:
On Fri, 29 May 2020, Patrick Schratz wrote:
The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is present. It does not even install the package if this condition is not met but errors during linking.
Please document with the output of 00install.out from R CMD check rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo --version with system details. Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and the old metadata structure with Proj4 string degradation; GDAL < 3 with PROJ
= 6 must use the deprecated API and the new metadata structure.
I could not find any sources in the web or in the announcement post of v1.5.8 why this condition was enforced so strictly. Does it cause unwanted results?
Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
complete break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with
PROJ > = 6) after GDAL barnraising. So:
PROJ
< 6 >= 6 GDAL < 3 OK NO
> = 3 NO OK
If so, then there is a big issue since the ?gdal2 and proj >= 6? combination has been used by many people in the past already.
No good precedent, just binary packagers protecting slow downstream adaptation.
I haven?t tracked down for how long this situation was on the market
already.
Also the {sf} package is still installable with this combination
and I
am not aware of any warnings/issues that came up due to this so far.
rgdal is a much older package and has much more older code that needs this protection. It is possible to accommodate the no-go areas, but I'd really value patches to R-forge, as I cannot check multiple PROJ/GDAL version combinations just because someone isn't up to speed.
It further causes complete breakage for people relying on the homebrew package manager on macOS since current versions are gdal v2.2.4 and proj 7.0.1.
They should never have gone beyond PROJ 5 if they are stuck on GDAL. PROJ 7 is a very different animal.
The update to gdal3 is blocked since months due to an incompatibility with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading the issue, it looks unclear when this issue will be resolved. The alternative osgeo4mac tap holds formula that is unstable and somewhat broken. One cannot link rgdal with the current gdal v3.0.1 from osgeo4mac. This incompatibility with homebrew-core formula leads to further issues for CI tests that rely on the spatial homebrew stack for macOS builds. All of this is really unfortunate and I am wondering if this was a known factor when introducing this condition in the new release. Unfortunately rgdal is not hosted on any Git* provider (there is just the CRAN mirror with not much information about the recent changes) and it is unclear/untransparent to me if there is a continuous integration setup at all for this package.
Do not require me to leave the excellent SVN service on R-forge. Provide patches there.
I hope this is not another case in the series ?I do not like operating system XY and hence I do not test on it? which was seen recently in another package. That?s what CI is for and the responsibility of package authors.
I run and develop on Fedora, I have no access to OSX, and the current release was fully checked with ~ 900 revdeps repeatedly since November on successive older and newer versions of PROJ and GDAL. I blocked new PROJ with old GDAL recently to simplify development. CI is only as good as the scripts, I am completely sure that repeated revdep testing and reading the output is superior.
Also one should be aware that not every rgdal user is member of this mailing list (I guess not even 5%) and many people in production will face this error during install, most of them without any clue what to do or an idea why this was raised.
There has been plenty of information given about the CRS changes. rgdal could simply have been abandoned by me, and all those production volunteers might have fixed things, but I never hear anything at all.
In summary it would be great if - There would be more detailed information on the introduction of the new condition - Awareness of state-of-the-art platform related library versions before making such a release - Transparency/Existence of a cross-platform build process - If the incompatibility is just due to some features not being accessible with this configuration, I suggest to raise a warning during package load but not prevent the package from installing in the first place - A version-based NEWS file would be available. Currently one only sees a revision-based Changelog: https://cran.r-project.org/web/packages/rgdal/ChangeLog I am well aware that I am indirectly criticising (hopefully in a constructive way) the transparency and release process here, for which I will probably get a rough response. However, if that leads to more robustness and transparency in future release, I am fine with this. A quick patch release resolving the breakage would be highly appreciated.
Only with community imput. what you ask is not needed, just extra complexity. Please provide patches, or accept my invitation to join the R-forge project and commit your fixes directly. I can see how to do it, but I don't think it makes sense, and your messsage has not motivated me, to be honest. I'm prioritizing working with CRAN to iron out reverse dependency problems. Roger
Best, Patrick [[alternative HTML version deleted]]
_______________________________________________ R-sig-Geo mailing list R-sig-Geo at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo
-- Roger Bivand Department of Economics, Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no https://orcid.org/0000-0003-2392-6140 https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
Roger Bivand Department of Economics, Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no https://orcid.org/0000-0003-2392-6140 https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en