The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is
present.
It does not even install the package if this condition is not met but
errors during linking.
I could not find any sources in the web or in the announcement post of
v1.5.8 why this condition was enforced so strictly.
Does it cause unwanted results?
If so, then there is a big issue since the ?gdal2 and proj >= 6?
combination has been used by many people in the past already.
I haven?t tracked down for how long this situation was on the market
already.
Also the {sf} package is still installable with this combination and I
am not aware of any warnings/issues that came up due to this so far.
It further causes complete breakage for people relying on the homebrew
package manager on macOS since current versions are gdal v2.2.4 and proj
7.0.1.
The update to gdal3 is blocked since months due to an incompatibility
with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading the
issue, it looks unclear when this issue will be resolved.
The alternative osgeo4mac tap holds formula that is unstable and
somewhat broken. One cannot link rgdal with the current gdal v3.0.1 from
osgeo4mac.
This incompatibility with homebrew-core formula leads to further issues
for CI tests that rely on the spatial homebrew stack for macOS builds.
All of this is really unfortunate and I am wondering if this was a known
factor when introducing this condition in the new release.
Unfortunately rgdal is not hosted on any Git* provider (there is just
the CRAN mirror with not much information about the recent changes) and
it is unclear/untransparent to me if there is a continuous integration
setup at all for this package.
I hope this is not another case in the series ?I do not like operating
system XY and hence I do not test on it? which was seen recently in
another package. That?s what CI is for and the responsibility of
package authors.
Also one should be aware that not every rgdal user is member of this
mailing list (I guess not even 5%) and many people in production will
face this error during install, most of them without any clue what to do
or an idea why this was raised.
In summary it would be great if
- There would be more detailed information on the introduction of the
new condition
- Awareness of state-of-the-art platform related library versions before
making such a release
- Transparency/Existence of a cross-platform build process
- If the incompatibility is just due to some features not being
accessible with this configuration, I suggest to raise a warning during
package load but not prevent the package from installing in the first
place
- A version-based NEWS file would be available. Currently one only sees
a revision-based Changelog:
https://cran.r-project.org/web/packages/rgdal/ChangeLog
I am well aware that I am indirectly criticising (hopefully in a
constructive way) the transparency and release process here, for which I
will probably get a rough response.
However, if that leads to more robustness and transparency in future
release, I am fine with this.
A quick patch release resolving the breakage would be highly
appreciated.
Best, Patrick
rgdal: About the new gdal3 and proj >=6 condition
9 messages · Patrick Schratz, Roger Bivand, Roy Mendelssohn - NOAA Federal +1 more
On Fri, 29 May 2020, Patrick Schratz wrote:
The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is present. It does not even install the package if this condition is not met but errors during linking.
Please document with the output of 00install.out from R CMD check rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo --version with system details. Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and the old metadata structure with Proj4 string degradation; GDAL < 3 with PROJ
= 6 must use the deprecated API and the new metadata structure.
I could not find any sources in the web or in the announcement post of v1.5.8 why this condition was enforced so strictly. Does it cause unwanted results?
Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the complete
break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with PROJ >= 6) after
GDAL barnraising. So:
PROJ
< 6 >= 6
GDAL < 3 OK NO
>= 3 NO OK
If so, then there is a big issue since the ?gdal2 and proj >= 6? combination has been used by many people in the past already.
No good precedent, just binary packagers protecting slow downstream adaptation.
I haven?t tracked down for how long this situation was on the market
already.
Also the {sf} package is still installable with this combination and I
am not aware of any warnings/issues that came up due to this so far.
rgdal is a much older package and has much more older code that needs this protection. It is possible to accommodate the no-go areas, but I'd really value patches to R-forge, as I cannot check multiple PROJ/GDAL version combinations just because someone isn't up to speed.
It further causes complete breakage for people relying on the homebrew package manager on macOS since current versions are gdal v2.2.4 and proj 7.0.1.
They should never have gone beyond PROJ 5 if they are stuck on GDAL. PROJ 7 is a very different animal.
The update to gdal3 is blocked since months due to an incompatibility with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading the issue, it looks unclear when this issue will be resolved. The alternative osgeo4mac tap holds formula that is unstable and somewhat broken. One cannot link rgdal with the current gdal v3.0.1 from osgeo4mac. This incompatibility with homebrew-core formula leads to further issues for CI tests that rely on the spatial homebrew stack for macOS builds. All of this is really unfortunate and I am wondering if this was a known factor when introducing this condition in the new release. Unfortunately rgdal is not hosted on any Git* provider (there is just the CRAN mirror with not much information about the recent changes) and it is unclear/untransparent to me if there is a continuous integration setup at all for this package.
Do not require me to leave the excellent SVN service on R-forge. Provide patches there.
I hope this is not another case in the series ?I do not like operating system XY and hence I do not test on it? which was seen recently in another package. That?s what CI is for and the responsibility of package authors.
I run and develop on Fedora, I have no access to OSX, and the current release was fully checked with ~ 900 revdeps repeatedly since November on successive older and newer versions of PROJ and GDAL. I blocked new PROJ with old GDAL recently to simplify development. CI is only as good as the scripts, I am completely sure that repeated revdep testing and reading the output is superior.
Also one should be aware that not every rgdal user is member of this mailing list (I guess not even 5%) and many people in production will face this error during install, most of them without any clue what to do or an idea why this was raised.
There has been plenty of information given about the CRS changes. rgdal could simply have been abandoned by me, and all those production volunteers might have fixed things, but I never hear anything at all.
In summary it would be great if - There would be more detailed information on the introduction of the new condition - Awareness of state-of-the-art platform related library versions before making such a release - Transparency/Existence of a cross-platform build process - If the incompatibility is just due to some features not being accessible with this configuration, I suggest to raise a warning during package load but not prevent the package from installing in the first place - A version-based NEWS file would be available. Currently one only sees a revision-based Changelog: https://cran.r-project.org/web/packages/rgdal/ChangeLog I am well aware that I am indirectly criticising (hopefully in a constructive way) the transparency and release process here, for which I will probably get a rough response. However, if that leads to more robustness and transparency in future release, I am fine with this. A quick patch release resolving the breakage would be highly appreciated.
Only with community imput. what you ask is not needed, just extra complexity. Please provide patches, or accept my invitation to join the R-forge project and commit your fixes directly. I can see how to do it, but I don't think it makes sense, and your messsage has not motivated me, to be honest. I'm prioritizing working with CRAN to iron out reverse dependency problems. Roger
Best, Patrick [[alternative HTML version deleted]]
_______________________________________________ R-sig-Geo mailing list R-sig-Geo at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Roger Bivand Department of Economics, Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no https://orcid.org/0000-0003-2392-6140 https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
I have put a tarball, built without vignettes with GDAL 2.2.4 and PROJ 7.0.1 that passes CMD check (with warnings for missing vignettes) on: http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz It involves code copying to provide duplicated functions in three settings: PROJ < 6 & GDAL < 3 PROJ >= 6 & GDAL >= 3 PROJ >= 6 & GDAL < 3 (your homebrew case) You need the configure argument: --configure-args=--with-proj_api=proj_api.h. Without the argument, you now get directed to use it. For now I've dropped the configure tests for sqlite3, curl and tiff; works for me but may not work for you. Please make the output of: R CMD check --install-args="--configure-args=--with-proj_api=proj_api.h" rgdal_1.5-9.tar.gz available ASAP. I count on an immediate response. Roger
On Fri, 29 May 2020, Roger Bivand wrote:
On Fri, 29 May 2020, Patrick Schratz wrote:
The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is present. It does not even install the package if this condition is not met but errors during linking.
Please document with the output of 00install.out from R CMD check rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo --version with system details. Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and the old metadata structure with Proj4 string degradation; GDAL < 3 with PROJ
= 6 must use the deprecated API and the new metadata structure.
I could not find any sources in the web or in the announcement post of v1.5.8 why this condition was enforced so strictly. Does it cause unwanted results?
Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the complete
break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with PROJ >= 6) after
GDAL barnraising. So:
PROJ
< 6 >= 6
GDAL < 3 OK NO
> = 3 NO OK
If so, then there is a big issue since the ?gdal2 and proj >= 6? combination has been used by many people in the past already.
No good precedent, just binary packagers protecting slow downstream adaptation.
I haven?t tracked down for how long this situation was on the market
already.
Also the {sf} package is still installable with this combination and I
am not aware of any warnings/issues that came up due to this so far.
rgdal is a much older package and has much more older code that needs this protection. It is possible to accommodate the no-go areas, but I'd really value patches to R-forge, as I cannot check multiple PROJ/GDAL version combinations just because someone isn't up to speed.
It further causes complete breakage for people relying on the homebrew package manager on macOS since current versions are gdal v2.2.4 and proj 7.0.1.
They should never have gone beyond PROJ 5 if they are stuck on GDAL. PROJ 7 is a very different animal.
The update to gdal3 is blocked since months due to an incompatibility with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading the issue, it looks unclear when this issue will be resolved. The alternative osgeo4mac tap holds formula that is unstable and somewhat broken. One cannot link rgdal with the current gdal v3.0.1 from osgeo4mac. This incompatibility with homebrew-core formula leads to further issues for CI tests that rely on the spatial homebrew stack for macOS builds. All of this is really unfortunate and I am wondering if this was a known factor when introducing this condition in the new release. Unfortunately rgdal is not hosted on any Git* provider (there is just the CRAN mirror with not much information about the recent changes) and it is unclear/untransparent to me if there is a continuous integration setup at all for this package.
Do not require me to leave the excellent SVN service on R-forge. Provide patches there.
I hope this is not another case in the series ?I do not like operating system XY and hence I do not test on it? which was seen recently in another package. That?s what CI is for and the responsibility of package authors.
I run and develop on Fedora, I have no access to OSX, and the current release was fully checked with ~ 900 revdeps repeatedly since November on successive older and newer versions of PROJ and GDAL. I blocked new PROJ with old GDAL recently to simplify development. CI is only as good as the scripts, I am completely sure that repeated revdep testing and reading the output is superior.
Also one should be aware that not every rgdal user is member of this mailing list (I guess not even 5%) and many people in production will face this error during install, most of them without any clue what to do or an idea why this was raised.
There has been plenty of information given about the CRS changes. rgdal could simply have been abandoned by me, and all those production volunteers might have fixed things, but I never hear anything at all.
In summary it would be great if - There would be more detailed information on the introduction of the new condition - Awareness of state-of-the-art platform related library versions before making such a release - Transparency/Existence of a cross-platform build process - If the incompatibility is just due to some features not being accessible with this configuration, I suggest to raise a warning during package load but not prevent the package from installing in the first place - A version-based NEWS file would be available. Currently one only sees a revision-based Changelog: https://cran.r-project.org/web/packages/rgdal/ChangeLog I am well aware that I am indirectly criticising (hopefully in a constructive way) the transparency and release process here, for which I will probably get a rough response. However, if that leads to more robustness and transparency in future release, I am fine with this. A quick patch release resolving the breakage would be highly appreciated.
Only with community imput. what you ask is not needed, just extra complexity. Please provide patches, or accept my invitation to join the R-forge project and commit your fixes directly. I can see how to do it, but I don't think it makes sense, and your messsage has not motivated me, to be honest. I'm prioritizing working with CRAN to iron out reverse dependency problems. Roger
Best, Patrick [[alternative HTML version deleted]]
_______________________________________________ R-sig-Geo mailing list R-sig-Geo at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Roger Bivand Department of Economics, Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no https://orcid.org/0000-0003-2392-6140 https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
It?s gdal 2.4.4 instead of 2.2.4 as written before. Should not make a difference though. ``` pkg-config proj --modversion 7.0.1 gdalinfo --version GDAL 2.4.4, released 2020/01/08 ``` With revision 993 checked out from svn I see ``? checking for unistd.h... yes checking proj.h usability... yes checking proj.h presence... yes checking for proj.h... yes checking for proj_context_create in -lproj... yes checking Using GDAL < 3 with PROJ >= 6... yes configure: error: GDAL >= 3 should be used with PROJ >= 6, use --with-proj_api="proj_api.h" for deprecated API ```
Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and the old metadata structure with Proj4 string degradation; GDAL < 3 with PROJ
What does ?makes no sense? mean here? I am interested whether this combination produces significant bugs when being used together. If its just ?the user won?t be able to make use of proj7 features without gdal3? but apart from this everything is fine, then I do not see **any** reason for this strict installation blocker. While both gdal and proj are coupled tightly, I would expect assertions in the upstream libraries to account for such clashes. It seems unlikely that client packages like rgdal need to prevent installation if this undesired version mixture of both libraries is found.
No good precedent, just binary packagers protecting slow downstream adaptation.
Not sure what you mean here exactly. My point is that the gdal2 + proj7
combination exists since March 6 in homebrew. Meaning that since then
users on macOS used this combination.
commit 7234fc2073e5475931238281493ae3367b4dcdf6
Author: Stephan H?gel <shugel at tcd.ie>
Date: Fri Mar 6 12:56:51 2020 +0000
proj 7.0.0
rgdal is a much older package and has much more older code that needs this protection. It is possible to accommodate the no-go areas, but I'd really value patches to R-forge, as I cannot check multiple PROJ/GDAL version combinations just because someone isn't up to speed.
I cannot argue about any needed protection here. However, if that is the case then there was a danger zone of almost 3 months on macOS now. IF that combination causes unwanted side-effects (which is still not fully answered). I am not saying that you should check every gdal/proj combination on every platform/distribution, especially not on the niche ones. However, there are three main platforms (with Ubuntu being the main Linux one) and it is part of the game (imo) for client packages to check valid installations at least on those three platforms, using their default package manager. If that would have happened, we would not be writing here probably because you would have recognised the issue and maybe enforced the update to GDAL3 on macOS or postponed the rgdal update. It would be great if that kind of coverage could be done in the future.
Do not require me to leave the excellent SVN service on R-forge. Provide patches there.
I am just sharing my user experience, which is probably the view of many others. More contributions would be made if contributing would be easier. R-forge and svn do not make it attractive. It will always be your choice, no question that. But if you want to provide a service to the community, sometimes there is more to that than just code/releases.
CI is only as good as the scripts, I am completely sure that repeated revdep testing and reading the output is superior.
CI and revdep checks are two different things. CI happens at every commit (in the best scenario) and on multiple platforms. It aims at package AND platform integrity. Revdepchecks are on the package side, tackling a completely different layer of security. As of today (but actually since at least 5 years) there are no excuses for not being able to check on all main platforms despite arguing with ?I do not want to do it but rely on my very own local test setup?.
There has been plenty of information given about the CRS changes. rgdal could simply have been abandoned by me, and all those production volunteers might have fixed things, but I never hear anything at all.
Not sure what this should tell me/us. If you need help maintaining the package or porting its infrastructure to the 21st century then there would be a lot of people willing to help.# However, replying to (useful) criticism with ?I could have simply abandoned it, be thankful (to me), guys? (that?s my interpretation of this sentence) does not help anyone. No one questions your contributions to the r-spatial community. However, adapting to new (development) changes and making it easier for others to contribute is also a very valuable contribution that is not really measurable in any unit.
Only with community imput. what you ask is not needed, just extra complexity. Please provide patches, or accept my invitation to join the R-forge project and commit your fixes directly. I can see how to do it, but I don't think it makes sense, and your messsage has not motivated me, to be honest. I'm prioritizing working with CRAN to iron out reverse dependency problems.
Ok, I understood. Sadly I have not enough time fighting to break up old structures and changing things to the better/making things easier if there is no real support from the other side. The r-spatial community is not really a ?community? in my eyes but a conglomerate of three (four) main persons doing it their way, each completely different, with few motivation to change things. Each semester students ask me how the relations between the packages and their authors are and why things are so different. I am not going to expand on what I tell them usually, but often enough they come to me with their personal view after some time and all I need to say is ?yes? or ?no? (sadly most often ?yes?). What this means is left as an exercise to the reader. Regards, Patrick
On 29 May 2020, at 14:45, Roger Bivand wrote:
On Fri, 29 May 2020, Patrick Schratz wrote:
The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is present. It does not even install the package if this condition is not met but errors during linking.
Please document with the output of 00install.out from R CMD check rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo --version with system details. Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and the old metadata structure with Proj4 string degradation; GDAL < 3 with PROJ
= 6 must use the deprecated API and the new metadata structure.
I could not find any sources in the web or in the announcement post of v1.5.8 why this condition was enforced so strictly. Does it cause unwanted results?
Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
complete break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with
PROJ >= 6) after GDAL barnraising. So:
PROJ
< 6 >= 6 GDAL < 3 OK NO
>= 3 NO OK
If so, then there is a big issue since the ?gdal2 and proj >= 6? combination has been used by many people in the past already.
No good precedent, just binary packagers protecting slow downstream adaptation.
I haven?t tracked down for how long this situation was on the
market
already.
Also the {sf} package is still installable with this combination and
I
am not aware of any warnings/issues that came up due to this so far.
rgdal is a much older package and has much more older code that needs this protection. It is possible to accommodate the no-go areas, but I'd really value patches to R-forge, as I cannot check multiple PROJ/GDAL version combinations just because someone isn't up to speed.
It further causes complete breakage for people relying on the homebrew package manager on macOS since current versions are gdal v2.2.4 and proj 7.0.1.
They should never have gone beyond PROJ 5 if they are stuck on GDAL. PROJ 7 is a very different animal.
The update to gdal3 is blocked since months due to an incompatibility with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading the issue, it looks unclear when this issue will be resolved. The alternative osgeo4mac tap holds formula that is unstable and somewhat broken. One cannot link rgdal with the current gdal v3.0.1 from osgeo4mac. This incompatibility with homebrew-core formula leads to further issues for CI tests that rely on the spatial homebrew stack for macOS builds. All of this is really unfortunate and I am wondering if this was a known factor when introducing this condition in the new release. Unfortunately rgdal is not hosted on any Git* provider (there is just the CRAN mirror with not much information about the recent changes) and it is unclear/untransparent to me if there is a continuous integration setup at all for this package.
Do not require me to leave the excellent SVN service on R-forge. Provide patches there.
I hope this is not another case in the series ?I do not like operating system XY and hence I do not test on it? which was seen recently in another package. That?s what CI is for and the responsibility of package authors.
I run and develop on Fedora, I have no access to OSX, and the current release was fully checked with ~ 900 revdeps repeatedly since November on successive older and newer versions of PROJ and GDAL. I blocked new PROJ with old GDAL recently to simplify development. CI is only as good as the scripts, I am completely sure that repeated revdep testing and reading the output is superior.
Also one should be aware that not every rgdal user is member of this mailing list (I guess not even 5%) and many people in production will face this error during install, most of them without any clue what to do or an idea why this was raised.
There has been plenty of information given about the CRS changes. rgdal could simply have been abandoned by me, and all those production volunteers might have fixed things, but I never hear anything at all.
In summary it would be great if - There would be more detailed information on the introduction of the new condition - Awareness of state-of-the-art platform related library versions before making such a release - Transparency/Existence of a cross-platform build process - If the incompatibility is just due to some features not being accessible with this configuration, I suggest to raise a warning during package load but not prevent the package from installing in the first place - A version-based NEWS file would be available. Currently one only sees a revision-based Changelog: https://cran.r-project.org/web/packages/rgdal/ChangeLog I am well aware that I am indirectly criticising (hopefully in a constructive way) the transparency and release process here, for which I will probably get a rough response. However, if that leads to more robustness and transparency in future release, I am fine with this. A quick patch release resolving the breakage would be highly appreciated.
Only with community imput. what you ask is not needed, just extra complexity. Please provide patches, or accept my invitation to join the R-forge project and commit your fixes directly. I can see how to do it, but I don't think it makes sense, and your messsage has not motivated me, to be honest. I'm prioritizing working with CRAN to iron out reverse dependency problems. Roger
Best, Patrick [[alternative HTML version deleted]]
_______________________________________________ R-sig-Geo mailing list R-sig-Geo at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo
-- Roger Bivand Department of Economics, Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no https://orcid.org/0000-0003-2392-6140 https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
The URL returns a 404. FWIW I?ve tried it locally using the latest svn revision but have no clue where I should find `proj_api.h` (I?ve looked in the complete proj installation directory). I see ``` ls /usr/local/Cellar/proj/7.0.1/lib/ libproj.19.dylib libproj.a libproj.dylib@ libproj.la* pkgconfig/ ``` Even if this custom installation might work at some point, making such adjustments until gdal3 arrives in homebrew will be a major pain, also for CI builds. It would still be highly appreciated if this internal assertion would be removed again.
On 29 May 2020, at 18:37, Roger Bivand wrote:
I have put a tarball, built without vignettes with GDAL 2.2.4 and PROJ 7.0.1 that passes CMD check (with warnings for missing vignettes) on: http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz It involves code copying to provide duplicated functions in three settings: PROJ < 6 & GDAL < 3 PROJ >= 6 & GDAL >= 3 PROJ >= 6 & GDAL < 3 (your homebrew case) You need the configure argument: --configure-args=--with-proj_api=proj_api.h. Without the argument, you now get directed to use it. For now I've dropped the configure tests for sqlite3, curl and tiff; works for me but may not work for you. Please make the output of: R CMD check --install-args="--configure-args=--with-proj_api=proj_api.h" rgdal_1.5-9.tar.gz available ASAP. I count on an immediate response. Roger On Fri, 29 May 2020, Roger Bivand wrote:
On Fri, 29 May 2020, Patrick Schratz wrote:
The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is present. It does not even install the package if this condition is not met but errors during linking.
Please document with the output of 00install.out from R CMD check rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo --version with system details. Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and the old metadata structure with Proj4 string degradation; GDAL < 3 with PROJ
= 6 must use the deprecated API and the new metadata structure.
I could not find any sources in the web or in the announcement post of v1.5.8 why this condition was enforced so strictly. Does it cause unwanted results?
Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
complete break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with
PROJ >= 6) after GDAL barnraising. So:
PROJ
< 6 >= 6 GDAL < 3 OK NO
> = 3 NO OK
If so, then there is a big issue since the ?gdal2 and proj >= 6? combination has been used by many people in the past already.
No good precedent, just binary packagers protecting slow downstream adaptation.
I haven?t tracked down for how long this situation was on the
market
already.
Also the {sf} package is still installable with this combination
and I
am not aware of any warnings/issues that came up due to this so
far.
rgdal is a much older package and has much more older code that needs this protection. It is possible to accommodate the no-go areas, but I'd really value patches to R-forge, as I cannot check multiple PROJ/GDAL version combinations just because someone isn't up to speed.
It further causes complete breakage for people relying on the homebrew package manager on macOS since current versions are gdal v2.2.4 and proj 7.0.1.
They should never have gone beyond PROJ 5 if they are stuck on GDAL. PROJ 7 is a very different animal.
The update to gdal3 is blocked since months due to an incompatibility with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading the issue, it looks unclear when this issue will be resolved. The alternative osgeo4mac tap holds formula that is unstable and somewhat broken. One cannot link rgdal with the current gdal v3.0.1 from osgeo4mac. This incompatibility with homebrew-core formula leads to further issues for CI tests that rely on the spatial homebrew stack for macOS builds. All of this is really unfortunate and I am wondering if this was a known factor when introducing this condition in the new release. Unfortunately rgdal is not hosted on any Git* provider (there is just the CRAN mirror with not much information about the recent changes) and it is unclear/untransparent to me if there is a continuous integration setup at all for this package.
Do not require me to leave the excellent SVN service on R-forge. Provide patches there.
I hope this is not another case in the series ?I do not like operating system XY and hence I do not test on it? which was seen recently in another package. That?s what CI is for and the responsibility of package authors.
I run and develop on Fedora, I have no access to OSX, and the current release was fully checked with ~ 900 revdeps repeatedly since November on successive older and newer versions of PROJ and GDAL. I blocked new PROJ with old GDAL recently to simplify development. CI is only as good as the scripts, I am completely sure that repeated revdep testing and reading the output is superior.
Also one should be aware that not every rgdal user is member of this mailing list (I guess not even 5%) and many people in production will face this error during install, most of them without any clue what to do or an idea why this was raised.
There has been plenty of information given about the CRS changes. rgdal could simply have been abandoned by me, and all those production volunteers might have fixed things, but I never hear anything at all.
In summary it would be great if - There would be more detailed information on the introduction of the new condition - Awareness of state-of-the-art platform related library versions before making such a release - Transparency/Existence of a cross-platform build process - If the incompatibility is just due to some features not being accessible with this configuration, I suggest to raise a warning during package load but not prevent the package from installing in the first place - A version-based NEWS file would be available. Currently one only sees a revision-based Changelog: https://cran.r-project.org/web/packages/rgdal/ChangeLog I am well aware that I am indirectly criticising (hopefully in a constructive way) the transparency and release process here, for which I will probably get a rough response. However, if that leads to more robustness and transparency in future release, I am fine with this. A quick patch release resolving the breakage would be highly appreciated.
Only with community imput. what you ask is not needed, just extra complexity. Please provide patches, or accept my invitation to join the R-forge project and commit your fixes directly. I can see how to do it, but I don't think it makes sense, and your messsage has not motivated me, to be honest. I'm prioritizing working with CRAN to iron out reverse dependency problems. Roger
Best, Patrick [[alternative HTML version deleted]]
_______________________________________________ R-sig-Geo mailing list R-sig-Geo at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo
-- Roger Bivand Department of Economics, Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no https://orcid.org/0000-0003-2392-6140 https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
On Fri, 29 May 2020, Patrick Schratz wrote:
The URL returns a 404.
Do you mean http://spatial.nhh.no/R/rgdal/? Please do say what you mean. If your browser blocks http: rather than https:, that is your choice. I imagine that it silently inserts an "s".
FWIW I?ve tried it locally using the latest svn revision but have no clue where I should find `proj_api.h` (I?ve looked in the complete proj installation directory).
If PROJ 7.0.1 is properly installed, the header will be in $prefix/include. If your packager put all the headers somewhere else, use the command argument: $ ls /usr/local/include/proj* /usr/local/include/proj_api.h /usr/local/include/proj.h /usr/local/include/proj_constants.h /usr/local/include/proj_symbol_rename.h /usr/local/include/proj_experimental.h /usr/local/include/proj: common.hpp crs.hpp metadata.hpp coordinateoperation.hpp datum.hpp nn.hpp coordinatesystem.hpp io.hpp util.hpp You should not need to know. What does pkg-config proj --cflags say (once you've set PKG_CONFIG_PATH, of course.
I see ``` ls /usr/local/Cellar/proj/7.0.1/lib/ libproj.19.dylib libproj.a libproj.dylib@ libproj.la* pkgconfig/ ```
In your earlier post, you negected to say that you used the necessary configure argument. The error message says that you did not use it. Please report back when you have done so. Futher, I quote from your message: "It further causes complete breakage for people relying on the homebrew package manager on macOS since current versions are gdal v2.2.4 and proj 7.0.1." Please accept that I pay attention to what you write.
Even if this custom installation might work at some point, making such adjustments until gdal3 arrives in homebrew will be a major pain, also for CI builds. It would still be highly appreciated if this internal assertion would be removed again.
Getting things right may be painful; GDAL 2 with PROJ 7 is wrong, because it mixes two incompatible metadata storage systems. Migrating all users to GDAL >= 3 & PROJ >= 6 is a matter of urgency to avoid silent positional error of about 125m, or about 4 satellite image cells. See the vignette here http://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html and this r-spatial blog https://www.r-spatial.org/r/2020/03/17/wkt.html. I am not willing to let silent error creep in because some people find it inconvenient to avoid the problem. Both sf and sp/rgdal are covering your backs even if it feels rough. We've already helped dozens of maintainers of reverse dependencies future-proof their users from the consequences of changes made in PROJ and GDAL, where we have also contributed to making the transition easier by exposing helper functions in their APIs, which we use. I expect you to be able to CMD check a tarball from the commmand line, no more than that, to report back, and to try to grasp that these external software components are really important and completely non-trivial. Roger
On 29 May 2020, at 18:37, Roger Bivand wrote:
I have put a tarball, built without vignettes with GDAL 2.2.4 and PROJ 7.0.1 that passes CMD check (with warnings for missing vignettes) on: http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz It involves code copying to provide duplicated functions in three settings: PROJ < 6 & GDAL < 3 PROJ > = 6 & GDAL >= 3 PROJ > = 6 & GDAL < 3 (your homebrew case) You need the configure argument: --configure-args=--with-proj_api=proj_api.h. Without the argument, you now get directed to use it. For now I've dropped the configure tests for sqlite3, curl and tiff; works for me but may not work for you. Please make the output of: R CMD check --install-args="--configure-args=--with-proj_api=proj_api.h" rgdal_1.5-9.tar.gz available ASAP. I count on an immediate response. Roger On Fri, 29 May 2020, Roger Bivand wrote:
On Fri, 29 May 2020, Patrick Schratz wrote:
The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is present. It does not even install the package if this condition is not met but errors during linking.
Please document with the output of 00install.out from R CMD check rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo --version with system details. Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and the old metadata structure with Proj4 string degradation; GDAL < 3 with PROJ
= 6 must use the deprecated API and the new metadata structure.
I could not find any sources in the web or in the announcement post of v1.5.8 why this condition was enforced so strictly. Does it cause unwanted results?
Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
complete break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with
PROJ > = 6) after GDAL barnraising. So:
PROJ
< 6 >= 6 GDAL < 3 OK NO
> = 3 NO OK
If so, then there is a big issue since the ?gdal2 and proj >= 6? combination has been used by many people in the past already.
No good precedent, just binary packagers protecting slow downstream adaptation.
I haven?t tracked down for how long this situation was on the market
already.
Also the {sf} package is still installable with this combination
and I
am not aware of any warnings/issues that came up due to this so far.
rgdal is a much older package and has much more older code that needs this protection. It is possible to accommodate the no-go areas, but I'd really value patches to R-forge, as I cannot check multiple PROJ/GDAL version combinations just because someone isn't up to speed.
It further causes complete breakage for people relying on the homebrew package manager on macOS since current versions are gdal v2.2.4 and proj 7.0.1.
They should never have gone beyond PROJ 5 if they are stuck on GDAL. PROJ 7 is a very different animal.
The update to gdal3 is blocked since months due to an incompatibility with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading the issue, it looks unclear when this issue will be resolved. The alternative osgeo4mac tap holds formula that is unstable and somewhat broken. One cannot link rgdal with the current gdal v3.0.1 from osgeo4mac. This incompatibility with homebrew-core formula leads to further issues for CI tests that rely on the spatial homebrew stack for macOS builds. All of this is really unfortunate and I am wondering if this was a known factor when introducing this condition in the new release. Unfortunately rgdal is not hosted on any Git* provider (there is just the CRAN mirror with not much information about the recent changes) and it is unclear/untransparent to me if there is a continuous integration setup at all for this package.
Do not require me to leave the excellent SVN service on R-forge. Provide patches there.
I hope this is not another case in the series ?I do not like operating system XY and hence I do not test on it? which was seen recently in another package. That?s what CI is for and the responsibility of package authors.
I run and develop on Fedora, I have no access to OSX, and the current release was fully checked with ~ 900 revdeps repeatedly since November on successive older and newer versions of PROJ and GDAL. I blocked new PROJ with old GDAL recently to simplify development. CI is only as good as the scripts, I am completely sure that repeated revdep testing and reading the output is superior.
Also one should be aware that not every rgdal user is member of this mailing list (I guess not even 5%) and many people in production will face this error during install, most of them without any clue what to do or an idea why this was raised.
There has been plenty of information given about the CRS changes. rgdal could simply have been abandoned by me, and all those production volunteers might have fixed things, but I never hear anything at all.
In summary it would be great if - There would be more detailed information on the introduction of the new condition - Awareness of state-of-the-art platform related library versions before making such a release - Transparency/Existence of a cross-platform build process - If the incompatibility is just due to some features not being accessible with this configuration, I suggest to raise a warning during package load but not prevent the package from installing in the first place - A version-based NEWS file would be available. Currently one only sees a revision-based Changelog: https://cran.r-project.org/web/packages/rgdal/ChangeLog I am well aware that I am indirectly criticising (hopefully in a constructive way) the transparency and release process here, for which I will probably get a rough response. However, if that leads to more robustness and transparency in future release, I am fine with this. A quick patch release resolving the breakage would be highly appreciated.
Only with community imput. what you ask is not needed, just extra complexity. Please provide patches, or accept my invitation to join the R-forge project and commit your fixes directly. I can see how to do it, but I don't think it makes sense, and your messsage has not motivated me, to be honest. I'm prioritizing working with CRAN to iron out reverse dependency problems. Roger
Best, Patrick [[alternative HTML version deleted]]
_______________________________________________ R-sig-Geo mailing list R-sig-Geo at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo
-- Roger Bivand Department of Economics, Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no https://orcid.org/0000-0003-2392-6140 https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
Roger Bivand Department of Economics, Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no https://orcid.org/0000-0003-2392-6140 https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
Hi All: It appears that Anaconda has the requisite versions, as well as gdal-config and the appropriate include files etc etc, though I haven?t tried to compile yet, as I have to be on a different computer. So I have two questions for now: 1. Other than the rgdal source and setting everything to point to the correct locations, is there anything else I need to have installed (I have the compilers). If I can get it to work, this might be for R4.0 only, I don?t keep two versions around, and I don?t have Catalina. 2. I assume there are other packages on the Mac that are in a similar situation, would I be correct in assuming sp and sf? Thanks, -Roy
On May 29, 2020, at 1:44 PM, Roger Bivand <Roger.Bivand at nhh.no> wrote: On Fri, 29 May 2020, Patrick Schratz wrote:
The URL returns a 404.
Do you mean http://spatial.nhh.no/R/rgdal/? <http://spatial.nhh.no/R/rgdal/?> Please do say what you mean. If your browser blocks http: rather than https:, that is your choice. I imagine that it silently inserts an "s".
FWIW I?ve tried it locally using the latest svn revision but have no clue where I should find `proj_api.h` (I?ve looked in the complete proj installation directory).
If PROJ 7.0.1 is properly installed, the header will be in $prefix/include. If your packager put all the headers somewhere else, use the command argument: $ ls /usr/local/include/proj* /usr/local/include/proj_api.h /usr/local/include/proj.h /usr/local/include/proj_constants.h /usr/local/include/proj_symbol_rename.h /usr/local/include/proj_experimental.h /usr/local/include/proj: common.hpp crs.hpp metadata.hpp coordinateoperation.hpp datum.hpp nn.hpp coordinatesystem.hpp io.hpp util.hpp You should not need to know. What does pkg-config proj --cflags say (once you've set PKG_CONFIG_PATH, of course.
I see ``` ls /usr/local/Cellar/proj/7.0.1/lib/ libproj.19.dylib libproj.a libproj.dylib@ libproj.la <http://libproj.la/>* pkgconfig/ ```
In your earlier post, you negected to say that you used the necessary configure argument. The error message says that you did not use it. Please report back when you have done so. Futher, I quote from your message: "It further causes complete breakage for people relying on the homebrew package manager on macOS since current versions are gdal v2.2.4 and proj 7.0.1." Please accept that I pay attention to what you write.
Even if this custom installation might work at some point, making such adjustments until gdal3 arrives in homebrew will be a major pain, also for CI builds. It would still be highly appreciated if this internal assertion would be removed again.
Getting things right may be painful; GDAL 2 with PROJ 7 is wrong, because it mixes two incompatible metadata storage systems. Migrating all users to GDAL >= 3 & PROJ >= 6 is a matter of urgency to avoid silent positional error of about 125m, or about 4 satellite image cells. See the vignette here http://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html <http://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html> and this r-spatial blog https://www.r-spatial.org/r/2020/03/17/wkt.html <https://www.r-spatial.org/r/2020/03/17/wkt.html>. I am not willing to let silent error creep in because some people find it inconvenient to avoid the problem. Both sf and sp/rgdal are covering your backs even if it feels rough. We've already helped dozens of maintainers of reverse dependencies future-proof their users from the consequences of changes made in PROJ and GDAL, where we have also contributed to making the transition easier by exposing helper functions in their APIs, which we use. I expect you to be able to CMD check a tarball from the commmand line, no more than that, to report back, and to try to grasp that these external software components are really important and completely non-trivial. Roger
On 29 May 2020, at 18:37, Roger Bivand wrote:
I have put a tarball, built without vignettes with GDAL 2.2.4 and PROJ 7.0.1 that passes CMD check (with warnings for missing vignettes) on: http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz It involves code copying to provide duplicated functions in three settings: PROJ < 6 & GDAL < 3 PROJ > = 6 & GDAL >= 3 PROJ > = 6 & GDAL < 3 (your homebrew case) You need the configure argument: --configure-args=--with-proj_api=proj_api.h. Without the argument, you now get directed to use it. For now I've dropped the configure tests for sqlite3, curl and tiff; works for me but may not work for you. Please make the output of: R CMD check --install-args="--configure-args=--with-proj_api=proj_api.h" rgdal_1.5-9.tar.gz available ASAP. I count on an immediate response. Roger On Fri, 29 May 2020, Roger Bivand wrote:
On Fri, 29 May 2020, Patrick Schratz wrote:
The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is present. It does not even install the package if this condition is not met but errors during linking.
Please document with the output of 00install.out from R CMD check rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo --version with system details. Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and the old metadata structure with Proj4 string degradation; GDAL < 3 with PROJ
= 6 must use the deprecated API and the new metadata structure. I could not find any sources in the web or in the announcement post of v1.5.8 why this condition was enforced so strictly. Does it cause unwanted results?
Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
complete break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with PROJ > = 6) after GDAL barnraising. So:
PROJ
< 6 >= 6 GDAL < 3 OK NO
> = 3 NO OK
If so, then there is a big issue since the ?gdal2 and proj >= 6? combination has been used by many people in the past already.
No good precedent, just binary packagers protecting slow downstream adaptation.
I haven?t tracked down for how long this situation was on the market
already.
Also the {sf} package is still installable with this combination
and I
am not aware of any warnings/issues that came up due to this so far.
rgdal is a much older package and has much more older code that needs this protection. It is possible to accommodate the no-go areas, but I'd really value patches to R-forge, as I cannot check multiple PROJ/GDAL version combinations just because someone isn't up to speed.
It further causes complete breakage for people relying on the homebrew package manager on macOS since current versions are gdal v2.2.4 and proj 7.0.1.
They should never have gone beyond PROJ 5 if they are stuck on GDAL. PROJ 7 is a very different animal.
The update to gdal3 is blocked since months due to an incompatibility with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading the issue, it looks unclear when this issue will be resolved. The alternative osgeo4mac tap holds formula that is unstable and somewhat broken. One cannot link rgdal with the current gdal v3.0.1 from osgeo4mac. This incompatibility with homebrew-core formula leads to further issues for CI tests that rely on the spatial homebrew stack for macOS builds. All of this is really unfortunate and I am wondering if this was a known factor when introducing this condition in the new release. Unfortunately rgdal is not hosted on any Git* provider (there is just the CRAN mirror with not much information about the recent changes) and it is unclear/untransparent to me if there is a continuous integration setup at all for this package.
Do not require me to leave the excellent SVN service on R-forge. Provide patches there.
I hope this is not another case in the series ?I do not like operating system XY and hence I do not test on it? which was seen recently in another package. That?s what CI is for and the responsibility of package authors.
I run and develop on Fedora, I have no access to OSX, and the current release was fully checked with ~ 900 revdeps repeatedly since November on successive older and newer versions of PROJ and GDAL. I blocked new PROJ with old GDAL recently to simplify development. CI is only as good as the scripts, I am completely sure that repeated revdep testing and reading the output is superior.
Also one should be aware that not every rgdal user is member of this mailing list (I guess not even 5%) and many people in production will face this error during install, most of them without any clue what to do or an idea why this was raised.
There has been plenty of information given about the CRS changes. rgdal could simply have been abandoned by me, and all those production volunteers might have fixed things, but I never hear anything at all.
In summary it would be great if - There would be more detailed information on the introduction of the new condition - Awareness of state-of-the-art platform related library versions before making such a release - Transparency/Existence of a cross-platform build process - If the incompatibility is just due to some features not being accessible with this configuration, I suggest to raise a warning during package load but not prevent the package from installing in the first place - A version-based NEWS file would be available. Currently one only sees a revision-based Changelog: https://cran.r-project.org/web/packages/rgdal/ChangeLog I am well aware that I am indirectly criticising (hopefully in a constructive way) the transparency and release process here, for which I will probably get a rough response. However, if that leads to more robustness and transparency in future release, I am fine with this. A quick patch release resolving the breakage would be highly appreciated.
Only with community imput. what you ask is not needed, just extra complexity. Please provide patches, or accept my invitation to join the R-forge project and commit your fixes directly. I can see how to do it, but I don't think it makes sense, and your messsage has not motivated me, to be honest. I'm prioritizing working with CRAN to iron out reverse dependency problems. Roger
Best, Patrick [[alternative HTML version deleted]]
_______________________________________________ R-sig-Geo mailing list R-sig-Geo at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo
-- Roger Bivand Department of Economics, Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no https://orcid.org/0000-0003-2392-6140 https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
-- Roger Bivand Department of Economics, Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no <mailto:Roger.Bivand at nhh.no> https://orcid.org/0000-0003-2392-6140 <https://orcid.org/0000-0003-2392-6140> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________ <https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________> R-sig-Geo mailing list R-sig-Geo at r-project.org <mailto:R-sig-Geo at r-project.org>
rgdal-1.5-9 fixed all my install problems on ubuntu 20.04. My mother has a needle point pillow at her summer home which declares "Guests of Guests May Not Bring Guests". Analogously, 'dependencies of dependencies will not check dependencies' is applicable here. rgdal-1.5-9 facilitated:
library(rgdal)
Loading required package: sp
rgdal: version: 1.5-9, (SVN revision 991M)
Geospatial Data Abstraction Library extensions to R successfully loaded
Loaded GDAL runtime: GDAL 3.1.0, released 2020/05/03
Path to GDAL shared files: /usr/local/share/gdal
GDAL binary built with GEOS: TRUE
Loaded PROJ runtime: Rel. 7.0.1, May 1st, 2020, [PJ_VERSION: 701]
Path to PROJ shared files:
/root/.local/share/proj:/usr/local/share/proj:/usr/local/share/proj
PROJ CDN enabled:FALSE
Linking to sp version:1.4-1
To mute warnings of possible GDAL/OSR exportToProj4() degradation,
use options("rgdal_show_exportToProj4_warnings"="none") before loading
rgdal.
GEOS version is 3.9.0dev
I still have some work to do on the PostgreSQL end of things, but I am back
to working
spatial. I still received errors unless I started the Install session in
sudo R, which leads me
to think that my /usr/local/(bin,lib,include) are symlinks from
one/some/all installs of GEOS,
PROJ, GDAL, which are not the fault of R developers, but I'd forgotten this
issue and rediscovered
that I pestered Roger and Edzer about it in 2019. So, sudo R.
I'm back to having a working spatial environment. Thank you Roger!
Chris
On Fri, May 29, 2020 at 5:16 PM Roy Mendelssohn - NOAA Federal via
R-sig-Geo <r-sig-geo at r-project.org> wrote:
Hi All: It appears that Anaconda has the requisite versions, as well as gdal-config and the appropriate include files etc etc, though I haven?t tried to compile yet, as I have to be on a different computer. So I have two questions for now: 1. Other than the rgdal source and setting everything to point to the correct locations, is there anything else I need to have installed (I have the compilers). If I can get it to work, this might be for R4.0 only, I don?t keep two versions around, and I don?t have Catalina. 2. I assume there are other packages on the Mac that are in a similar situation, would I be correct in assuming sp and sf? Thanks, -Roy
On May 29, 2020, at 1:44 PM, Roger Bivand <Roger.Bivand at nhh.no> wrote: On Fri, 29 May 2020, Patrick Schratz wrote:
The URL returns a 404.
Do you mean http://spatial.nhh.no/R/rgdal/? <
http://spatial.nhh.no/R/rgdal/?> Please do say what you mean. If your browser blocks http: rather than https:, that is your choice. I imagine that it silently inserts an "s".
FWIW I?ve tried it locally using the latest svn revision but have no
clue where I should find `proj_api.h` (I?ve looked in the complete proj installation directory).
If PROJ 7.0.1 is properly installed, the header will be in
$prefix/include. If your packager put all the headers somewhere else, use the command argument:
$ ls /usr/local/include/proj* /usr/local/include/proj_api.h /usr/local/include/proj.h /usr/local/include/proj_constants.h
/usr/local/include/proj_symbol_rename.h
/usr/local/include/proj_experimental.h /usr/local/include/proj: common.hpp crs.hpp metadata.hpp coordinateoperation.hpp datum.hpp nn.hpp coordinatesystem.hpp io.hpp util.hpp You should not need to know. What does pkg-config proj --cflags say
(once you've set PKG_CONFIG_PATH, of course.
I see ``` ls /usr/local/Cellar/proj/7.0.1/lib/ libproj.19.dylib libproj.a libproj.dylib@ libproj.la <
http://libproj.la/>* pkgconfig/
```
In your earlier post, you negected to say that you used the necessary
configure argument. The error message says that you did not use it. Please report back when you have done so. Futher, I quote from your message: "It further causes complete breakage for people relying on the homebrew package manager on macOS since current versions are gdal v2.2.4 and proj 7.0.1." Please accept that I pay attention to what you write.
Even if this custom installation might work at some point, making such
adjustments until gdal3 arrives in homebrew will be a major pain, also for CI builds.
It would still be highly appreciated if this internal assertion would
be removed again.
Getting things right may be painful; GDAL 2 with PROJ 7 is wrong,
because it mixes two incompatible metadata storage systems. Migrating all users to
GDAL >= 3 & PROJ >= 6 is a matter of urgency to avoid silent positional
error of about 125m, or about 4 satellite image cells. See the vignette here http://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html < http://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html> and this r-spatial blog https://www.r-spatial.org/r/2020/03/17/wkt.html < https://www.r-spatial.org/r/2020/03/17/wkt.html>.
I am not willing to let silent error creep in because some people find
it inconvenient to avoid the problem. Both sf and sp/rgdal are covering your backs even if it feels rough. We've already helped dozens of maintainers of reverse dependencies future-proof their users from the consequences of changes made in PROJ and GDAL, where we have also contributed to making the transition easier by exposing helper functions in their APIs, which we use.
I expect you to be able to CMD check a tarball from the commmand line,
no more than that, to report back, and to try to grasp that these external software components are really important and completely non-trivial.
Roger
On 29 May 2020, at 18:37, Roger Bivand wrote:
I have put a tarball, built without vignettes with GDAL 2.2.4 and PROJ 7.0.1 that passes CMD check (with warnings for missing vignettes) on: http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz It involves code copying to provide duplicated functions in three settings: PROJ < 6 & GDAL < 3 PROJ > = 6 & GDAL >= 3 PROJ > = 6 & GDAL < 3 (your homebrew case) You need the configure argument: --configure-args=--with-proj_api=proj_api.h. Without the argument, you now get directed to use it. For now I've dropped the configure tests for sqlite3, curl and tiff;
works
for me but may not work for you. Please make the output of: R CMD check
--install-args="--configure-args=--with-proj_api=proj_api.h"
rgdal_1.5-9.tar.gz available ASAP. I count on an immediate response. Roger On Fri, 29 May 2020, Roger Bivand wrote:
On Fri, 29 May 2020, Patrick Schratz wrote:
The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is present. It does not even install the package if this condition is not met but errors during linking.
Please document with the output of 00install.out from R CMD check rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo
--version
with system details. Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and
the
old metadata structure with Proj4 string degradation; GDAL < 3 with
PROJ
= 6 must use the deprecated API and the new metadata structure. I could not find any sources in the web or in the announcement post
of
v1.5.8 why this condition was enforced so strictly. Does it cause unwanted results?
Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the complete break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with
PROJ > = 6) after GDAL barnraising. So:
PROJ
< 6 >= 6 GDAL < 3 OK NO
> = 3 NO OK
If so, then there is a big issue since the ?gdal2 and proj >= 6? combination has been used by many people in the past already.
No good precedent, just binary packagers protecting slow downstream adaptation.
I haven?t tracked down for how long this situation was on the market
already.
Also the {sf} package is still installable with this combination
and I
am not aware of any warnings/issues that came up due to this so far.
rgdal is a much older package and has much more older code that needs this protection. It is possible to accommodate the no-go areas, but
I'd
really value patches to R-forge, as I cannot check multiple PROJ/GDAL version combinations just because someone isn't up to speed.
It further causes complete breakage for people relying on the
homebrew
package manager on macOS since current versions are gdal v2.2.4 and
proj
7.0.1.
They should never have gone beyond PROJ 5 if they are stuck on GDAL.
PROJ
7 is a very different animal.
The update to gdal3 is blocked since months due to an incompatibility with `liblas` (https://github.com/libLAS/libLAS/issues/164).
Reading the
issue, it looks unclear when this issue will be resolved. The alternative osgeo4mac tap holds formula that is unstable and somewhat broken. One cannot link rgdal with the current gdal v3.0.1 from osgeo4mac. This incompatibility with homebrew-core formula leads to further
issues
for CI tests that rely on the spatial homebrew stack for macOS
builds.
All of this is really unfortunate and I am wondering if this was a
known
factor when introducing this condition in the new release. Unfortunately rgdal is not hosted on any Git* provider (there is just the CRAN mirror with not much information about the recent changes)
and
it is unclear/untransparent to me if there is a continuous
integration
setup at all for this package.
Do not require me to leave the excellent SVN service on R-forge.
Provide
patches there.
I hope this is not another case in the series ?I do not like
operating
system XY and hence I do not test on it? which was seen recently in another package. That?s what CI is for and the responsibility of package authors.
I run and develop on Fedora, I have no access to OSX, and the current release was fully checked with ~ 900 revdeps repeatedly since
November on
successive older and newer versions of PROJ and GDAL. I blocked new
PROJ
with old GDAL recently to simplify development. CI is only as good as
the
scripts, I am completely sure that repeated revdep testing and reading the output is superior.
Also one should be aware that not every rgdal user is member of this mailing list (I guess not even 5%) and many people in production will face this error during install, most of them without any clue what
to do
or an idea why this was raised.
There has been plenty of information given about the CRS changes.
rgdal
could simply have been abandoned by me, and all those production volunteers might have fixed things, but I never hear anything at all.
In summary it would be great if - There would be more detailed information on the introduction of the new condition - Awareness of state-of-the-art platform related library versions before making such a release - Transparency/Existence of a cross-platform build process - If the incompatibility is just due to some features not being accessible with this configuration, I suggest to raise a warning during package load but not prevent the package from installing in the first place - A version-based NEWS file would be available. Currently one only sees a revision-based Changelog: https://cran.r-project.org/web/packages/rgdal/ChangeLog I am well aware that I am indirectly criticising (hopefully in a constructive way) the transparency and release process here, for which I will probably get a rough response. However, if that leads to more robustness and transparency in future release, I am fine with this. A quick patch release resolving the breakage would be highly appreciated.
Only with community imput. what you ask is not needed, just extra complexity. Please provide patches, or accept my invitation to join
the
R-forge project and commit your fixes directly. I can see how to do
it,
but I don't think it makes sense, and your messsage has not motivated
me,
to be honest. I'm prioritizing working with CRAN to iron out reverse dependency problems. Roger
Best, Patrick [[alternative HTML version deleted]]
_______________________________________________ R-sig-Geo mailing list R-sig-Geo at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo
-- Roger Bivand Department of Economics, Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no https://orcid.org/0000-0003-2392-6140 https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
-- Roger Bivand Department of Economics, Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no <mailto:
Roger.Bivand at nhh.no>
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________ < https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________
R-sig-Geo mailing list R-sig-Geo at r-project.org <mailto:R-sig-Geo at r-project.org>
https://stat.ethz.ch/mailman/listinfo/r-sig-geo> [[alternative HTML version deleted]] _______________________________________________ R-sig-Geo mailing list R-sig-Geo at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo
1 day later
Fresh version of rgdal rev 995 on link and R-forge, with more tweaks to the configure file. Please test. Roger
On Fri, 29 May 2020, Roger Bivand wrote:
I have put a tarball, built without vignettes with GDAL 2.2.4 and PROJ 7.0.1 that passes CMD check (with warnings for missing vignettes) on: http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz It involves code copying to provide duplicated functions in three settings: PROJ < 6 & GDAL < 3 PROJ > = 6 & GDAL >= 3 PROJ > = 6 & GDAL < 3 (your homebrew case) You need the configure argument: --configure-args=--with-proj_api=proj_api.h. Without the argument, you now get directed to use it. For now I've dropped the configure tests for sqlite3, curl and tiff; works for me but may not work for you. Please make the output of: R CMD check --install-args="--configure-args=--with-proj_api=proj_api.h" rgdal_1.5-9.tar.gz available ASAP. I count on an immediate response. Roger On Fri, 29 May 2020, Roger Bivand wrote:
On Fri, 29 May 2020, Patrick Schratz wrote:
The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is present. It does not even install the package if this condition is not met but errors during linking.
Please document with the output of 00install.out from R CMD check rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo --version with system details. Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and the old metadata structure with Proj4 string degradation; GDAL < 3 with PROJ
= 6 must use the deprecated API and the new metadata structure.
I could not find any sources in the web or in the announcement post of v1.5.8 why this condition was enforced so strictly. Does it cause unwanted results?
Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the complete
break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with PROJ >= 6) after
GDAL barnraising. So:
PROJ
< 6 >= 6
GDAL < 3 OK NO
> = 3 NO OK
If so, then there is a big issue since the ?gdal2 and proj >= 6? combination has been used by many people in the past already.
No good precedent, just binary packagers protecting slow downstream adaptation.
I haven?t tracked down for how long this situation was on the market
already.
Also the {sf} package is still installable with this combination and I
am not aware of any warnings/issues that came up due to this so far.
rgdal is a much older package and has much more older code that needs this protection. It is possible to accommodate the no-go areas, but I'd really value patches to R-forge, as I cannot check multiple PROJ/GDAL version combinations just because someone isn't up to speed.
It further causes complete breakage for people relying on the homebrew package manager on macOS since current versions are gdal v2.2.4 and proj 7.0.1.
They should never have gone beyond PROJ 5 if they are stuck on GDAL. PROJ 7 is a very different animal.
The update to gdal3 is blocked since months due to an incompatibility with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading the issue, it looks unclear when this issue will be resolved. The alternative osgeo4mac tap holds formula that is unstable and somewhat broken. One cannot link rgdal with the current gdal v3.0.1 from osgeo4mac. This incompatibility with homebrew-core formula leads to further issues for CI tests that rely on the spatial homebrew stack for macOS builds. All of this is really unfortunate and I am wondering if this was a known factor when introducing this condition in the new release. Unfortunately rgdal is not hosted on any Git* provider (there is just the CRAN mirror with not much information about the recent changes) and it is unclear/untransparent to me if there is a continuous integration setup at all for this package.
Do not require me to leave the excellent SVN service on R-forge. Provide patches there.
I hope this is not another case in the series ?I do not like operating system XY and hence I do not test on it? which was seen recently in another package. That?s what CI is for and the responsibility of package authors.
I run and develop on Fedora, I have no access to OSX, and the current release was fully checked with ~ 900 revdeps repeatedly since November on successive older and newer versions of PROJ and GDAL. I blocked new PROJ with old GDAL recently to simplify development. CI is only as good as the scripts, I am completely sure that repeated revdep testing and reading the output is superior.
Also one should be aware that not every rgdal user is member of this mailing list (I guess not even 5%) and many people in production will face this error during install, most of them without any clue what to do or an idea why this was raised.
There has been plenty of information given about the CRS changes. rgdal could simply have been abandoned by me, and all those production volunteers might have fixed things, but I never hear anything at all.
In summary it would be great if - There would be more detailed information on the introduction of the new condition - Awareness of state-of-the-art platform related library versions before making such a release - Transparency/Existence of a cross-platform build process - If the incompatibility is just due to some features not being accessible with this configuration, I suggest to raise a warning during package load but not prevent the package from installing in the first place - A version-based NEWS file would be available. Currently one only sees a revision-based Changelog: https://cran.r-project.org/web/packages/rgdal/ChangeLog I am well aware that I am indirectly criticising (hopefully in a constructive way) the transparency and release process here, for which I will probably get a rough response. However, if that leads to more robustness and transparency in future release, I am fine with this. A quick patch release resolving the breakage would be highly appreciated.
Only with community imput. what you ask is not needed, just extra complexity. Please provide patches, or accept my invitation to join the R-forge project and commit your fixes directly. I can see how to do it, but I don't think it makes sense, and your messsage has not motivated me, to be honest. I'm prioritizing working with CRAN to iron out reverse dependency problems. Roger
Best, Patrick [[alternative HTML version deleted]]
_______________________________________________ R-sig-Geo mailing list R-sig-Geo at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Roger Bivand Department of Economics, Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no https://orcid.org/0000-0003-2392-6140 https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en