Skip to content

gdalUtils 0.2.0 now on CRAN

11 messages · Tim Keitt, Jonathan Greenberg, Rainer M Krug +1 more

#
I would like to announce a new R package "gdalUtils", now on CRAN.
gdalUtils is a set of R wrappers for most of the GDAL utility programs
(http://gdal.org/gdal_utilities.html).  gdalUtils is a collaboration
between Matteo Mattiuzzi and me.

gdalUtils requires an already-installed GDAL on your system:
http://trac.osgeo.org/gdal/wiki/DownloadingGdalBinaries

For Windows, I recommend installing QGIS Standalone
(http://www.qgis.org/en/site/forusers/download.html) which appears to
have the most up-to-date binaries of GDAL for the Windows operating
system including support for HDF4/5 files.

Note that this is NOT a replacement for Roger Bivand's fantastic rgdal
package, it is a complementary package that simply provides R-wrappers
for functions like gdalwarp, gdal_translate, ogr2ogr, etc.  We've
tried to make the interface more R-like in terms of input parameters,
and have provided some additional features such as (when relevant)
returning outputs in Robert Hijman's raster format.  The parameter
naming and the documentation follows GDAL, with permission from Frank
Warmerdam (lead GDAL developer).

We have also provided some value-added functions such as
batch_gdal_translate and get_subdatasets (for extracting subdataset
names from HDF4/5 and NetCDF files).

I will note the primary motivation for developing this package was
initially to provide (finally) HDF4/5 file support to Windows users
(for Landsat and MODIS processing, among other things).  However, many
of the GDAL functions provide additional capabilities that R functions
do not currently have, and they are quite a bit faster than most of
their R equivalents (compare projectRaster_rigorous in my
spatial.tools package to gdalwarp(...,method="mode") for an example).

When you first fire it up, it may take a bit to search for a valid
GDAL install on your computer.  If you have more than one (Windows
users may find this), it will use the latest version.  We tried to
make this system-agnostic, but let us know if you have any problems
getting the functions to work.  The best way to test is:
library(gdalUtils)
gdalinfo(version=TRUE)
gdalinfo(formats=TRUE)

Cheers!

--j
#
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 01/08/14, 23:42 , Jonathan Greenberg wrote:
Very nice - haven't tried any working examples, but it installs on a
mac without problems and finds the gdal installation installed via
homebrew.

But I have some questions:

The automatic search is nice - but I unlinked gdal via homebrew, i.e.
the links to the binaries and libraries are not in the path anymore,
and I could not load gdalUtils anymore, as the gdalUtils did not find
the libraries anymore (understandable). But it seams, that gdalUtils
did not search for gdal, which is installed as a Framework as well.
Now I removed gdalUtils again and installed it again, with gdal still
unlinked, but it did not install as it did not find the gdal
libraries, despite gdal being available in a framework (see
http://www.kyngchaos.com/software/frameworks for the ones installed -
they are quite popular, and required, among GRASS and QGIS users on Mac).

Question 1:

Would it be possible, to include the gdal Frameworks in the search path?

At the end is a layout of the directory structure of the gdal frameworks.

Question 2:

Is it (or would it) be possible to manually set the installation of
gdal to be used? This would make comparison of versions of gdal as
well as reproducible research much easier.

Question 3:

I can't test it right now, but I assume that gdalUtils does search for
a new gdal installation if it can't find the one used before? Is there
a way of initiating the search (and selection) if a newer version has
been installed?

Thanks for a very neat package,

Rainer

Directory structure of the GDAL.Framework on a MAC:

/Library/Frameworks/GDAL.framework/
??? Headers -> Versions/Current/Headers
??? Programs -> Versions/Current/Programs
??? Resources -> Versions/Current/Resources
??? Versions
?   ??? 1.10
?   ?   ??? Headers
?   ?   ??? Libraries
?   ?   ?   ??? ogdi
?   ?   ??? PlugIns
?   ?   ??? Programs
?   ?   ??? Python
?   ?   ?   ??? 2.6
?   ?   ?   ?   ??? site-packages
?   ?   ?   ?       ??? osgeo
?   ?   ?   ??? 2.7
?   ?   ?       ??? site-packages
?   ?   ?           ??? osgeo
?   ?   ??? Resources
?   ?   ?   ??? doc
?   ?   ?   ?   ??? gdal
?   ?   ?   ?       ??? java
?   ?   ?   ?       ?   ??? org
?   ?   ?   ?       ?   ?   ??? gdal
?   ?   ?   ?       ?   ?       ??? gdal
?   ?   ?   ?       ?   ?       ??? gdalconst
?   ?   ?   ?       ?   ?       ??? ogr
?   ?   ?   ?       ?   ?       ??? osr
?   ?   ?   ?       ?   ??? resources
?   ?   ?   ?       ??? ogr
?   ?   ?   ??? gdal
?   ?   ??? unix
?   ?       ??? bin
?   ?       ??? include -> ../Headers
?   ?       ??? lib
?   ??? 1.9
?   ?   ??? Headers
?   ?   ??? Libraries
?   ?   ?   ??? ogdi
?   ?   ??? PlugIns
?   ?   ??? Programs
?   ?   ??? Python
?   ?   ?   ??? 2.6
?   ?   ?   ?   ??? site-packages
?   ?   ?   ?       ??? osgeo
?   ?   ?   ??? 2.7
?   ?   ?       ??? site-packages
?   ?   ?           ??? osgeo
?   ?   ??? Resources
?   ?   ?   ??? doc
?   ?   ?   ?   ??? gdal
?   ?   ?   ?       ??? java
?   ?   ?   ?       ?   ??? org
?   ?   ?   ?       ?   ?   ??? gdal
?   ?   ?   ?       ?   ?       ??? gdal
?   ?   ?   ?       ?   ?       ??? gdalconst
?   ?   ?   ?       ?   ?       ??? ogr
?   ?   ?   ?       ?   ?       ??? osr
?   ?   ?   ?       ?   ??? resources
?   ?   ?   ?       ??? ogr
?   ?   ?   ??? gdal
?   ?   ??? unix
?   ?       ??? bin
?   ?       ??? include -> ../Headers
?   ?       ??? lib
?   ??? Current -> 1.10
??? unix -> Versions/Current/unix
- -- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer at krugs.de

Skype:      RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSzl2YAAoJENvXNx4PUvmCp6sH/3KD9/vKJlBms+OOuRbD2MjS
6PmSXT4e0J2MpUDQbm3xBDMFNPa7EtlWMeYHvAQHdKfRF87/n6ZTcW0wIxIhmIcC
/3q0lx/yipjXScR2EerMyX3B1bVCRNHFgi5BjNnDUrX6lfNm4DD8HqV6Rn5AxJZE
NkSZ+mQPJ7UgqV9awNtd2QlK+qyjq7TeFrprNrKuoXZ8ALJjZSXn70WDh7jARvP6
zD/rjHFB83Kk61+A0TRBeWfXWtVy1CJZCR0cCXoofBNjT8kYu2EUsWtYqj6EmKBX
1ThW6/yTst9t1cs+j6kDhBG3c+rJ14mx5nkOUXRzCk6wfu/bdGMGoVbEPjW4PII=
=kSdZ
-----END PGP SIGNATURE-----
#
Rainer:

Responses below!
On Thu, Jan 9, 2014 at 2:28 AM, Rainer M Krug <Rainer at krugs.de> wrote:
Quick question: did you try restarting R AFTER you unlinked the
homebrew version?  Here's why I ask: the first time you run ANY
gdalUtils in a session, what it does is spiders your system for
working GDAL installations (in fact, it looks for the frameworks
first).  After this first time, it won't re-scan the drive unless you
do one of two things: 1) restart R and re-load GDALUtils, or 2) run
gdal_setInstallation(rescan=TRUE)
These are the common locations it searches for, before it attempts a
brute-force search of your whole drive:

if (.Platform$OS=="unix")
{
common_locations <- c(
# UNIX systems
"/usr/bin",
"/usr/local/bin",
# Mac
# Kyngchaos frameworks:
"/Library/Frameworks/GDAL.framework/Programs",
# MacPorts:
"/opt/local/bin"
)
}
if (.Platform$OS=="windows")
{
common_locations <- c(
"C:\\Program Files",
"C:\\Program Files (x86)",
"C:\\OSGeo4W"
)
}

I use those frameworks, and the function worked for me, but let me
know if it failed to find yours (perhaps I'll strip the /Programs from
the search path?)  It is easy for me to add new search locations, so
if there are other common locations for ANY OS just let me know.
Yes, we can add in this functionality at a future date.  Right now,
you can check to see what your installs are by:
gdal_setInstallation(rescan=TRUE)
getOption("gdalUtils_gdalPath")

In general, gdalUtils will use the first element of the
getOption("gdalUtils_gdalPath"), which is chosen by the most recent
version (by date).
Yep, as I said either restart R or run:
gdal_setInstallation(rescan=TRUE)

  
    
#
Tim: thanks! We have something similar, where the package uses a
Sys.which() to see if gdalinfo is available in the PATH as one of its
attempts to find a valid install.

Rainer et al: I pushed version 0.2.3 to R-forge (you'll need to SVN it
until R-forge builds the new package later today/tomorrow) which now
supports a fixed search_path parameter.  To use this:

gdal_setInstallation(search_path="/pathto/your/favorite/GDAL/",rescan=TRUE)

If it finds a valid gdalinfo in that path, it will use that one (thus
forcing gdalUtils to use a specific install).  If it doesn't find it,
it will search as usual.  Note that I updated the help for
gdal_setInstallation to better explain how gdalUtils searches for a
valid install.

Here's the R-forge site:
https://r-forge.r-project.org/projects/gdalutils/

--j
On Thu, Jan 9, 2014 at 12:10 PM, Tim Keitt <tkeitt at utexas.edu> wrote:

  
    
#
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 01/09/14, 18:46 , Jonathan Greenberg wrote:
Hi
Yes, I restarted R afterwards - see transcript below:

####################################
Rainers-MacBook-Pro-2:~ rainerkrug$ brew unlink gdal
Unlinking /usr/local/Cellar/gdal/1.10.1... 163 links removed
Rainers-MacBook-Pro-2:~ rainerkrug$ R

R version 3.0.2 Patched (2014-01-07 r64692) -- "Frisbee Sailing"
Copyright (C) 2014 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin10.8.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

[Previously saved workspace restored]
Error in dyn.load(file, DLLpath = DLLpath, ...) :
  unable to load shared object
'/Users/rainerkrug/Library/R/3.0/library/rgdal/libs/rgdal.so':
  dlopen(/Users/rainerkrug/Library/R/3.0/library/rgdal/libs/rgdal.so,
6): Library not loaded: /usr/local/lib/libgdal.1.dylib
  Referenced from:
/Users/rainerkrug/Library/R/3.0/library/rgdal/libs/rgdal.so
  Reason: image not found
Error: package or namespace load failed for ?gdalUtils?
####################################

and I obviously can not call the command, as the package is not loaded.

But I just realized what the problem is: the importing on rgdal. If I
unlink, rgdal does not load anymore as it is compiled using the
homebrew gdal version, and consequently gdalUtils does not load
anymore. So the problem is not directly with gdalUtils. But this
raises an important question: what happens if rgdal and gdalUtils are
using different gdal versions? I think this could call for
inconsistencies.

As gdalUtils depends on rgdal, I would actually suggest to use the
same gdal installation as rgdal, if this is possible.
As mentioned above, the problem is the importing of rgdal.
As mentioned above, due to the importing of rgdal, I would suggest to
use the same version as rgdal, if possible, to keep it consistent and
reproducible.
OK.

Hope my comments are useful,

Rainer
- -- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer at krugs.de

Skype:      RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSzwGOAAoJENvXNx4PUvmCNcUH/RHSZJgaIbMsipZUD2dCL+Fx
dTquYOrt7v1w1JZ+nrVNcU65o9Dp91GmGQ2w61EF9l+59y3e91UynFu94/OLEZIb
37aZvss28lsRW1MokJZRBWor8z3EQNY2kq8c4GOZoMBIRTT+LK9Mvv/EQXcntP0P
LXyTb3fXdXE/GePOteiG2vyCzu36maENbfbY5nqVIiu+qO5H60HcFmJR7FcX1Ty/
Q8BQ4vMkJQm6tJ2ozUJNHccgb0klNqUEp1Kwmiufen1m4jN3Qv9v1epAFLz6KArx
7j5O4pGLzdQXbnEjLSCwS1GL8P6+jWXzbiZA1KX+FI2xypu762b5tKXcw5eVw6w=
=tCRS
-----END PGP SIGNATURE-----
#
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 01/09/14, 19:43 , Jonathan Greenberg wrote:
This sounds very useful. But to come back to the importing of rgdal,
which was causing my problem: As gdalUtils is quite useful without
rgdal, would it be possible to move rgdal to Enhances and to disable
the functions which require rgdal when rgdal is not installed?
And if rgdal is installed, to use the same gdal installation (I don't
know if this is the case already)?

Just a ciosmetic suggestion: it would be nice if, when loadu=ing the
package via library(gdalUtils), that the gdal version and path could
be printed.

Thanks,

Rainer
- -- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer at krugs.de

Skype:      RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSzwNMAAoJENvXNx4PUvmCjHAIAJQG7cMz0BgzTQEYdgMl3/xi
1TH+8rgLFREc/J5DTm24ohpS+SwMDFVVsDeBoPYWX+RWyImN54OuozW4dnAO3Z7t
8xt15eZaBi/u8PncYlqht9END7nBi1RmIznb5VIMdkqCtT1FfhycsUuSRqTeJaX8
0ocaDFIkJTY76UJQ2w9TYVMLuYByNMbr8Y5B6PDCqHiNG0JTT46CfKReP3JlVbPd
gEnlSA4FD3h8ACBfJUP7/pSkNj+AzmufBqZPczIJSHYL8dxC03FvAn2FTQcYwL2M
p376qaThGvgOjN1QrebNffzPm96nyh4DYZPT2lCFeqYwFP0+JANGreEaNzO+flA=
=Jeqi
-----END PGP SIGNATURE-----
#
Ok, pushed a new version (0.2.4) to R-forge following these
suggestions (rgdal and raster are now moved to "Suggests", and the
examples should hopefully be updated to support it).

Rainer: would you mind testing this out?  Try it with and without
outputRaster=TRUE (it should fail on outputRaster=TRUE if you don't
have rgdal installed).

--j
On Thu, Jan 9, 2014 at 2:15 PM, Rainer M Krug <Rainer at krugs.de> wrote:

  
    
#
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 01/09/14, 22:13 , Jonathan Greenberg wrote:
OK - It installs and loads with disfunctional rgdal (I assume the
worst case scenario).

I then linked gdal again, and it is using the homebrew version (which
is good).

But I get the following:

###################################
List of 2
 $ :List of 5
  ..$ path            : chr "/usr/local/bin/"
  ..$ version         :List of 1
  .. ..$ version: chr "1.10.1"
  ..$ date            :List of 1
  .. ..$ date: chr "2013-08-26"
.
.SNIP
.
$ :List of 5
  ..$ path            : chr
"/Library/Frameworks/GDAL.framework/Versions/1.10/Programs/"
  ..$ version         :List of 1
  .. ..$ version: chr "1.10.1"
  ..$ date            :List of 1
  .. ..$ date: chr "2013-08-26"
.
.SNIP
.
###################################

but I have 1.9 installed as well - is it on purpose that gdalUtils
does not find this version as it is older?
Haven't worked with gdalUtils yet - could you tell me which command I
should try with the outputRaster argument?

Cheers,

Rainer
If it finds a valid gdalinfo in that path, it will use that one
- -- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer at krugs.de

Skype:      RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSz7RqAAoJENvXNx4PUvmCXVgH/0/sTGd62EFLyNV+3P0p8WuJ
nZJQrIOJtQqq8B5Ds7gXY5yvaWxbXqEPgqe1a7sldybrcux7c7srTbslV1aX0rAg
oZHaNvBYwdPfZZFM9JQfZn9XSa2kGYo9VMkLCcjCBo/Bdn4V3ITeoUpuuRqLpsZ5
bl85aR3mOCeKXpQII+1kqWZGgOg6BfnmGWriKy96U2Mi4DwHSpD2qXqr5wkevItZ
ADb8sTJvfmrsUHrzJyOMU2qO1G779Zy4/QrbT/9bQe5fJotw0Rv/pm0091ESxncE
sJZcis1HLauD+i1V74aZmDG8lDGisVaDOqkTtdzgjOUAIeKs7zEcd46UoAmzpOs=
=lofs
-----END PGP SIGNATURE-----
5 days later
#
Hi Thiago:

In general, this should have a small overhead compared to running GDAL
purely command line, but it really is just the GDAL binaries you are
running anyway (the wrapper calls those binaries directly).  The
overhead comes from the initial search/validation of the GDAL install
(a once-per-session occurrence).

Cheers!

--j

On Wed, Jan 15, 2014 at 3:22 PM, Thiago V. dos Santos
<thi_veloso at yahoo.com.br> wrote: