MODIS package's runGdal() returns error: dataFormat='GTiff', format not supported
Thanks for all your help Chris, the original error has been corrected, and now there is another error when trying to run runGdal(). I removed the previous version of MODIS and installed version 0.10-34 (develop branch) as per Florian's suggestion. Everything runs fine until it gets to accessing the server and downloading the file. The same happens to my colleague who is running a 64-bit Ubuntu 14.04 machine with R 3.3.0. The issue seems to stem from the fact that in the getStruc.R file FtpDayDirs[1]==FALSE. In fact FtpDayDirs[1] is NULL, which in turn is due to the fact that getStruc.R failed to connect to e4ftl01.cr.usgs.gov (port 80: Timed out). Is anyone else having this issue? I tried accessing http://e4ftl01.cr.usgs.gov/ on my browser and it does seems to be down. Running getStruc.R and selecting LAADS instead of LPDAAC seems to work, but I don't know how to incorporate this into the runGdal() command. Cheers, Hakim Abdi R version 3.2.5 (2016-04-14) -- "Very, Very Secure Dishes" Copyright (C) 2016 The R Foundation for Statistical Computing Platform: x86_64-w64-mingw32/x64 (64-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under certain conditions. Type 'license()' or 'licence()' for distribution details. R is a collaborative project with many contributors. Type 'contributors()' for more information and 'citation()' on how to cite R or R packages in publications. Type 'demo()' for some demos, 'help()' for on-line help, or 'help.start()' for an HTML browser interface to help. Type 'q()' to quit R.
remove.packages("MODIS")
Removing package from ?C:/Program Files/R/R-3.2.5/library? (as ?lib? is unspecified)
install.packages("C:/Users/Hakim/Downloads/MODIS-develop", repos = NULL,
type = "source") * installing *source* package 'MODIS' ... ** R ** inst ** byte-compile and prepare package for lazy loading ** help *** installing help indices ** building package indices ** testing if installed package can be loaded * DONE (MODIS)
require(MODIS)
Loading required package: MODIS Loading required package: raster Loading required package: sp MODIS_manual: https://ivfl-rio.boku.ac.at/owncloud/public.php?service=files&t=660dc830afb091237cc40b3dea2fdf6b Attaching package: ?MODIS? The following object is masked from ?package:base?: file.size
MODISoptions()
All suggested packages are installed Detecting available write drivers! Found: 64 candidate drivers, detecting file extensions... 43 usable drivers detected! STORAGE: _______________ localArcPath : C:/Users/Hakim/Documents/MODIS_ARC/ outDirPath : C:/Users/Hakim/Documents/MODIS_ARC/PROCESSED/ DOWNLOAD: _______________ MODISserverOrder : LPDAAC, LAADS dlmethod : auto stubbornness : 5 PROCESSING: _______________ GDAL : GDAL 2.0.2, released 2016/01/26 MRT : Version 4.1 (March 2011) pixelSize : asIn outProj : GEOGRAPHIC resamplingType : NN dataFormat : GTiff DEPENDENCIES: _______________
dates <- as.POSIXct( as.Date(c("01/04/2016","15/04/2016"),format =
"%d/%m/%Y") ) # specify your beginning and end dates
dates <- transDate(dates[1],dates[2]) # transform them into a readable
format
product <- "MOD13Q1" # MODIS vegetation indices
bands <- "010" # here I select the 2nd (EVI)
h = c("22") # your tile horizontal value
v = c("07","08") # your tile vertical value
runGdal(product=product,begin=dates$beginDOY,end = dates$endDOY,tileH =
h,tileV = v, SDSstring = bands, outProj="4326")
Loading required package: rgdal
rgdal: version: 1.1-8, (SVN revision 616)
Geospatial Data Abstraction Library extensions to R successfully loaded
Loaded GDAL runtime: GDAL 2.0.1, released 2015/09/15
Path to GDAL shared files: C:/Program Files/R/R-3.2.5/library/rgdal/gdal
GDAL does not use iconv for recoding strings.
Loaded PROJ.4 runtime: Rel. 4.9.1, 04 March 2015, [PJ_VERSION: 491]
Path to PROJ.4 shared files: C:/Program Files/R/R-3.2.5/library/rgdal/proj
Linking to sp version: 1.2-3
Loading required package: rgeos
rgeos version: 0.3-19, (SVN revision 524)
GEOS runtime version: 3.5.0-CAPI-1.9.0 r4084
Linking to sp version: 1.2-3
Polygon checking: TRUE
########################
outProj = +init=epsg:4326 +proj=longlat +datum=WGS84 +no_defs
+ellps=WGS84 +towgs84=0,0,0
pixelSize = asIn
resamplingType = near
Output directory =
C:/Users/Hakim/Documents/MODIS_ARC/PROCESSED//MOD13Q1.005_20160525102653
(no 'job' name specified, generated (date/time based))
########################
Loading required package: RCurl
Loading required package: bitops
Downloading structure on 'LPDAAC' for: MOD13Q1.005
Try: 1
*Error in if (FtpDayDirs[1] == FALSE) { : *
* missing value where TRUE/FALSE needed*
On Wed, May 25, 2016 at 12:34 AM, chris english <
englishchristophera at gmail.com> wrote:
Progress has certainly been made regards MODIS.
> sessionInfo()
R version 3.2.2 (2015-08-14) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 14.04.4 LTS locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 [4] LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 [7] LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C [10] LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] MODIS_0.10-34 raster_2.5-2 sp_1.2-3 devtools_1.9.1 loaded via a namespace (and not attached): [1] rsconnect_0.4.1.11 tools_3.2.2 Rcpp_0.12.5 memoise_0.2.1 grid_3.2.2 [6] digest_0.6.8 lattice_0.20-33 Thomas Nauss, University of Marburg, examined the problem and proposed a fix (https://github.com/MatMatt/MODIS/issues/5). It has not actually been applied to the 'develop' branch currently the MODIS_0.10-34. But cloning the develop branch and editing minorFuns.R with his suggested fix and doing a local install from source gives me:
install.packages("/home/chris/MODIS_tst/MODIS", repos = NULL, type =
"source")
require(MODIS)
Loading required package: MODIS Loading required package: raster Loading required package: sp MODIS_manual: https://ivfl-rio.boku.ac.at/owncloud/public.php?service=files&t=660dc830afb091237cc40b3dea2fdf6b Attaching package: ?MODIS? The following object is masked from ?package:base?: file.size
MODISoptions()
All suggested packages are installed Detecting available write drivers! Found: 64 candidate drivers, detecting file extensions... 44 usable drivers detected! This is a darn sight better than zero drivers. STORAGE:
_______________
localArcPath : /home/chris/MODIS_ARC
outDirPath : /home/chris/MODIS_ARC/PROCESSED
DOWNLOAD:
_______________
MODISserverOrder : LPDAAC, LAADS
dlmethod : auto
stubbornness : high
PROCESSING:
_______________
GDAL : GDAL 2.1.0dev, released 2015/99/99
MRT : Version 4.1 (March 2011)
pixelSize : asIn
outProj : asIn
resamplingType : NN
dataFormat : GTiff
Returning to Hakim's initial post:
dates <- as.POSIXct( as.Date(c("01/01/2010","01/05/2016"),format
="%d/%m/%Y") )
dates <- transDate(dates[1],dates[2])
product <- "MOD13Q1"
bands <- "010"
h = c("21","22")
v = c("07","08")
runGdal(product=product,begin=dates$beginDOY,end = dates$endDOY,tileH =
+ h,tileV = v, SDSstring = bands, outProj="4326")
Loading required package: rgdal
rgdal: version: 1.1-3, (SVN revision 594)
Geospatial Data Abstraction Library extensions to R successfully loaded
Loaded GDAL runtime: GDAL 2.1.0dev, released 2015/99/99
Path to GDAL shared files: /usr/local/share/gdal
Loaded PROJ.4 runtime: Rel. 4.9.1, 04 March 2015, [PJ_VERSION: 491]
Path to PROJ.4 shared files: (autodetected)
WARNING: no proj_defs.dat in PROJ.4 shared files
Linking to sp version: 1.2-1
Loading required package: rgeos
rgeos version: 0.3-15, (SVN revision 515)
GEOS runtime version: 3.4.2-CAPI-1.8.2 r3921
Linking to sp version: 1.2-1
Polygon checking: TRUE
########################
NOTE: rgdal::checkCRSArgs: no proj_defs.dat in PROJ.4 shared files
outProj = +init=epsg:4326 +proj=longlat +datum=WGS84
+no_defs +ellps=WGS84 +towgs84=0,0,0
pixelSize = asIn
resamplingType = near
Output directory =
/home/chris/MODIS_ARC/PROCESSED/MOD13Q1.005_20160525005632 (no 'job'
name specified, generated (date/time based))
########################
Loading required package: RCurl
Loading required package: bitops
Downloading structure on 'LPDAAC' for: MOD13Q1.005
Try: 1
Error in if (FtpDayDirs[1] == FALSE) { :
missing value where TRUE/FALSE needed
Hummph. Well, I'm very happy to have drivers. Uncertain if my 64
candidates vs 44 represents additional name changes as Thomas
suggested and the 44 apply to his fix and an additional 20 are
differences between Frank's and Evan's naming conventions as to those
other 20. And uncertain as to how to ask MODIS which 44 are viable vs
20 that were not viable, though this would probably be useful.
It appears, given Hakim's request code, that he won't be getting MODIS
data as Error after Try: 1 suggests a certain lack of stubbornness or
further problem. Nor will I of course. I tried a debugonce(runGdal),
but the implications of the runGdal() are substantially above and
beyond my pay grade and current level in R.
So back to eyetracking data and Bagidis semi distances. I'm sure MODIS
will be sorted out by those more familiar with expected results.
Chris
On Thu, May 19, 2016 at 1:31 PM, chris english
<englishchristophera at gmail.com> wrote:
Playing with MODISoptions.R lines 155-162:
gdalPath = '/usr/local/bin' # take out correctPath as can't find where
defined at present ln 157
if(length(grep(dir(gdalPath),pattern="gdalinfo"))==0)
+ {
+ stop(paste0("The 'gdalPath' you have provided
'",normalizePath(gdalPath,"/",FALSE) ,
+ "' does not contain any gdal utilities, make sure to address the
+ folder with GDAL executables (ie: gdalinfo)!"))
+ }
The preceding executes without complaint or stop as gdalinfo was found in
/usr/local/bin to have length >0.
gdalDrivers()
47 GTiff GeoTIFF TRUE TRUE FALSE
51 HDF4Image HDF4 Dataset TRUE FALSE FALSE
We have the necessary drivers. Hard coding MODIS_Opts.R gdalPath doesn't
work. Ah well.
On Thu, May 19, 2016 at 7:10 AM, chris english
<englishchristophera at gmail.com> wrote:
Logging out (Log out?!?, do people still do that?), which is mentioned
in
the manual, though actually means more than closing and opening a new
terminal, actually restarting the kernel, results in system update where
after MRT is recognized but still, in my case, drivers are not:
library(MODIS)
MODISoptions()
All suggested packages are installed
Detecting available write drivers!
Found: 64 candidate drivers, detecting file extensions...
ERROR 1: --format option given with format 'VRT-raster-', but that
format
not
recognised. Use the --formats option to get a list of available
formats,
and use the short code (ie. GTiff or HFA) as the format identifier.
ERROR 1: --format option given with format 'GTiff-raster-', but that
format not
recognised. Use the --formats option to get a list of available
formats,
and use the short code (ie. GTiff or HFA) as the format identifier.
0 usable drivers detected!
STORAGE:
_______________
localArcPath : /home/chris/MODIS_ARC
outDirPath : /home/chris/MODIS_ARC/PROCESSED
DOWNLOAD:
_______________
MODISserverOrder : LPDAAC, LAADS
dlmethod : auto
stubbornness : high
PROCESSING:
_______________
GDAL : GDAL 2.1.0dev, released 2015/99/99
MRT : Version 4.1 (March 2011)
pixelSize : asIn
outProj : asIn
resamplingType : NN
dataFormat : GTiff
DEPENDENCIES:
_______________
So, a little more digging.
Chris
On Wed, May 18, 2016 at 2:13 PM, chris english
<englishchristophera at gmail.com> wrote:
Hakim,
Your installation and mine share the same defect:
MODIS:::checkGdalDriver()
[1] TRUE
MODIS:::checkGdalDriver(GTiff)
Error in correctPath(path) : object 'GTiff' not found
MODIS:::checkGdalDriver
function (path = NULL)
{
inW <- getOption("warn")
on.exit(options(warn = inW))
options(warn = -1)
path <- correctPath(path)
cmd <- paste0(path, "gdalinfo --formats")
if (.Platform$OS == "windows") {
driver <- try(shell(cmd, intern = TRUE), silent = TRUE)
}
else {
driver <- try(system(cmd, intern = TRUE), silent = TRUE)
}
if (class(driver) == "try-error") {
options(warn = inW)
warning("No gdal installation found please install 'gdal' on
your
system first!")
return(FALSE)
}
if (length(grep(driver, pattern = "HDF4")) == 0) {
return(FALSE)
}
else {
return(TRUE)
}
}
<bytecode: 0x7ddc970>
<environment: namespace:MODIS>
If we hard code:
cmd <- "/usr/local/bin/gdalinfo --formats"
driver <- try(system(cmd, intern = TRUE), silent = TRUE)
driver
[1] "Supported Formats:"
[2] " VRT -raster- (rw+v): Virtual Raster"
[3] " GTiff -raster- (rw+vs): GeoTIFF"
[4] " NITF -raster- (rw+vs): National Imagery Transmission Format"
[5] " RPFTOC -raster- (rovs): Raster Product Format TOC format"
[6] " ECRGTOC -raster- (rovs): ECRG TOC format"
[7] " HFA -raster- (rw+v): Erdas Imagine Images (.img)"
[8] " SAR_CEOS -raster- (rov): CEOS SAR Image"
[9] " CEOS -raster- (rov): CEOS Image"
[10] " JAXAPALSAR -raster- (rov): JAXA PALSAR Product Reader (Level
1.1/1.5)"
I am still looking for where correctPath() comes from as well as trying
to get my MRT recognized, but slowly slowly.
Seems we're looking at environment variables (MRT, and likely
gdal_config) and some other special sause that we've overlooked in
installation of MODIS.
Chris
On Wed, May 18, 2016 at 8:53 AM, chris english
<englishchristophera at gmail.com> wrote:
?MODISoptions
MODIS:::checkTools("GDAL")
Checking availabillity of GDAL:
OK, GDAL 2.1.0dev, released 2015/99/99 found!
getOption("MODIS_gdalOutDriver")
[1] name description extension
<0 rows> (or 0-length row.names)
Interestingly not finding even GTiff default. Well, more digging, but
these outputs explain runGdal error output.
On Tue, May 17, 2016 at 10:07 AM, Hakim Abdi <hakim.abdi at nateko.lu.se
wrote:
Same here. Loading rgdal didn't make a difference.
On Tue, May 17, 2016 at 9:05 AM, chris english
<englishchristophera at gmail.com> wrote:
I suspect it is an incomplete Modis build even though library(Modis)
loads without complaint, but will have to check into this later. I
recall
not seeing MRT complete...which I think is inscribing a folder
though is
probably much more.
Having rgdal loaded as against your sessionInfo() didn't make a
difference.
Chris
On Tue, May 17, 2016 at 9:52 AM, chris english
<englishchristophera at gmail.com> wrote:
Hi Hakim,
Interesting and unexpected as I am on a linux box but I have the
same
problem, or I can reproduce your's:
sessionInfo()
R version 3.2.2 (2015-08-14)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 14.04.4 LTS
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
LC_TIME=en_US.UTF-8
[4] LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8
LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8
LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods
base
other attached packages:
[1] rgdal_1.1-3 MODIS_0.10-34 raster_2.5-2 sp_1.2-1
loaded via a namespace (and not attached):
[1] rsconnect_0.4.1.11 tools_3.2.2 Rcpp_0.12.4
grid_3.2.2 lattice_0.20-33
dates <- as.POSIXct( as.Date(c("01/01/2010","01/05/2016"),format
="%d/%m/%Y") )
dates <- transDate(dates[1],dates[2])
product <- "MOD13Q1"
bands <- "010"
h = c("21","22")
v = c("07","08")
runGdal(product=product,begin=dates$beginDOY,end =
dates$endDOY,tileH =
+ h,tileV = v, SDSstring = bands, outProj="4326")
Error in runGdal(product = product, begin = dates$beginDOY, end =
dates$endDOY, :
in argument dataFormat='GTiff', format not supported by GDAL
type:
'gdalWriteDriver()' (column 'name') to list available inputs
debugonce(runGdal)
#output
Browse[2]> n
debug: stop("in argument dataFormat='", opts$dataFormat, "', format
not supported by GDAL type: 'gdalWriteDriver()' (column 'name') to
list
available inputs")
Browse[2]> n
Error in runGdal(product = product, begin = dates$beginDOY, end =
dates$endDOY, :
in argument dataFormat='GTiff', format not supported by GDAL
type:
'gdalWriteDriver()' (column 'name') to list available inputs
Hmmmm. More digging. Sorry to not be helpful to this point.
Chris
On Mon, May 16, 2016 at 11:47 AM, Hakim Abdi
<hakim.abdi at nateko.lu.se> wrote:
Hi Alex,
The OS is a 64 bit Windows 7 as specified in the sessionInfo()
output I posted. When I upgraded QGIS, I did so by uninstalling
the previous
one and reinstalling the new version. I'm not quite sure what it
means to
upgrade GDAL since I installed it fresh with OSGeo4W. In R, I
reinstall all
the packages I need fresh off the repository using the
install.views
command.
gdalDrivers() produces the output that's attached.
Cheers,
Hakim
On Wed, May 11, 2016 at 5:09 PM, Alex Mandel
<tech_dev at wildintellect.com> wrote:
What operating system?
When you upgraded QGIS did you also upgrade GDAL?
When you upgraded R, did you update your packages (or rebuild any
packages)?
What do you get from the following command?
gdalDrivers()
Thanks,
Alex
On 05/02/2016 02:11 PM, Hakim Abdi wrote:
Hello everyone,
I'm having problems with the MODIS package, specifically, the
runGdal()
command after the recent update to R 3.2.5. The error I get is
this:
Error in runGdal(product = product, begin = dates$beginDOY,
end =
dates$endDOY, : in argument dataFormat='GTiff', format not
supported by
GDAL type: 'gdalWriteDriver()' (column 'name') to list
available
inputs
I'm not quite sure why this is happening, and I haven't found a
solution
online. Does anyone have an idea? The package was working fine
before I
upgraded R from 3.1.1. to 3.2.5, QGIS from 2.12.3 to 2.14.1 and
RStudio to
version 0.99.467.
My details are below, thanks for the assistance:
R version 3.2.5 (2016-04-14) -- "Very, Very Secure Dishes"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-w64-mingw32/x64 (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
require(MODIS)
Loading required package: MODIS
Loading required package: raster
Loading required package: sp
MODIS_manual:
https://ivfl-rio.boku.ac.at/owncloud/public.php?service=files&t=660dc830afb091237cc40b3dea2fdf6b
Attaching package: ?MODIS?
The following object is masked from ?package:base?:
file.size
MODISoptions()
All suggested packages are installed
Detecting available write drivers!
Found: 64 candidate drivers, detecting file extensions...
0 usable drivers detected!
STORAGE:
_______________
localArcPath : C:/Users/Hakim/Documents/MODIS_ARC/
outDirPath : C:/Users/Hakim/Documents/MODIS_ARC/PROCESSED/
DOWNLOAD:
_______________
MODISserverOrder : LPDAAC, LAADS
dlmethod : auto
stubbornness : 5
PROCESSING:
_______________
GDAL : GDAL 2.0.2, released 2016/01/26
MRT : Version 4.1 (March 2011)
pixelSize : asIn
outProj : GEOGRAPHIC
resamplingType : NN
dataFormat : GTiff
DEPENDENCIES:
_______________
dates <- as.POSIXct(
as.Date(c("01/01/2010","01/05/2016"),format
=
"%d/%m/%Y") )
dates <- transDate(dates[1],dates[2])
product <- "MOD13Q1"
bands <- "010"
h = c("21","22")
v = c("07","08")
*> runGdal(product=product,begin=dates$beginDOY,end =
dates$endDOY,tileH =
h,tileV = v, SDSstring = bands, outProj="4326")*
*Error in runGdal(product = product, begin = dates$beginDOY,
end
=
dates$endDOY, : **in argument dataFormat='GTiff', format not
supported by
GDAL type: 'gdalWriteDriver()' (column 'name') to list
available
inputs*
sessionInfo()
R version 3.2.5 (2016-04-14)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1
locale:
[1] LC_COLLATE=English_United States.1252
LC_CTYPE=English_United
States.1252
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods
base
other attached packages:
[1] MODIS_0.10-34 raster_2.5-2 sp_1.2-3
loaded via a namespace (and not attached):
[1] tools_3.2.5 Rcpp_0.12.4 grid_3.2.5
lattice_0.20-33
__________________________________________________
Hakim Abdi | PhD Candidate
Center for Geobiosphere Science
Department of Physical Geography and Ecosystem Science
Lund University, S?lvegatan 12, SE-223 62 Lund, Sweden
[[alternative HTML version deleted]]
_______________________________________________
R-sig-Geo mailing list
R-sig-Geo at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
_______________________________________________
R-sig-Geo mailing list
R-sig-Geo at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo