Skip to content

Problem reading a HDF5 file with readGDAL in Windows

11 messages · Oscar Perpiñan, Michael Sumner, John Baumgartner +1 more

Original
Oscar Perpiñan
# 
Hello,

I am trying to read a HDF5 file whose variable name contains '//'. I have no
problem in a Linux machine, but 'readGDAL' throws an error in Windows. If I
am not wrong, it is because the initialize method of the 'GDALReadOnlyDataset'
class includes a call to 'normalizePath'.

For example, the next code works in Linux but fails in Windows (it uses this
file
ftp://atrain.sci.gsfc.nasa.gov/data/s4pa//OMI/OMCLDO2_CPR.003/2015/125/OMI.L2.CloudOMCLDO2Strip200kmAlongCloudSat.2015.05.05.020752Z.v003.he5
)

x <- readGDAL("HDF5:OMI.L2.CloudOMCLDO2Strip200kmAlongClo
udSat.2015.05.05.020752Z.v003.he5://HDFEOS/SWATHS/
CloudFractionAndPressure/Data_Fields/ChiSquaredOfFit")

Is there any way to circumvent this problem?

Thanks in advance.

Oscar.
-----------------------------------------------------------------
Oscar Perpi??n Lamigueiro
Dpto. Ing. El?ctrica, Electr?nica, Autom?tica y F?sica Aplicada (ETSIDI-UPM)
Grupo de Sistemas Fotovoltaicos (IES-UPM)
URL: http://oscarperpinan.github.io
Michael Sumner
# 
Are you sure you have the HDF5 driver in Windows?  Do you build yourself
against your own GDAL or against OSGeo4W?  (It's not in the CRAN
windows-build).

This is the error the CRAN build gives (though it's actually not
recognizing the driver):

"... does not exist in the file system, and is not recognised as a
supported dataset name."

(gdalinfo works fine with your subdataset string here on my Windows8 with
OSGeo4W. )

Cheers, Mike.
On Tue, 5 May 2015 at 21:35 Oscar Perpi?an <oscar.perpinan at gmail.com> wrote:

            

      
      
      
    

        

      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      John Baumgartner
    

    

      
      #
    

  


  

      
Not a remedy, but the following might be a workable alternative...

ncdf4::nc_open("OMI.L2.CloudOMCLDO2Strip200kmAlongCloudSat.2015.05.05.020752Z.v003.he5")$var$`Data
Fields/ChiSquaredOfFit`

I'm not familiar with HDF5 structure, so not sure of possible downsides.

Cheers,
John

On Tue, May 5, 2015 at 9:34 PM, Oscar Perpi?an <oscar.perpinan at gmail.com>
wrote:

            

      
      
      
    

        

      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Robert J. Hijmans
    

    

      
      #
    

  


  

      
In which case you can change the extension to '.nc' and do

library(raster)
library(ncdf4)
x <- raster("E:/downloads/OMI.L2.CloudOMCLDO2Strip200kmAlongCloudSat.2015.05.05.020752Z.v003.nc",
var='Data Fields/ChiSquaredOfFit')

Robert

        
On Tue, May 5, 2015 at 5:28 AM, John Baumgartner <johnbaums at gmail.com> wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Robert J. Hijmans
    

    

      
      #
    

  


  

      
Even better (renaming not necessary):

x <- raster("E:/downloads/OMI.L2.CloudOMCLDO2Strip200kmAlongCloudSat.2015.05.05.020752Z.v003.he5",
var='Data Fields/ChiSquaredOfFit', ncdf=TRUE)

Best, Robert

        
On Tue, May 5, 2015 at 10:24 AM, Robert J. Hijmans <r.hijmans at gmail.com> wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Michael Sumner
    

    

      
      #
    

  


  

      
On Wed, 6 May 2015 at 03:28 Robert J. Hijmans <r.hijmans at gmail.com> wrote:

        

            

      
      
      
    

        
But only if you build and install ncdf4 yourself (or use DP's one off-CRAN:
http://cirrus.ucsd.edu/~pierce/ncdf/)

There's no ncdf4 for Windows on CRAN.

Robert's solution doesn't work for me on Windows with ncdf (though I am
sure it it would if I build ncdf with NetCDF4).

(I feel like I've missed something here . . .)

Cheers, Mike.

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Robert J. Hijmans
    

    

      
      #
    

  


  

      
Yes, you need the ncdf4 package (apart from build your own options
that I would not generally recommend). That's why I explicitly did
`library(ncdf4)`. On windows that indeed means downloading it from
http://cirrus.ucsd.edu/~pierce/ncdf/ and unzipping the content to the
right folder ("install from local zip file" in Rgui). Just a tiny bit
of additional effort.

Robert

        
On Tue, May 5, 2015 at 2:20 PM, Michael Sumner <mdsumner at gmail.com> wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Oscar Perpiñan
    

    

      
      #
    

  


  

      
Hi,

            

      
      
      
    

        
I will check it later.

This is the error the CRAN build gives (though it's actually not

            

      
      
      
    

        
Exactly. And if I run traceback() after it, I find that it's the
'GDAL.open' function that is giving the error. In the error message I see
that the 'normalizePath' function is changing the variable name. For
example, if I use a file with a variable named "//DSSF", it gives:
"C:\Users\xxxx\AppData\Local\Temp\RtmpsD81tO\HDF5:201407291200:\DSSF"

Thanks

Oscar.

  
  


  


      

    

    

      
      

        


  

        

      Oscar Perpiñan
    

    

      
      #
    

  


  

          

      
      
      
    

        
Thanks. It works with the file of the example. But it fails with this one
(even on a Linux machine):
https://www.dropbox.com/s/3zmhbtcpf7ly4nw/201407291200

            

      
      
      
    

        
Error in R_nc4_open: NetCDF: Can't open HDF5 attribute
....

            

      
      
      
    

        
Error in R_nc4_open: NetCDF: Can't open HDF5 attribute
....

However, I can open this file in Linux with:

x <- readGDAL('HDF5:201407291200://DSSF')

Best,

Oscar.

  
  


  


      

    

    

      
      

        


  

        

      Robert J. Hijmans
    

    

      
      #
    

  


  

      
It works for me. I get:

            

      
      
      
    

        
Loading required package: sp

            

      
      
      
    

        
class       : RasterLayer
dimensions  : 651, 1701, 1107351  (nrow, ncol, ncell)
resolution  : 1, 1  (x, y)
extent      : 0.5, 1701.5, 0.5, 651.5  (xmin, xmax, ymin, ymax)
coord. ref. : NA
data source : E:\downloads\201407291200
names       : DSSF
zvar        : DSSF

            

      
      
      
    

        
R version 3.2.0 RC (2015-04-08 r68161)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United
States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United
States.1252

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] ncdf4_1.12    raster_2.3-41 sp_1.0-17

loaded via a namespace (and not attached):
[1] grid_3.2.0      lattice_0.20-31

            

      
      
      
    

        
On Tue, May 5, 2015 at 10:53 PM, Oscar Perpi?an

        
<oscar.perpinan at gmail.com> wrote:

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Oscar Perpiñan
    

    

      
      #
    

  


  

      
Not for me, and I cannot understand why:

            

      
      
      
    

        
Error in R_nc4_open: NetCDF: Can't open HDF5 attribute
Error in ncdf4::nc_open(filename) :
  Error in nc_open trying to open file
/home/oscar/Dropbox/share/201407291200

            

      
      
      
    

        
R version 3.2.0 (2015-04-16)
Platform: i586-pc-linux-gnu (32-bit)
Running under: Debian GNU/Linux 8 (jessie)

locale:
 [1] LC_CTYPE=es_ES.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=es_ES.UTF-8        LC_COLLATE=es_ES.UTF-8
 [5] LC_MONETARY=es_ES.UTF-8    LC_MESSAGES=es_ES.UTF-8
 [7] LC_PAPER=es_ES.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=es_ES.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] ncdf4_1.13    raster_2.3-40 sp_1.0-14

loaded via a namespace (and not attached):
[1] compiler_3.2.0  tools_3.2.0     grid_3.2.0      lattice_0.20-27