readGDAL does not close connections

I suspect that readGDAL opens connections to the files, but does not 
close them and the program fails because too many files have been 
openened simultaneously. I do not get messages like "Closing GDAL 
dataset handle 0x8ff7900...  destroyed ... done." when I run the script. 
Does anybody on the r-sig-geo list have similar experiences?

  

Dear r-sig-geo list,

I'm struggeling with readGDAL (rgdal package) not closing its file
connections (I suspect). I am trying to construct a timeseries of
rainfall using the following loop (conceptual code):

pt = SpatialPoint()
rainfall = numeric(length = length(listofiles))
for(i in 1:length(listofiles)) {
   grid = readGDAL(listofiles[i])
   rainfall[i] = grd$value[overlay(grid, pt)]
}

There are more than 35000 grids (HDF5) that I loop over (2007 - 15
minutes interval). After reading about 1000 grids my program fails:

HDF5:"radar/2007/01/11/RAD_NL21_PCP_NA_200701111515.h5"://image1/image_data
has GDAL driver HDF5Image
and has 256 rows and 256 columns
Error in gzfile(file, "r") : unable to open connection
In addition: Warning message:
In gzfile(file, "r") :
 cannot open compressed file '/home/hiemstra/lib/R/rgdal/DESCRIPTION'

I suspect that readGDAL opens connections to the files, but does not
close them and the program fails because too many files have been
openened simultaneously. I do not get messages like "Closing GDAL
dataset handle 0x8ff7900...  destroyed ... done." when I run the script.
Does anybody on the r-sig-geo list have similar experiences?

kind regards,
Paul

ps Additional information on my session

sessionInfo()

R version 2.6.1 (2007-11-26)
i486-pc-linux-gnu

locale:
LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.UTF-8;LC_MONETARY=en_US.UTF-8;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-8;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=en_US.UTF-8;LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] rgdal_0.5-24   maptools_0.7-7 foreign_0.8-23 automap_0.4-3
classInt_0.1-9
[6] e1071_1.5-17   class_7.2-41   gstat_0.9-43   sp_0.9-23

loaded via a namespace (and not attached):
[1] grid_2.6.1      lattice_0.17-6  rcompgen_0.1-17

I run Debian Lenny

Paul Hiemstra wrote:

I suspect that readGDAL opens connections to the files, but does not
close them and the program fails because too many files have been
openened simultaneously. I do not get messages like "Closing GDAL
dataset handle 0x8ff7900...  destroyed ... done." when I run the script.
Does anybody on the r-sig-geo list have similar experiences?

Looking at readGDAL, it seems that GDAL.close is called on the open file
handle before the data are returned. This should give you the handle ...
destroyed ... messages for each file you open.

In the C code of package rgdal, I see reference to GDALOpen(), but not
to GDALClose(). I guess it should?

--
Edzer

On Thu, 6 Mar 2008, Edzer Pebesma wrote:

  

Paul Hiemstra wrote:
    

I suspect that readGDAL opens connections to the files, but does not
close them and the program fails because too many files have been
openened simultaneously. I do not get messages like "Closing GDAL
dataset handle 0x8ff7900...  destroyed ... done." when I run the script.
Does anybody on the r-sig-geo list have similar experiences?


      

Looking at readGDAL, it seems that GDAL.close is called on the open file
handle before the data are returned. This should give you the handle ...
destroyed ... messages for each file you open.

In the C code of package rgdal, I see reference to GDALOpen(), but not
to GDALClose(). I guess it should?
    

Where? In general, handles to GDAL datasets are all that are open and 
visible in R, they are external pointers, and are either destroyed by 
design, calling the C++ function RGDAL_CloseHandle in 
src/gdal-bindings.cpp, l. 357, or by the finalizer - see the beginning of 
R/gdal.R, and uses of .setCollectorFun(), which accompany calls to 
RGDAL_CloseHandle in class definitions.

I don't think that this is a problem in compiled code, because connections 
only exist in R code, and are not used directly by readGDAL().

Roger

  

--
Edzer