Dear all,
I'm trying to fix a subtle bug in the hdf5 package. This package provides an
interfaces to the HDF5 library and hence allows one to load data into R from
files in the HDF5 format. The bug appeared during a period in which R changed
but the package did not.
I include below both the R and C code, stripped of everything except what is
needed to show the bug. What is supposed to happen is
(*) the user calls R function hdf5load()
(*) hdf5load() calls C function do_hdf5load()
(*) do_hdf5load() opens the HDF5 file recording its HDF5 file id (fid)
(*) do_hdf5load() calls C function setup_onexit, passing fid to it
(*) setup_onexit sets up the on.exit call to be R function hdf5cleanup with
fid as its argument
(*) C function do_hdf5load() walks the HDF5 file's tree structure of groups
of groups of [...] of datasets, mapping them to an R list of lists of [...]
of array variables. This recursive procedure may have a variety of exit
points buried inside itself.
(*) C function do_hdf5load() exits for some reason. R function hdf5load()
therefore exits but before doing so it calls its on.exit code (which is
hdf5cleanup(fid) with the right value of fid), closing the file.
The problem is that when do_hdf5load() and hdf5load() exit, hdf5cleanup() is
usually not called, meaning that the file is left open. You might not notice
this, but if you are processing a few year's worth of data, which is stored
at 1 file per day, you may end up with the system limit number of files open
and be unable to open any more.
I have a suspicion that the problem dates to a change in R at 2.8.0. If you
do help(on.exit) it notes under "Details" that: "Where ?expr? was evaluated
changed in R 2.8.0 ..." But it is not clear how I should modify the C code to
force hdf5cleanup() to be reliably called when do_hdf5load() exits.
Any help appreciated.
Hugh (possibly the nearest thing to a maintainer that the hdf5 package
currently has)
(R and C code follow)
#----------------------------------------------------------------
"hdf5load" <- function (file, load = TRUE, verbosity = 0, tidy = FALSE)
{
call <- sys.call()
.External("do_hdf5load", call, sys.frame(sys.parent()), file, load,
as.integer (verbosity), as.logical(tidy),
PACKAGE="hdf5")
}
"hdf5cleanup" <- function (fid)
{
call <- sys.call()
print("In hdf5cleanup: calling do_hdf5cleanup")
invisible(.External("do_hdf5cleanup", call, sys.frame(sys.parent()), fid,
PACKAGE="hdf5"))
}
#----------------------------------------------------------------
/*---------------------------------------------------------------*/
SEXP do_hdf5load (SEXP args)
{
/* Code to process args snipped */
if ((fid = H5Fopen (path, H5F_ACC_RDONLY, H5P_DEFAULT)) < 0)
errorcall (call, "unable to open HDF file: %s", path);
setup_onexit (fid, env);
/* Messy code to walk tree structure of file snipped */
}
/* The following function shown in its entirety */
setup_onexit (hid_t fid, SEXP env)
{
eval (lang2 (install ("on.exit"),
lang2 (install ("hdf5cleanup"),
ScalarInteger (fid))),
env);
}
SEXP
do_hdf5cleanup (SEXP args)
{
/* Code to process args snipped */
/* various cleanup things done including this: */
H5Fclose(fid)
}
/*---------------------------------------------------------------*/