raster/rgdal- problem: Too many open files (Linux)

Dear list,

We have a problem which appears to be a bug in either rgdal or raster,
although it could also be a bug in base R or in our understanding of how
to deal with connections.

We have a process which is writing a rather large (~10-20.000) number of
geoTiffs via writeRaster. However, the process has frequently stopped
with an error of the type:
Error in .local(.Object, ...) :
    TIFFOpen:/local0/skoiejo/hri/test.tif: Too many open files
The issue seems to be the creation of temp-files in the temp directory
which is given by tempdir(), not by raster:::.tmpdir(). These temp-files
seem to be created by the call
    transient <- new("GDALTransientDataset", driver=driver, rows=r at nrows,
cols=r at ncols, bands=nbands, type=dataformat, fname=filename,
options=options, handle=NULL)
from raster:::.getGDALtransient
The temp-files are deleted after writing the geoTiff, but are not
removed from the list of open files in Linux, which on our system was
limited to 1024 files (ulimit -n) per process. Below is a script which
can replicate the issue (takes a few minutes to reach 1024) and
sessionInfo().

Currently we are trying to solve the issue by increasing the limit of
file connections, but we would prefer a solution where the connections
are properly deleted, either before writeRaster finishes, or a command
which we can include in our script, either R-code or a call to System().
The connections are not visible via showConnections(), and
closeAllConnections() does not help.

Thanks,
Jon

raster/rgdal- problem: Too many open files (Linux)

Thread (5 messages)