Skip to content

readRAST6: too many open files

8 messages · Roger Bivand, Agustin Lobo

Original
Agustin Lobo
# 
We're using
rastdum <- readRAST6("nomdum",cat=F,ignore.stderr = T)

(on a ubuntu 8.10 machine and R 2.9)
within a for() loop of hundreds of iterations.

At some point (ca. 199) we get an error
"Too many open files"

and suspect that this is because readRAST6() is opening
and not closing files. Is this a bug? I remember having
a similar problem with writeGDAL() on a win machine a long ago,
but that problem was fixed.

Thanks

Agus
Roger Bivand
#
On Tue, 5 May 2009, Agustin Lobo wrote:

            

      
      
      
    

        
Possibly, but a complete sessionInfo() would be necessary - the versions 
of the packages involved may matter. Does this reproduce with readRAST6() 
in a loop reading say the same raster? I've run 300 reads of spearfish 
elevation.dem without seeing problems for:

            

      
      
      
    

        
R version 2.9.0 (2009-04-17)
i386-pc-mingw32

locale:
LC_COLLATE=Norwegian (Bokm?l)_Norway.1252;LC_CTYPE=Norwegian 
(Bokm?l)_Norway.1252;LC_MONETARY=Norwegian 
(Bokm?l)_Norway.1252;LC_NUMERIC=C;LC_TIME=Norwegian (Bokm?l)_Norway.1252

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] spgrass6_0.6-5 XML_2.3-0      rgdal_0.6-8    sp_0.9-36

loaded via a namespace (and not attached):
[1] grid_2.9.0      lattice_0.17-22

Roger

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Agustin Lobo
    

    

      
      #
    

  


  

      
Here it is:

R version 2.9.0 (2009-04-17)
i486-pc-linux-gnu

locale:
LC_CTYPE=es_ES.UTF-8;LC_NUMERIC=C;LC_TIME=es_ES.UTF-8;LC_COLLATE=es_ES.UTF-8;LC_MONETARY=C;LC_MESSAGES=es_ES.UTF-8;LC_PAPER=es_ES.UTF-8;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=es_ES.UTF-8;LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base    

other attached packages:
[1] spgrass6_0.6-4 XML_1.95-3     rgdal_0.6-7    sp_0.9-34    

loaded via a namespace (and not attached):
[1] grid_2.9.0      lattice_0.17-22


We are upgrading to match your versions, will retry and let you know

Agus

        
Roger Bivand wrote:

            

      
      
      
    

        

      
      
    

        
-------------- next part --------------
A non-text attachment was scrubbed...
Name: alobolistas.vcf
Type: text/x-vcard
Size: 251 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-geo/attachments/20090505/5975ab65/attachment.vcf>

  
  


  


      

    

    

      
      

        


  

        

      Roger Bivand
    

    

      
      #
    

  


  

      
On Tue, 5 May 2009, Agustin Lobo wrote:

        

            

      
      
      
    

        
Is it possible that this only happens with the default plugin=NULL 
argument to readRAST6(), and you have a GRASS GDAL plugin, so it gets 
auto-found and used? Does it still happen when plugin=FALSE?

Roger

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Agustin Lobo
    

    

      
      #
    

  


  

      
Roger,

After upgrading, the test works with plugin=F,
while plugin=T produces the same result.

Also, ignore.stderr=T yields:

ERROR 6: SetColorTable() only supported for Byte or UInt16 bands in TIFF 
format.
WARNING: Input raster map constains cells with NULL-value (no-data). The
          value 999 was used to represent no-data values in the input
          map.You can specify nodata value by nodata parameter.
/media/Transcend/Montseny/GrassData//carlos/A_TOTAL/.tmp/caminoccg/nomdum 
has GDAL driver GTiff
and has 94 rows and 116 columns

which is not including any more the first lines that we had with the 
previous version:

raster map/current region mismatch detected in components:
            cols            rows origin.northing  origin.easting
            TRUE            TRUE           FALSE            TRUE
set plugin=TRUE to override; continuing with plugin=FALSE

Now we have:

 > sessionInfo()
R version 2.9.0 (2009-04-17)
i486-pc-linux-gnu

locale:
LC_CTYPE=es_ES.UTF-8;LC_NUMERIC=C;LC_TIME=es_ES.UTF-8;LC_COLLATE=es_ES.UTF-8;LC_MONETARY=C;LC_MESSAGES=es_ES.UTF-8;LC_PAPER=es_ES.UTF-8;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=es_ES.UTF-8;LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] spgrass6_0.6-4 XML_1.95-3     rgdal_0.6-7    sp_0.9-36

loaded via a namespace (and not attached):
[1] grid_2.9.0      lattice_0.17-22
 >

We are trying now with our process, that takes much longer than the 
test, we'll let you know.

Thanks!

Agus


-------------- next part --------------
A non-text attachment was scrubbed...
Name: alobolistas.vcf
Type: text/x-vcard
Size: 251 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-geo/attachments/20090505/fd026f53/attachment.vcf>

  
  


  


      

    

    

      
      

        


  

        

      Roger Bivand
    

    

      
      #
    

  


  

      
On Tue, 5 May 2009, Agustin Lobo wrote:

        

            

      
      
      
    

        
I find this too on Fedora 10, plugin=TRUE fails after a hundred or so 
calls, and then crashes R on a further command. With plugin=FALSE, there 
is no problem at all. The problem then lies in the plugin driver, which 
isn't anything that spgrass6 can do anything about. It looks as though the 
plugin is opening file handles to GRASS files but not closing them. I'll 
try to ask on the GDAL and grass-dev lists. For the time being, it seems 
best to use plugin=FALSE in readRAST6().

            

      
      
      
    

        
This is a GRASS version issue, more recent GRASS support a flag to avoid 
r.out.gdal trying to write non-existent colour tables. Could you repeat 
your GRASS version:

execGRASS("g.version")

should do it.

            

      
      
      
    

        
Maybe set a NODATA= value in the readRAST6() call to avoid heuristics 
trying to find a possible value?

Roger

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Roger Bivand
    

    

      
      #
    

  


  

      
On Tue, 5 May 2009, Roger Bivand wrote:

        

            

      
      
      
    

        
A follow-up. The previous results were using a GDAL 1.5.3 RPM, linking 
against GRASS 6.3 libraries. Having built a fresh GDAL 1.6.0 from source, 
linked against a fresh GRASS 6.4.0 RC4, the plugin now works and does not 
crash R. Browsing the GDAL GRASS raster plugin, there seem to have been 
important commits between GDAL 1.5.3 and 1.6.0 (possibly also 1.5.4, 
untested). GRASS has also changed between 6.3 and 6.4.0 release 
candidates. So a possible solution is to upgrade both GDAL and GRASS, or 
at least GDAL.

Roger

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Agustin Lobo
    

    

      
      #
    

  


  

      
ok, thanks, we are using

 > system("g.version")
GRASS 6.3.0 (2008)

which is what comes for binaries for ubuntu 8.04 by now.
We should be compiling qgis, grass and gdal soon and will
try again and let you know.

Anyway, by now our process works using plugin=F and ignore.stderr=T
and the newest spgrass6, rgdal and sp packages.

Thanks a lot

Agus

        
Roger Bivand wrote: