Skip to content

GDAL.close

12 messages · Oliver Soong, Roger Bivand

Original
Roger Bivand
#
On Mon, 28 Oct 2013, Oliver Soong wrote:

            

      
      
      
    

        
Could you please provide a working example? I have looked at this, but 
need a baseline to know whether I'm looking at the same thing. I'm very 
unsure that this is a robust solution, and need an instrumented example, 
including listings of the temporary directory during the process, to see 
the consequences. Thanks for looking into this, but I'd prefer to be sure 
that a Windows-specific fix doesn't make things worse for others too. 
Please also report on the source of your Windows rgdal binary - is it from 
CRAN or locally built dynamically linking your own GDAL?

Best wishes,

Roger

            

      
      
      
    

        

      
      
    

        

  
    

      
      
    

  


  


    

    
2 days later

    

      
      

        
      

    

    
1 day later

    

      
      

        


  

        

      Roger Bivand
    

    

      
      #
    

  


  

      
Which Windows? XP, Vista, 7, 8, 8.1? 32 and 64 bit? If Vista/7/8, run as 
administrator or not? I agree that the code in those parts of rgdal is not 
well-designed - it was well-designed, but has been modified so that it 
works for most people cross-platform, and has had to accommodate changes 
that have taken place in GDAL over more than 10 years, not least the 
error-handler.

The simple solution to your practical problem is to for you to use a 
larger temporary drive under Windows, or change to an operating system 
that does not have these side-effects.

Assisting you is not just a matter of doing what you think works for you, 
but making sure it doesn't break anything else for anybody else 
cross-platform.

Your script does not check for other files in tempdir, so I prepended a 
listing of prior content:

pc <- dir(tempdir())

and dropped them from the list for unlinking:

now <- dir(tempdir())
unlink(paste(tempdir(), now[!(now %in% pc)], sep=.Platform$file.sep))

I do not see how your script exercises the problem. It creates a new 
transient file, but does not close it, which was the behaviour you are 
unhappy with. If I add

GDAL.close(r3)

on Linux, the transient dataset is removed. On Windows 7 64-bit with the 
CRAN rgdal binary run as user, temporary files are left in tempdir for r1, 
r2, and r3. The same three temporary files are left when run as 
administrator.

The earliest version of GDAL.close was:

GDAL.close <- function(dataset) {
             .setCollectorFun(slot(dataset, 'handle'), NULL)
             .Call('RGDAL_CloseDataset', dataset, PACKAGE="rgdal")
             invisible()
}

with a version in 2007 in the THK branch calling a closeDataset method, 
containing:

             handle <- slot(dataset, "handle")
             unreg.finalizer(handle)
             .Call("RGDAL_DeleteHandle", handle, PACKAGE="rgdal")

with:

unreg.finalizer <- function(obj) reg.finalizer(obj, function(x) x)

and by 2010 was:

GDAL.close <- function(dataset) {
             .setCollectorFun(slot(dataset, 'handle'), NULL)
             .Call('RGDAL_CloseDataset', dataset, PACKAGE="rgdal")
             invisible(gc())
}

Special handling of GDALTransientDataset was added in revision 433 in 
Janual 2013, and modified in revision 462 in April 2013.

It has seemed IIRC that Windows can treat arbitrary files as open. It is 
also possible that there is an interaction between Windows and 
rgdal:::.setCollectorFun(), which does what it should, when given the NULL 
argument, setting:

.setCollectorFun <- function(object, fun) {

   if (is.null(fun)) fun <- function(obj) obj
   reg.finalizer(object, fun, onexit=TRUE)

}

so incorporating the THK branch logic. It could possibly also vary across 
drivers, so finding a robust fix means setting up a test rig with multiple 
Windows machines and testing for multiple drivers to see why some temprary 
files are being treated as open when other operating systems don't have 
problems in their removal. Windows users with too small temporary 
directories. I welcome contributions from people who understand Windows 
and can actually explain why we see the consequences we see.

One candidate may be to branch to .Call("RGDAL_DeleteHandle", handle, 
PACKAGE="rgdal") for the GDALTransientDataset case; I'll report back once 
the package has gone through win-builder.

Hope this doesn't muddle too much, clarification doesn't seem like the 
right expression.

Roger

        
On Sun, 3 Nov 2013, Oliver Soong wrote:

        

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Roger Bivand
    

    

      
      #
    

  


  

      
On Mon, 4 Nov 2013, Roger Bivand wrote:

        

            

      
      
      
    

        
The Windows binary with this modification is at:

http://win-builder.r-project.org/TO95cIM24UVL

but I do not see that it has altered behaviour under Windows 7 running as 
user. For some reason Windows sees the transient files as open. Please try 
other drivers to ensure that this isn't driver-specific.

Roger

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        
      

    

    

      
      

        
      

    

    

      
      

        
      

    

    

      
      

        


  

        

      Roger Bivand
    

    

      
      #
    

  


  

      
On Tue, 5 Nov 2013, Oliver Soong wrote:

        

            

      
      
      
    

        
Here is the next version:

http://win-builder.r-project.org/0X0318s8iW0C

with:

             .setCollectorFun(slot(dataset, 'handle'), NULL)
             .Call('RGDAL_CloseDataset', dataset, PACKAGE="rgdal")
             .Call("RGDAL_CloseHandle", slot(dataset, 'handle'),
                 PACKAGE="rgdal")

Running under Windows, and using:

http://download.sysinternals.com/files/Handle.zip

to check, the three transient datasets are open:

   3B8: File  (RW-)   C:\Users\rsb\AppData\Local\Temp\Rtmp8kZk0N\awlr2.tif
   3C0: File  (RW-)   C:\Users\rsb\AppData\Local\Temp\Rtmp8kZk0N\tfur1.tif
   498: File  (RW-)   C:\Users\rsb\AppData\Local\Temp\Rtmp8kZk0N\mxbr3.tif

I'm assuming that RW- means read and write open, the third character is D, 
which is probably directory.

Roger

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        
      

    

    

      
      

        


  

        

      Roger Bivand
    

    

      
      #
    

  


  

      
On Tue, 5 Nov 2013, Oliver Soong wrote:

        

            

      
      
      
    

        
Thanks to Even Rouault on gdal-dev, I think that:

http://win-builder.r-project.org/jJIzeDT1C100

contains a fixed version, with RGDAL_DeleteHandle now closing the dataset 
before deleting it. This worked on Linux and Windows in a rough cut, and 
this one after removing the unneeded unlink() call in GDAL.close() now 
works on Linux, and should also work on Windows. Behaviour on Linux and 
Windows with respect to transient datasets being deleted is now the same. 
It was helpful that you stayed with this, it was far from simple, as Even 
explained on the gdal-dev thread.

Roger

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        
      

    

    

      
      

        


  

        

      Roger Bivand
    

    

      
      #
    

  


  

      
On Wed, 6 Nov 2013, Oliver Soong wrote:

        

            

      
      
      
    

        
Good, thanks for checking. The bug fix will be in the next release of 
rgdal on CRAN.

Roger