An embedded and charset-unspecified text was scrubbed... Name: not available URL: <https://stat.ethz.ch/pipermail/r-sig-geo/attachments/20140323/56d34a21/attachment.pl>
Managing temporary files created when using the Raster package
4 messages · Julie Lee-Yaw, Jonathan Greenberg
Julie: Why not just build a ?file.remove into your script? Also, most raster functions allow you to explicitly set a filename (see e.g. ?calc --> the filename= parameter), rather than letting it generate a random filename -- this would likely make it easier for you to figure out the appropriate file to delete. --j
On Sun, Mar 23, 2014 at 4:44 PM, Julie Lee-Yaw <julleeyaw at yahoo.ca> wrote:
Hi
I am doing a lot of raster calculations repeatedly (in a loop of sorts). As might be expected, this process is generating a lot of temporary files and filing up my hard drive before my script finishes. The first step in my script is to stack a set of very large rasters. This stack is then the input for the function that I'm using (e.g. the loop). Can I add in a line in my loop (e.g. removeTmpFiles) to delete temporary files created during the loop without ruining the integrity of the raster stack?
For example:
files<-list.files(pattern=".tif")
myStack<-stack(files)
myFunction<-function(x, rasterStack, y, z){
# several calculations using the raster stack and x, y, z
# non-raster results from the calculations get saved to file using write.csv()
removeTmpFiles(h=0) # remove temp. raster files from disk???
}
lapply(x=list_of_x_values, myFunction, rasterStack=myStack, y=something, z=somethingelse)
Is the removeTmpFiles line appropriate here?
Thanks for the help!
[[alternative HTML version deleted]]
_______________________________________________ R-sig-Geo mailing list R-sig-Geo at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Jonathan A. Greenberg, PhD Assistant Professor Global Environmental Analysis and Remote Sensing (GEARS) Laboratory Department of Geography and Geographic Information Science University of Illinois at Urbana-Champaign 259 Computing Applications Building, MC-150 605 East Springfield Avenue Champaign, IL 61820-6371 Phone: 217-300-1924 http://www.geog.illinois.edu/~jgrn/ AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007
6 days later
An embedded and charset-unspecified text was scrubbed... Name: not available URL: <https://stat.ethz.ch/pipermail/r-sig-geo/attachments/20140330/2a3a3e33/attachment.pl>
Hi Julie:
Glad it worked out! Just for the sake of preserving this info on the
interwebs and to add some "tweaks" to your solutions, first, raster's
temporary files are stored in:
rasterOptions()$tmpdir
This folder can be changed to:
rasterOptions(tmpdir="/pathto/new/temp/dir")
Re: list files, if you want to full path, I use:
normalizePath(list.files(pattern=glob2rx("temp*"),full.names =TRUE))
full.names=TRUE will give you the full path to the file, although it
will be a relative path -- normalizePath "fixes" this to the absolute
path. The pattern parameter for list.files is a "regular expression"
(http://en.wikipedia.org/wiki/Regular_expression), which many people
aren't terribly familiar with -- if you are more familiar with the
wildcard approach (your example suggests you are -- I am too), to be
safe, wrap that in a glob2rx() function so it it created properly:
glob2rx("temp*")
The pattern="temp*", since it is interpreted as a regular expression,
is not interpreted the way you probably thought it was, and when you
remove files you should be careful about that. You could end up
deleting different files than you thought. "*" in regular expressions
means something different than it does in wildcard patterns.
Cheers!
--j
On Sun, Mar 30, 2014 at 10:59 AM, Julie Lee-Yaw <julleeyaw at yahoo.ca> wrote:
Hi Jonathan, Thanks for this suggestion. It took me awhile to find all the places where temporary rasters were being generated. After finding them, I used filename in all commands that were generating the files, pointing them all to a temporary directory and calling them all "temp_" something. I then used the following lines of code to remove them at the end of each loop (note file.remove seemed to require me to be in the directory with the files to be removed...) path<-"./myPermanentDirectory" # inside this directory, I had created a temp directory to dump temp files in setwd(paste(path,"/tmpdir",sep="")) # change directory to where the temporary files are do.call(file.remove,list(list.files(pattern="temp*))) setwd(path) # go back to permanent directory Thanks! Julie
________________________________
From: Jonathan Greenberg <jgrn at illinois.edu>
To: Julie Lee-Yaw <julleeyaw at yahoo.ca>
Cc: "r-sig-geo at r-project.org" <r-sig-geo at r-project.org>
Sent: Sunday, March 23, 2014 11:58:50 PM
Subject: Re: [R-sig-Geo] Managing temporary files created when using the
Raster package
Julie:
Why not just build a ?file.remove into your script? Also, most raster
functions allow you to explicitly set a filename (see e.g. ?calc -->
the filename= parameter), rather than letting it generate a random
filename -- this would likely make it easier for you to figure out the
appropriate file to delete.
--j
On Sun, Mar 23, 2014 at 4:44 PM, Julie Lee-Yaw <julleeyaw at yahoo.ca> wrote:
Hi
I am doing a lot of raster calculations repeatedly (in a loop of sorts).
As might be expected, this process is generating a lot of temporary files
and filing up my hard drive before my script finishes. The first step in my
script is to stack a set of very large rasters. This stack is then the input
for the function that I'm using (e.g. the loop). Can I add in a line in my
loop (e.g. removeTmpFiles) to delete temporary files created during the loop
without ruining the integrity of the raster stack?
For example:
files<-list.files(pattern=".tif")
myStack<-stack(files)
myFunction<-function(x, rasterStack, y, z){
# several calculations using the raster stack and x, y, z
# non-raster results from the calculations get saved to file using
write.csv()
removeTmpFiles(h=0) # remove temp. raster files from disk???
}
lapply(x=list_of_x_values, myFunction, rasterStack=myStack, y=something,
z=somethingelse)
Is the removeTmpFiles line appropriate here?
Thanks for the help!
[[alternative HTML version deleted]]
_______________________________________________
R-sig-Geo mailing list
R-sig-Geo at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
--
Jonathan A. Greenberg, PhD
Assistant Professor
Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
Department of Geography and Geographic Information Science
University of Illinois at Urbana-Champaign
259 Computing Applications Building, MC-150
605 East Springfield Avenue
Champaign, IL 61820-6371
Phone: 217-300-1924
http://www.geog.illinois.edu/~jgrn/
AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007
Jonathan A. Greenberg, PhD Assistant Professor Global Environmental Analysis and Remote Sensing (GEARS) Laboratory Department of Geography and Geographic Information Science University of Illinois at Urbana-Champaign 259 Computing Applications Building, MC-150 605 East Springfield Avenue Champaign, IL 61820-6371 Phone: 217-300-1924 http://www.geog.illinois.edu/~jgrn/ AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007