Converting large RasterStack to CSVs fast

Hello Comunello!
Thanks for the reply. Your vectorized code is faster than the one I have, by a margin of 5%, which is an improvement. I think the limiting step is clipping. Is there any paralleled? version of clip?
Mohammad
?
From: "r-sig-geo-request at r-project.org" <r-sig-geo-request at r-project.org>
 To: r-sig-geo at r-project.org 
 Sent: Saturday, May 16, 2015 12:00 PM
 Subject: R-sig-Geo Digest, Vol 141, Issue 16

Send R-sig-Geo mailing list submissions to
??? r-sig-geo at r-project.org

To subscribe or unsubscribe via the World Wide Web, visit
??? https://stat.ethz.ch/mailman/listinfo/r-sig-geo
or, via email, send a message with subject or body 'help' to
??? r-sig-geo-request at r-project.org

You can reach the person managing the list at
??? r-sig-geo-owner at r-project.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of R-sig-Geo digest..."

Today's Topics:

? 1. Re: Issue with ogrInfo (?der Comunello)
? 2. Re: Converting large RasterStack to CSVs fast (?der Comunello)
? 3. create a shapefile (Gustavo Dalposso)
? 4. Re: create a shapefile (Felinto COSTA)

----------------------------------------------------------------------

Message: 1
Date: Fri, 15 May 2015 07:41:57 -0400
From: ?der Comunello <comunello.eder at gmail.com>
To: r-sig-geo at r-project.org
Subject: Re: [R-sig-Geo] Issue with ogrInfo
Message-ID:
??? <CABmC8g=ZAFGTu=RJ6BaZjD76gstYwrRJ=rRPqPCuE6WDKVkaqQ at mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"

Hello,

I think the problem is that your link is recovering only html code from
github site. It's necessary to modify it to get the "real" JSON file.

### <code r>
sapply(c("rgeos", "maptools", "rgdal"), require, char=T)
url0 <- "
https://github.com/kjhealy/uk-elections/blob/master/maps/topo_wpc.json"
download.file(url0, dest=basename(url0), mode="wb")
# downloaded 26 KB

uk.map <- readOGR(basename(url0), "wpc")
# Error in ogrInfo(dsn = dsn, layer = layer, encoding = encoding, use_iconv
= use_iconv,? :
#? Cannot open file

url1 <- "
https://raw.githubusercontent.com/kjhealy/uk-elections/master/maps/topo_wpc.json
"
download.file(url1, dest=basename(url0), mode="wb")
# downloaded 2.3 MB

uk.map <- readOGR(basename(url1), "wpc")
# OGR data source with driver: GeoJSON
# Source: "topo_wpc.json", layer: "wpc"
# with 632 features
# It has 2 fields

plot(uk.map)
### </code>

?der Comunello <c <comunello.eder at gmail.com>omunello.eder at gmail.com>
Dourados, MS - [22 16.5'S, 54 49'W]

??? [[alternative HTML version deleted]]

------------------------------

Message: 2
Date: Fri, 15 May 2015 12:35:44 -0400
From: ?der Comunello <comunello.eder at gmail.com>
To: "r-sig-geo at r-project.org" <r-sig-geo at r-project.org>
Subject: Re: [R-sig-Geo] Converting large RasterStack to CSVs fast
Message-ID:
??? <CABmC8gmd8aV6WBRpXRGnwbmRht8Z_MX3WWd9=N999RzoaZVoRA at mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"

Hello, Mohammad!

You may have some improvement in performance avoiding "for statements" and
using a "vectorized" code. You could try something like the code below.

If you can test with your data, i would appreciate if you inform the
results.

### <code r>
require(rgdal); require(raster)

getwd()
### download some data to test
getData("worldclim", var = "tmin", res = 10) ### tmin
fn <- dir("wc10", patt=".bil$", full=T)
fn <- fn[order(nchar(fn), fn)]; fn
#? [1] "wc10/tmin1.bil"? "wc10/tmin2.bil"? "wc10/tmin3.bil"? ...

### read images
s <- stack(fn) ### dimensions? : 900, 2160, 1944000, 12? (nrow, ncol,
ncell, nlayers)
fromDisk(s)

### extents of subsets
bor <- extent(s); bor
res <- 45 ### subsets resolution
X? <- unique(c(seq(bor at xmin, bor at xmax, by=res), bor at xmax)); X
Y? <- unique(c(seq(bor at ymin, bor at ymax, by=res), bor at ymax)); Y
ext <- cbind(expand.grid(Xmin=X[-length(X)], Ymin=Y[-length(Y)]),
? ? ? ? ? ? expand.grid(Xmax=X[-1], Ymax=Y[-1]))[,c(1,3,2,4)]
head(ext); nrow(ext)

plot(s, 1)
system.time(
sapply(1:nrow(ext), function(i) {
? ? mask <- ext[i,]
? ? subset <- with(mask, extent(c(Xmin, Xmax, Ymin, Ymax)))
? ? plot(subset, add=T)
? ? text(rowMeans(mask[,1:2]), rowMeans(mask[,3:4]), lab=i)
? ? c <- crop(s, subset)
? ? write.table(as.data.frame(rasterToPoints(c)), paste0("p",i,".txt"), )
}))
#? ? user? system elapsed
#? 213.79? ? 7.00? 224.94

txt <- dir(patt="^p[0-9]+.txt$")
txt <- txt[order(nchar(txt), txt)]; txt
#? [1] "p1.txt"? "p2.txt"? "p3.txt"? "p4.txt"? ...

### </code>

Cheers,

?der Comunello <c <comunello.eder at gmail.com>omunello.eder at gmail.com>
Dourados, MS - [22 16.5'S, 54 49'W]

?der Comunello <c <comunello.eder at gmail.com>omunello.eder at gmail.com>
Dourados, MS - [22 16.5'S, 54 49'W]

2015-05-15 5:43 GMT-04:00 Mohammad Abdel-Razek via R-sig-Geo <
r-sig-geo at r-project.org>: