foreach package - Parallel Apply (Section 5.2 in Vignettes foreach Manual)

library(iterators)
x <- matrix(rnorm(16), 4, 4)
it <- iter(x, chunksize=2)
print(as.list(it))

On 10 December 2009 at 09:12, Debabrata Midya wrote:
| Dear hpc users,
|
| Thanks in advance.
|
| I am using R 2.10.0 on Windows XP, Intel(R) Core(TM) 2 Duo CPU E8400 @3.00GHz 2.99GHz, 1.96 of RAM.
|
| I need you assistance to solve the problem below:
|
| foreach package - Parallel Apply (Section 5.2 in Vignettes foreach Manual)
|
| "Now we're only sending any given column of the matrix to one parallel execution worker. But it
| would be even more efficient if we sent the matrix in bigger chunks. To do that, we use a function
| called iblkcol that returns an iterator that will return multiple columns of the original matrix.
| That means that the R expression will need to execute the user's function once for every column in
| its submatrix".
|
| The function below is from the above Vigenette:
|
| applyKernel <- function(newX, FUN, d2, d.call, dn.call = NULL, ...) {
| ?foreach(x = iblkcol(newX, 3), .combine = "c", .packages = "foreach") %dopar%
| ?{
| ? ? foreach(i = 1:ncol(x)) %do% FUN(array(x[, i], d.call, dn.call), ...)
| ?}
| }
|
| > applyKernel(matrix(1:16, 4), mean, 4, 4)
|
| Error in eval(expr, envir, enclos) : could not find function "iblkcol"
|
| How can I get this function?

I am not sure how familiar you are with vignettes but the key is that they
contain all the code. ?So here it is, straight from the package sources. The
<<...>> and @ lines are the Sweave markers, R code is in-between.

<<ex13.iter, results=hide, echo=FALSE>>=
iblkcol <- function(a, chunks) {
?n <- ncol(a)
?i <- 1

?nextElem <- function() {
? ?if (chunks <= 0 || n <= 0) stop('StopIteration')
? ?m <- ceiling(n / chunks)
? ?r <- seq(i, length=m)
? ?i <<- i + m
? ?n <<- n - m
? ?chunks <<- chunks - 1
? ?a[,r, drop=FALSE]
?}

?structure(list(nextElem=nextElem), class=c('iblkcol', 'iter'))
}
nextElem.iblkcol <- function(obj) obj$nextElem()
@


Use the sources for R and its packages. They are a much-underused treasure.

Dirk

|
| Once again, thank you very much for the time you have given.
|
|
| I am looking forward for your reply.
|
| Regards,
|
| Deb
|
| ? ? ? [[alternative HTML version deleted]]
|
| _______________________________________________
| R-sig-hpc mailing list
| R-sig-hpc at r-project.org
| https://stat.ethz.ch/mailman/listinfo/r-sig-hpc

--
Three out of two people have difficulties with fractions.

library(iterators)
x <- matrix(rnorm(16), 4, 4)
it <- iter(x, by='row', chunksize=2)
print(as.list(it))

Stephen,

Thanks for such a quick response. Your advice will assist me a lot.

One clarification please.

"... allows you to iterate over multiple columns at a time. ...".

Is it possible to do over multiple rows at a time?

Once again, thank you very much for the time you have given.
I am looking forward for your reply.

Regards,
Deb

Stephen Weston <stephen.b.weston at gmail.com> 10/12/2009 10:07 am >>>

It's worth noting that the iter method for matrices also allows you to
iterate
over multiple columns at a time.? But instead of specifying the number
of blocks, you have to specify the number of columns in each block.

Here's an example:

library(iterators)
x <- matrix(rnorm(16), 4, 4)
it <- iter(x, chunksize=2)
print(as.list(it))

[[1]]
?????????? [,1]?????? [,2]
[1,]? 0.1811217? 1.7126129
[2,]? 0.1415969? 0.6980985
[3,] -1.3921071 -0.9700261
[4,] -1.2338734 -0.5530707

[[2]]
?????????? [,1]??????? [,2]
[1,] -1.1162171? 1.09920729
[2,]? 0.9893546 -0.37677605
[3,] -0.5508948 -0.02057267
[4,] -0.3817569 -1.53422901

The other method is more convenient when you want to
split up the work evenly between all of the parallel workers,
however.

- Steve


On Wed, Dec 9, 2009 at 5:38 PM, Dirk Eddelbuettel <edd at debian.org> wrote:

On 10 December 2009 at 09:12, Debabrata Midya wrote:
| Dear hpc users,
|
| Thanks in advance.
|
| I am using R 2.10.0 on Windows XP, Intel(R) Core(TM) 2 Duo CPU E8400
@3.00GHz 2.99GHz, 1.96 of RAM.
|
| I need you assistance to solve the problem below:
|
| foreach package - Parallel Apply (Section 5.2 in Vignettes foreach
Manual)
|
| "Now we're only sending any given column of the matrix to one parallel
execution worker. But it
| would be even more efficient if we sent the matrix in bigger chunks. To
do that, we use a function
| called iblkcol that returns an iterator that will return multiple
columns of the original matrix.
| That means that the R expression will need to execute the user's
function once for every column in
| its submatrix".
|
| The function below is from the above Vigenette:
|
| applyKernel <- function(newX, FUN, d2, d.call, dn.call = NULL, ...) {
|? foreach(x = iblkcol(newX, 3), .combine = "c", .packages = "foreach")
%dopar%
|? {
|???? foreach(i = 1:ncol(x)) %do% FUN(array(x[, i], d.call, dn.call), ...)
|? }
| }
|
| > applyKernel(matrix(1:16, 4), mean, 4, 4)
|
| Error in eval(expr, envir, enclos) : could not find function "iblkcol"
|
| How can I get this function?

I am not sure how familiar you are with vignettes but the key is that they
contain all the code.? So here it is, straight from the package sources.
The
<<...>> and @ lines are the Sweave markers, R code is in-between.

<<ex13.iter, results=hide, echo=FALSE>>=
iblkcol <- function(a, chunks) {
? n <- ncol(a)
? i <- 1

? nextElem <- function() {
??? if (chunks <= 0 || n <= 0) stop('StopIteration')
??? m <- ceiling(n / chunks)
??? r <- seq(i, length=m)
??? i <<- i + m
??? n <<- n - m
??? chunks <<- chunks - 1
??? a[,r, drop=FALSE]
? }

? structure(list(nextElem=nextElem), class=c('iblkcol', 'iter'))
}
nextElem.iblkcol <- function(obj) obj$nextElem()
@


Use the sources for R and its packages. They are a much-underused
treasure.

Dirk

|
| Once again, thank you very much for the time you have given.
|
|
| I am looking forward for your reply.
|
| Regards,
|
| Deb
|
|?????? [[alternative HTML version deleted]]
|
| _______________________________________________
| R-sig-hpc mailing list
| R-sig-hpc at r-project.org
| https://stat.ethz.ch/mailman/listinfo/r-sig-hpc

--
Three out of two people have difficulties with fractions.