Rmpi long vector support

n=6e8
x=integer(n)
a=serialize(x,NULL)
length(a)

n=6e8
x=integer(n)
a=serialize(x,NULL)

Jim,

On Aug 7, 2013, at 11:59 AM, Jim Gattiker <j.gattiker at gmail.com> wrote:

To bcast.Robj, Rmpi uses serialize() to pack the object into "raw",
which
operates as a vector of bytes. R supports vectors up to 2^31 elements.

that is not true. R supports vectors up to 2^52 elements. That is way
beyond current RAM sizes and certainly more than 2^31:

n=6e9
x=integer(n)
a=serialize(x,NULL)
length(a)

[1] 2.4e+10

log2(length(a))

[1] 34.48232

However Rmpi does not. You have to use XLENGTH and R_xlen_t in the C code
if you want to go beyond 2^31.

Cheers,
Simon

Your
object, serialized, is close to that. I'm a little puzzled though,
350x350000 matrix works for me; perhaps there's more to your actual
call.

Setting up the slave environment explicitly seems to me to be better
practice, i.e. using mpi.bcast.Robj2Slave and related calls, then the
applyLB() doesn't contain the data. As I'm reading it, I think the way
this
is set up currently will send the data to a slave for each application
of
the apply.

If you're facing larger data sizes, a solution is to explicitly cut up
the
object, mpi.bcast.Robj2Slave the pieces in turn, and then use an
mpi.bcast.cmd to collect them together. Another approach would be to
write
to data to a file, and direct the slaves to read it into their
environments
with an mpi.bcast.cmd.

As a comment: I can't tell your application from the code, but if the
slaves are to be working each on only part of the data, it's better to
send
the slave just the part of the data it needs.

   cheers,
      jim




On Tue, Aug 6, 2013 at 6:37 PM, Ei-ji Nakama <nakama at ki.rim.or.jp>
wrote:

Hi,

<WARNING>
It is not yet completed...
</WARNING>
http://prs.ism.ac.jp/~nakama/Rhpc/

2013/8/7, Xiaochun Sun <xiaoch.sun at gmail.com>:

The same code worked fine for smaller dataset, such as 350x25000. Does

that

mean the current Rmpi don't allow long vectors to be passed to slaves?

Any

idea on that or any alternatives to Rmpi?

Its can treat the considerably big data.

--
EI-JI Nakama  <nakama (a) ki.rim.or.jp>
"\u4e2d\u9593\u6804\u6cbb"  <nakama (a) ki.rim.or.jp>

	[[alternative HTML version deleted]]