Skip to content

[Rcpp-devel] Filling a big.matrix in Rcpp

9 messages · Shraddha Pai, Dirk Eddelbuettel, Michael Kane +2 more

Original
Shraddha Pai
# 
Hi all,
I'm trying unsuccessfully to process a big.matrix in a c++ function in the
simple example below.

I first tried to pass the object itself ("bigmat") and got an "Error: not
a matrix" message during compile time. So I'm now trying to pass the
matrix descriptor ("matDesc"). From previous posts on rcpp-devel I *think*
I need to pass it as an external pointer, but this isn't working either.
The compile gives me an error about using matrix operations (nrow()) on a
non-matrix. 

How can I correctly pass a big.matrix reference in a way that the C++
function can dereference it inside the function and use matrix operations?

I've just started using Rcpp (yesterday!), and haven't programmed in C++
for over 10 years now, so my constructs may not be quite right. A related
question: if I wanted to parallel process matrix operations, is it better
to use doMC in the R layer, or something in the C++ layer? Especially if
we're interacting with a big.matrix?
Are there better options (R/Rcpp libraries) for dealing with large vectors
(6M) or matrices (6M x 200), where CPU and memory mgmt are both a concern?

Thanks in advance,
Shraddha
-----
Shraddha Pai
Krembil Family Epigenetic Research Laboratory
Centre for Addiction and Mental Health, Toronto
---------------------------------------------------------------------------
----------


Example:
-----
matFn <- '
Rcpp::XPtr<*NumericMatrix> Am(A);
int nrows = Am.nrow();
int ncolumns = Am.ncol();
for (int j = 0; j < ncolumns; j++){
	for (int i = 1; i < nrows; i++){
		Am(i,j) = Am(i,j) + Am(i-1,j);
	}
}
return Am;
';
# same function declaration as in example 2.
funx <- cxxfunction(signature(A="externalptr"), body=matFn, plugin =
"Rcpp" )

# set up big.matrix
nrows <- 10000
bkFile <- "bigmat.bk"
descFile <- "bigmatk.desc"
suppressMessages(library(bigmemory))
bigmat <- filebacked.big.matrix(nrow=nrows, ncol=3,type="integer", init=1,
	backingfile=bkFile, backingpath=".",descriptorfile=descFile,
	dimnames=c(NULL,NULL))
matDesc <- bigmemory::describe(bigmat)

bigmat2 <- funx(matDesc)



______________________________________________________________________
This email has been scanned by the CAMH Email Security System.
______________________________________________________________________
Michael Kane
# 
Shradda,

A big.matrix is not a regular R matrix. It is a completely separate type.
The definition can be found in BigMatrix.h. I've rewritten your C++ code
but there is still a problem. To build the function the compiler needs to
know where the bigmemory header files are. Basically, you need to #include
"MatrixAccessor". I'm not sure how to do this but it is almost certainly
somewhere in the documentation or maybe someone else on the list will nice
enough to give you a pointer.

Thanks,
Mike

matFn <- '
Rcpp::XPtr<BigMatrix*> bigMat(A);
MatrixAccessor Am(bigMat);

int nrows = bigMat.nrow();
int ncolumns = bigMat.ncol();
for (int j = 0; j < ncolumns; j++){
        for (int i = 1; i < nrows; i++){
                Am[i,j] = Am[i,j] + Am[i-1,j];
        }
}
return A;
';
# same function declaration as in example 2.
funx <- cxxfunction(signature(A="externalptr"), body=matFn,
  plugin = "Rcpp")

# set up big.matrix
nrows <- 10000
bkFile <- "bigmat.bk"
descFile <- "bigmatk.desc"
suppressMessages(library(bigmemory))
bigmat <- filebacked.big.matrix(nrow=nrows, ncol=3,type="integer", init=1,
        backingfile=bkFile, backingpath=".",descriptorfile=descFile,
        dimnames=c(NULL,NULL))
matDesc <- bigmemory::describe(bigmat)

bigmat2 <- funx(matDesc)
On Thu, Mar 14, 2013 at 11:57 AM, Shraddha Pai <Shraddha.Pai at camh.ca> wrote:

            

      
      
      
    

        

      
      
    

        
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20130314/b0e91d57/attachment.html>

  
  


  


      

    

    

      
      

        


  

        

      Dirk Eddelbuettel
    

    

      
      #
    

  


  

      
I do not think we have support for bigmemory's big.matrix objects. So you
can't just assum it works.

Contributions welcome.  This is both 'an obvious one' as it would make sense
at the C++ and R levels, and something that should not be so hard as we
support XPtr. 

Dirk

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Michael Kane
    

    

      
      #
    

  


  

      
OK, I've got it working.

You need to add the bigmemory inst directory to the PKG_CPPFLAGS, include
MatrixAccessor.hpp, and pass the pointer to the BigMatrix object to your
function. The following code should do the trick but note that it does not
create a new big.matrix object it simply modifies the big.matrix object
that is passed to your function.

require(Rcpp)
require(inline)
require(bigmemory)

Sys.setenv(PKG_CPPFLAGS=paste(
  paste("-I", system.file(package="bigmemory"), "/include/bigmemory",
sep=""),
  Sys.getenv("PKG_CPPFLAGS")))

matFn <- '
Rcpp::XPtr<BigMatrix> bigMat(A);
MatrixAccessor<int> Am(*bigMat);

int nrows = bigMat->nrow();
int ncolumns = bigMat->ncol();
for (int j = 0; j < ncolumns; j++){
        for (int i = 1; i < nrows; i++){
                Am[j][i] = Am[j][i] + Am[j][i-1];
        }
}
';
# same function declaration as in example 2.
funx <- cxxfunction(signature(A="externalptr"), body=matFn,   plugin =
"Rcpp", includes='#include "MatrixAccessor.hpp"')

# set up big.matrix
nrows <- 10000
bkFile <- "bigmat.bk"
descFile <- "bigmatk.desc"
suppressMessages(library(bigmemory))
bigmat <- filebacked.big.matrix(nrow=nrows, ncol=3,type="integer", init=1,
        backingfile=bkFile, backingpath=".",descriptorfile=descFile,
        dimnames=c(NULL,NULL))
matDesc <- bigmemory::describe(bigmat)

funx(bigmat at address)

        
On Thu, Mar 14, 2013 at 12:40 PM, Dirk Eddelbuettel <edd at debian.org> wrote:

        

            

      
      
      
    

        

      
      
    

        
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20130314/a2b2e90b/attachment.html>

  
  


  


      

    

    

      
      

        


  

        

      Smith, Dale (Norcross)
    

    

      
      #
    

  


  

      
Write it up for Rcpp Gallery.

 

Dale Smith, Ph.D.

Senior Financial Quantitative Analyst

Risk & Compliance

Fiserv

Office: 678-375-5315

www.fiserv.com <http://www.fiserv.com/> 

 

From: rcpp-devel-bounces at r-forge.wu-wien.ac.at
[mailto:rcpp-devel-bounces at r-forge.wu-wien.ac.at] On Behalf Of Michael
Kane
Sent: Thursday, March 14, 2013 1:19 PM
To: Dirk Eddelbuettel
Cc: rcpp-devel at lists.r-forge.r-project.org
Subject: Re: [Rcpp-devel] Filling a big.matrix in Rcpp

 

OK, I've got it working.

You need to add the bigmemory inst directory to the PKG_CPPFLAGS,
include MatrixAccessor.hpp, and pass the pointer to the BigMatrix object
to your function. The following code should do the trick but note that
it does not create a new big.matrix object it simply modifies the
big.matrix object that is passed to your function.

require(Rcpp)
require(inline)
require(bigmemory)

Sys.setenv(PKG_CPPFLAGS=paste(
  paste("-I", system.file(package="bigmemory"), "/include/bigmemory",
sep=""),
  Sys.getenv("PKG_CPPFLAGS")))

matFn <- '
Rcpp::XPtr<BigMatrix> bigMat(A);
MatrixAccessor<int> Am(*bigMat);

int nrows = bigMat->nrow();
int ncolumns = bigMat->ncol();
for (int j = 0; j < ncolumns; j++){
        for (int i = 1; i < nrows; i++){
                Am[j][i] = Am[j][i] + Am[j][i-1];
        }
}
';
# same function declaration as in example 2.
funx <- cxxfunction(signature(A="externalptr"), body=matFn,   plugin =
"Rcpp", includes='#include "MatrixAccessor.hpp"')

# set up big.matrix
nrows <- 10000
bkFile <- "bigmat.bk"
descFile <- "bigmatk.desc"
suppressMessages(library(bigmemory))
bigmat <- filebacked.big.matrix(nrow=nrows, ncol=3,type="integer",
init=1,
        backingfile=bkFile, backingpath=".",descriptorfile=descFile,
        dimnames=c(NULL,NULL))
matDesc <- bigmemory::describe(bigmat)

funx(bigmat at address)



On Thu, Mar 14, 2013 at 12:40 PM, Dirk Eddelbuettel <edd at debian.org>
wrote:


I do not think we have support for bigmemory's big.matrix objects. So
you
can't just assum it works.

Contributions welcome.  This is both 'an obvious one' as it would make
sense
at the C++ and R levels, and something that should not be so hard as we
support XPtr.

Dirk

--
Dirk Eddelbuettel | edd at debian.org | http://dirk.eddelbuettel.com

_______________________________________________
Rcpp-devel mailing list
Rcpp-devel at lists.r-forge.r-project.org
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20130314/2ba10cdb/attachment-0001.html>

  
  


  


      

    

    

      
      

        


  

        

      Dirk Eddelbuettel
    

    

      
      #
    

  


  

      
On 14 March 2013 at 12:24, Smith, Dale wrote:

        
| Write it up for Rcpp Gallery.

Yup. Our thoughts too.  Here is a slightly modified version which works as a
single cpp file thanks to the wonders of sourceCpp() --- bigmemory headers
are found automagically, and the example is executed too.

Dirk



// -*- mode: C++; c-indent-level: 4; c-basic-offset: 4; indent-tabs-mode: nil; -*-

#include <Rcpp.h>

// The next line is all it takes to find the bigmemory 
// headers -- thanks to the magic of Rcpp attributes

// [[Rcpp::depends(bigmemory)]]
#include <bigmemory/MatrixAccessor.hpp>


// We define a simple function, and pass the incoming XPtr as a SEXP;
// we could also pass a templated XPtr. Function returns only a bool.

// [[Rcpp::export]]
bool fun(SEXP A) {
    Rcpp::XPtr<BigMatrix> bigMat(A);
    MatrixAccessor<int> Am(*bigMat);
    
    int nrows = bigMat->nrow();
    int ncolumns = bigMat->ncol();
    for (int j = 0; j < ncolumns; j++){
	for (int i = 1; i < nrows; i++){
            Am[j][i] = Am[j][i] + Am[j][i-1];
        }
    }
    return Rcpp::wrap(true);
}

// R code for testing below

/*** R
require(bigmemory)

# set up big.matrix
nrows <- 10000
bkFile <- "bigmat.bk"
descFile <- "bigmatk.desc"
bigmat <- filebacked.big.matrix(nrow=nrows, ncol=3,type="integer", init=1,
				backingfile=bkFile, backingpath=".",descriptorfile=descFile,
				dimnames=c(NULL,NULL))
matDesc <- bigmemory::describe(bigmat)

fun(bigmat at address)

cat("Done\n")
*/

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Romain Francois
    

    

      
      #
    

  


  

      
Le 14 mars 2013 ? 19:33, Dirk Eddelbuettel <edd at debian.org> a ?crit :

            

      
      
      
    

        
Why do you use wrap here ? Wrap will create a SEXP, not a bool !

Actually why is this not void ?

Romain

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Dirk Eddelbuettel
    

    

      
      #
    

  


  

      
On 15 March 2013 at 00:51, Romain Francois wrote:

        
| Why do you use wrap here ?

Reflex, because I returned an atomistic type.

| Wrap will create a SEXP, not a bool !

Quite right but was in a rush while doing other things too. "Forgot" that I
was using sourceCpp() which give one additional layer of indirection.
 
| Actually why is this not void ?

Quite right.  Better version below. Thanks!  

Ultimately, we do of course want a version that takes an XPtr to a big.matrix
and does something sensible and returns it.

Dirk


// -*- mode: C++; c-indent-level: 4; c-basic-offset: 4; indent-tabs-mode: nil; -*-

#include <Rcpp.h>

// The next line is all it takes to find the bigmemory 
// headers -- thanks to the magic of Rcpp attributes

// [[Rcpp::depends(bigmemory)]]
#include <bigmemory/MatrixAccessor.hpp>


// We define a simple function, and pass the incoming XPtr as a SEXP;
// we could also pass a templated XPtr. Function returns only a bool.

// [[Rcpp::export]]
void fun(SEXP A) {
    Rcpp::XPtr<BigMatrix> bigMat(A);
    MatrixAccessor<int> Am(*bigMat);
    
    int nrows = bigMat->nrow();
    int ncolumns = bigMat->ncol();
    for (int j = 0; j < ncolumns; j++){
	for (int i = 1; i < nrows; i++){
            Am[j][i] = Am[j][i] + Am[j][i-1];
        }
    }
    return;
}

// R code for testing below

/*** R
require(bigmemory)

# set up big.matrix
nrows <- 10000
bkFile <- "bigmat.bk"
descFile <- "bigmatk.desc"
bigmat <- filebacked.big.matrix(nrow=nrows, ncol=3,type="integer", init=1,
				backingfile=bkFile, backingpath=".",descriptorfile=descFile,
				dimnames=c(NULL,NULL))
matDesc <- bigmemory::describe(bigmat)

fun(bigmat at address)

cat("Done\n")
*/

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Shraddha Pai
    

    

      
      #
    

  


  

      
Hi all,
Thanks for all the responses and for clarifying what big matrices are.

I had actually looked through the rcpp-devel site before attempting to use
big.matrix with Rcpp. I probably misunderstood this thread about
bigmemory; it sounded like bigmemory objects worked "out-of-the-box" with
Rcpp (and maybe Xptr could play a role).
http://r.789695.n4.nabble.com/looking-for-adice-on-bigmemory-framework-with
-C-and-java-interoperability-td4610333.html

I presume the below is consistent with Michael's earlier post suggesting
use of a BigMatrix pointer, and that the old thread was just discussed the
data structure of big.matrix objects (rather than that they can be used as
"just doubles").

            

      
      
      
    

        
I'll try the code posted below and confirm.

Regards,
Shraddha

        
On 13-03-14 8:57 PM, "Dirk Eddelbuettel" <edd at debian.org> wrote:

        

            

      
      
      
    

        

      
      
    

        
______________________________________________________________________
This email has been scanned by the CAMH Email Security System.
______________________________________________________________________