[Rcpp-devel] float for BigMatrix

Dear List,

I wonder if there is a way to convert a big matrix to ?float? instead of
?double? within a Rcpp program. The reason for using float is mainly for
performance improvement.

For instance, I have a simple function named ?print_bigmat? as shown below.

As shown in the output, ?double_bigmat' will save the correct values of
the original matrix ?x? but not ?float_bigmat?.

// [[Rcpp::export]]
int print_bigmat(SEXP pBigMat) {


    XPtr<BigMatrix> xpMat(pBigMat);


    const mat& double_bigmat = arma::Mat<double>((double
*)xpMat->matrix(), xpMat->nrow(), xpMat->ncol(), false);


    const fmat& float_bigmat = arma::Mat<float>((float *)xpMat->matrix(),
xpMat->nrow(), xpMat->ncol(), false);


    Rcout << double_bigmat << endl;


    Rcout << float_bigmat << endl;


    return 0;
}


Output:

x <- matrix(rnorm(10), nrow=2)
x

            [,1]        [,2]      [,3]       [,4]       [,5]
[1,] -0.05514382 -0.03943825 1.4145593 -0.1161918  2.3282466
[2,] -1.22023371 -0.35592125 0.7714512  0.6865120 -0.3504811

print_bigmat(as.big.matrix(x)@address)

  -0.0551  -0.0394   1.4146  -0.1162   2.3282
  -1.2202  -0.3559   0.7715   0.6865  -0.3505

  -3.3865e-14  -8.6552e+04   4.5441e-07  -5.0912e+23  -1.5184e+34
  -1.3456e+00  -1.9025e+00  -1.2828e+00  -1.6780e+00   1.9268e+00

[1] 0

Thanks much,
Yue

On Jun 2, 2015, at 12:27 AM, Scott Ritchie <sritchie73 at gmail.com> wrote:

Hi Yue,

The call (float *)xpMat->matrix() is simply telling C++ to interpret the stored memory as a float, so it?s simply breaking up the stored binary data into float-sized chunks instead of double-sized chunks, so you get nonsense numbers. To store as a float, you would have to cast double_bigmat to a float type, which I believe makes a new copy of the object after casting each value, so would defeat the purpose. Also big.matrix objects in R only support char, short, int, and double as the underlying storage type (see help(?big.matrix?, ?bigmemory?)), so you wouldn?t be able to access the float matrix from R as a big.matrix (without casting it back to a double) anyway.

Regards,

Scott


On 2 June 2015 at 12:58, Yue Li <gorillayue at gmail.com <mailto:gorillayue at gmail.com>> wrote:
Dear List,

I wonder if there is a way to convert a big matrix to ?float? instead of ?double? within a Rcpp program. The reason for using float is mainly for performance improvement.

For instance, I have a simple function named ?print_bigmat? as shown below.

As shown in the output, ?double_bigmat' will save the correct values of the original matrix ?x? but not ?float_bigmat?.

// [[Rcpp::export]]
int print_bigmat(SEXP pBigMat) {
    
    XPtr<BigMatrix> xpMat(pBigMat);
    
    const mat& double_bigmat = arma::Mat<double>((double *)xpMat->matrix(), xpMat->nrow(), xpMat->ncol(), false);
    
    const fmat& float_bigmat = arma::Mat<float>((float *)xpMat->matrix(), xpMat->nrow(), xpMat->ncol(), false);
    
    Rcout << double_bigmat << endl;
    
    Rcout << float_bigmat << endl;
    
    return 0;
}


Output:

x <- matrix(rnorm(10), nrow=2)
x

            [,1]        [,2]      [,3]       [,4]       [,5]
[1,] -0.05514382 -0.03943825 1.4145593 -0.1161918  2.3282466
[2,] -1.22023371 -0.35592125 0.7714512  0.6865120 -0.3504811

print_bigmat(as.big.matrix(x)@address)

  -0.0551  -0.0394   1.4146  -0.1162   2.3282
  -1.2202  -0.3559   0.7715   0.6865  -0.3505

  -3.3865e-14  -8.6552e+04   4.5441e-07  -5.0912e+23  -1.5184e+34
  -1.3456e+00  -1.9025e+00  -1.2828e+00  -1.6780e+00   1.9268e+00

[1] 0

Thanks much,
Yue

I see. Thanks for the explanation Scott. Wish the bigmatrix will get
native float type. I found a related post here:
https://github.com/kaneplusplus/bigmemory/issues/4

Yue



On Jun 2, 2015, at 12:27 AM, Scott Ritchie <sritchie73 at gmail.com> wrote:

Hi Yue,

The call (float *)xpMat->matrix() is simply telling C++ to interpret the
stored memory as a float, so it?s simply breaking up the stored binary data
into float-sized chunks instead of double-sized chunks, so you get nonsense
numbers. To store as a float, you would have to cast double_bigmat to a
float type, which I believe makes a new copy of the object after casting
each value, so would defeat the purpose. Also big.matrix objects in R only
support char, short, int, and double as the underlying storage type (see
help(?big.matrix?, ?bigmemory?)), so you wouldn?t be able to access the
float matrix from R as a big.matrix (without casting it back to a double)
anyway.

Regards,

Scott
?

On 2 June 2015 at 12:58, Yue Li <gorillayue at gmail.com> wrote:

Dear List,

I wonder if there is a way to convert a big matrix to ?float? instead of
?double? within a Rcpp program. The reason for using float is mainly for
performance improvement.

For instance, I have a simple function named ?print_bigmat? as shown
below.

As shown in the output, ?double_bigmat' will save the correct values of
the original matrix ?x? but not ?float_bigmat?.

// [[Rcpp::export]]
int print_bigmat(SEXP pBigMat) {

    XPtr<BigMatrix> xpMat(pBigMat);

    const mat& double_bigmat = arma::Mat<double>((double
*)xpMat->matrix(), xpMat->nrow(), xpMat->ncol(), false);

    const fmat& float_bigmat = arma::Mat<float>((float
*)xpMat->matrix(), xpMat->nrow(), xpMat->ncol(), false);

    Rcout << double_bigmat << endl;

    Rcout << float_bigmat << endl;

    return 0;
}


Output:

x <- matrix(rnorm(10), nrow=2)
x

            [,1]        [,2]      [,3]       [,4]       [,5]
[1,] -0.05514382 -0.03943825 1.4145593 -0.1161918  2.3282466
[2,] -1.22023371 -0.35592125 0.7714512  0.6865120 -0.3504811

print_bigmat(as.big.matrix(x)@address)

  -0.0551  -0.0394   1.4146  -0.1162   2.3282
  -1.2202  -0.3559   0.7715   0.6865  -0.3505

  -3.3865e-14  -8.6552e+04   4.5441e-07  -5.0912e+23  -1.5184e+34
  -1.3456e+00  -1.9025e+00  -1.2828e+00  -1.6780e+00   1.9268e+00

[1] 0

Thanks much,
Yue

On Jun 2, 2015, at 12:39 AM, Scott Ritchie <sritchie73 at gmail.com> wrote:

Oh nice catch! It?s not in the latest version on CRAN, but if you install the development version from github (devtools::install_github("kaneplusplus/bigmemory")) you will be able to create a big matrix with a type float. You will just need to explicitly tell R to store the data as a float, i.e. as.big.matrix(x, type="float"). Be warned though, your print code will probably segfault when you create the arma::Mat and try to print the matrix.


On 2 June 2015 at 14:30, Yue Li <gorillayue at gmail.com <mailto:gorillayue at gmail.com>> wrote:
I see. Thanks for the explanation Scott. Wish the bigmatrix will get native float type. I found a related post here: https://github.com/kaneplusplus/bigmemory/issues/4 <https://github.com/kaneplusplus/bigmemory/issues/4>

Yue

On Jun 2, 2015, at 12:27 AM, Scott Ritchie <sritchie73 at gmail.com <mailto:sritchie73 at gmail.com>> wrote:

Hi Yue,

The call (float *)xpMat->matrix() is simply telling C++ to interpret the stored memory as a float, so it?s simply breaking up the stored binary data into float-sized chunks instead of double-sized chunks, so you get nonsense numbers. To store as a float, you would have to cast double_bigmat to a float type, which I believe makes a new copy of the object after casting each value, so would defeat the purpose. Also big.matrix objects in R only support char, short, int, and double as the underlying storage type (see help(?big.matrix?, ?bigmemory?)), so you wouldn?t be able to access the float matrix from R as a big.matrix (without casting it back to a double) anyway.

Regards,

Scott


On 2 June 2015 at 12:58, Yue Li <gorillayue at gmail.com <mailto:gorillayue at gmail.com>> wrote:
Dear List,

I wonder if there is a way to convert a big matrix to ?float? instead of ?double? within a Rcpp program. The reason for using float is mainly for performance improvement.

For instance, I have a simple function named ?print_bigmat? as shown below.

As shown in the output, ?double_bigmat' will save the correct values of the original matrix ?x? but not ?float_bigmat?.

// [[Rcpp::export]]
int print_bigmat(SEXP pBigMat) {
    
    XPtr<BigMatrix> xpMat(pBigMat);
    
    const mat& double_bigmat = arma::Mat<double>((double *)xpMat->matrix(), xpMat->nrow(), xpMat->ncol(), false);
    
    const fmat& float_bigmat = arma::Mat<float>((float *)xpMat->matrix(), xpMat->nrow(), xpMat->ncol(), false);
    
    Rcout << double_bigmat << endl;
    
    Rcout << float_bigmat << endl;
    
    return 0;
}


Output:

x <- matrix(rnorm(10), nrow=2)
x

            [,1]        [,2]      [,3]       [,4]       [,5]
[1,] -0.05514382 -0.03943825 1.4145593 -0.1161918  2.3282466
[2,] -1.22023371 -0.35592125 0.7714512  0.6865120 -0.3504811

print_bigmat(as.big.matrix(x)@address)

  -0.0551  -0.0394   1.4146  -0.1162   2.3282
  -1.2202  -0.3559   0.7715   0.6865  -0.3505

  -3.3865e-14  -8.6552e+04   4.5441e-07  -5.0912e+23  -1.5184e+34
  -1.3456e+00  -1.9025e+00  -1.2828e+00  -1.6780e+00   1.9268e+00

[1] 0

Thanks much,
Yue