[Rcpp-devel] Timings for a program

I am new to Rcpp. I have converted my R program into an Rcpp program.
I want to ask a general question. Using Rcpp, my code is taking about 3
minutes to produce output for 100 observations. But it is taking about 2.45
hours to produce output for 1000 observations. Any suggestions or technical
hint for this type of problem.

First off, please include a minimal working example with your
question. Without more information, there's not much we can do to
help.

I agree with Christian that a reproducible working example is necessary
for detailed help with your problem. But I'm also bold enough to make a
guess about what is going on.

Your program is implementing an algorithm that is not linear in the
sample size but polynomial. You can verify this by recording run times
for sample sizes of 50, 60, 70, 80, 90, and 100. Plotting the run time
against the sample size should show the non-linear relationship. At a
sample size of 1000 the non-linear effects dominate, giving you a
roughly 5-fold increase in the run time over what you expect from your
smaller example.

It was pointed out many years ago by Joel Spolsky that this phenomenon
can arise from using a high level interface without understanding the
low-level implementation details. See his Wikipedia entry:

https://en.wikipedia.org/wiki/Joel_Spolsky#Schlemiel_the_Painter.27s_al
gorithm

Frequent reallocations of existing memory are a typical cause of this
problem.

All of this means that finding the underlying problem might be quite
hard for you to find, which is all the more reason to strip down your
program to find the smallest example that shows this behaviour.

Martyn

In the meantime, you might find this to be an interesting and helpful
read:
http://www.burns-stat.com/documents/books/the-r-inferno/

Best,
Christian-----------------------------------------------------------------------

This message and its attachments are strictly confidential. If you are
not the intended recipient of this message, please immediately notify 
the sender and delete it. Since its integrity cannot be guaranteed, 
its content cannot involve the sender's responsibility. Any misuse, 
any disclosure or publication of its content, either whole or partial, 
is prohibited, exception made of formally approved use
-----------------------------------------------------------------------

Hello Dear

Your suggestions are quite useful. I have deleted two for loops from my
code and trying to use vectors in place of them. It is hard for me to
produce a minimal example. However, now I am trying to use vectorization. I
am having some error. Following is an example code. My basic purpose is
subcube element wise multiplication with matrix's row. The operand % is not
working in this case. Any suggestion would be appreciated.

Here a is a cube of dimension (3, 3, 1) and b is a matrix of dimension (3,
3).

#include <RcppArmadillo.h>
using namespace Rcpp;
using namespace RcppArmadillo;
using namespace arma;
//[[Rcpp::depends(RcppArmadillo)]]
//[[Rcpp::export]]

arma::cube sub(arma::cube a, arma::mat b)
{
        a.subcube(2,0,0,2,2,0) = a.subcube(1, 0, 0, 1, 2, 0) .* b.row(0);
        return a;
}


The error is


subcube.cpp: In function ?arma::cube sub(arma::cube, arma::mat)?:
subcube.cpp:11:65: error: ?arma::Mat<eT>::row(arma::uword) [with eT =
double; arma::uword = unsigned int](0u)? cannot be used as a member
pointer, since it is of type  arma::subview_row<double>?
  a.subcube(2,0,0,2,2,0) = a.subcube(1, 0, 0, 1, 2, 0) .* b.row(0);






Thank You so much again for suggesting how to think about to produce a
speedy code.

Shaami

On Thu, 2017-01-05 at 05:39 -0700, Christian Gunning wrote:

I am new to Rcpp. I have converted my R program into an Rcpp program.
I want to ask a general question. Using Rcpp, my code is taking about 3
minutes to produce output for 100 observations. But it is taking about

2.45

hours to produce output for 1000 observations. Any suggestions or

technical

hint for this type of problem.

First off, please include a minimal working example with your
question. Without more information, there's not much we can do to
help.

I agree with Christian that a reproducible working example is necessary
for detailed help with your problem. But I'm also bold enough to make a
guess about what is going on.

Your program is implementing an algorithm that is not linear in the
sample size but polynomial. You can verify this by recording run times
for sample sizes of 50, 60, 70, 80, 90, and 100. Plotting the run time
against the sample size should show the non-linear relationship. At a
sample size of 1000 the non-linear effects dominate, giving you a
roughly 5-fold increase in the run time over what you expect from your
smaller example.

It was pointed out many years ago by Joel Spolsky that this phenomenon
can arise from using a high level interface without understanding the
low-level implementation details. See his Wikipedia entry:

https://en.wikipedia.org/wiki/Joel_Spolsky#Schlemiel_the_Painter.27s_al
gorithm

Frequent reallocations of existing memory are a typical cause of this
problem.

All of this means that finding the underlying problem might be quite
hard for you to find, which is all the more reason to strip down your
program to find the smallest example that shows this behaviour.

Martyn

In the meantime, you might find this to be an interesting and helpful
read:
http://www.burns-stat.com/documents/books/the-r-inferno/

Best,
Christian---------------------------------------------------

--------------------
This message and its attachments are strictly confidential. If you are
not the intended recipient of this message, please immediately notify
the sender and delete it. Since its integrity cannot be guaranteed,
its content cannot involve the sender's responsibility. Any misuse,
any disclosure or publication of its content, either whole or partial,
is prohibited, exception made of formally approved use
-----------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20170107/f4ea38da/attachment.html>