[Rcpp-devel] Exporting rcpp-based function into parLapply workers in an R package

clusterExport works just fine if you put your Rcpp code into your own
package and make that package available in your worker environment. Given
the need for compilation in possibly a variety of computing environments
for parallel processing this is definitely recommended.

On May 14, 2021 10:35:25 AM PDT, Naeem Khoshnevis <
khoshnevis.naeem at gmail.com> wrote:

Hi Michael,


Thank you so much for your response. That is correct. One method for
exporting required variables/functions is using the clusterExport
function,
which does not work for Rcpp-based functions. Another option is using
clusterEvalQ (as mentioned in the shared post); however, I am not sure
if
CRAN likes to see the library(package name) inside the codebase. What
are
your thoughts?

Best regards,
Naeem

On Fri, May 14, 2021 at 11:57 AM Michael Weylandt <
michael.weylandt at gmail.com> wrote:

Hi Naeem,

My (very quick) guess is that this isn't an Rcpp problem per se, but

a

parLapply problem. You need to explicitly load your package on each
worker so that functions from it are available.

See, e.g., the brief discussion here:

https://stackoverflow.com/questions/18357788/parallel-parlapply-setup#18358875

The "parallel" packages do not exactly replicate your environment on
each worker node (to avoid expensive set-up / communication costs) so
you need to do a bit more set-up.

Best,
Michael

On Fri, May 14, 2021 at 11:49 AM Naeem Khoshnevis
<khoshnevis.naeem at gmail.com> wrote:

Dear Rcpp developers:

Thanks for developing and maintaining the Rcpp package.
I wrote a function in Rcpp. It is available throughout the package

and

works as expected; however, it is not available for praLapply

workers. A

temporary fix is just using Rcpp::cppFunction inside the function

that

parLapply workers call and copy the entire function over there.

However,

this does not seem right for bigger and more complicated functions.

I would be grateful if you could let me know whether there is a

better

long-term solution. Here is the package and three functions that you

might

want to take a look at.

Original cpp function:

https://github.com/fasrc/CausalGPS/blob/master/src/compute_closest_wgps_helper.cpp

Wrapper function that calls this function + temporal fix:

https://github.com/fasrc/CausalGPS/blob/master/R/compute_closest_wgps.R

The function that uses parLapply (please see line 63-89) to run the

c++

code:

https://github.com/fasrc/CausalGPS/blob/master/R/compute_closest_wgps.R

Best regards,
Naeem

https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel

--
Sent from my phone. Please excuse my brevity.