Skip to content

[Rcpp-devel] sourceCpp issue: code vs. file

17 messages · Kevin Ushey, Dirk Eddelbuettel, JJ Allaire +2 more

#
Hi!

I am investigating some of Armadillo's new sparse matrix capabilities.
But, I ran into a problem with sourceCpp that I don't understand.
Here is an example right out of the Rcpp gallery
<http://gallery.rcpp.org/articles/armadillo-sparse-matrix/>

With code= it works as advertised, but it fails with file=
(and this is true with 4 different R installations that I tried):

suppressMessages(library(Matrix))
i <- c(1,3:8)
j <- c(2,9,6:10)
x <- 7 * (1:7)
A <- sparseMatrix(i, j, x = x)
print(A)

require(RcppArmadillo)

sourceCpp(code='
#include <RcppArmadillo.h>
// [[Rcpp::depends(RcppArmadillo)]]

using namespace Rcpp ;

// [[Rcpp::export]]
void convertSparse(S4 mat) { // slight improvement with two non-nested loops
     IntegerVector dims = mat.slot("Dim");
     IntegerVector i = mat.slot("i");
     IntegerVector p = mat.slot("p");
     NumericVector x = mat.slot("x");

     int nrow = dims[0], ncol = dims[1];
     arma::sp_mat res(nrow, ncol);

     // create space for values, and copy
     arma::access::rw(res.values) =
         arma::memory::acquire_chunked<double>(x.size() + 1);
     arma::arrayops::copy(arma::access::rwp(res.values),
                          x.begin(), x.size() + 1);

     // create space for row_indices, and copy -- so far in a lame loop
     arma::access::rw(res.row_indices) =
         arma::memory::acquire_chunked<arma::uword>(x.size() + 1);
     for (int j=0; j<i.size(); j++)
         arma::access::rwp(res.row_indices)[j] = i[j];

     // create space for col_ptrs, and copy -- so far in a lame loop
     arma::access::rw(res.col_ptrs) =
         arma::memory::acquire<arma::uword>(p.size() + 2);
     for (int j=0; j<p.size(); j++)
         arma::access::rwp(res.col_ptrs)[j] = p[j];

     // important: set the sentinel as well
     arma::access::rwp(res.col_ptrs)[p.size()+1] =
         std::numeric_limits<arma::uword>::max();

     // set the number of non-zero elements
     arma::access::rw(res.n_nonzero) = x.size();

     Rcout << "SpMat res:" << res << std::endl;
}')

convertSparse(A)
#Works!
## > SpMat res:[matrix size: 8x10; n_nonzero: 7; density: 8.75%]

##      (0, 1)          7.0000
##      (3, 5)         21.0000
##      (4, 6)         28.0000
##      (5, 7)         35.0000
##      (2, 8)         14.0000
##      (6, 8)         42.0000
##      (7, 9)         49.0000

sourceCpp(verbose=TRUE, rebuild=TRUE, file="~/arma-sp.cxx")
#Fails!
Generated extern "C" functions
--------------------------------------------------------


#include <Rcpp.h>

RcppExport SEXP sourceCpp_72326_convertSparse(SEXP matSEXP) {
BEGIN_RCPP
     Rcpp::RNGScope __rngScope;
     S4 mat = Rcpp::as<S4 >(matSEXP);
     convertSparse(mat);
     return R_NilValue;
END_RCPP
}

Generated R functions
-------------------------------------------------------

`.sourceCpp_72326_DLLInfo` <- 
dyn.load('/tmp/RtmpSgRD0G/sourcecpp_f97a2e266e27/sourceCpp_930.so')

convertSparse <- Rcpp:::sourceCppFunction(function(mat) {}, TRUE, 
`.sourceCpp_72326_DLLInfo`, 'sourceCpp_72326_convertSparse')

rm(`.sourceCpp_72326_DLLInfo`)

Building shared library
--------------------------------------------------------

DIR: /tmp/RtmpSgRD0G/sourcecpp_f97a2e266e27

/opt/local/lib64/R/bin/R CMD SHLIB -o 'sourceCpp_930.so' --preclean 
'arma-sp.cxx'
Error in 
dyn.load("/tmp/RtmpSgRD0G/sourcecpp_f97a2e266e27/sourceCpp_930.so") 
(from arma-sp.cxx.R#1) :
   unable to load shared object 
'/tmp/RtmpSgRD0G/sourcecpp_f97a2e266e27/sourceCpp_930.so':
   /tmp/RtmpSgRD0G/sourcecpp_f97a2e266e27/sourceCpp_930.so: cannot open 
shared object file: No such file or directory
#
Rodney and I have been over this a bit, without resolution. At my urging, he
posted now on the list (as _everybody_ should: more eyeballs are in fact
better as you can tell from the fact that we still have not found this).

On my side, I just works.  The second command is specific to my machine, but
this is just a git checkout of the gallery and the actual file.



R> setwd("/tmp")
R> system("cp -vax ~/git/rcpp-gallery/src/2012-12-25-armadillo-sparse-matrix.cpp .")
`/home/edd/git/rcpp-gallery/src/2012-12-25-armadillo-sparse-matrix.cpp' -> `./2012-12-25-armadillo-sparse-matrix.cpp'
R> sourceCpp("2012-12-25-armadillo-sparse-matrix.cpp")

R>   suppressMessages(library(Matrix))

R>   i <- c(1,3:8) 

R>   j <- c(2,9,6:10) 

R>   x <- 7 * (1:7)

R>   A <- sparseMatrix(i, j, x = x) 

R>   print(A)
8 x 10 sparse Matrix of class "dgCMatrix"
                             
[1,] . 7 . . .  .  .  .  .  .
[2,] . . . . .  .  .  .  .  .
[3,] . . . . .  .  .  . 14  .
[4,] . . . . . 21  .  .  .  .
[5,] . . . . .  . 28  .  .  .
[6,] . . . . .  .  . 35  .  .
[7,] . . . . .  .  .  . 42  .
[8,] . . . . .  .  .  .  . 49

R>   convertSparse(A)
SpMat res:
[matrix size: 8x10; n_nonzero: 7; density: 8.75%]

     (0, 1)          7.0000
     (3, 5)         21.0000
     (4, 6)         28.0000
     (5, 7)         35.0000
     (2, 8)         14.0000
     (6, 8)         42.0000
     (7, 9)         49.0000


R> 


Dirk, puzzled
#
Oops!  I forgot the version info...

 > sessionInfo()
R version 2.15.3 (2013-03-01)
Platform: x86_64-unknown-linux-gnu (64-bit)

locale:
  [1] LC_CTYPE=en_US.ISO8859-1       LC_NUMERIC=C
  [3] LC_TIME=en_US.ISO8859-1        LC_COLLATE=en_US.ISO8859-1
  [5] LC_MONETARY=en_US.ISO8859-1    LC_MESSAGES=en_US.ISO8859-1
  [7] LC_PAPER=C                     LC_NAME=C
  [9] LC_ADDRESS=C                   LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.ISO8859-1 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] RcppArmadillo_0.3.800.1 Rcpp_0.10.3          Matrix_1.0-12
[4] lattice_0.20-15

loaded via a namespace (and not attached):
[1] grid_2.15.3  tools_2.15.3
#
And for completeness, here is the invocation that fails for Rodney:


R> sourceCpp(verbose=TRUE, rebuild=TRUE, file="2012-12-25-armadillo-sparse-matrix.cpp")

Generated extern "C" functions 
--------------------------------------------------------


#include <Rcpp.h>

RcppExport SEXP sourceCpp_44697_convertSparse(SEXP matSEXP) {
BEGIN_RCPP
    Rcpp::RNGScope __rngScope;
    S4 mat = Rcpp::as<S4 >(matSEXP);
    convertSparse(mat);
    return R_NilValue;
END_RCPP
}

Generated R functions 
-------------------------------------------------------

`.sourceCpp_44697_DLLInfo` <- dyn.load('/tmp/Rtmpr8HLMP/sourcecpp_38624d1a5d0a/sourceCpp_73760.so')

convertSparse <- Rcpp:::sourceCppFunction(function(mat) {}, TRUE, `.sourceCpp_44697_DLLInfo`, 'sourceCpp_44697_convertSparse')

rm(`.sourceCpp_44697_DLLInfo`)

Building shared library
--------------------------------------------------------

DIR: /tmp/Rtmpr8HLMP/sourcecpp_38624d1a5d0a

/usr/lib/R/bin/R CMD SHLIB -o 'sourceCpp_73760.so' --preclean '2012-12-25-armadillo-sparse-matrix.cpp' 
ccache g++-4.7 -I/usr/share/R/include -DNDEBUG    -I"/usr/local/lib/R/site-library/Rcpp/include" -I"/usr/local/lib/R/site-library/RcppArmadillo/include" -I"/usr/local/lib/R/site-library/Rcpp/include"   -fpic  -g0 -O3 -Wall -pipe -Wno-variadic-macros -pedantic -c 2012-12-25-armadillo-sparse-matrix.cpp -o 2012-12-25-armadillo-sparse-matrix.o
g++-4.7 -shared -o sourceCpp_73760.so 2012-12-25-armadillo-sparse-matrix.o -llapack -lblas -lgfortran -lm -lquadmath -L/usr/local/lib/R/site-library/Rcpp/lib -lRcpp -Wl,-rpath,/usr/local/lib/R/site-library/Rcpp/lib -lQuantLib -L/usr/lib/R/lib -lR

R>   suppressMessages(library(Matrix))

R>   i <- c(1,3:8) 

R>   j <- c(2,9,6:10) 

R>   x <- 7 * (1:7)

R>   A <- sparseMatrix(i, j, x = x) 

R>   print(A)
8 x 10 sparse Matrix of class "dgCMatrix"
                             
[1,] . 7 . . .  .  .  .  .  .
[2,] . . . . .  .  .  .  .  .
[3,] . . . . .  .  .  . 14  .
[4,] . . . . . 21  .  .  .  .
[5,] . . . . .  . 28  .  .  .
[6,] . . . . .  .  . 35  .  .
[7,] . . . . .  .  .  . 42  .
[8,] . . . . .  .  .  .  . 49

R>   convertSparse(A)
SpMat res:
[matrix size: 8x10; n_nonzero: 7; density: 8.75%]

     (0, 1)          7.0000
     (3, 5)         21.0000
     (4, 6)         28.0000
     (5, 7)         35.0000
     (2, 8)         14.0000
     (6, 8)         42.0000
     (7, 9)         49.0000


R> 

Methinks there may be something particular to your box.

Dirk
#
On 05/22/2013 12:23 PM, Dirk Eddelbuettel wrote:
Possible, but I don't think it is likely.  I found this issue
on the following OSes:  RHEL 5, Oracle 6, CentOS 6 and OS X ML...
with various versions of GCC
#
And here is the failure on OS X ML...
Generated extern "C" functions 
--------------------------------------------------------


#include <Rcpp.h>

RcppExport SEXP sourceCpp_41979_convertSparse(SEXP matSEXP) {
BEGIN_RCPP
    Rcpp::RNGScope __rngScope;
    S4 mat = Rcpp::as<S4 >(matSEXP);
    convertSparse(mat);
    return R_NilValue;
END_RCPP
}

Generated R functions 
-------------------------------------------------------

`.sourceCpp_41979_DLLInfo` <- dyn.load('/var/folders/nb/62qyw7056rj7lwfqbqgmh6lw0000gn/T//RtmpZ7Msbu/sourcecpp_1ade06077c/sourceCpp_88708.so')

convertSparse <- Rcpp:::sourceCppFunction(function(mat) {}, `.sourceCpp_41979_DLLInfo`, 'sourceCpp_41979_convertSparse')

rm(`.sourceCpp_41979_DLLInfo`)

Building shared library
--------------------------------------------------------

DIR: /var/folders/nb/62qyw7056rj7lwfqbqgmh6lw0000gn/T//RtmpZ7Msbu/sourcecpp_1ade06077c

/Library/Frameworks/R.framework/Resources/bin/R CMD SHLIB -o 'sourceCpp_88708.so' --preclean 'arma-sp.cxx' 
Error in dyn.load("/var/folders/nb/62qyw7056rj7lwfqbqgmh6lw0000gn/T//RtmpZ7Msbu/sourcecpp_1ade06077c/sourceCpp_88708.so") (from arma-sp.cxx.R#1) : 
  unable to load shared object '/var/folders/nb/62qyw7056rj7lwfqbqgmh6lw0000gn/T//RtmpZ7Msbu/sourcecpp_1ade06077c/sourceCpp_88708.so':
  dlopen(/var/folders/nb/62qyw7056rj7lwfqbqgmh6lw0000gn/T//RtmpZ7Msbu/sourcecpp_1ade06077c/sourceCpp_88708.so, 6): image not found
R version 2.15.2 (2012-10-26)
Platform: i386-apple-darwin9.8.0/i386 (32-bit)

locale:
[1] C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] RcppArmadillo_0.3.6.1 Rcpp_0.10.2           Matrix_1.0-10         lattice_0.20-13      

loaded via a namespace (and not attached):
[1] compiler_2.15.2 grid_2.15.2     tools_2.15.2   

Thanks for any hints.

Rodney
#
Hi Rodney,

I have the same problem as you on Mac OSX.

I'm able to successfully compile the file through sourceCpp if I use a file
extension of .cpp rather than .cxx. Could this somehow be the reason why?
Let me know if that works for you.

-Kevin
On Wed, May 22, 2013 at 4:27 PM, Sparapani, Rodney <rsparapa at mcw.edu> wrote:

            

  
    
#
I had mentioned earlier to you to maybe look into how TMP, TMPDIR, ... are
used or set.  

My working example has

`.sourceCpp_44697_DLLInfo` <- dyn.load('/tmp/Rtmpr8HLMP/sourcecpp_38624d1a5d0a/sourceCpp_73760.so')

which makes sense -- Rtmp* inside /tmp.  Yours shows

`.sourceCpp_41979_DLLInfo` <- dyn.load('/var/folders/nb/62qyw7056rj7lwfqbqgmh6lw0000gn/T//RtmpZ7Msbu/sourcecpp_1ade06077c/sourceCpp_88708.so')

and that makes less sense. Why 

  /var/folders/nb/62qyw7056rj7lwfqbqgmh6lw0000gn/T//RtmpZ7Msbu

Any idea?

Dirk
#
I also can build successfully on Ubuntu and OSX with the .cpp extension
however the .cxx extension fails. I think the file extension is the likely
culprit.

J.J.
On Wed, May 22, 2013 at 5:13 PM, Kevin Ushey <kevinushey at gmail.com> wrote:

            
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20130522/56d2f73e/attachment-0001.html>
#
On 22 May 2013 at 17:59, JJ Allaire wrote:
| I also can build successfully on Ubuntu and OSX with the .cpp extension however
| the .cxx extension fails. I think the file extension is the likely culprit.

R governs the behaviour of R CMD <whatever> based on the extension.

The .cxx form was /never/ supported by R as far as I know.  

And previously, I had not noticed that Rodney had used this extension; I had
repeatedly suggested he create smaller working examples. I will also note
that ALL files in the Rcpp Gallery are .Rmd (for markdown) or .cpp.  That is
just the way it is -- if you want another extension you may have to create
your own build system.  R hands us a very powerful platform, but asks us to
play by its rules.

Dirk
#
On Wed, May 22, 2013 at 6:41 PM, Dirk Eddelbuettel <edd at debian.org> wrote:

            
Indeed. According to "Package subdirectories" in Writing R Extensions (
http://cran.r-project.org/doc/manuals/R-exts.html#Package-subdirectories)
the only supported extensions are .c, .cc, .cpp, .f, .f90, .f95, .m, and
.mm.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20130522/77714513/attachment.html>
#
On 05/22/2013 08:41 PM, Dirk Eddelbuettel wrote:
Aargh!!!!!!  Thanks guys!
#
So maybe sourceCpp could provide a warning if the extension is not in that list?

Hadley

--
Chief Scientist, RStudio
http://had.co.nz/
#
On 23 May 2013 at 09:03, Hadley Wickham wrote:
| >> R governs the behaviour of R CMD <whatever> based on the extension.
| >>
| >> The .cxx form was /never/ supported by R as far as I know.
| >
| > Indeed. According to "Package subdirectories" in Writing R Extensions
| > (http://cran.r-project.org/doc/manuals/R-exts.html#Package-subdirectories)
| > the only supported extensions are .c, .cc, .cpp, .f, .f90, .f95, .m, and
| > .mm.
| 
| So maybe sourceCpp could provide a warning if the extension is not in that list?

Hmpd. I am not exactly sure why you advocate supporting .c or .f, .f90, .f95,
.m and .mm none of which are likely to work _in any way_ with the parsing
done and scaffolding added by sourceCpp.  When you read 'sourceCpp', mentally
stress the last three letters.

This is for R and C++.  And the file extension is .cpp (with a bit of luck
you may get .cc).  

The rest is user error.  You can't try and catch everything.

Dirk
#
You can't catch everything, but can you try ;)  Given that people are
used to writing C++ with a variety of extensions, it seems like a
simple check here would be useful, especially given the otherwise
opaque nature of the error.

Since

Hadley

--
Chief Scientist, RStudio
http://had.co.nz/
#
On 05/23/2013 09:26 AM, Hadley Wickham wrote:
Hi!

As the original bugbear, I would also like to see this feature.  Or
possibly, just add ".cpp" to any file that does not already have it
before copying it to TEMP.
#
On 05/23/2013 03:01 PM, Rodney Sparapani wrote:
Actually, this latter suggestion is rather easy to implement
and, I believe, simplifies life for spaces in filenames, etc.
It adds 5 lines, but eliminates 7: a win of -2 lines!

sourceCpp <-function (file = "", code = NULL, env = globalenv(), rebuild 
= FALSE,
     showOutput = verbose, verbose = getOption("verbose"))
{
     if (!missing(code)) {
         file <- tempfile(fileext = ".cpp")
         con <- file(file, open = "w")
         writeLines(code, con)
         close(con)
     }
     ## 05/24/13 workaround to ensure a .cpp extension
     else {
       file[2] <- tempfile(fileext = ".cpp")
       file.copy(file[1], file[2])
       file <- file[2]
     }
     ## 05/24/13 as a side effect, no longer needed with this fix
     ## file <- normalizePath(file, winslash = "/")
     ## if (.Platform$OS.type == "windows") {
     ##     if (grepl(" ", basename(file), fixed = TRUE)) {
     ##         stop("The filename '", basename(file), "' contains 
spaces. This ",
     ##             "is not permitted.")
     ##     }
     ## }
     context <- .Call("sourceCppContext", PACKAGE = "Rcpp", file,
         code, rebuild, .Platform)
     if (context$buildRequired || rebuild) {
         if (verbose)
             .printVerboseOutput(context)
         succeeded <- FALSE
         output <- NULL
         depends <- .getSourceCppDependencies(context$depends,
             file)
         .validatePackages(depends, context$cppSourceFilename)
         envRestore <- .setupBuildEnvironment(depends, context$plugins,
             file)
         cwd <- getwd()
         setwd(context$buildDirectory)
         fromCode <- !missing(code)
         if (!.callBuildHook(context$cppSourcePath, fromCode,
             showOutput)) {
             .restoreEnvironment(envRestore)
             setwd(cwd)
             return(invisible(NULL))
         }
         on.exit({
             if (!succeeded) .showBuildFailureDiagnostics()
             .callBuildCompleteHook(succeeded, output)
             setwd(cwd)
             .restoreEnvironment(envRestore)
         })
         if (file.exists(context$previousDynlibPath)) {
             try(silent = T, dyn.unload(context$previousDynlibPath))
             file.remove(context$previousDynlibPath)
         }
         cmd <- paste(R.home(component = "bin"), .Platform$file.sep,
             "R ", "CMD SHLIB ", "-o ", shQuote(context$dynlibFilename),
             " ", ifelse(rebuild, "--preclean ", ""), 
shQuote(context$cppSourceFilename),
             sep = "")
         if (showOutput)
             cat(cmd, "\n")
         result <- suppressWarnings(system(cmd, intern = !showOutput))
         if (!showOutput) {
             output <- result
             attributes(output) <- NULL
             status <- attr(result, "status")
             if (!is.null(status)) {
                 cat(result, "\n")
                 succeeded <- FALSE
                 stop("Error ", status, " occurred building shared 
library.")
             }
             else if (!file.exists(context$dynlibFilename)) {
                 cat(result, "\n")
                 succeeded <- FALSE
                 stop("Error occurred building shared library.")
             }
             else {
                 succeeded <- TRUE
             }
         }
         else if (!identical(as.character(result), "0")) {
             succeeded <- FALSE
             stop("Error ", result, " occurred building shared library.")
         }
         else {
             succeeded <- TRUE
         }
     }
     else {
         if (verbose)
             cat("\nNo rebuild required (use rebuild = TRUE to ",
                 "force a rebuild)\n\n", sep = "")
     }
     if (length(context$exportedFunctions) > 0 || length(context$modules) >
         0) {
         exports <- c(context$exportedFunctions, context$modules)
         removeObjs <- exports[exports %in% ls(envir = env, all.names = T)]
         remove(list = removeObjs, envir = env)
         scriptPath <- file.path(context$buildDirectory, 
context$rSourceFilename)
         source(scriptPath, local = env)
     }
     else if (getOption("rcpp.warnNoExports", default = TRUE)) {
         warning("No Rcpp::export attributes or RCPP_MODULE declarations ",
             "found in source")
     }
     if (length(context$embeddedR) > 0) {
         srcConn <- textConnection(context$embeddedR)
         source(file = srcConn, echo = TRUE)
     }
     invisible(list(functions = context$exportedFunctions, modules = 
context$modules))
}

environment(sourceCpp) <- asNamespace("Rcpp")