[Rcpp-devel] Use size_t rather than int for vector si zes and indexing?
Thanks for the pointer Dirk (or in this case Rcpp::XPtr), I'll definitely take a look at this. Have to go back to the drawing board a bit as I'd currently been using a lot of built-in R functions so switching to bigmemory will be a bit non-trivial. I promise I'll make a demo if I get that far though :) Thanks again for a great package and all the hard work! -Chuck On Wed, 13 Apr 2011 13:38:06 -0500, Dirk Eddelbuettel <edd at debian.org> wrote:
Hi Chuck, Thanks for posting here. On 13 April 2011 at 12:22, chuck at chuckcode.com wrote: | Thanks Jay for the lightning fast insightful response! Didn't realize that | this limitation was intrinsic to R itself. I'll have to go look at | bigmemory and some of the other projects. Yes, Rcpp uses 'proxy model' classes around the underlying SEXP objects.
It
really is just R under it, and we inherit all its advantages and
oddities.
Such as int32 indexing for vectors. The bigmemory package can be a way out, and as it interfaces R using external pointers, you should also be able to hook into this via the Rcpp::XPtr class. That would make for a great new demo or example. Feel like cooking one up while you work through this? ;-) Cheers, Dirk | | Thanks, | -Chuck | | On Wed, 13 Apr 2011 14:11:25 -0400, Jay Emerson <jayemerson at gmail.com> | wrote: | > Chuck, | > | > Internally, R is using 4-byte integers for indexing, and the length
of
| > a vector is thus constrained to 2-billion-ish elements. Two ways | > around this include packages ff and | > bigmemory, for example, or relying on database-like queries.
However,
| > the resulting objects can't be used with standard R functions (with | > some special exceptions). | > | > Jay | > | > | > On Wed, Apr 13, 2011 at 2:05 PM, <chuck at chuckcode.com> wrote: | >> | >> Hi All, | >> | >> Thanks to Dirk Eddelbuettel and the other contributors for such a | >> wonderful package. Rcpp really transforms the way that I've been
able
| to | >> incorporate c++ code into R. Makes it possible to speed up the critical | >> computations while keeping all the great flexibility and features
of
R. | >> | >> I've run into a problem lately with particularly large vectors in Rcpp. | I | >> seem to be overflowing when my vectors get larger than 2^31
elements
on | a | >> 64 bit system. It looks from the code of both the classic (included | >> below) | >> and the newer versions as though this is due to using ints rather
than
| >> something like size_t (http://en.wikipedia.org/wiki/Size_t) as the type | >> for | >> size and indexing into the vector. It looks like RcppResultsSet's add() | >> functions may also be using ints. | >> | >> Curious to know if there is a particular reason for using ints
rather
| >> than
| >> something like size_t and if the project managers would be open to
| >> changing
| >> to accomodate larger vectors.
| >>
| >> Thanks,
| >> -Chuck Sugnet
| >>
| >> template <typename T>
| >> class RcppVector {
| >> public:
| >> ? ? ? ?typedef T* iterator ;
| >> ? ? ? ?typedef const T* const_iterator ;
| >>
| >> ? ?RcppVector(SEXP vec);
| >> ? ?RcppVector(int len);
| >> ? ?int size() const;
| >> ? ?T& operator()(int i) const;
| >> ? ?T *cVector() const;
| >> ? ?std::vector<T> stlVector() const;
| >>
| >> ? ?inline const_iterator begin() const { return v ; }
| >> ? ?inline const_iterator end() const { return v + len ; }
| >>
| >> ? ?inline iterator begin(){ return v ; }
| >> ? ?inline iterator end(){ return v + len ; }
| >>
| >> private:
| >> ? ?int len;
| >> ? ?T *v;
| >> };
| >> _______________________________________________
| >> Rcpp-devel mailing list
| >> Rcpp-devel at lists.r-forge.r-project.org
| >>
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
| >>
| _______________________________________________
| Rcpp-devel mailing list
| Rcpp-devel at lists.r-forge.r-project.org
|
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel