I use Rcpp::RawVector to allow sending any type of R data-set through network socket after serializing the input data. As I use Rcpp::RawVector, I notice that the "size" data type of the class is defined as int, and I got an error when the input raw byte size is larger than MAX_INT - the error message: "long vectors not supported yet:" Any one has suggestions when I want to process a raw object size that is larger than MAX_INT? thanks,
[Rcpp-devel] using Rcpp::RawVector whose object size is larger than MAX_INT
3 messages · ricky l, Dirk Eddelbuettel, r l
On 10 April 2014 at 11:34, ricky l wrote:
| I use Rcpp::RawVector to allow sending any type of R data-set through | network socket after serializing the input data. | As I use Rcpp::RawVector, I notice that the "size" data type of the | class is defined as int, and I got an error when the input raw byte | size is larger than MAX_INT - the error message: "long vectors not | supported yet:" Any one has suggestions when I want to process a raw | object size that is larger than MAX_INT? thanks, With R 3.0.0, these limits internal to R moved beyond this int barrier. It is entirely possibly that our end has not yet adjusted. If you actually have such a machine with that much RAM, I'd welcome tests from your end :) It should be reasonably straightforward to dive into the code and relax that restriction by going from MAX_INT to the double type and the limit R now uses (which AFAICR is 2^35 rather than 2^31 but I may be off). But I'd be more comfortable making such a change if it was properly tested. Dirk
Dirk Eddelbuettel | edd at debian.org | http://dirk.eddelbuettel.com
That's valid, Dirk. I will check how far I can make it work first. thanks.
On Thu, Apr 10, 2014 at 11:53 AM, Dirk Eddelbuettel <edd at debian.org> wrote:
On 10 April 2014 at 11:34, ricky l wrote: | I use Rcpp::RawVector to allow sending any type of R data-set through | network socket after serializing the input data. | As I use Rcpp::RawVector, I notice that the "size" data type of the | class is defined as int, and I got an error when the input raw byte | size is larger than MAX_INT - the error message: "long vectors not | supported yet:" Any one has suggestions when I want to process a raw | object size that is larger than MAX_INT? thanks, With R 3.0.0, these limits internal to R moved beyond this int barrier. It is entirely possibly that our end has not yet adjusted. If you actually have such a machine with that much RAM, I'd welcome tests from your end :) It should be reasonably straightforward to dive into the code and relax that restriction by going from MAX_INT to the double type and the limit R now uses (which AFAICR is 2^35 rather than 2^31 but I may be off). But I'd be more comfortable making such a change if it was properly tested. Dirk -- Dirk Eddelbuettel | edd at debian.org | http://dirk.eddelbuettel.com
-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20140410/b43fdcda/attachment.html>