looking for adice on bigmemory framework with C++ and java interoperability
Andre,
On May 4, 2012, at 9:50 PM, andre zege wrote:
I work with problems that have rather large data requirements -- typically a bunch of multigig arrays. Given how generous R is with using memory, the only way for me to work with R has been to use bigmatrices from bigmemory package. One thing that is missing a bit is interoperability of bigmatrices with C++ and possibly java. What i mean by that is API that would allow read and write filebacked matrices from C++, and ideally java without being called from R. Having ability to save armadillo matrices into filebacked matrices and load them back into armadillo would be another very useful thing. This would allow really smooth cooperation between various pieces of software. I would prefer to avoid using Rinside for that. I guess i could hack bigmemory C++ code a bit, compile it into a C++ shared library and it'll do. I guess i could hack it a bit to work with armadillo matrices as well. I don't want however to reinvent the wheel and if there is something like that already somewhere i would rather use it for the moment. Looking very much for suggestions. If there is truly nothing like that and someone with C++ or especially java development experience is interested and want to cooperate on this, let me know too.
bigmemory matrices are simply arrays of native types (typically doubles, but bm supports other types, too) so they are trivially readable/writable from both C++ (just read into memory and cast to the array type) and Java (e.g, DoubleBuffer view on a ByteBuffer). So the question is what exactly is the problem? Cheers, Simon
Best Andre NB. I guess something like what i want -- access to the same disc caches from R, C++, and java (and python) exists in HDF world. I, however, don't know how performance of HDF compares with bigmemory matrices, which i come to like and appreciate a lot. If there is someone who could address simplicity of use and performance of HDF vs bigmemory, it'd be very interesting. [[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel