R_alloc with more than 2GB (PR#7721)
On Thu, 10 Mar 2005, Wolfgang Huber wrote:
Dear Prof Ripley,
It is a feature. Other parts of R expect a CHARSXP to have length less than or equal to 2^31 - 1.
OK, after looking closer at the code and comments in memory.c and Rinternals.h (typedef int R_len_t;) I realized that.
Could you not use x = allocVector(REALSXP, vs) and REAL(x)[i]? That will get you up to 2^31 - 1 elements, which is the R limit AFAIK.
Thanks, that is an excellent idea. It should be fine for my immediate needs, and better what I've just been doing with Calloc!
Because of the use of Fortran, it is hard to see how to allow internal lengths (in elements, not necessarily bytes) to exceed that value. We need to return to that, but it is not straightforward and last time we discussed it we agreed to defer it.
We can manage a better error message, but I am afraid nothing else in the near future.
In the application that triggered this posting, the memory is for a C array of doubles within a user-defined C function, not for anything that needs to become an R object, so maybe a suggestion would be to make R_alloc go directly to malloc without the detour over allocString or allocVector; or something along that line?
R_alloc makes use of garbage collection to avoid the need for explicit free()ing. Otherwise you might as well use Calloc. Given that all memory allocated via the heap is aligned to doubles, there seems to me to be little or no loss in using a REALSXP rather than a CHARSXP, and certainly negligible loss for large vectors. That will buy us a factor of 8 for the present. Brian
Brian D. Ripley, ripley@stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UK Fax: +44 1865 272595