Hello, The Writing R extensions manual section 6.1.1 describes the transient memory allocation function R_alloc, and states that memory allocated by R_alloc is automatically freed after the .C or .Call function is completed. However, based on my understanding of R's memory handling, as well as some test functions I have written, I suspect that this is not quite accurate. If the .Call function returns an external pointer to something created with R_alloc, then this object seems to stick around after the .Call function is completed, and is subject to garbage collection once the external pointer object is removed. Does anyone know, can I count on this behavior on any platform? It would certainly be useful for me. ie, Can I create an external pointer to something created with R_alloc, and trust that it will not be free'd until the external pointer object is removed? And if so, should the manual be edited to describe this behavior? Thanks, Melissa Hubisz
transient memory allocation and external pointers
8 messages · Melissa Jane Hubisz, Simon Urbanek, Luke Tierney +1 more
On Apr 19, 2010, at 10:39 AM, Melissa Jane Hubisz wrote:
Hello, The Writing R extensions manual section 6.1.1 describes the transient memory allocation function R_alloc, and states that memory allocated by R_alloc is automatically freed after the .C or .Call function is completed. However, based on my understanding of R's memory handling, as well as some test functions I have written, I suspect that this is not quite accurate. If the .Call function returns an external pointer to something created with R_alloc, then this object seems to stick around after the .Call function is completed, and is subject to garbage collection once the external pointer object is removed.
Yes, because the regular rules for the lifetime of an R object apply since it is in fact an R object. It is subject to garbage collection so if you assign it anywhere its lifetime will be tied to that object (in your example EXTPTRSXP). Although this is true in general (because that is the only way how it can be safely managed by the memory system), I'm not sure it is guaranteed by the API - i.e. it could be changed at any point to an arbitrary memory location which does not necessarily have that semantics. So you can decide to run with it but the fact that this is undocumented means it is not guaranteed to stay that way forever so you may need to change your code if it does. Cheers, Simon
Does anyone know, can I count on this behavior on any platform? It would certainly be useful for me. ie, Can I create an external pointer to something created with R_alloc, and trust that it will not be free'd until the external pointer object is removed? And if so, should the manual be edited to describe this behavior? Thanks, Melissa Hubisz
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
On 4/19/10 8:59 AM, Simon Urbanek wrote:
On Apr 19, 2010, at 10:39 AM, Melissa Jane Hubisz wrote:
Hello, The Writing R extensions manual section 6.1.1 describes the transient memory allocation function R_alloc, and states that memory allocated by R_alloc is automatically freed after the .C or .Call function is completed. However, based on my understanding of R's memory handling, as well as some test functions I have written, I suspect that this is not quite accurate. If the .Call function returns an external pointer to something created with R_alloc, then this object seems to stick around after the .Call function is completed, and is subject to garbage collection once the external pointer object is removed.
Yes, because the regular rules for the lifetime of an R object apply since it is in fact an R object. It is subject to garbage collection so if you assign it anywhere its lifetime will be tied to that object (in your example EXTPTRSXP).
I may be misunderstanding the question, but I think the answer is
actually that it is *not* safe to put memory allocated via R_alloc into
the external pointer address of an EXTPTRSXP.
Here's what I think Melissa is doing:
SEXP make_test_xp(SEXP s)
{
SEXP ans;
const char *s0 = CHAR(STRING_ELT(s, 0));
char *buf = (char *)R_alloc(strlen(s0) + 1, sizeof(char));
memcpy(buf, s0, strlen(s0) + 1);
ans = R_MakeExternalPtr(buf, R_NilValue, R_NilValue);
return ans;
}
The memory allocated by R_alloc is "released" at the end of the .Call
via vmaxset(vmax). Using R_alloc in this way will lead to memory
corruption (it does for me when I made a simple test case).
For memory that really is external (not SEXP), then you should instead
use Calloc and register a finalizer for the external pointer that will
do any required cleanup and then call Free.
If instead you want to have an externally managed SEXP, you could put it
in the protected slot of the external pointer, but then you should
allocate it using standard R allocation functions.
+ seth
Seth Falcon | @sfalcon | http://userprimary.net/
Thanks for the responses. Seth's example is indeed what I was trying (hoping) to do, it seems to work on my system fine (ubuntu x86_64, R 2.10.1). But if it doesn't work for him, then that definitely answers my question. I guess I'll have to go the Calloc/Free route. Thanks, Melissa
On Mon, Apr 19, 2010 at 1:22 PM, Seth Falcon <seth at userprimary.net> wrote:
On 4/19/10 8:59 AM, Simon Urbanek wrote:
On Apr 19, 2010, at 10:39 AM, Melissa Jane Hubisz wrote:
Hello, The Writing R extensions manual section 6.1.1 describes the transient memory allocation function R_alloc, and states that memory allocated by R_alloc is automatically freed after the .C or .Call function is completed. ?However, based on my understanding of R's memory handling, as well as some test functions I have written, I suspect that this is not quite accurate. ?If the .Call function returns an external pointer to something created with R_alloc, then this object seems to stick around after the .Call function is completed, and is subject to garbage collection once the external pointer object is removed.
Yes, because the regular rules for the lifetime of an R object apply since it is in fact an R object. It is subject to garbage collection so if you assign it anywhere its lifetime will be tied to that object (in your example EXTPTRSXP).
I may be misunderstanding the question, but I think the answer is actually
that it is *not* safe to put memory allocated via R_alloc into the external
pointer address of an EXTPTRSXP.
Here's what I think Melissa is doing:
SEXP make_test_xp(SEXP s)
{
? ?SEXP ans;
? ?const char *s0 = CHAR(STRING_ELT(s, 0));
? ?char *buf = (char *)R_alloc(strlen(s0) + 1, sizeof(char));
? ?memcpy(buf, s0, strlen(s0) + 1);
? ?ans = R_MakeExternalPtr(buf, R_NilValue, R_NilValue);
? ?return ans;
}
The memory allocated by R_alloc is "released" at the end of the .Call via
vmaxset(vmax). ?Using R_alloc in this way will lead to memory corruption (it
does for me when I made a simple test case).
For memory that really is external (not SEXP), then you should instead use
Calloc and register a finalizer for the external pointer that will do any
required cleanup and then call Free.
If instead you want to have an externally managed SEXP, you could put it in
the protected slot of the external pointer, but then you should allocate it
using standard R allocation functions.
+ seth
--
Seth Falcon | @sfalcon | http://userprimary.net/
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
On Apr 19, 2010, at 1:22 PM, Seth Falcon wrote:
On 4/19/10 8:59 AM, Simon Urbanek wrote:
On Apr 19, 2010, at 10:39 AM, Melissa Jane Hubisz wrote:
Hello, The Writing R extensions manual section 6.1.1 describes the transient memory allocation function R_alloc, and states that memory allocated by R_alloc is automatically freed after the .C or .Call function is completed. However, based on my understanding of R's memory handling, as well as some test functions I have written, I suspect that this is not quite accurate. If the .Call function returns an external pointer to something created with R_alloc, then this object seems to stick around after the .Call function is completed, and is subject to garbage collection once the external pointer object is removed.
Yes, because the regular rules for the lifetime of an R object apply since it is in fact an R object. It is subject to garbage collection so if you assign it anywhere its lifetime will be tied to that object (in your example EXTPTRSXP).
I may be misunderstanding the question, but I think the answer is actually that it is *not* safe to put memory allocated via R_alloc into the external pointer address of an EXTPTRSXP.
Here's what I think Melissa is doing:
SEXP make_test_xp(SEXP s)
{
SEXP ans;
const char *s0 = CHAR(STRING_ELT(s, 0));
char *buf = (char *)R_alloc(strlen(s0) + 1, sizeof(char));
memcpy(buf, s0, strlen(s0) + 1);
ans = R_MakeExternalPtr(buf, R_NilValue, R_NilValue);
return ans;
}
The memory allocated by R_alloc is "released" at the end of the .Call via vmaxset(vmax). Using R_alloc in this way will lead to memory corruption (it does for me when I made a simple test case).
Can you elaborate on that? (It's really tricky to test this since you cannot attach a finalizer to the allocated memory). AFAICT the R_alloc allocates a regular R vector (raw or real depending on size) so the usual R object rules apply. Then it is attached to the VStack. If you also assign it to any other object accessible from the GC roots (before the VStack goes away) then even removing the VStack entry won't cause de-allocation because it will be flagged from the other root at mark time so it won't be garbage collected. VStack is not released blindly it is simply pruned and left to garbage collection to decide whether to release the objects or not. That said, the lesson to Melissa is that you can simply allocate a raw vector with the same effect - there is no need to use R_alloc() in her case (is user code PROTECTing is sort of equivalent to the VStack used internally). Cheers, Simon
For memory that really is external (not SEXP), then you should instead use Calloc and register a finalizer for the external pointer that will do any required cleanup and then call Free. If instead you want to have an externally managed SEXP, you could put it in the protected slot of the external pointer, but then you should allocate it using standard R allocation functions. + seth -- Seth Falcon | @sfalcon | http://userprimary.net/
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
On Apr 20, 2010, at 10:12 AM, Simon Urbanek wrote:
On Apr 19, 2010, at 1:22 PM, Seth Falcon wrote:
On 4/19/10 8:59 AM, Simon Urbanek wrote:
On Apr 19, 2010, at 10:39 AM, Melissa Jane Hubisz wrote:
Hello, The Writing R extensions manual section 6.1.1 describes the transient memory allocation function R_alloc, and states that memory allocated by R_alloc is automatically freed after the .C or .Call function is completed. However, based on my understanding of R's memory handling, as well as some test functions I have written, I suspect that this is not quite accurate. If the .Call function returns an external pointer to something created with R_alloc, then this object seems to stick around after the .Call function is completed, and is subject to garbage collection once the external pointer object is removed.
Yes, because the regular rules for the lifetime of an R object apply since it is in fact an R object. It is subject to garbage collection so if you assign it anywhere its lifetime will be tied to that object (in your example EXTPTRSXP).
I may be misunderstanding the question, but I think the answer is actually that it is *not* safe to put memory allocated via R_alloc into the external pointer address of an EXTPTRSXP.
Here's what I think Melissa is doing:
SEXP make_test_xp(SEXP s)
{
SEXP ans;
const char *s0 = CHAR(STRING_ELT(s, 0));
char *buf = (char *)R_alloc(strlen(s0) + 1, sizeof(char));
memcpy(buf, s0, strlen(s0) + 1);
ans = R_MakeExternalPtr(buf, R_NilValue, R_NilValue);
return ans;
}
The memory allocated by R_alloc is "released" at the end of the .Call via vmaxset(vmax). Using R_alloc in this way will lead to memory corruption (it does for me when I made a simple test case).
Can you elaborate on that? (It's really tricky to test this since you cannot attach a finalizer to the allocated memory). AFAICT the R_alloc allocates a regular R vector (raw or real depending on size) so the usual R object rules apply. Then it is attached to the VStack. If you also assign it to any other object accessible from the GC roots (before the VStack goes away) then even removing the VStack entry won't cause de-allocation because it will be flagged from the other root at mark time so it won't be garbage collected. VStack is not released blindly it is simply pruned and left to garbage collection to decide whether to release the objects or not.
Ah, I now see the issue - I missed the part that you're NOT using it as SEXP (tag/prot) in the EXTPTR but as void pointer in which case it is not traversed at GC time - point taken. If you assign it as SEXP anywhere (list, vector, etc.) then my point remains ;). But, again, use PROTECT(allocVector(RAWSXP, ..)) for the same yet safe effect. Cheers, Simon
That said, the lesson to Melissa is that you can simply allocate a raw vector with the same effect - there is no need to use R_alloc() in her case (is user code PROTECTing is sort of equivalent to the VStack used internally). Cheers, Simon
For memory that really is external (not SEXP), then you should instead use Calloc and register a finalizer for the external pointer that will do any required cleanup and then call Free. If instead you want to have an externally managed SEXP, you could put it in the protected slot of the external pointer, but then you should allocate it using standard R allocation functions. + seth -- Seth Falcon | @sfalcon | http://userprimary.net/
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
On Tue, 20 Apr 2010, Simon Urbanek wrote:
On Apr 19, 2010, at 1:22 PM, Seth Falcon wrote:
On 4/19/10 8:59 AM, Simon Urbanek wrote:
On Apr 19, 2010, at 10:39 AM, Melissa Jane Hubisz wrote:
Hello, The Writing R extensions manual section 6.1.1 describes the transient memory allocation function R_alloc, and states that memory allocated by R_alloc is automatically freed after the .C or .Call function is completed. However, based on my understanding of R's memory handling, as well as some test functions I have written, I suspect that this is not quite accurate. If the .Call function returns an external pointer to something created with R_alloc, then this object seems to stick around after the .Call function is completed, and is subject to garbage collection once the external pointer object is removed.
Yes, because the regular rules for the lifetime of an R object apply since it is in fact an R object. It is subject to garbage collection so if you assign it anywhere its lifetime will be tied to that object (in your example EXTPTRSXP).
I may be misunderstanding the question, but I think the answer is actually that it is *not* safe to put memory allocated via R_alloc into the external pointer address of an EXTPTRSXP.
Here's what I think Melissa is doing:
SEXP make_test_xp(SEXP s)
{
SEXP ans;
const char *s0 = CHAR(STRING_ELT(s, 0));
char *buf = (char *)R_alloc(strlen(s0) + 1, sizeof(char));
memcpy(buf, s0, strlen(s0) + 1);
ans = R_MakeExternalPtr(buf, R_NilValue, R_NilValue);
return ans;
}
The memory allocated by R_alloc is "released" at the end of the .Call via vmaxset(vmax). Using R_alloc in this way will lead to memory corruption (it does for me when I made a simple test case).
Can you elaborate on that? (It's really tricky to test this since you cannot attach a finalizer to the allocated memory). AFAICT the R_alloc allocates a regular R vector (raw or real depending on size) so the usual R object rules apply. Then it is attached to the VStack. If you also assign it to any other object accessible from the GC roots (before the VStack goes away) then even removing the VStack entry won't cause de-allocation because it will be flagged from the other root at mark time so it won't be garbage collected. VStack is not released blindly it is simply pruned and left to garbage collection to decide whether to release the objects or not.
But R_alloc returns the pointer to the data associated with the SEXPR that goes in the vstack, and there is no official way to get from that data pointer to the SEXPR. So the allocation can't be GC protected by anything done in the code that calls R_alloc. In any case the implementation of R_alloc is not intended to be public and could change. luke
That said, the lesson to Melissa is that you can simply allocate a raw vector with the same effect - there is no need to use R_alloc() in her case (is user code PROTECTing is sort of equivalent to the VStack used internally). Cheers, Simon
For memory that really is external (not SEXP), then you should instead use Calloc and register a finalizer for the external pointer that will do any required cleanup and then call Free. If instead you want to have an externally managed SEXP, you could put it in the protected slot of the external pointer, but then you should allocate it using standard R allocation functions. + seth -- Seth Falcon | @sfalcon | http://userprimary.net/
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Luke Tierney
Chair, Statistics and Actuarial Science
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa Phone: 319-335-3386
Department of Statistics and Fax: 319-335-3017
Actuarial Science
241 Schaeffer Hall email: luke at stat.uiowa.edu
Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu
On 4/20/10 6:24 AM, Melissa Jane Hubisz wrote:
Thanks for the responses. Seth's example is indeed what I was trying (hoping) to do, it seems to work on my system fine (ubuntu x86_64, R 2.10.1). But if it doesn't work for him, then that definitely answers my question. I guess I'll have to go the Calloc/Free route.
I expect that you could get your approach to not work on your system as well, you just have to try harder ;-) Memory related bugs can be quite tricky, because incorrect code may run fine most of the time. To trigger a problem, you need to have the right pattern of allocation such that data will be written over the memory that your invalid external pointer points to. + seth
Seth Falcon | @sfalcon | http://userprimary.net/