Small changes to big objects (1)

Martin Morgan commented in email to me that a change to any slot of an
object that has other, large slot(s) does substantial computation,
presumably from copying the whole object.  Is there anything to be done?

There are in fact two possible changes, one automatic but only partial,
the other requiring some action on the programmer's part.  Herewith the
first; I'll discuss the second in a later email.

Some context:  The notion is that our object has some big data and some
additional smaller things.  We need to change the small things but would
rather not copy the big things all the time.  (With long vectors, this
becomes even more relevant.)

There are three likely scenarios: slots, attributes and named list
components.  Suppose our object has "little" and "BIG" encoded in one of
these.

The three relevant computations are:

x at little <- other
attr(x, "little") <- other
x$little <- other

It turns out that these are all similar in behavior with one important
exception--fixing that is the automatic change.

I need to review what R does here. All these are replacement functions,
`@<-`, `attr<-`, `$<-`.  The evaluator checks before calling any
replacement whether the object needs to be duplicated (in a routine
EnsureLocal()).  It does that by examining a special field that holds
the reference status of the object.

Some languages, such as Python (and S) keep reference counts for each
object, de-allocating the object when the reference count drops back to
zero.  R uses a different strategy. Its NAMED() field is 0, 1 or 2
according to whether the object has been assigned never, once or more
than once.  The field is not a reference count and is not
decremented--relevant for this issue.  Objects are de-allocated only
when garbage collection occurs and the object does not appear in any
current frame or other context.
(I did not write any of this code, so apologies if I'm misrepresenting it.)

When any of these replacement operations first occurs for a particular
object in a particular function call, it's very likely that the
reference status will be 2 and EnsureLocal will duplicate it--all of it.
Regardless of which of the three forms is used.

Here the non-level-playing-field aspect comes in.  `@<-` is a normal R
function (a "closure") but the other two are primitives in the main code
for R.  Primitives have no frame in which arguments are stored.  As a
result the new version of x is normally stored with status 1.

If one does a second replacement in the same call (in a loop, e.g.) that
should not normally copy again.  But the result of `@<-` will be an
object from its frame and will have status 2 when saved, forcing a copy
each time.

So the change, naturally, is that R 3.0.0 will have a primitive
implementation of `@<`.  This has been implemented in r-devel (rev. 61544).

Please try it out _before_ we issue that version, especially if you own
a package that does things related to this question.

John

PS:  Some may have noticed that I didn't mention a fourth approach:
fields in a reference class object.  The assumption was that we wanted
classical, functional behavior here.  Reference classes don't have the
copy problem but don't behave functionally either.  But that is in fact
the direction for the other approach.  I'll discuss that later, when the
corresponding code is available.

Is there a difference in the copying behavior of

x at little <- other

and

x at little[] <- other

I was using the second form in (yet another!) modification of the internal
representation of mixed-effects models in the lme4 package in the hopes
that it would not trigger copying of the entire object.  The object
representing the model is quite large but the changes during iterations are
to small vectors representing parameters and coefficients.



On Thu, Jan 3, 2013 at 1:08 PM, John Chambers <jmc at r-project.org> wrote:

Martin Morgan commented in email to me that a change to any slot of an
object that has other, large slot(s) does substantial computation,
presumably from copying the whole object.  Is there anything to be done?

There are in fact two possible changes, one automatic but only partial,
the other requiring some action on the programmer's part.  Herewith the
first; I'll discuss the second in a later email.

Some context:  The notion is that our object has some big data and some
additional smaller things.  We need to change the small things but would
rather not copy the big things all the time.  (With long vectors, this
becomes even more relevant.)

There are three likely scenarios: slots, attributes and named list
components.  Suppose our object has "little" and "BIG" encoded in one of
these.

The three relevant computations are:

x at little <- other
attr(x, "little") <- other
x$little <- other

It turns out that these are all similar in behavior with one important
exception--fixing that is the automatic change.

I need to review what R does here. All these are replacement functions,
`@<-`, `attr<-`, `$<-`.  The evaluator checks before calling any
replacement whether the object needs to be duplicated (in a routine
EnsureLocal()).  It does that by examining a special field that holds the
reference status of the object.

Some languages, such as Python (and S) keep reference counts for each
object, de-allocating the object when the reference count drops back to
zero.  R uses a different strategy. Its NAMED() field is 0, 1 or 2
according to whether the object has been assigned never, once or more than
once.  The field is not a reference count and is not decremented--relevant
for this issue.  Objects are de-allocated only when garbage collection
occurs and the object does not appear in any current frame or other context.
(I did not write any of this code, so apologies if I'm misrepresenting it.)

When any of these replacement operations first occurs for a particular
object in a particular function call, it's very likely that the reference
status will be 2 and EnsureLocal will duplicate it--all of it. Regardless
of which of the three forms is used.

Here the non-level-playing-field aspect comes in.  `@<-` is a normal R
function (a "closure") but the other two are primitives in the main code
for R.  Primitives have no frame in which arguments are stored.  As a
result the new version of x is normally stored with status 1.

If one does a second replacement in the same call (in a loop, e.g.) that
should not normally copy again.  But the result of `@<-` will be an object
from its frame and will have status 2 when saved, forcing a copy each time.

So the change, naturally, is that R 3.0.0 will have a primitive
implementation of `@<`.  This has been implemented in r-devel (rev. 61544).

Please try it out _before_ we issue that version, especially if you own a
package that does things related to this question.

John

PS:  Some may have noticed that I didn't mention a fourth approach: fields
in a reference class object.  The assumption was that we wanted classical,
functional behavior here.  Reference classes don't have the copy problem
but don't behave functionally either.  But that is in fact the direction
for the other approach.  I'll discuss that later, when the corresponding
code is available.

	[[alternative HTML version deleted]]

myLargeData <- LargeData("/path/to/file")
mySubSet <- myLargeData[1:10, seq(1,15,by=3)]

LargeData <- setClass("LargeData", representation(filename="character"))
setMethod("initialize","LargeData", function(.Object,filename) .Object at filename <- filename)

fullData <- myLargeData
class(fullData)

myEigen <- eigen(myLargeData)

Hi All,

I'm currently trying to write an S4 class that mimics a data.frame, but stores data on disc in HDF5 format.  The idea is that the dataset is likely to be too large to fit into a standard desktop machine, and by using subscripts, the user may load bits of the dataset at a time.  eg:

myLargeData <- LargeData("/path/to/file")
mySubSet <- myLargeData[1:10, seq(1,15,by=3)]

I've therefore defined by LargeData class thus

LargeData <- setClass("LargeData", representation(filename="character"))
setMethod("initialize","LargeData", function(.Object,filename) .Object at filename <- filename)

I've then defined the "[" method to call a C++ function (Rcpp), opening the HDF5 file, and returning the required rows/cols as a data.frame.

However, what if the user wants to load the entire dataset into memory?  Which method do I overload to achieve the following?

fullData <- myLargeData
class(fullData)

[1] "data.frame"

or apply transformations:

myEigen <- eigen(myLargeData)

In C++ I would normally overload the "double" or "float" operator to achieve this -- can I do the same thing in R?

Thanks,

Chris

Hi All,

I'm currently trying to write an S4 class that mimics a data.frame, but stores data on disc in HDF5 format.  The idea is that the dataset is likely to be too large to fit into a standard desktop machine, and by using subscripts, the user may load bits of the dataset at a time.  eg:

myLargeData <- LargeData("/path/to/file")
mySubSet <- myLargeData[1:10, seq(1,15,by=3)]

I've therefore defined by LargeData class thus

LargeData <- setClass("LargeData", representation(filename="character"))
setMethod("initialize","LargeData", function(.Object,filename) .Object at filename <- filename)

I've then defined the "[" method to call a C++ function (Rcpp), opening the HDF5 file, and returning the required rows/cols as a data.frame.

However, what if the user wants to load the entire dataset into memory?  Which method do I overload to achieve the following?

fullData <- myLargeData
class(fullData)

[1] "data.frame"

or apply transformations:

myEigen <- eigen(myLargeData)

In C++ I would normally overload the "double" or "float" operator to achieve this -- can I do the same thing in R?

Thanks,

Chris

Chris Jewell <chris.jewell at warwick.ac.uk>
    on Wed, 9 Jan 2013 13:28:49 +1300 writes:
"CJ" == Chris Jewell <chris.jewell at warwick.ac.uk>
    on Wed, 9 Jan 2013 13:28:49 +1300 writes: