I can work on this. Thanks Luke.
~G
On Wed, Jul 24, 2019 at 8:25 AM Tierney, Luke <luke-tierney at uiowa.edu>
wrote:
If one of you wanted to try to create a patch to support ALTREP
generic vectors here are some notes:
The main challenge I am aware of (there might be others): Allowing
DATAPTR to return a writable pointer would be too dangerous because
the GC write barrier needs to see all mutations. So it would be best
if Dataptr and Dataptr_or_null methods were not allowed to be
defined. The default methods in altrep.c should do the right think.
A reasonable name for the abstract class would be 'altlist'.
'altrep' methods that a class can provide:
Unserialize or UnserializeEX
Serialized_state
Duplicate or DuplicateEx
Coerce
Inspect
Length
'altvec' methods a class should provide:
Extract_subset
not Dataptr
not Dataptr_or_null
'altlist' specific methods:
Elt
Set_elt
Best,
luke
On Tue, 23 Jul 2019, Gabriel Becker wrote:
Hi Kylie,
Is it a list with only numerics in it? (I only see REALSXPs there, but
obviously inspect isn't showing all of them). If so, you could load it
into one big vector and then also keep partitioning information around.
Bioconductor does this (see ?IRanges::CompressedList ). The potential
benefit here being that the underlying large vector could then be a big
out-of-memory altrep. How helpful this would be depends somewhat on what
you want to do with it, of course, but it is something that comes to
Also, I would expect some overhead but that seems like a lot (without
having done super much in the way of benchmarking). What exactly is
as.altrep doing?
Best,
~G
On Tue, Jul 23, 2019 at 9:54 AM Michael Lawrence via R-devel <
r-devel at r-project.org> wrote:
Hi Kylie,
As an alternative in the short term, you could consider deriving from
S4Vector's List class, implementing the getListElement() method to
lazily create the objects.
Michael
On Tue, Jul 23, 2019 at 9:09 AM Bemis, Kylie <k.bemis at northeastern.edu
Hello,
I was wondering if there were any plans for ALTREP lists (VECSXP)?
It seems to me that they could be supported in a similar way to how
ALTSTRING works, with Elt() and Set_elt() methods, or would there be
problems with that I?m not seeing due to lists not being atomic
I was taking an approach of converting each list element (of a
file-based list data structure) to an ALTREP representation to build
This seems fine for shorter lists with large elements, but I noticed
that for longer lists with smaller elements, this could be far more
time-consuming than simply reading the entire list into memory and
returning a non-ALTREP list:
<34840 length> matter_list :: out-of-memory list
(1.1 MB real | 543.3 MB virtual)
system.time(y <- as.list(x))
user system elapsed
1.116 2.175 5.053
system.time(z <- as.altrep(x))
user system elapsed
36.295 4.717 41.216
@108255000 19 VECSXP g1c7 [MARK,NAM(7)] (len=34840, tl=0)
@7f9044d9fc00 14 REALSXP g1c7 [MARK] (len=1129, tl=0)
404.093,404.096,404.099,404.102,404.105,...
@7f9044d25e00 14 REALSXP g1c7 [MARK] (len=890, tl=0)
409.924,409.927,409.931,409.934,409.937,...
@7f9044da6000 14 REALSXP g1c7 [MARK] (len=1878, tl=0)
400.3,400.303,400.306,400.309,400.312,...
@7f9031a6b000 14 REALSXP g1c7 [MARK] (len=2266, tl=0)
402.179,402.182,402.185,402.188,402.191,...
@7f9031a77a00 14 REALSXP g1c7 [MARK] (len=1981, tl=0)
403.021,403.024,403.027,403.03,403.033,...
@108210000 19 VECSXP g1c7 [MARK,NAM(7)] (len=34840, tl=0)
@7f904eea7660 14 REALSXP g1c0 [MARK,NAM(7)] matter vector (mode=4,
@7f9050347498 14 REALSXP g1c0 [MARK,NAM(7)] matter vector (mode=4,
@7f904d286b20 14 REALSXP g1c0 [MARK,NAM(7)] matter vector (mode=4,
@7f904fd38820 14 REALSXP g1c0 [MARK,NAM(7)] matter vector (mode=4,
@7f904c75ce90 14 REALSXP g1c0 [MARK,NAM(7)] matter vector (mode=4,
...
In this situation, it would be much faster and simpler for me to
a theoretical ALTREP list that serves SEXP elements on-demand, similar
how ALTSTRING seems to be implemented.
I don?t know how many other people would get a use out of ALTREP
Are there any plans for this?
Thanks!
~~~
Kylie Ariel Bemis
Khoury College of Computer Sciences
Northeastern University
kuwisdelu.github.io<https://kuwisdelu.github.io>
[[alternative HTML version deleted]]