Clearing attributes returns ALTREP, serialize still saves them
Hi all, I don't have a solution yet, but a bit more here:
.Internal(inspect(x2b))
@7f913826d590 14 REALSXP g0c0 [REF(1)] wrapper [srt=-2147483648,no_na=0]
@7f9137500320 14 REALSXP g0c7 [REF(2),ATT] (len=100, tl=0)
0.45384,0.926371,0.838637,-1.71485,-0.719073,...
ATTRIB:
@7f913826dc20 02 LISTSXP g0c0 [REF(1)]
TAG: @7f91378538d0 01 SYMSXP g0c0 [MARK,REF(460)] "data"
@7f9118310000 14 REALSXP g0c7 [REF(2)] (len=1000000, tl=0)
0.66682,0.480576,-1.13229,0.453313,-0.819498,...
attr(x2b, "data") <- "small"
.Internal(inspect(x2b))
@7f913826d590 14 REALSXP g0c0 [REF(1),ATT] wrapper
[srt=-2147483648,no_na=0]
@7f9137500320 14 REALSXP g0c7 [REF(2),ATT] (len=100, tl=0)
0.45384,0.926371,0.838637,-1.71485,-0.719073,...
ATTRIB:
@7f913826dc20 02 LISTSXP g0c0 [REF(1)]
TAG: @7f91378538d0 01 SYMSXP g0c0 [MARK,REF(461)] "data"
@7f9118310000 14 REALSXP g0c7 [REF(2)] (len=1000000, tl=0)
0.66682,0.480576,-1.13229,0.453313,-0.819498,...
ATTRIB:
@7f913826c870 02 LISTSXP g0c0 [REF(1)]
TAG: @7f91378538d0 01 SYMSXP g0c0 [MARK,REF(461)] "data"
@7f9120580850 16 STRSXP g0c1 [REF(3)] (len=1, tl=0)
@7f91205808c0 09 CHARSXP g0c1 [REF(3),gp=0x60] [ASCII] [cached]
"small"
So we can see that the assignment of attr(x2b, "data") IS doing something,
but it isn't doing the right thing. The fact that the above code assigned
null instead of a value was hiding this.
I will dig into this more if someone doesn't get it fixed before me, but it
won't be until after useR, because I'm preparing multiple talks for that
and it is this coming week.
Best,
~G
On Fri, Jul 2, 2021 at 9:15 PM Zafer Barutcuoglu <
zafer.barutcuoglu at gmail.com> wrote:
Hi all,
Setting names/dimnames on vectors/matrices of length>=64 returns an ALTREP
wrapper which internally still contains the names/dimnames, and calling
base::serialize on the result writes them out. They are unserialized in the
same way, with the names/dimnames hidden in the ALTREP wrapper, so the
problem is not obvious except in wasted time, bandwidth, or disk space.
Example:
v1 <- setNames(rnorm(64), paste("element name", 1:64))
v2 <- unname(v1)
names(v2)
# NULL
length(serialize(v1, NULL))
# [1] 2039
length(serialize(v2, NULL))
# [1] 2132
length(serialize(v2[TRUE], NULL))
# [1] 543
con <- rawConnection(raw(), "w")
serialize(v2, con)
v3 <- unserialize(rawConnectionValue(con))
names(v3)
# NULL
length(serialize(v3, NULL))
# 2132
# Similarly for matrices:
m1 <- matrix(rnorm(64), 8, 8, dimnames=list(paste("row name", 1:8),
paste("col name", 1:8)))
m2 <- unname(m1)
dimnames(m2)
# NULL
length(serialize(m1, NULL))
# [1] 918
length(serialize(m2, NULL))
# [1] 1035
length(serialize(m2[TRUE, TRUE], NULL))
# 582
Previously discussed here, too:
https://r.789695.n4.nabble.com/Invisible-names-problem-td4764688.html
This happens with other attributes as well, but less predictably:
x1 <- structure(rnorm(100), data=rnorm(1000000))
x2 <- structure(x1, data=NULL)
length(serialize(x1, NULL))
# [1] 8000952
length(serialize(x2, NULL))
# [1] 924
x1b <- rnorm(100)
attr(x1b, "data") <- rnorm(1000000)
x2b <- x1b
attr(x2b, "data") <- NULL
length(serialize(x1b, NULL))
# [1] 8000863
length(serialize(x2b, NULL))
# [1] 8000956
This is pretty severe, trying to track down why serializing a small object
kills the network, because of which large attributes it may have once had
during its lifetime around the codebase that are still secretly tagging
along.
Is there a plan to resolve this? Any suggestions for maybe a C++
workaround until then? Or an alternative performant serialization solution?
Best,
--
Zafer
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel