R Bug: write.table for matrix of more than 2, 147, 483, 648 elements

On 18/04/2018 5:08 PM, Tousey, Colton wrote:

Hello,

I want to report a bug in R that is limiting my capabilities to 
export a matrix with write.csv or write.table with over 2,147,483,648 
elements (C's int limit). I found this bug already reported about 
before: https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17182. 
However, there appears to be no solution or fixes in upcoming R 
version releases.

The error message is coming from the writetable part of the utils 
package in the io.c source 
code(https://svn.r-project.org/R/trunk/src/library/utils/src/io.c):
/* quick integrity check */
???????????????? if(XLENGTH(x) != (R_len_t)nr * nc)
???????????????????? error(_("corrupt matrix -- dims not not match 
length"));

The issue is that nr*nc is an integer and the size of my matrix, 2.8 
billion elements, exceeds C's limit, so the check forces the code to 
fail.

Yes, looks like a typo:? R_len_t is an int, and that's how nr was 
declared.? It should be R_xlen_t, which is bigger on machines that 
support big vectors.

I haven't tested the change; there may be something else in that 
function that assumes short vectors.

Duncan Murdoch

My version:

R.Version()

$platform
[1] "x86_64-w64-mingw32"

$arch
[1] "x86_64"

$os
[1] "mingw32"

$system
[1] "x86_64, mingw32"

$status
[1] ""

$major
[1] "3"

$minor
[1] "4.3"

$year
[1] "2017"

$month
[1] "11"

$day
[1] "30"

$`svn rev`
[1] "73796"

$language
[1] "R"

$version.string
[1] "R version 3.4.3 (2017-11-30)"

$nickname
[1] "Kite-Eating Tree"

Thank you,
Colton


Colton Tousey
Research Associate II
P: 816.585.0300?? E: colton.tousey at kc.frb.org
FEDERAL RESERVE BANK OF KANSAS CITY
1 Memorial Drive?? *?? Kansas City, Missouri 64198?? * 
www.kansascityfed.org

????[[alternative HTML version deleted]]

On 04/19/2018 02:06 AM, Duncan Murdoch wrote:

On 18/04/2018 5:08 PM, Tousey, Colton wrote:

Hello,

I want to report a bug in R that is limiting my capabilities to 
export a matrix with write.csv or write.table with over 
2,147,483,648 elements (C's int limit). I found this bug already 
reported about before: 
https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17182. However, 
there appears to be no solution or fixes in upcoming R version 
releases.

The error message is coming from the writetable part of the utils 
package in the io.c source 
code(https://svn.r-project.org/R/trunk/src/library/utils/src/io.c):
/* quick integrity check */
???????????????? if(XLENGTH(x) != (R_len_t)nr * nc)
???????????????????? error(_("corrupt matrix -- dims not not match 
length"));

The issue is that nr*nc is an integer and the size of my matrix, 2.8 
billion elements, exceeds C's limit, so the check forces the code to 
fail.

Yes, looks like a typo:? R_len_t is an int, and that's how nr was 
declared.? It should be R_xlen_t, which is bigger on machines that 
support big vectors.

I haven't tested the change; there may be something else in that 
function that assumes short vectors.

Indeed, I think the function won't work for long vectors because of 
EncodeElement2 and EncodeElement0. EncodeElement2/0 would have to be 
changed, including their signatures

Le 19/04/2018 ? 09:30, Tomas Kalibera a ?crit?:

On 04/19/2018 02:06 AM, Duncan Murdoch wrote:

On 18/04/2018 5:08 PM, Tousey, Colton wrote:

Hello,

I want to report a bug in R that is limiting my capabilities to 
export a matrix with write.csv or write.table with over 
2,147,483,648 elements (C's int limit). I found this bug already 
reported about before: 
https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17182. However, 
there appears to be no solution or fixes in upcoming R version 
releases.

The error message is coming from the writetable part of the utils 
package in the io.c source 
code(https://svn.r-project.org/R/trunk/src/library/utils/src/io.c):
/* quick integrity check */
???????????????? if(XLENGTH(x) != (R_len_t)nr * nc)
???????????????????? error(_("corrupt matrix -- dims not not match 
length"));

The issue is that nr*nc is an integer and the size of my matrix, 
2.8 billion elements, exceeds C's limit, so the check forces the 
code to fail.

Yes, looks like a typo:? R_len_t is an int, and that's how nr was 
declared.? It should be R_xlen_t, which is bigger on machines that 
support big vectors.

I haven't tested the change; there may be something else in that 
function that assumes short vectors.

Indeed, I think the function won't work for long vectors because of 
EncodeElement2 and EncodeElement0. EncodeElement2/0 would have to be 
changed, including their signatures

That would be a definite fix but before such deep rewriting is 
undertaken may the following small fix (in addition to "(R_xlen_t)nr * 
nc") will be sufficient for cases where nr and nc are in int range but 
their product can reach long vector limit:

replace
??? tmp = EncodeElement2(x, i + j*nr, quote_col[j], qmethod,
??? ??? ??? ??? ??? &strBuf, sdec);
by
??? tmp = EncodeElement2(VECTOR_ELT(x, (R_xlen_t)i + j*nr), 0, 
quote_col[j], qmethod,
??? ??? ??? ??? ??? &strBuf, sdec);

Serguei

On 04/19/2018 11:47 AM, Serguei Sokol wrote:

replace
??? tmp = EncodeElement2(x, i + j*nr, quote_col[j], qmethod,
??? ??? ??? ??? ??? &strBuf, sdec);
by
??? tmp = EncodeElement2(VECTOR_ELT(x, (R_xlen_t)i + j*nr), 0, 
quote_col[j], qmethod,
??? ??? ??? ??? ??? &strBuf, sdec);

Unfortunately we can't do that, x is a matrix of an atomic vector 
type. VECTOR_ELT is taking elements of a generic vector, so it cannot 
be applied to "x". But even if we extracted a single element from "x" 
(e.g. via a type-switch etc), we would not be able to pass it to 
EncodeElement0 which expects a full atomic vector (that is, including 
its header). Instead we would have to call functions like 
EncodeInteger, EncodeReal0, etc on the individual elements. Which is 
then the same as changing EncodeElement0 or implementing a new version 
of it. This does not seem that hard to fix, just is not as trivial as 
changing the cast..

Serguei Sokol <sokol at insa-toulouse.fr>
    on Thu, 19 Apr 2018 13:29:54 +0200 writes:

Serguei Sokol <sokol at insa-toulouse.fr>
     on Thu, 19 Apr 2018 13:29:54 +0200 writes:

  [...............]

     > Thanks Tomas for this detailed explanation.

     > I would like also to signal a problem with the list. It must be
     > corrupted in some way because beside the Tomas'? response I've got five
     > or six (so far) dating spam. All of them coming from two emails:
     > Kristina Oliynik <kristinaoliynik604324 at kw.taluss.com> and
     > Samantha Smith <samanthasmith317260 at kw.fefty.com>.

Well, that's the current ones for you.  They change over time,
and in my experience you get about 10--20 (about once per hour;
on purpose not exactly every 60 minutes) and then it stops.

I've replied to the thread  "Hacked" on R-help yesterday:
   https://stat.ethz.ch/pipermail/r-help/2018-April/452423.html

This has started ca 2 weeks ago on R-help already, and today
we've learned that even  R-SIG-Mixed-Models  is affected.

I think I don't see them anymore at all because my spam filters have adapted.

Note that

1. This is *NOT* from regular mailing list subscribers, and none
    of these spam come via the R mailing list servers.

2. It's still a huge pain and disreputable to the R lists of course.

3. I had hoped we could wait and see it go away, but I may be wrong.

4. We have re-started discussing what could be done.

    One drastic measure would make mailing list usage
    *less* attractive by "munging" all poster's e-mail addresses.

-----

For now use your mail providers spam filters to quickly get rid
of this. .. or more interestingly and clearly less legally: use R to
write "mail bombs".  Write an R function sending ca 10 e-mails per
hour randomly to that address   ... ;-)  I did something like
that (with a shell script, not R) at the end of last millennium
when I was younger and the internet was a much much smaller
space than now...

Martin