R Bug: write.table for matrix of more than 2, 147, 483, 648 elements

Hi Colton,

You could divide your write task into chunks that do not violate the 2^31-1 limit.

write.table has an append argument (default FALSE).

Figure out a row chunk size nri < nr such that nri * nc is under 2^31-1 and use
write.table() to write that out.

Then use
write.table(  append = TRUE, )
for the next chunk of rows, looping over chunks until done.  Two chunks will get your 2.8 billion entries done.

Best

Steve



Steven McKinney, Ph.D.

Statistician
Molecular Oncology and Breast Cancer Program
British Columbia Cancer Research Centre

-----Original Message-----
From: R-devel [mailto:r-devel-bounces at r-project.org] On Behalf Of Tousey,
Colton
Sent: April-18-18 2:08 PM
To: R-core at r-project.org; simon.urbanek at R-project.org; R-devel at r-
project.org
Subject: [Rd] R Bug: write.table for matrix of more than 2, 147, 483, 648
elements

Hello,

I want to report a bug in R that is limiting my capabilities to export a
matrix with write.csv or write.table with over 2,147,483,648 elements (C's
int limit). I found this bug already reported about before: https://bugs.r-
project.org/bugzilla/show_bug.cgi?id=17182. However, there appears to be no
solution or fixes in upcoming R version releases.

The error message is coming from the writetable part of the utils package
in the io.c source code(https://svn.r-
project.org/R/trunk/src/library/utils/src/io.c):
/* quick integrity check */
                 if(XLENGTH(x) != (R_len_t)nr * nc)
                     error(_("corrupt matrix -- dims not not match
length"));

The issue is that nr*nc is an integer and the size of my matrix, 2.8
billion elements, exceeds C's limit, so the check forces the code to fail.

My version:

R.Version()

$platform
[1] "x86_64-w64-mingw32"

$arch
[1] "x86_64"

$os
[1] "mingw32"

$system
[1] "x86_64, mingw32"

$status
[1] ""

$major
[1] "3"

$minor
[1] "4.3"

$year
[1] "2017"

$month
[1] "11"

$day
[1] "30"

$`svn rev`
[1] "73796"

$language
[1] "R"

$version.string
[1] "R version 3.4.3 (2017-11-30)"

$nickname
[1] "Kite-Eating Tree"

Thank you,
Colton


Colton Tousey
Research Associate II
P: 816.585.0300   E: colton.tousey at kc.frb.org
FEDERAL RESERVE BANK OF KANSAS CITY
1 Memorial Drive   *   Kansas City, Missouri 64198   *
www.kansascityfed.org

	[[alternative HTML version deleted]]