4 messages · Ivan Krylov, Simon Urbanek, Jan van der Laan +1 more
? Wed, 19 Jun 2024 09:52:20 +0200 Jan van der Laan <rhelp at eoos.dds.nl> ?????:
What is the status of supporting long vectors in data.frames (e.g. data.frames with more than 2^31 records)? Is this something that is being worked on? Is there a time line for this? Is this something I can contribute to?
Apologies if you've already received a better answer off-list.
The second point is not really an issue - R already uses numerics for larger-than-32-bit indexing at R level and it works just fine for objects up to ca. 72 petabytes. However, the first one is a bit more relevant than one would think. At one point I have experimented with allowing data frames with more than 2^31 rows, but it breaks in many places - some quite unexpected. Beside dim() there is also the issue with (non-expanded) row names. Overall, it is a lot more work - some would have to be done in R but some would require changes to packages as well. (In practice I use sharded data frames for large data which removes the limit and allows parallel processing - but requires support from the methods that will be applied to them). Cheers, Simon
On Jul 2, 2024, at 16:04, Ivan Krylov via R-devel <r-devel at r-project.org> wrote: ? Wed, 19 Jun 2024 09:52:20 +0200 Jan van der Laan <rhelp at eoos.dds.nl> ?????:
What is the status of supporting long vectors in data.frames (e.g. data.frames with more than 2^31 records)? Is this something that is being worked on? Is there a time line for this? Is this something I can contribute to?
Apologies if you've already received a better answer off-list. From from my limited understanding, the problem with supporting larger-than-(2^31-1) dimensions has multiple facets: - In many parts of R code, there's the assumption that dim() is of integer type. That wouldn't be a problem by itself, except... - R currently lacks a native 64-bit integer type. About a year ago Gabe Becker mentioned that Luke Tierney has been considering improvements in this direction, but it's hard to introduce 64-bit integers without making the user worry even more about data types (numeric != integer != 64-bit integer) or introducing a lot of overhead (64-bit integers being twice as large as 32-bit ones and, depending on the workload, frequently redundant). - Two-dimensional objects eventually get transformed into matrices and handed to LAPACK for linear algebra operations. Currently, the interface used by R to talk to BLAS and LAPACK only supports 32-bit signed integers for lengths. 64-bit BLASes and LAPACKs do exist (e.g. OpenBLAS can be compiled with 64-bit lengths), but we haven't taught R to use them. (This isn't limited to array dimensions, by the way. If you try to svd() a 40000 by 40000 matrix, it'll try to ask for temporary memory with length that overflows a signed 32-bit integer, get a much shorter allocation instead, promptly overflow the buffer and crash the process.) As you see, it's interconnected; work on one thing will involve the other two. -- Best regards, Ivan
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Ivan, Simon, Thanks for the replies. I can work around the limitation. I currently either divide the data into shards or use a list with (long) vectors depending on what I am trying to do. But I have to transform between the two representations which takes time and memory and often need more code than I would have if I could have used data.frames. Being able to create large (> 2^31-1 rows) data.frames and doing some basic things like selecting rows and columns, would already be really nice. That would also allow package maintainers to start supporting these data.frames. I imagine getting large data.frames working in functions like lm, is not trivial and lm might not support this any time soon. However, a package like biglm might. But from what you are saying, I get the impression that this is not something that is being actively worked on. I must say, my hands a kind of itching to try. Best, Jan
The second point is not really an issue - R already uses numerics for larger-than-32-bit indexing at R level and it works just fine for objects up to ca. 72 petabytes. However, the first one is a bit more relevant than one would think. At one point I have experimented with allowing data frames with more than 2^31 rows, but it breaks in many places - some quite unexpected. Beside dim() there is also the issue with (non-expanded) row names. Overall, it is a lot more work - some would have to be done in R but some would require changes to packages as well. (In practice I use sharded data frames for large data which removes the limit and allows parallel processing - but requires support from the methods that will be applied to them). Cheers, Simon
On Jul 2, 2024, at 16:04, Ivan Krylov via R-devel <r-devel at r-project.org> wrote: ? Wed, 19 Jun 2024 09:52:20 +0200 Jan van der Laan <rhelp at eoos.dds.nl> ?????:
What is the status of supporting long vectors in data.frames (e.g. data.frames with more than 2^31 records)? Is this something that is being worked on? Is there a time line for this? Is this something I can contribute to?
Apologies if you've already received a better answer off-list. From from my limited understanding, the problem with supporting larger-than-(2^31-1) dimensions has multiple facets: - In many parts of R code, there's the assumption that dim() is of integer type. That wouldn't be a problem by itself, except... - R currently lacks a native 64-bit integer type. About a year ago Gabe Becker mentioned that Luke Tierney has been considering improvements in this direction, but it's hard to introduce 64-bit integers without making the user worry even more about data types (numeric != integer != 64-bit integer) or introducing a lot of overhead (64-bit integers being twice as large as 32-bit ones and, depending on the workload, frequently redundant). - Two-dimensional objects eventually get transformed into matrices and handed to LAPACK for linear algebra operations. Currently, the interface used by R to talk to BLAS and LAPACK only supports 32-bit signed integers for lengths. 64-bit BLASes and LAPACKs do exist (e.g. OpenBLAS can be compiled with 64-bit lengths), but we haven't taught R to use them. (This isn't limited to array dimensions, by the way. If you try to svd() a 40000 by 40000 matrix, it'll try to ask for temporary memory with length that overflows a signed 32-bit integer, get a much shorter allocation instead, promptly overflow the buffer and crash the process.) As you see, it's interconnected; work on one thing will involve the other two. -- Best regards, Ivan
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Unfortunately, as has been noted, some changes require many parties to change at once and can cause huge problems when an unchanged part is reached. If integers are a fixed size, an implementation can be straightforward and you can patch in libraries and parts already used and tested and in languages like C. Python is an example where they went another way and the built-in integer type has an indefinite length integer. But that can mess with efficiency so some extensions commonly used for their versions of Dataframe often allow you to specify one of several types of fixed length integer for efficiency. -----Original Message----- From: R-devel <r-devel-bounces at r-project.org> On Behalf Of Jan van der Laan Sent: Thursday, July 4, 2024 2:38 AM To: r-devel at r-project.org Subject: Re: [Rd] Large vector support in data.frames Ivan, Simon, Thanks for the replies. I can work around the limitation. I currently either divide the data into shards or use a list with (long) vectors depending on what I am trying to do. But I have to transform between the two representations which takes time and memory and often need more code than I would have if I could have used data.frames. Being able to create large (> 2^31-1 rows) data.frames and doing some basic things like selecting rows and columns, would already be really nice. That would also allow package maintainers to start supporting these data.frames. I imagine getting large data.frames working in functions like lm, is not trivial and lm might not support this any time soon. However, a package like biglm might. But from what you are saying, I get the impression that this is not something that is being actively worked on. I must say, my hands a kind of itching to try. Best, Jan
The second point is not really an issue - R already uses numerics for larger-than-32-bit indexing at R level and it works just fine for objects up to ca. 72 petabytes. However, the first one is a bit more relevant than one would think. At one point I have experimented with allowing data frames with more than 2^31 rows, but it breaks in many places - some quite unexpected. Beside dim() there is also the issue with (non-expanded) row names. Overall, it is a lot more work - some would have to be done in R but some would require changes to packages as well. (In practice I use sharded data frames for large data which removes the limit and allows parallel processing - but requires support from the methods that will be applied to them). Cheers, Simon
On Jul 2, 2024, at 16:04, Ivan Krylov via R-devel <r-devel at r-project.org> wrote: ? Wed, 19 Jun 2024 09:52:20 +0200 Jan van der Laan <rhelp at eoos.dds.nl> ?????:
What is the status of supporting long vectors in data.frames (e.g. data.frames with more than 2^31 records)? Is this something that is being worked on? Is there a time line for this? Is this something I can contribute to?
Apologies if you've already received a better answer off-list. From from my limited understanding, the problem with supporting larger-than-(2^31-1) dimensions has multiple facets: - In many parts of R code, there's the assumption that dim() is of integer type. That wouldn't be a problem by itself, except... - R currently lacks a native 64-bit integer type. About a year ago Gabe Becker mentioned that Luke Tierney has been considering improvements in this direction, but it's hard to introduce 64-bit integers without making the user worry even more about data types (numeric != integer != 64-bit integer) or introducing a lot of overhead (64-bit integers being twice as large as 32-bit ones and, depending on the workload, frequently redundant). - Two-dimensional objects eventually get transformed into matrices and handed to LAPACK for linear algebra operations. Currently, the interface used by R to talk to BLAS and LAPACK only supports 32-bit signed integers for lengths. 64-bit BLASes and LAPACKs do exist (e.g. OpenBLAS can be compiled with 64-bit lengths), but we haven't taught R to use them. (This isn't limited to array dimensions, by the way. If you try to svd() a 40000 by 40000 matrix, it'll try to ask for temporary memory with length that overflows a signed 32-bit integer, get a much shorter allocation instead, promptly overflow the buffer and crash the process.) As you see, it's interconnected; work on one thing will involve the other two. -- Best regards, Ivan
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel