How to create a high-dimensional matrix
On Tue, Oct 16, 2012 at 10:56 PM, lrl <liurl1221 at gmail.com> wrote:
Should it be .Machine$integer.max- 429497 ^2?
6 of one; half dozen of the other. Your matrix is still absolutely massive: the development version of R does allow for lager matrices so I think you can store a 429497 by 429497 in memory then, but it won't be hugely performant.
The matrix is not a sparse matrix, but a symmetric matrix. It's a process of feature selection. I am choosing the most important variables from 439497 variables. Now I am considering divide the whole dataset into several part and process a small part first.
The use of a training set is generally good practice, but I'm terribly
skeptical of any automated feature selection from a pool of ~400k
selectors. To wit,
library(fortunes)
fortune("fancy random")
Cheers,
Michael