building a formula for glm() with 30,000 independent variables
On Sun, 10 Nov 2002 06:28:51 -0800
Ben Liblit <liblit at eecs.berkeley.edu> wrote:
I would like to use R to perform a logistic regression with about 30,000 independent variables. That's right, thirty thousand. Most will be irrelevant: the intent is to use the regression to identify the few that actually matter. Among other things, this calls for giving glm() a colossal "y ~ ..." formula with thirty thousand summed terms on its right hand side. I build up the formula as a string and then call as.formula() to convert it. Unfortunately, the conversion fails. The parser reports that it has overflowed its stack. :-( Is there any way to pull this off in R? Can anyone suggest alternatives to glm() or to R itself that might be capable of handling a problem of this size? Or am I insane to even be considering an analysis like this? Thanks! It would be worth doing a simulation first to see if ANY statistical properties of the resulting estimates or P-values work as advertised with your setup. I would expect severe biases, lack of preservation of type I error, and low probability of selecting the "correct" variables. The "few that actually matter" will likely be those whose estimates are made with the most error.
Frank E Harrell Jr Prof. of Biostatistics & Statistics Div. of Biostatistics & Epidem. Dept. of Health Evaluation Sciences U. Virginia School of Medicine http://hesweb1.med.virginia.edu/biostat -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._