R help-classification accuracy of DFA and RF using caret

An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20131106/fd03094b/attachment.pl>

Hi,

I am a graduate student applying published R scripts to compare the classification accuracy of 2 predictive models, one built using discriminant function analysis and one using random forests (webpage link for these scripts is provided below).  The purpose of these models is to predict the biotic integrity of streams.  Specifically, I am trying to compare the classification accuracy (i.e., prediction of group membership)of both the DFA and RF models using k-fold crossvalidation for the following metrics: AUC ROC, percent correctly classified, specificity, sensitivity, and Kappa.
Sensitivity, "accuracy" (= percent correct), and specificity are only defined when you establish a particular threshold for decision. The is no "sensitivity" or "specificity" that will accrue to a classification model. AUC is an effort at presenting such an overall value, but it has deficiencies and is insensitive to statistically significant differences in models.
I would also like to obtain the F statistic, Wilks lambda, MSE or RMSE for the random forest models as the script does not contain code to get this data.
I doubt very much that is by accident or oversight on the part of the randomForest developers.
 I think I need to use the caret package to obtain the classification accuracy, but I keep getting error messages when I apply the train function to my data.  As I am relatively new to R and my thesis committee is unable to help as they are also unfamiliar with R, I thought it best to ask for help.
I think you need to add a statistician to your committee. The difficulties you are facing (of which you appear to be unaware) are not just related to being new to R.
 Would someone be willing to help me?

Thanks,
Robin

http://www.epa.gov/wed/pages/models/rivpacs/rivpacs.htm

TrainDataDFAgrps2 <-predcal
TrainClassesDFAgrps2 <-grp.2;
DFAgrps2Fit1 <- train(TrainDataDFAgrps2, TrainClassesDFAgrps2,
+  method = "lda",
+ tuneLength = 10,
+ trControl = trainControl(method = "cv"));
Error in train.default(TrainDataDFAgrps2, TrainClassesDFAgrps2, method = "lda",  :
 wrong model type for regression
That error is pointing out that you are choosing a method that expects a particular form of outcome (continuous) and does not accept a categorical (possibly an R factor?) outcome. I suspect you may be using the `caret` package, but it's unclear. I think this is further evidence of the need for competent statistical consultation. You would be advised to study further in Venables and Ripley's MASS(v4) or in Hastie, Tibshirani, and Freidmans ESL(v2).

This link, found with a simple google search, suggests that the author of the cited code is at an academic institution only one state away from you: fw.oregonstate.edu/system/files/Van%20Sickle%20CV%20consult.pdf?. He may be willing to offer assistance.
David.

> 
>> RFgrps2Fit1 <- train(TrainDataRFgrps2, TrainClassesRFgrps2,
> +  method = "rf",
> + tuneLength = 10,
> + trControl = trainControl(method = "cv"));
> There were 50 or more warnings (use warnings() to see the first 50)
> 
> Clip of predcal (same length as grp.2, but too much data to display all):
>> predcal
>          Reference_Test HUC12_AREA_HA_log10 ELEV_m M_Slp_sqt Precip_mm Temp_CX10
> 2370                   R                 3.7  588.0       2.2      1751       148
> 559                    R                 4.0  643.1       1.8      1674       141
> 2062                   R                 4.0  643.1       1.8      1674       141
> 2467                   R                 4.0  643.1       1.8      1674       141
> 1176                   R                 3.9  694.3       2.4      1534       131
> 1840                   R                 3.9  694.3       2.4      1534       131
> 2052                   R                 3.9  694.3       2.4      1534       131
> 1174                   R                 4.1  605.0       2.1      1382       138
> 1841                   R                 4.1  605.0       2.1      1382       138
> 2051                   R                 4.1  605.0       2.1      1382       138
> 1831                   R                 4.1  363.9       1.7       937       156
> 
> 
> Grps.2:
> grp.2
>  [1] 1 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 1 2 1 1
> [45] 2 2 1 1 1 1 1 1 1 2 2 1 1 1 2 2 1 2 2 1 1 1 2 2 2 2 2 2 1 1 1 2 2 2 1 2 2 2 2 2 2 2 2 1
> [89] 1 2 2 2 2 2 1 1 2 2 2 1 2 1 2 2 1 2 1 1 2
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

David Winsemius
Alameda, CA, USA
<SNIP>
I think you need to add a statistician to your committee. The 
difficulties you are facing (of which you appear to be unaware) are 
not just related to being new to R.
<SNIP>

Fortune?

     cheers,

     Rolf
Second (perhaps with the slight addition indicated)

-- Bert

... And **amen!** to the sentiment expressed.
On 11/07/13 10:57, David Winsemius wrote:

    <SNIP>
I think you need to add a statistician to your [PhD] committee. The difficulties
you are facing (of which you appear to be unaware) are not just related to
being new to R.
    <SNIP>

Fortune?

    cheers,

    Rolf

______________________________________________
R-help at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Bert Gunter
Genentech Nonclinical Biostatistics

(650) 467-7374
Seen on StackOverflow. Any seconds?

"I would heed the warnings and diagnostics. They are there for a reason. The Ostrich algorithm does not help you."

  -- Dirk Eddelbuettel commenting on StackOverflow when a questioner said he had not run R CMD check because he suspected other problems would be found (Nov 2013)

Achim: I also have a fortunes spelling correction (at least for v 1.5-0):
Looking at:  fortune("calibration")  == fortune(277),  The author's last name needs to be Beleites.
David Winsemius
Alameda, CA, USA
Seen on StackOverflow. Any seconds?

"I would heed the warnings and diagnostics. They are there for a reason. The Ostrich algorithm does not help you."

   -- Dirk Eddelbuettel commenting on StackOverflow when a questioner said he had not run R CMD check because he suspected other problems would be found (Nov 2013)
Right on!!!

     cheers,

     Rolf

On 11/07/13 13:19, David Winsemius wrote:
Seen on StackOverflow. Any seconds?

"I would heed the warnings and diagnostics. They are there for a reason. 
The Ostrich algorithm does not help you."

   -- Dirk Eddelbuettel commenting on StackOverflow when a questioner said 
he had not run R CMD check because he suspected other problems would be 
found (Nov 2013)
Right on!!!
Yes! :-)

Now also in the devel-version on R-Forge.

Thanks,
Z
   cheers,

   Rolf