Modeling precision and recall with GLMMs

Hi Ramon,

I'm not sure that I fully understand the details of what you
want to accomplish. But I do want to ask: you jump right into your email
assuming that of course you want to model precision and recall, but what
about modelling the data directly (i.e., individual classification
decisions) rather than summaries of the data? Then you could work
backward (forward?) from the model results to compute what the implied
precision and recall would be

If you decided that modelling the data directly would work for your
purposes, then one way of doing this would be to regress classification
decisions ("P" or "N") on actual classifications ("P" or "N").

If this is done in a probit model, it is equivalent to the equal-variance
signal detection model studied at length in psychology, with the
intercept being the "criterion" in signal detection language (denoted c),
and the slope being "sensitivity" (denoted d' or d-prime). It should
definitely be possible to compute precision and recall from c and
d'.

This might be simpler with a logit rather than probit link
function.

Let me know if I have misunderstood what you are trying to
accomplish

Jake

From: rdiaz02 at gmail.com.> To:
r-sig-mixed-models at r-project.org.> Date: Tue, 11 Mar 2014 11:48:57
+0100.> CC: ramon.diaz at iib.uam.es.> Subject: [R-sig-ME] Modeling
precision and recall with GLMMs.> .

 Dear All,. .

 I am examining the performance of a couple of classification-like
 methods. under different scenarios. Two of the metrics I am using are
 precision and. recall (TP/(TP + FP) and TP/(TP + FN), where TP, FP, and
 FN are "true. positives", "false positives", and "false negatives" in a
 simple two-way. confusion matrix). Some of the combinations of methods
 have been used on. exactly the same data sets. So it is easy to set up a
 binomial model (or. multinomial2 if using MCMCglmm) such as.

cbind(TP, FP) ~ fixed effects + (1|dataset)

However, the left hand side sounds questionable, specially with
precision:. the expression TP/(TP + FP) has, in the denominator, a (TP +
FP) [the. number of results returned, or retrieved instances, etc] that,
itself, can. be highly method-dependent (i.e., affected by the fixed
effects). So rather. than a true proportion, this seems more like a
ratio, where each of TP and. FP have their own variance, a covariance,
etc, and thus the error. distribution is a mess (not the tidy thing of a
binomial).

I've looked around in the literature and have not found much (maybe
the. problem are my searching skills :-). Most people use rankings of
methods,. not directly modeling precision or recall in the left-hand
side of a. (generalized) linear model. A couple of papers use a linear
model on the. log-transformed response (which I think is even worse than
the above. binomial model, specially with lots of 0s or 1s). Some other
people use a. single measure, such as the F-measure or Matthews
correlation coefficient,. and I am using something similar too, but I
specifically wanted to also. model precision and recall.. . .

An option would be a multi-response model with MCMCglmm, but I am not
sure if this is appropriate either (dependence of the sum of FP and TP
on the. fixed effects).. . .

Best,

R-sig-mixed-models at r-project.org mailing list