Dear All,
I have collected data from primary studies where an assessment aid
(diagrams) was developed and tested for improvements in accuracy and
precision of visual assessments (% leaf area affected - disease symptoms)
compared with unaided ones.
A sample of raters was used in each study (varied across studies). Each
rater made two assessments (estimates of % leaf affected area) for a same
sample of leaves (a range of values from 0 to 100%), first unaided (und)
and then aided (aid) (pre-post test).
Visual estimates from each assessment (e.g. 50 ratings) were regressed
against ?actual? values. Regression coefficients (beta1 and beta1) are
measures of accuracy and the correlation coefficient (r) is a measure of
precision. So, I have three variables for each rater.
The data look like this:
Study Rater r_und r_aid
1 1 0.65 0.76
1 2 0.76 0.90
1 3 0.80 0.90
. . . .
Wolfgang has kindly helped me (more than two years ago!) to preliminary
fit a multi-level model in metafor to summarize the gains precision
(correlation coefficients). The effect-size was the absolute difference
(r_aided - r_unaided) and there was a way to calculate sampling variance.
This worked fine, but I have been struggling to define what would be the
appropriate approach to analyze gains in accuracy using beta1 and beta1. In
a primary study, vote-couting was used to infer on the value of the aid
based on number of raters with significant (P = 0.05) departures of beta1
and beta2 from 0 and 1, respectively, both unaided and aided.
My problem is that I cannot calculate an index (such as concordance
correlation) for accuracy because raw data was not available (I have the
correlation coefficient, which explains in part the overall accuracy or
concordance). So, I don?t think that an estimate of absolute difference in
beta1 and beta2 between aided and unaided estimate for each rater make
sense? I can see (histograms) that, in general, b1 and b2 are closer to 0
and 1, respectively when using the aid.
What I did so far was to aggregate (means) each coefficients by study to
then obtain the sampling variance (raters as samples) for the study. Then,
a bivariate model was fitted to these data from k studies and the estimates
of b1 and b2 were obtained for each condition. The data looks exactly like
shown above, but with b1_und and b1_aid, etc.
I am not sure if there is a better way to analyze these data. Ideally, I
would like to be able to calculate the concordance coefficient (which
includes the correlation coefficient) but this seems not possible without
the raw data. What I have allows me to analyze separately precision and
accuracy, but while the former seems OK (I have an effect-size and sampling
variance for each rater), I am not sure how to better estimate gains in
accuracy using the regression coefficients.
Any thoughts or examples on this?
Thanks,
Emerson
Prof. Emerson M. Del Ponte
Departamento de Fitopatologia
Universidade Federal de Vi?osa
Vi?osa, MG - Brasil
+55 (31) 3899-1103 <+55%2031%203899-1103>
Twitter: @edelponte