Skip to content

comparing 2 models

6 messages · P Greenwood, Voeten, C.C., Emmanuel Curis +3 more

Original
P Greenwood
# 
Hello

Although alpha-band EEG is something of a gold standard in predicting processing of stimuli (pre-stimulus alpha), it is not easy to measure EEG outside the lab (e.g. in a vehicle).  In contrast, heart-rate and eye gaze are more reliably obtained in the field using low-cost wearable sensors. 

We sought to compare the ability of 3 sensors to model human reaction time (RT) to a signal from automation.  We compared these measures: alpha-band (PzAlpha), heart-rate variability (HRV), and eye gaze (lnX).  Each person has 10 trials in each of 5 drives. Pnum is subject number.

We sought to model RT from  alpha-band,  heart-rate variability (HRV), and eye gaze (lnX) and Drive.

sumModelInteraction4 <- lmer(RT ~ 1 + (PzAlpha+ HRV+lnX+Drive)^3 +  (1 | Pnum) + (1 | Trial), data = INFAST_Behavioralnew, REML = FALSE)

The output, pasted below, reveals interactions: PzAlpha x HRV, PzAlpha x Drive, and HRV x Drive. Also pasted below is some of the raw data.

An LRT comparing this model with an additive model (PzAlpha+RMSSD+lnX+Drive+Trial) yields a significant difference.  This suggests that the interactions  PzAlpha x Drive and HRV x Drive are meaningful predictors. 

We would like to determine whether PzAlpha x Drive or  HRV x Drive is the better predictor of RT.  What is the best way to compare those 2 models?  The measures are scaled and centered.

Thank you very much.

Pam Greenwood

Fixed effects:
                      Estimate Std. Error         df t value Pr(>|t|)   
(Intercept)         -1.908e-01  1.368e-01  3.310e+01  -1.395  0.17235   
PzAlpha             -1.098e-01  6.042e-02  2.339e+02  -1.818  0.07041 . 
HRV               -1.127e-01  6.402e-02  1.112e+03  -1.760  0.07868 . 
lnX                  7.921e-02  7.547e-02  1.102e+03   1.050  0.29416   
Drive                5.382e-02  1.661e-02  1.080e+03   3.241  0.00123 **
PzAlpha: HRV       -1.586e-01  6.424e-02  7.035e+02  -2.468  0.01382 * 
PzAlpha:lnX          1.116e-01  7.400e-02  8.456e+02   1.508  0.13199   
PzAlpha:Drive        4.405e-02  1.730e-02  1.049e+03   2.546  0.01103 * 
HRV:lnX           -4.723e-02  6.905e-02  1.100e+03  -0.684  0.49407   
HRV:Drive          3.652e-02  1.747e-02  1.098e+03   2.091  0.03677 * 
lnX:Drive           -1.275e-02  1.992e-02  1.100e+03  -0.640  0.52217   
PzAlpha: HRV:lnX    1.473e-02  2.567e-02  8.244e+02   0.574  0.56621   
PzAlpha: HRV:Drive  3.308e-02  1.878e-02  1.084e+03   1.761  0.07845 . 
PzAlpha:lnX:Drive   -2.325e-02  2.005e-02  9.166e+02  -1.160  0.24640   
HRV:lnX:Drive      9.729e-03  1.769e-02  1.097e+03   0.550  0.58247   


	
	Pnum
	Drive		Trial	RT	ACC	FzAlpha	CzAlpha	PzAlpha	FzTheta	CzTheta	PzTheta	MeanPupil	lnX	lnY	MeanRR	
HRV
20	1	1	1480.8931	1	7.9928	10.216	7.6254	3.4916	4.8657	6.4977	4.280969072	-3.208115816	-2.423813328	0.7336  		0.074666667
20	1	2	1983.254	1	-8.2609	0.62018	0.32812	4.2257	6.0181	6.5564	4.360414101	-1.926558582	-2.364252526	0.7336		0.074666667
20	1	3	1588.0317	1	1.2572	5.5394	9.0619	4.322	6.7421	7.2778	4.429370379	-2.510514134	-2.890876402	0.734		0.073333333
20	1	4	2600	0	-2.0822	-3.5216	2.597	7.6632	9.4505	9.2404	3.994177574	-2.121340179	-3.875411777	0.7408		0.050666667
20	1	5	1268.9969	1	2.463	4.5837	4.0916	3.4363	4.2989	-2.1573	3.927884406	-1.754642861	-2.737213207	0.7516		0.014666667




















































0





























Editorial Board, NeuroImage
David King Hall 2052
George Mason University
MSN 3F5, 4400 University Drive
Fairfax, VA 22030-4444

Ph: 703 993-4268
fax: 703 993-1359
email: Pgreenw1 at gmu.edu
http://psychology.gmu.edu/people/pgreenw1
Voeten, C.C.
# 
Hi Pam,

I don't know if it is the 'best' way, but 'a' way could be to simply compare the AICs of these two models?

lmer(RT ~ 1 + PzAlpha+ HRV+lnX+Drive + PzAlpha:Drive +  (1 | Pnum) + (1 | Trial), data = INFAST_Behavioralnew, REML = FALSE)
lmer(RT ~ 1 + PzAlpha+ HRV+lnX+Drive + HRV:Drive +  (1 | Pnum) + (1 | Trial), data = INFAST_Behavioralnew, REML = FALSE)

Best,
Cesko

-----Oorspronkelijk bericht-----
Van: R-sig-mixed-models <r-sig-mixed-models-bounces at r-project.org> Namens P Greenwood
Verzonden: woensdag 25 april 2018 03:18
Aan: r-sig-mixed-models at r-project.org
Onderwerp: [R-sig-ME] comparing 2 models

Hello

Although alpha-band EEG is something of a gold standard in predicting processing of stimuli (pre-stimulus alpha), it is not easy to measure EEG outside the lab (e.g. in a vehicle).  In contrast, heart-rate and eye gaze are more reliably obtained in the field using low-cost wearable sensors. 

We sought to compare the ability of 3 sensors to model human reaction time (RT) to a signal from automation.  We compared these measures: alpha-band (PzAlpha), heart-rate variability (HRV), and eye gaze (lnX).  Each person has 10 trials in each of 5 drives. Pnum is subject number.

We sought to model RT from  alpha-band,  heart-rate variability (HRV), and eye gaze (lnX) and Drive.

sumModelInteraction4 <- lmer(RT ~ 1 + (PzAlpha+ HRV+lnX+Drive)^3 +  (1 | Pnum) + (1 | Trial), data = INFAST_Behavioralnew, REML = FALSE)

The output, pasted below, reveals interactions: PzAlpha x HRV, PzAlpha x Drive, and HRV x Drive. Also pasted below is some of the raw data.

An LRT comparing this model with an additive model (PzAlpha+RMSSD+lnX+Drive+Trial) yields a significant difference.  This suggests that the interactions  PzAlpha x Drive and HRV x Drive are meaningful predictors. 

We would like to determine whether PzAlpha x Drive or  HRV x Drive is the better predictor of RT.  What is the best way to compare those 2 models?  The measures are scaled and centered.

Thank you very much.

Pam Greenwood

Fixed effects:
                      Estimate Std. Error         df t value Pr(>|t|)   
(Intercept)         -1.908e-01  1.368e-01  3.310e+01  -1.395  0.17235   
PzAlpha             -1.098e-01  6.042e-02  2.339e+02  -1.818  0.07041 . 
HRV               -1.127e-01  6.402e-02  1.112e+03  -1.760  0.07868 . 
lnX                  7.921e-02  7.547e-02  1.102e+03   1.050  0.29416   
Drive                5.382e-02  1.661e-02  1.080e+03   3.241  0.00123 **
PzAlpha: HRV       -1.586e-01  6.424e-02  7.035e+02  -2.468  0.01382 * 
PzAlpha:lnX          1.116e-01  7.400e-02  8.456e+02   1.508  0.13199   
PzAlpha:Drive        4.405e-02  1.730e-02  1.049e+03   2.546  0.01103 * 
HRV:lnX           -4.723e-02  6.905e-02  1.100e+03  -0.684  0.49407   
HRV:Drive          3.652e-02  1.747e-02  1.098e+03   2.091  0.03677 * 
lnX:Drive           -1.275e-02  1.992e-02  1.100e+03  -0.640  0.52217   
PzAlpha: HRV:lnX    1.473e-02  2.567e-02  8.244e+02   0.574  0.56621   
PzAlpha: HRV:Drive  3.308e-02  1.878e-02  1.084e+03   1.761  0.07845 . 
PzAlpha:lnX:Drive   -2.325e-02  2.005e-02  9.166e+02  -1.160  0.24640   
HRV:lnX:Drive      9.729e-03  1.769e-02  1.097e+03   0.550  0.58247   


	
	Pnum
	Drive		Trial	RT	ACC	FzAlpha	CzAlpha	PzAlpha	FzTheta	CzTheta	PzTheta	MeanPupil	lnX	lnY	MeanRR	
HRV
20	1	1	1480.8931	1	7.9928	10.216	7.6254	3.4916	4.8657	6.4977	4.280969072	-3.208115816	-2.423813328	0.7336  		0.074666667
20	1	2	1983.254	1	-8.2609	0.62018	0.32812	4.2257	6.0181	6.5564	4.360414101	-1.926558582	-2.364252526	0.7336		0.074666667
20	1	3	1588.0317	1	1.2572	5.5394	9.0619	4.322	6.7421	7.2778	4.429370379	-2.510514134	-2.890876402	0.734		0.073333333
20	1	4	2600	0	-2.0822	-3.5216	2.597	7.6632	9.4505	9.2404	3.994177574	-2.121340179	-3.875411777	0.7408		0.050666667
20	1	5	1268.9969	1	2.463	4.5837	4.0916	3.4363	4.2989	-2.1573	3.927884406	-1.754642861	-2.737213207	0.7516		0.014666667




















































0





























Editorial Board, NeuroImage
David King Hall 2052
George Mason University
MSN 3F5, 4400 University Drive
Fairfax, VA 22030-4444

Ph: 703 993-4268
fax: 703 993-1359
email: Pgreenw1 at gmu.edu
http://psychology.gmu.edu/people/pgreenw1



_______________________________________________
R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
Emmanuel Curis
# 
Hello,

I think if your interest is more in predictive power of your model
than in its ability to reproduce the dataset, your best choice would
be using cross-validation.  And to summarize the closeness of your
predicted results to the real ones, you may use all the concordance
tools, either descriptive like Bland-Altman plots, or more
quantititive like concordance correlation coefficients (but beware
that their interpretation beyond ? the closest to one, the better ? is
not easy) or similar agreement measures. Obviously, the second part
can be done without the cross-validation step, on your single, whole
dataset.

The ? best ? model will be the one witht the highest concordance
correlation coefficient; to test that it is significantly better is
more tricky, but should be done with carefully crafted simulations,
fitting one model to simulated data generated by either one model or
the other and vice-versa...

Note also that if all your predictors are continuous, as seems to be
in your description, the syntax (x+y+...)^n is misleading, because it
forgets some important terms.

Indeed, let consider the two-variables case, x and y. The first order
model is z = ?0 + alpha * x + beta * y, a plane. The second order
model would then be a paraboloid,

z = ?0 + alpha * x + beta * y + gamma * x? + delta * y? + a * x * y

However, the syntax (x+y)^2 expends to x + y + x:y, that is it forgets
the I(x^2) and I(y^2) terms. So, unless you have strong belief that
both gamma and delta are 0, you're model is incomplete.

Hope all of this will help,
Best regards,
Emmanuel
On Tue, Apr 24, 2018 at 09:17:32PM -0400, P Greenwood wrote:
? Hello
? 
? Although alpha-band EEG is something of a gold standard in predicting processing of stimuli (pre-stimulus alpha), it is not easy to measure EEG outside the lab (e.g. in a vehicle).  In contrast, heart-rate and eye gaze are more reliably obtained in the field using low-cost wearable sensors. 
? 
? We sought to compare the ability of 3 sensors to model human reaction time (RT) to a signal from automation.  We compared these measures: alpha-band (PzAlpha), heart-rate variability (HRV), and eye gaze (lnX).  Each person has 10 trials in each of 5 drives. Pnum is subject number.
? 
? We sought to model RT from  alpha-band,  heart-rate variability (HRV), and eye gaze (lnX) and Drive.
? 
? sumModelInteraction4 <- lmer(RT ~ 1 + (PzAlpha+ HRV+lnX+Drive)^3 +  (1 | Pnum) + (1 | Trial), data = INFAST_Behavioralnew, REML = FALSE)
? 
? The output, pasted below, reveals interactions: PzAlpha x HRV, PzAlpha x Drive, and HRV x Drive. Also pasted below is some of the raw data.
? 
? An LRT comparing this model with an additive model (PzAlpha+RMSSD+lnX+Drive+Trial) yields a significant difference.  This suggests that the interactions  PzAlpha x Drive and HRV x Drive are meaningful predictors. 
? 
? We would like to determine whether PzAlpha x Drive or  HRV x Drive is the better predictor of RT.  What is the best way to compare those 2 models?  The measures are scaled and centered.
? 
? Thank you very much.
? 
? Pam Greenwood
? 
? Fixed effects:
?                       Estimate Std. Error         df t value Pr(>|t|)   
? (Intercept)         -1.908e-01  1.368e-01  3.310e+01  -1.395  0.17235   
? PzAlpha             -1.098e-01  6.042e-02  2.339e+02  -1.818  0.07041 . 
? HRV               -1.127e-01  6.402e-02  1.112e+03  -1.760  0.07868 . 
? lnX                  7.921e-02  7.547e-02  1.102e+03   1.050  0.29416   
? Drive                5.382e-02  1.661e-02  1.080e+03   3.241  0.00123 **
? PzAlpha: HRV       -1.586e-01  6.424e-02  7.035e+02  -2.468  0.01382 * 
? PzAlpha:lnX          1.116e-01  7.400e-02  8.456e+02   1.508  0.13199   
? PzAlpha:Drive        4.405e-02  1.730e-02  1.049e+03   2.546  0.01103 * 
? HRV:lnX           -4.723e-02  6.905e-02  1.100e+03  -0.684  0.49407   
? HRV:Drive          3.652e-02  1.747e-02  1.098e+03   2.091  0.03677 * 
? lnX:Drive           -1.275e-02  1.992e-02  1.100e+03  -0.640  0.52217   
? PzAlpha: HRV:lnX    1.473e-02  2.567e-02  8.244e+02   0.574  0.56621   
? PzAlpha: HRV:Drive  3.308e-02  1.878e-02  1.084e+03   1.761  0.07845 . 
? PzAlpha:lnX:Drive   -2.325e-02  2.005e-02  9.166e+02  -1.160  0.24640   
? HRV:lnX:Drive      9.729e-03  1.769e-02  1.097e+03   0.550  0.58247   
? 
? 
? 	
? 	Pnum
? 	Drive		Trial	RT	ACC	FzAlpha	CzAlpha	PzAlpha	FzTheta	CzTheta	PzTheta	MeanPupil	lnX	lnY	MeanRR	
? HRV
? 20	1	1	1480.8931	1	7.9928	10.216	7.6254	3.4916	4.8657	6.4977	4.280969072	-3.208115816	-2.423813328	0.7336  		0.074666667
? 20	1	2	1983.254	1	-8.2609	0.62018	0.32812	4.2257	6.0181	6.5564	4.360414101	-1.926558582	-2.364252526	0.7336		0.074666667
? 20	1	3	1588.0317	1	1.2572	5.5394	9.0619	4.322	6.7421	7.2778	4.429370379	-2.510514134	-2.890876402	0.734		0.073333333
? 20	1	4	2600	0	-2.0822	-3.5216	2.597	7.6632	9.4505	9.2404	3.994177574	-2.121340179	-3.875411777	0.7408		0.050666667
? 20	1	5	1268.9969	1	2.463	4.5837	4.0916	3.4363	4.2989	-2.1573	3.927884406	-1.754642861	-2.737213207	0.7516		0.014666667
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 0
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? 
? Editorial Board, NeuroImage
? David King Hall 2052
? George Mason University
? MSN 3F5, 4400 University Drive
? Fairfax, VA 22030-4444
? 
? Ph: 703 993-4268
? fax: 703 993-1359
? email: Pgreenw1 at gmu.edu
? http://psychology.gmu.edu/people/pgreenw1
? 
? 
? 	[[alternative HTML version deleted]]
? 
? _______________________________________________
? R-sig-mixed-models at r-project.org mailing list
? https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
Jonathan Judge
# 
I think the best option would be to refit these models in Stan (a front end like rstanarm or brms makes it easy with similar lme4 syntax) and use the LOO function to compare their out of sample predictive power. 

Otherwise, if you wish to stick with lme4, you need some way to properly count the number of effective parameters. AIC doesn?t do that because it doesn?t understand / account for the shrinkage of random effects. You might consider WAIC, which does a better job of this. There is an implementation for lme4 models in the blmeco package. 

Jonathan

Sent from my iPhone
Phillip Alday
# 
In addition to the previous suggestions (CV/LOO and AIC comparison),
there is also Vuong's closeness test for non nested models. I haven't
used it myself, but a quick search suggests that a fair amount of
information about its use and misuse (in e.g. checking for
zero-inflation in Poisson models) is available.

One small note here: PzAlpha and HRV are both continuous predictors, so
they both contribute the same number of degrees of freedom to a model,
so you can compare (log) likelihoods directly instead of using AIC or
BIC because the number of parameters will be the same. (You still can't
do the LRT because amongst other things the chisq df will be 0.) This
also suggests the parameter counting concern  is largely moot, as the
parameters and data are the same. (I'll let someone else more qualified
address the broader issue of parameter counting for AIC/BIC in mixed
models.)

There are also a number of parameter selection methods that work within
one model. In frequentist land, there's elastic net and LASSO, which are
implemented for mixed models in packages such as lmmen and glmmLasso.
There's two ways to use those methods here:

(1) cross validation on the penalty for number of parameters so that the
model selects the 'ideal' number of predictors
(2) setting a particularly penalty to achieve a desired number of
parameters and letting the model choose which ones those are

Both should be typically well supported in good elastic net and LASSO
implementations.

In Bayesian terms, you can achieve the same thing by appropriate choice
of priors. brms has LASSO and horseshoe priors and uses early the same
syntax as lme4:

sumModelInteraction4 <- brm(RT ~ 1 + (PzAlpha+ HRV+lnX+Drive)^3 +  (1 |
Pnum) + (1 | Trial), data = INFAST_Behavioralnew,
prior=set_prior('horseshoe',class='b'))

(The author of brms, Paul B?rkner, also contributes to this list and
might be able to say more about ways of addressing this issue.)

Finally, I see that you're using untransformed RTs in a linear/gaussian
model. There's a lot of debate on whether you should transform or use a
non linear/gaussian model (e.g. log transform, use the gamma transform
with identity or inverse link, etc.). I would check to see how well the
current model actually describes the data and see if non-gaussian models
or a suitable transformation improves fit before doing variable selection.

Best,
Phillip
On 04/25/2018 01:47 PM, Jonathan Judge wrote:
Ben Bolker
# 
WAIC may be better than cAIC (conditional AIC: Greven and Kneib), for
all I know, but just pointing out for completeness that there's a cAIC4
package that works with lme4.
On 2018-04-25 07:47 AM, Jonathan Judge wrote: