fixed effect testing again (but different)

Daniel,

I think you're running into a problem of model misspecification (where the population has effects that are omitted from your model in lmer).

1. If there aren't any fixed effects in the population and you omit the fixed effects from lmer but include the random effects in lmer (call that lmer model A) then lmer should correctly model the heteroscedasticity.

2. If there are fixed effects in the population and you include the fixed effects as well as the random effects (call that lmer model B) in lmer then lmer should correctly model the heteroscedasticity.

3. If there are fixed effects in the population and you omit the fixed effects from lmer but include the random effects in lmer (model A again) then lmer might not correctly model the data because of model misspecification. I think this is the
problem you're running into.

Assuming that you keep the heteroscedasticity in the lmer models, testing for the presence of fixed effects amounts to comparing the fit of model B against the fit of model A. If the fixed effects are present then you'll accept model B and use that
model's coefficients to estimate the heteroscedasticity; if the fixed effects are absent then you'll accept model A and use that model's coefficients to estimate the heteroscedasticity.

alan

What I'm observing is that this procedure only works if the fixed
effect corresponding to the subject split is also included in the
model. Then, the M|subject and F|subject terms are both distributed
around zero and the results do seem correct.

I was interested in testing the fixed effect term, so I was trying to
run a model estimating the subject variance separately for males and
females, but without a fixed effect term for the difference between
them.

I thought that difference would get absorbed into the random effects
(e.g. the dummy slopes for M distributed around -3, those for F
distributed around +3)  but that the variances would come out
accurately.

This seems to run into trouble, maybe because it's a violation of the
assumptions that the random effects are distributed around zero. What
seems to happen is that the intercept and random effects are
accurately estimated for one stratum, and then the other stratum isn't
right.

But if hypothesis testing isn't the issue and you always include a
fixed-effect term for the same outer variable that you're doing the
heteroscedasticity 'hack' for, then yes, it does seem to work.