Skip to content

lmer: No significant coefficients, but significant improvement of model fit?

5 messages · Ben Bolker, Gjalt-Jorn Peters, Jarrod Hadfield +1 more

Ben Bolker
# 
<r-sig-mixed-models at ...> writes:
thanks.  I will answer the short version and see how far I get
with the long version.
This is not really an lmer question, but a more general modeling
question.  There are a few things you could mean here, but I don't
think any of them have to do with the "p-value issue", which is
more one of how to deal with the unknown distribution of the test
statistic under the null hypothesis for not-large data sets
{see http://glmm.wikidot.com/faq for more links on the p-value stuff,
and others}

  * you could be asking about the difference between the results
of summary() [which uses Wald tests based on local curvature]
and anova() [which does a more precise test based on model comparison];
anova() is not perfect, but it's more accurate (and hence sometimes
different from) summary
  * you could be asking about multiple predictors, none of which
is individually significant at p<0.05, but their combined effects
(i.e. comparing a model with all predictors vs. none) are significant 
at p<0.05.  This is not really surprising, because the joint effect
of the predictors can be stronger than any one individually.  (Also,
if you're not working with a balanced, nested LMM, the effects of
the predictors can interact.)
A colon refers to an interaction, a slash refers to nesting (so
~a/b is equivalent to ~a+a:b, or "b nested within a"): there's more
on this at the wikidot FAQ as well.
* cannabisUse_bi: a factor with two levels, \"0\" and \"1\". \'0\'
  indicates no cannabis use in the past week; \'1\' indicates cannabis
  use in the past week. This is the dependent variable (i.e. the
  criterion).
* moment: a factor with two levels, \'before\' and \'after\'
* id.factor: a factor with 444 levels, the identification of each
  participants (note that there are quite a lot of missing values,
  only about 276 cases without missings)
* school: a factor with 8 levels, each representing the school that
  the participants attend
* cannabisShow: a factor with 2 levels, \'control\' and
 \'intervention\' - this reflects whether a participant received the
 \'intervention\', aimed to decrease cannabis use, or
 not. Participants in five schools received the intervention;
 participants in three other schools didn\'t.
Although others on this list disagree, I don't find "nesting" to be
very useful in the context of fixed effects, because the levels of
fixed effects almost always have identical meanings across different
levels of the random effect (i.e., "before" means the same for me as
for you)

 I would say the simplest sensible model would be

glmer(cannabisUse_bi ~ cannabisShow*moment + (1|school/id.factor), 
    family=binomial, data=dat.long)

which if your individuals are uniquely identified should be the same
as using (1|school) + (1|id.factor) as the random effects.

But I agree that you may very well want to try to take into account
whether the effects of the fixed effects differ among schools: you
might _like_ to see whether they differ among individuals as well, but
it is somewhere between impossible and very difficult to extract this
from binary data per individual (I'm sure you can't identify the
effects of cannabisShow, because each individual only gets one
intervention, and I'm pretty sure that you can't identify the effects
of before/after either, because all you have is binary data -- if you
had continuous data you *might* be able to detect variation in slope
among individuals, if it weren't confounded with residual error).

So I would try

glmer(cannabisUse_bi ~ cannabisShow*moment +
   (cannabisShow*moment|school) + (1|id.factor), family=binomial,
   data=dat.long)

(assuming that id.factor is unique across schools)
if I include that, I get:
Maybe the before/after variation among schools (moment|school) is
  doing a lot?  Also, see my comment above about Wald tests.
glmer provides likelihood ratio tests, which are good when the
sample size is large.  If you didn't have the school level I would say
not to worry about it, but 8 schools is not a large number ...
The second leaves out the school effect, as specified above.
14393-Need-help-with-lmer-model-specification-syntax-for-nested-mixed-model
Gjalt-Jorn Peters
# 
Dear Ben,

thank you very, very much for your extensive answer!

I have run both models you suggest, and they fit equally well (Chi^2 = 
.99). In addition, the effect of each predictor is the same in both 
models (which makes sense of course).

I will just look at each effects' significance level, as per your 
worries regarding the number of schools (i.e. 8 is not many school).

Before endeavoring more interpretation, though, I will first read up on 
mixed models using the valuable suggestions given earlier :-)

Again, thank you very, very much for your answer, kind regards,

Gjalt-Jorn

*Gjalt-Jorn Peters* | http://behaviorchange.eu

Behavior change research | Health psychology
Intervention development | Applied social psychology 	[ 	GG 
<http://greatergood.eu> 	OU <http://ou.nl> 	UM 
<http://maastrichtuniversity.nl> 	}
On 07-11-2012 16:00, Ben Bolker wrote:
Jarrod Hadfield
# 
Hi,

I've uploaded a new version of MCMCglmm to CRAN. Main additions are I)  
more flexible methods for multi-membership and related models II) bug  
fix for the predict function for certain types of random-effect  
marginalisation.  III) sir models reinstated  IV) proposal  
distribution for MH steps returned.

Cheers,

Jarrod



I) The addition of a linking.function that links different random  
effects together. For example imagine two random terms, mother and  
grandmother, for which some mothers also appear as grandmothers.  
Denoting the associated random effect for the mothers (m) and  
grandomothers (g)  we could:

a) fit the simple model ~mother+grandmother which estimates separate  
variances (VAR(m) and VAR(g)) and sets the covariance to zero  
(COV(m,g)=0)

b) use the linking function "str" to fit the model  
~str(mother+grandmother) which estimates separate variances (VAR(m)  
and VAR(g)) but also estimates the covariance (COV(m,g))

c) use the linking function "mm" to fit a multimembership model  
~mm(mother+grandmother) which forces the variances to be equal  
(VAR(m)= VAR(g)) and forces the correlation to be one  
(COV(m,g)=VAR(m)= VAR(g)). Multi-membership models can still be fit  
using idv(mult.memb(~mother+grandmother))

Terms within mm or str can be linked to a ginverse if the ginverse  
list name corresponds to the first term in the linking.function (i.e.  
ginverse=list(mother=A) in the models above). They can also be  
interacted with variance functions (i.e.  
us(sex):str(mother+grandmother) is possible)

II) The predict function did not obtain the correct contribution to  
the variance from the marginalised random effects when us(function)  
defined the marginalised terms and the function was such that a single  
datum was associated with >1 term. For example, in a random regression  
us(1+x) we have the variance for datum i as  
V[1,1]+2*x[i]*V[1,2]+(x[i]^2)*V[2,2]  where V[1,1] is the variance in  
intercept, V[2,2] is the variance in slopes and V[1,2] the covariance  
between intercept and slope.  The term 2*x[i]*V[1,2] was omitted in  
the older versions.  This may effect confidence intervals and fitted  
values on the data scale for non-gaussian data, and prediction  
intervals more generally.

III) sir models have gone back to a dense specification. This means  
that big data sets may run out of memory when setting up the  
equations, but at least it will run if this is not the case.

IV) Tune element in output gives the proposal distribution for the  
latent variables that was used (after the adaptive phase).
David Atkins
# 
[Posting to list as others might be interested...]

Jarrod--

Very cool to see the continued development of MCMCglmm.

My typical use of predict() functions (across various R regression-based 
commands) involves generating predictions on newdata -- typically to 
help interpret models involving non-linear terms and/or interactions. 
As far as I can tell, the predict function in v2.17 of MCMCglmm does not 
yet incorporate new data.

Any guess on when the newdata argument in predict.MCMCglmm might "come 
online"?

cheers, Dave
Jarrod Hadfield
# 
Hi Dave,

I did intend to make it part of the current version. The difficulty is  
that if the fixed predictors in newdata have less levels than those in  
data, then things like the intercept will have a different  
interpretation. If data and newdata could be guaranteed to have the  
same levels for fixed terms (and terms within a variance.function)  
then it would be more straightforward. I guess I could return an error  
if this was not the case, and allow predictions on newdata when these  
conditions were satisfied....

Cheers,

Jarrod


Quoting David Atkins <datkins at u.washington.edu> on Thu, 08 Nov 2012  
16:18:36 -0800: