Question about non-significant interactions

Dear Francesco,
-----Original Message-----
From: Francesco Romano [mailto:fbromano77 at gmail.com]
Sent: Thursday, July 11, 2019 9:50 AM
To: Fox, John <jfox at mcmaster.ca>
Cc: Thierry Onkelinx <thierry.onkelinx at inbo.be>; r-sig-mixed-models at r-
project.org
Subject: Re: [R-sig-ME] Fwd: Question about non-significant interactions

Dear John (and others),

I think I have solved this issue, if my understanding of the stats is correct.

First, wIth regards to (2), yes, I have taken that into account. I recoded my
factors as contr.sum, ran the Anova again (with and without the bayesian
wrapper), to find that results did not change.
The tests for the interaction should be identical but it would be odd if the type-III tests for the main effects didn't change at least slightly when you changed the contrast coding from contr.treatment() to contr.sum().
I also tried the other ways,
namely via the mixed function, which allows you to use mixed-effect models,
automatically recodes your factors to contr.sum, and uses LRT, and the
traditional anova function with lihelihood ratio tests comparing two models
at a time. In all cases, I never get a significant interaction but always get a
main effect for both fixed effects. By the way, regarding your (3), our original
contact a few years back was a query of mine trying to find a shortcut to
traditional likelihood ratio testing comparing models that would also allow
random effects, hence your suggestion to use your car::Anova, which for the
record, I love.
Thank you.
As far as my understanding of the results goes, but feel free to chime in if you
disagree, there is no main interaction because the simple effects that I find
(i.e. the pairwise comparisons) are the same across the two tests. To simplify
things a bit, in my GJT test, the first level of the task factor, I find the following
relationships between groups HL = L2, L1 > HL, and L1 > L2, where HL, L1, and
L2 are the three levels of the group factor, and = is just shorthand for no
significant difference. The relationships are exactly the same at the other level
of the task factor, namely priming, Same goes for the relationship between
tasks at each level of the group variable: every group is significantly more
likely to score correctly on the GJT than the priming task. I take these to
explain why the main interaction turns out not to be statistically significant.
If the simple effects of group are identical across the levels of task, then of course there is no interaction between group and task. If the simple effects are similar, then you probably won't detect an interaction unless, e.g., you have a lot of data. But to treat "no significant difference" as indicative of equality (even heuristically) could certainly lead to apparently paradoxical results, although that hasn't happened in your case.

Best,
 John
Best,

Frank

On Wed, Jul 10, 2019 at 3:43 PM Fox, John <jfox at mcmaster.ca
<mailto:jfox at mcmaster.ca> > wrote:

	Dear Francesco and Thierry,

	To elaborate slightly on Thierry's suggestion and also to address some
other points:

	(1) You (Francesco) could use the linearHypothesis() function in the
car package to test more specific hypotheses. It would probably be easier to
formulate such hypotheses if you fit an equivalent cell-mean model, defining a
factor with levels for the 9 = 3*3 combinations of levels of your two factors.

	(2) The type-III tests produced by Anova() still aren't sensible. Did you
look at the material from ?Anova that I included in my previous response?

	(3) You say that Anova() uses "a form of shortcut to the traditional
way of model-fitting/ reduction via the function anova() comparing models."
That's not quite true in general, because what Anova() does depends on the
model class and the type of test statistic selected. It's true that for a mixed-
effects model, Anova() produces Wald tests rather than likelihood-ratio tests.
Anova() doesn't have a "bglmerMod" method, and so I assume that the
"merMod" method is inherited or that the default method is being used.

	Best,
	 John

	-----------------------------
	John Fox, Professor Emeritus
	McMaster University
	Hamilton, Ontario
	Canada L8S 4M4
	web: socserv.mcmaster.ca/jfox <http://socserv.mcmaster.ca/jfox>

	________________________________________
	From: R-sig-mixed-models [r-sig-mixed-models-bounces at r-
project.org <mailto:r-sig-mixed-models-bounces at r-project.org> ] on behalf of
Thierry Onkelinx via R-sig-mixed-models [r-sig-mixed-models at r-project.org
<mailto:r-sig-mixed-models at r-project.org> ]
	Sent: July 10, 2019 2:51 AM
	To: Francesco Romano
	Cc: r-sig-mixed-models at r-project.org <mailto:r-sig-mixed-models at r-
project.org>
	Subject: Re: [R-sig-ME] Fwd: Question about non-significant
interactions

	Dear Francesco,

	To answer your question, you should convert your hypothesis in a set
of
	linear contrasts and test those.

	Best regards,

	ir. Thierry Onkelinx
	Statisticus / Statistician

	Vlaamse Overheid / Government of Flanders
	INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH
INSTITUTE FOR NATURE AND
	FOREST
	Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality
Assurance
	thierry.onkelinx at inbo.be <mailto:thierry.onkelinx at inbo.be>
	Havenlaan 88 bus 73, 1000 Brussel
	www.inbo.be <http://www.inbo.be>

	///////////////////////////////////////////////////////////////////////////
////////////////
	To call in the statistician after the experiment is done may be no more
	than asking him to perform a post-mortem examination: he may be
able to say
	what the experiment died of. ~ Sir Ronald Aylmer Fisher
	The plural of anecdote is not data. ~ Roger Brinner
	The combination of some data and an aching desire for an answer
does not
	ensure that a reasonable answer can be extracted from a given body
of data.
	~ John Tukey
	///////////////////////////////////////////////////////////////////////////
////////////////

	<https://www.inbo.be>

	Op di 9 jul. 2019 om 23:25 schreef Francesco Romano
<fbromano77 at gmail.com <mailto:fbromano77 at gmail.com> >:

	> ---------- Forwarded message ---------
	> From: Francesco Romano <fbromano77 at gmail.com
<mailto:fbromano77 at gmail.com> >
	> Date: Tue, Jul 9, 2019 at 11:24 PM
	> Subject: Re: [R-sig-ME] Question about non-significant interactions
	> To: Fox, John <jfox at mcmaster.ca <mailto:jfox at mcmaster.ca> >
	>
	>
	> Dear John,
	>
	> Thanks for the reply. One of my research entails examining the
relationship
	> between 3 groups of speakers, the 3 levels of the group categorical
	> variable previously mentioned, and two tasks. One prediction is that
one
	> group will perform better than other groups on one test but not the
other.
	>
	> I fit a maximal model using the bglmr function as shown previously,
then
	> used car::Anova to determine main effects. My understanding from
previous
	> interaction with you precisely here on r-sig-me is that the function
works
	> as a form of shortcut to the traditional way of model-fitting/
reduction
	> via the function anova() comparing models, eliminating terms one at
a time.
	>
	> I hope this is clearer now and yes, the question is more of a
statistical
	> one than an R one, even though I suspect the mixed-effect aspect of
the
	> regression may be relevant to answering it.
	>
	> Frank
	>
	>  Tue, Jul 9, 2019 at 11:03 PM Fox, John <jfox at mcmaster.ca
<mailto:jfox at mcmaster.ca> > wrote:
	>
	> > Dear Francesco,
	> >
	> > I didn't entirely follow your question and I expect that to answer it,
it
	> > would be necessary to know more about what your research
entails. As you
	> > imply, this seems to be more a statistics question than an R
question.
	> It's
	> > also not clear to me what function you used to fit the mixed-
effects
	> > logistic regression.
	> >
	> > But I did notice that you're apparently using Anova() for type-III
tests
	> > with the default contr.treatment() coding for factors. The main-
effect
	> > tests that result are not sensible. As it says in ?Anova:
	> >
	> > "Warning
	> > Be careful of type-III tests: For a traditional multifactor ANOVA
model
	> > with interactions, for example, these tests will normally only be
	> sensible
	> > when using contrasts that, for different terms, are orthogonal in
the
	> > row-basis of the model, such as those produced by contr.sum,
contr.poly,
	> or
	> > contr.helmert, but not by the default contr.treatment. In a model
that
	> > contains factors, numeric covariates, and interactions, main-effect
tests
	> > for factors will be for differences over the origin. In contrast (pun
	> > intended), type-II tests are invariant with respect to (full-rank)
	> contrast
	> > coding. If you don't understand this issue, then you probably
shouldn't
	> use
	> > Anova for type-III tests."
	> >
	> > I hope that this is of some help,
	> >  John
	> > -----------------------------
	> > John Fox, Professor Emeritus
	> > McMaster University
	> > Hamilton, Ontario
	> > Canada L8S 4M4
	> > web: socserv.mcmaster.ca/jfox <http://socserv.mcmaster.ca/jfox>
	> >
	> >
	> > ________________________________________
	> > From: R-sig-mixed-models [r-sig-mixed-models-bounces at r-
project.org <mailto:r-sig-mixed-models-bounces at r-project.org> ] on
	> > behalf of Francesco Romano [fbromano77 at gmail.com
<mailto:fbromano77 at gmail.com> ]
	> > Sent: July 9, 2019 9:49 AM
	> > To: r-sig-mixed-models at r-project.org <mailto:r-sig-mixed-
models at r-project.org>
	> > Subject: [R-sig-ME] Question about non-significant interactions
	> >
	> > Dear all,
	> >
	> >
	> > I have more of a theoretical than practical question for you. The
model I
	> > am using has two IVs, group (3 levels) and task (2 levels), and a
	> > categorical DV (correct versus incorrect), hence logistic regression.
	> > Random effects for subjects and items, as well as slopes for group
by
	> item
	> > and task by subject.
	> >
	> > I am interested in the effect of belonging any of three groups, the
	> levels
	> > of the group IV, in order to test some a priori predictions. The
bayesian
	> > wrapper is to help the model converge.
	> >
	> > Here is the output:
	> >
	> > > summary(paper2analysis1)
	> > Cov prior  : item ~ wishart(df = 5.5, scale = Inf, posterior.scale =
cov,
	> > common.scale = TRUE)
	> >            : Participant ~ wishart(df = 4.5, scale = Inf,
	> posterior.scale =
	> > cov, common.scale = TRUE)
	> > Prior dev  : 6.9466
	> >
	> > Generalized linear mixed model fit by maximum likelihood
(Laplace
	> > Approximation) ['bglmerMod']
	> >  Family: binomial  ( logit )
	> > Formula: correctness ~ task * group + (1 + task | Participant) + (1 +
	> >  group | item)
	> >    Data: data
	> > Control: glmerControl(optimizer = "bobyqa")
	> >
	> >      AIC      BIC   logLik deviance df.resid
	> >   3857.8   3957.2  -1913.9   3827.8     5570
	> >
	> > Scaled residuals:
	> >     Min      1Q  Median      3Q     Max
	> > -2.0196 -0.3744 -0.2312 -0.1368  6.9534
	> >
	> > Random effects:
	> >  Groups      Name        Variance Std.Dev. Corr
	> >  item        (Intercept) 1.1266   1.0614
	> >              groupL2     0.1311   0.3620   -0.12
	> >              groupNS     0.2029   0.4504   -0.31  0.17
	> >  Participant (Intercept) 0.7582   0.8708
	> >              taskpriming 1.2163   1.1029   -0.77
	> > Number of obs: 5585, groups:  item, 219; Participant, 46
	> >
	> > Fixed effects:
	> >                     Estimate Std. Error z value Pr(>|z|)
	> > (Intercept)         -2.49187    0.28318  -8.800  < 2e-16 ***
	> > taskpriming          1.30911    0.37367   3.503 0.000459 ***
	> > groupL2             -0.04042    0.38322  -0.105 0.916005
	> > groupNS             -1.01144    0.36607  -2.763 0.005727 **
	> > taskpriming:groupL2  0.04305    0.48693   0.088 0.929544
	> > taskpriming:groupNS -0.04942    0.46034  -0.107 0.914506
	> > ---
	> > Signif. codes:  0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1
	> >
	> > Correlation of Fixed Effects:
	> >             (Intr) tskprm gropL2 gropNS tsk:L2
	> > taskpriming -0.733
	> > groupL2     -0.660  0.482
	> > groupNS     -0.693  0.507  0.509
	> > tskprmng:L2  0.499 -0.632 -0.755 -0.386
	> > tskprmng:NS  0.530 -0.676 -0.390 -0.750  0.508
	> >
	> > The model was then subjected to car::Anova for ANOVA type III
analysis
	> with
	> > the following output:
	> >
	> > > car::Anova(paper2analysis1, type = "III")
	> > Analysis of Deviance Table (Type III Wald chisquare tests)
	> >
	> > Response: correctness
	> >               Chisq Df Pr(>Chisq)
	> > (Intercept) 77.4344  1  < 2.2e-16 ***
	> > task        12.2737  1  0.0004594 ***
	> > group        9.9237  2  0.0070000 **
	> > task:group   0.0391  2  0.9806462
	> > ---
	> > Signif. codes:  0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1
	> >
	> > I am not sure how to interpret the non-significant interaction in
this
	> > case. Does this mean that, although simple effects exist at group
level
	> > within one particular task or at task level within one particular
group,
	> I
	> > lack sufficient power to conclude those effects are real? If I look at
	> the
	> > simple effects, I do indeed find such effects but am not sure how to
	> > interpret them against the lack of a main interaction. At a practical
	> > level, the interaction, rather than the main effects, is the most
	> important
	> > part of the analysis.
	> >
	> > Thank you in advance for any advice.
	> >
	> > Francesco
	> >
	> >
	> >
	> >
	> >
	> > Best,
	> >
	> > Frank
	> >
	> >         [[alternative HTML version deleted]]
	> >
	> > _______________________________________________
	> > R-sig-mixed-models at r-project.org <mailto:R-sig-mixed-
models at r-project.org>  mailing list
	> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
	> >
	> --
	> Inviato da Gmail Mobile
	> --
	> Inviato da Gmail Mobile
	>
	>         [[alternative HTML version deleted]]
	>
	> _______________________________________________
	> R-sig-mixed-models at r-project.org <mailto:R-sig-mixed-models at r-
project.org>  mailing list
	> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
	>

	        [[alternative HTML version deleted]]

	_______________________________________________
	R-sig-mixed-models at r-project.org <mailto:R-sig-mixed-models at r-
project.org>  mailing list
	https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models

Question about non-significant interactions

Thread (4 messages)