Question about non-significant interactions
Francesco, To answer the question you originally asked as I understand it: the interaction test would be telling you whether the difference you find between tasks varies between groups. The result of the interaction test is not information about whether differences between tasks (or differences between groups) are legitimate. It's telling you if the result of comparing the tasks varied depending on which group you're comparing the task results in. The interaction test in telling you that you did not observe the pairwise difference between your tasks varying between different groups of subjects significantly. HTH, -Ch?
On Wed, Jul 10, 2019 at 9:43 AM Fox, John <jfox at mcmaster.ca> wrote:
Dear Francesco and Thierry, To elaborate slightly on Thierry's suggestion and also to address some other points: (1) You (Francesco) could use the linearHypothesis() function in the car package to test more specific hypotheses. It would probably be easier to formulate such hypotheses if you fit an equivalent cell-mean model, defining a factor with levels for the 9 = 3*3 combinations of levels of your two factors. (2) The type-III tests produced by Anova() still aren't sensible. Did you look at the material from ?Anova that I included in my previous response? (3) You say that Anova() uses "a form of shortcut to the traditional way of model-fitting/ reduction via the function anova() comparing models." That's not quite true in general, because what Anova() does depends on the model class and the type of test statistic selected. It's true that for a mixed-effects model, Anova() produces Wald tests rather than likelihood-ratio tests. Anova() doesn't have a "bglmerMod" method, and so I assume that the "merMod" method is inherited or that the default method is being used. Best, John ----------------------------- John Fox, Professor Emeritus McMaster University Hamilton, Ontario Canada L8S 4M4 web: socserv.mcmaster.ca/jfox
________________________________________ From: R-sig-mixed-models [r-sig-mixed-models-bounces at r-project.org] on behalf of Thierry Onkelinx via R-sig-mixed-models [ r-sig-mixed-models at r-project.org] Sent: July 10, 2019 2:51 AM To: Francesco Romano Cc: r-sig-mixed-models at r-project.org Subject: Re: [R-sig-ME] Fwd: Question about non-significant interactions Dear Francesco, To answer your question, you should convert your hypothesis in a set of linear contrasts and test those. Best regards, ir. Thierry Onkelinx Statisticus / Statistician Vlaamse Overheid / Government of Flanders INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE AND FOREST Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance thierry.onkelinx at inbo.be Havenlaan 88 bus 73, 1000 Brussel www.inbo.be /////////////////////////////////////////////////////////////////////////////////////////// To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey /////////////////////////////////////////////////////////////////////////////////////////// <https://www.inbo.be> Op di 9 jul. 2019 om 23:25 schreef Francesco Romano <fbromano77 at gmail.com : ---------- Forwarded message --------- From: Francesco Romano <fbromano77 at gmail.com> Date: Tue, Jul 9, 2019 at 11:24 PM Subject: Re: [R-sig-ME] Question about non-significant interactions To: Fox, John <jfox at mcmaster.ca> Dear John, Thanks for the reply. One of my research entails examining the relationship between 3 groups of speakers, the 3 levels of the group categorical variable previously mentioned, and two tasks. One prediction is that one group will perform better than other groups on one test but not the other. I fit a maximal model using the bglmr function as shown previously, then used car::Anova to determine main effects. My understanding from previous interaction with you precisely here on r-sig-me is that the function works as a form of shortcut to the traditional way of model-fitting/ reduction via the function anova() comparing models, eliminating terms one at a time. I hope this is clearer now and yes, the question is more of a statistical one than an R one, even though I suspect the mixed-effect aspect of the regression may be relevant to answering it. Frank Tue, Jul 9, 2019 at 11:03 PM Fox, John <jfox at mcmaster.ca> wrote: Dear Francesco, I didn't entirely follow your question and I expect that to answer it, it would be necessary to know more about what your research entails. As you imply, this seems to be more a statistics question than an R question. It's also not clear to me what function you used to fit the mixed-effects logistic regression. But I did notice that you're apparently using Anova() for type-III tests with the default contr.treatment() coding for factors. The main-effect tests that result are not sensible. As it says in ?Anova: "Warning Be careful of type-III tests: For a traditional multifactor ANOVA model with interactions, for example, these tests will normally only be sensible when using contrasts that, for different terms, are orthogonal in the row-basis of the model, such as those produced by contr.sum, contr.poly, or contr.helmert, but not by the default contr.treatment. In a model that contains factors, numeric covariates, and interactions, main-effect tests for factors will be for differences over the origin. In contrast (pun intended), type-II tests are invariant with respect to (full-rank) contrast coding. If you don't understand this issue, then you probably shouldn't use Anova for type-III tests." I hope that this is of some help, John ----------------------------- John Fox, Professor Emeritus McMaster University Hamilton, Ontario Canada L8S 4M4 web: socserv.mcmaster.ca/jfox ________________________________________ From: R-sig-mixed-models [r-sig-mixed-models-bounces at r-project.org] on behalf of Francesco Romano [fbromano77 at gmail.com] Sent: July 9, 2019 9:49 AM To: r-sig-mixed-models at r-project.org Subject: [R-sig-ME] Question about non-significant interactions Dear all, I have more of a theoretical than practical question for you. The model I am using has two IVs, group (3 levels) and task (2 levels), and a categorical DV (correct versus incorrect), hence logistic regression. Random effects for subjects and items, as well as slopes for group by item and task by subject. I am interested in the effect of belonging any of three groups, the levels of the group IV, in order to test some a priori predictions. The bayesian wrapper is to help the model converge. Here is the output: summary(paper2analysis1) Cov prior : item ~ wishart(df = 5.5, scale = Inf, posterior.scale = cov, common.scale = TRUE) : Participant ~ wishart(df = 4.5, scale = Inf, posterior.scale = cov, common.scale = TRUE) Prior dev : 6.9466 Generalized linear mixed model fit by maximum likelihood (Laplace Approximation) ['bglmerMod'] Family: binomial ( logit ) Formula: correctness ~ task * group + (1 + task | Participant) + (1 + group | item) Data: data Control: glmerControl(optimizer = "bobyqa") AIC BIC logLik deviance df.resid 3857.8 3957.2 -1913.9 3827.8 5570 Scaled residuals: Min 1Q Median 3Q Max -2.0196 -0.3744 -0.2312 -0.1368 6.9534 Random effects: Groups Name Variance Std.Dev. Corr item (Intercept) 1.1266 1.0614 groupL2 0.1311 0.3620 -0.12 groupNS 0.2029 0.4504 -0.31 0.17 Participant (Intercept) 0.7582 0.8708 taskpriming 1.2163 1.1029 -0.77 Number of obs: 5585, groups: item, 219; Participant, 46 Fixed effects: Estimate Std. Error z value Pr(>|z|) (Intercept) -2.49187 0.28318 -8.800 < 2e-16 *** taskpriming 1.30911 0.37367 3.503 0.000459 *** groupL2 -0.04042 0.38322 -0.105 0.916005 groupNS -1.01144 0.36607 -2.763 0.005727 ** taskpriming:groupL2 0.04305 0.48693 0.088 0.929544 taskpriming:groupNS -0.04942 0.46034 -0.107 0.914506 --- Signif. codes: 0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1 Correlation of Fixed Effects: (Intr) tskprm gropL2 gropNS tsk:L2 taskpriming -0.733 groupL2 -0.660 0.482 groupNS -0.693 0.507 0.509 tskprmng:L2 0.499 -0.632 -0.755 -0.386 tskprmng:NS 0.530 -0.676 -0.390 -0.750 0.508 The model was then subjected to car::Anova for ANOVA type III analysis with the following output: car::Anova(paper2analysis1, type = "III") Analysis of Deviance Table (Type III Wald chisquare tests) Response: correctness Chisq Df Pr(>Chisq) (Intercept) 77.4344 1 < 2.2e-16 *** task 12.2737 1 0.0004594 *** group 9.9237 2 0.0070000 ** task:group 0.0391 2 0.9806462 --- Signif. codes: 0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1 I am not sure how to interpret the non-significant interaction in this case. Does this mean that, although simple effects exist at group level within one particular task or at task level within one particular group, I lack sufficient power to conclude those effects are real? If I look at the simple effects, I do indeed find such effects but am not sure how to interpret them against the lack of a main interaction. At a practical level, the interaction, rather than the main effects, is the most important part of the analysis. Thank you in advance for any advice. Francesco Best, Frank [[alternative HTML version deleted]] _______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models -- Inviato da Gmail Mobile -- Inviato da Gmail Mobile [[alternative HTML version deleted]] _______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models [[alternative HTML version deleted]] _______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models _______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
PhD Student Experience and Cognition Lab Department of Human Development Cornell University Martha Van Rensselaer Hall Ithaca, NY 14853 acl249 at cornell.edu http://casasanto.com/chelucero/ [[alternative HTML version deleted]]