Dear all,
I have 3 related questions, probably already answered, but which I cannot find:
When computing a model with lmer I get the message
fixed-effect model matrix is rank deficient so dropping 1 column / coefficient
Then, my questions are, first, how can I see/compute/get the rank deficient fixed-effect model matrix, second how is that matrix computed, and third (these actually are 2 questions), if my model is yet valid (is it?) how can the dropped fixed effect explained in the results of a paper.
In the example in ?fixef
fm2 <- lmer(Reaction ~ Days + Days2 + (1|Subject),
data=transform(sleepstudy,Days2=Days))
fixef(fm2,add.dropped=TRUE)
the problem happens because 2 independent variables are equal, but in my model the numeric independent variables are not so highly correlated. In fact the problem happens with the interaction between a factor and a numeric variable, since it is one of the categories of the factor interacting with the numeric, which is dropped.
Thank you and stay safe!
Iago
fixed-effect model matrix is rank deficient so dropping 1 column / coefficient
5 messages · Iago Giné Vázquez, Thierry Onkelinx
Dear Iago, We only have your description of the data. It would be easier for us to help you if you provide a small dataset that illustrates the structure in your data and how you use the data in your model. Best regards, Thierry ir. Thierry Onkelinx Statisticus / Statistician Vlaamse Overheid / Government of Flanders INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE AND FOREST Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance thierry.onkelinx at inbo.be Havenlaan 88 bus 73, 1000 Brussel www.inbo.be /////////////////////////////////////////////////////////////////////////////////////////// To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey /////////////////////////////////////////////////////////////////////////////////////////// <https://www.inbo.be> Op wo 3 mrt. 2021 om 13:11 schreef IAGO GIN? V?ZQUEZ <i.gine at pssjd.org>:
Dear all,
I have 3 related questions, probably already answered, but which I cannot
find:
When computing a model with lmer I get the message
fixed-effect model matrix is rank deficient so dropping 1 column /
coefficient
Then, my questions are, first, how can I see/compute/get the rank
deficient fixed-effect model matrix, second how is that matrix computed,
and third (these actually are 2 questions), if my model is yet valid (is
it?) how can the dropped fixed effect explained in the results of a paper.
In the example in ?fixef
fm2 <- lmer(Reaction ~ Days + Days2 + (1|Subject),
data=transform(sleepstudy,Days2=Days))
fixef(fm2,add.dropped=TRUE)
the problem happens because 2 independent variables are equal, but in my
model the numeric independent variables are not so highly correlated. In
fact the problem happens with the interaction between a factor and a
numeric variable, since it is one of the categories of the factor
interacting with the numeric, which is dropped.
Thank you and stay safe!
Iago
[[alternative HTML version deleted]]
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
Dear Thierry, Thank you for the ask. I send you the data attached. The model I am using is lmer(outcome ~ YLD0 + YLD1 + YLD2 + YLD3 + YLD4 + SEV1 + SEV2 + SEV3 + SEV4 + SEV5 + SEV6 + x0*x1 + x2 + x0*x3 + year + (1|id), data=data2Sh). The factor variable is x0. The dropped coefficient is x0==2 : x3 Let me know any other information I can give. Thank you! Iago
De: Thierry Onkelinx <thierry.onkelinx at inbo.be>
Enviat el: dimecres, 3 de mar? de 2021 13:30 Per a: IAGO GIN? V?ZQUEZ <i.gine at pssjd.org> A/c: r-sig-mixed-models at r-project.org <r-sig-mixed-models at r-project.org> Tema: Re: [R-sig-ME] fixed-effect model matrix is rank deficient so dropping 1 column / coefficient Dear Iago, We only have your description of the data. It would be easier for us to help you if you provide a small dataset that illustrates the structure in your data and how you use the data in your model. Best regards, Thierry ir. Thierry Onkelinx Statisticus / Statistician Vlaamse Overheid / Government of Flanders INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE AND FOREST Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance thierry.onkelinx at inbo.be<mailto:thierry.onkelinx at inbo.be> Havenlaan 88 bus 73, 1000 Brussel www.inbo.be<http://www.inbo.be> /////////////////////////////////////////////////////////////////////////////////////////// To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey /////////////////////////////////////////////////////////////////////////////////////////// [https://inbo-website-prd-532750756126.s3-eu-west-1.amazonaws.com/inbologoleeuw_nl.png]<https://www.inbo.be> Op wo 3 mrt. 2021 om 13:11 schreef IAGO GIN? V?ZQUEZ <i.gine at pssjd.org<mailto:i.gine at pssjd.org>>: Dear all, I have 3 related questions, probably already answered, but which I cannot find: When computing a model with lmer I get the message fixed-effect model matrix is rank deficient so dropping 1 column / coefficient Then, my questions are, first, how can I see/compute/get the rank deficient fixed-effect model matrix, second how is that matrix computed, and third (these actually are 2 questions), if my model is yet valid (is it?) how can the dropped fixed effect explained in the results of a paper. In the example in ?fixef fm2 <- lmer(Reaction ~ Days + Days2 + (1|Subject), data=transform(sleepstudy,Days2=Days)) fixef(fm2,add.dropped=TRUE) the problem happens because 2 independent variables are equal, but in my model the numeric independent variables are not so highly correlated. In fact the problem happens with the interaction between a factor and a numeric variable, since it is one of the categories of the factor interacting with the numeric, which is dropped. Thank you and stay safe! Iago [[alternative HTML version deleted]] _______________________________________________ R-sig-mixed-models at r-project.org<mailto:R-sig-mixed-models at r-project.org> mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
Dear Iago, The reference category of x0 (x0 == "0") contains only zeros for x3. This is why you get the rank deficiency. The solution is to create the meaningful interactions as new variables. data2Sh$x3_1 <- ifelse(data2Sh$x0 == "1", data2Sh$x3, 0) data2Sh$x3_2 <- ifelse(data2Sh$x0 == "2", data2Sh$x3, 0) lmer(outcome ~ YLD0 + YLD1 + YLD2 + YLD3 + YLD4 + SEV1 + SEV2 + SEV3 + SEV4 + SEV5 + SEV6 + x0*x1 + x2 + x3 + x3_1 + x3_2 + year + (1|id), data = data2Sh) Best regards, ir. Thierry Onkelinx Statisticus / Statistician Vlaamse Overheid / Government of Flanders INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE AND FOREST Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance thierry.onkelinx at inbo.be Havenlaan 88 bus 73, 1000 Brussel www.inbo.be /////////////////////////////////////////////////////////////////////////////////////////// To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey /////////////////////////////////////////////////////////////////////////////////////////// <https://www.inbo.be> Op wo 3 mrt. 2021 om 14:23 schreef IAGO GIN? V?ZQUEZ <i.gine at pssjd.org>:
Dear Thierry, Thank you for the ask. I send you the data attached. The model I am using is lmer(outcome ~ YLD0 + YLD1 + YLD2 + YLD3 + YLD4 + SEV1 + SEV2 + SEV3 + SEV4 + SEV5 + SEV6 + x0*x1 + x2 + x0*x3 + year + (1|id), data=data2Sh). The factor variable is x0. The dropped coefficient is x0==2 : x3 Let me know any other information I can give. Thank you! *Iago * ------------------------------ *De:* Thierry Onkelinx <thierry.onkelinx at inbo.be> *Enviat el:* dimecres, 3 de mar? de 2021 13:30 *Per a:* IAGO GIN? V?ZQUEZ <i.gine at pssjd.org> *A/c:* r-sig-mixed-models at r-project.org <r-sig-mixed-models at r-project.org> *Tema:* Re: [R-sig-ME] fixed-effect model matrix is rank deficient so dropping 1 column / coefficient Dear Iago, We only have your description of the data. It would be easier for us to help you if you provide a small dataset that illustrates the structure in your data and how you use the data in your model. Best regards, Thierry ir. Thierry Onkelinx Statisticus / Statistician Vlaamse Overheid / Government of Flanders INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE AND FOREST Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance thierry.onkelinx at inbo.be Havenlaan 88 bus 73, 1000 Brussel www.inbo.be /////////////////////////////////////////////////////////////////////////////////////////// To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey /////////////////////////////////////////////////////////////////////////////////////////// <https://www.inbo.be> Op wo 3 mrt. 2021 om 13:11 schreef IAGO GIN? V?ZQUEZ <i.gine at pssjd.org>: Dear all, I have 3 related questions, probably already answered, but which I cannot find: When computing a model with lmer I get the message fixed-effect model matrix is rank deficient so dropping 1 column / coefficient Then, my questions are, first, how can I see/compute/get the rank deficient fixed-effect model matrix, second how is that matrix computed, and third (these actually are 2 questions), if my model is yet valid (is it?) how can the dropped fixed effect explained in the results of a paper. In the example in ?fixef fm2 <- lmer(Reaction ~ Days + Days2 + (1|Subject), data=transform(sleepstudy,Days2=Days)) fixef(fm2,add.dropped=TRUE) the problem happens because 2 independent variables are equal, but in my model the numeric independent variables are not so highly correlated. In fact the problem happens with the interaction between a factor and a numeric variable, since it is one of the categories of the factor interacting with the numeric, which is dropped. Thank you and stay safe! Iago [[alternative HTML version deleted]]
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
Note that you should only use the new interaction variable to avoid the rank deficiency (no x3). lmer(outcome ~ YLD0 + YLD1 + YLD2 + YLD3 + YLD4 + SEV1 + SEV2 + SEV3 + SEV4 + SEV5 + SEV6 + x0*x1 + x2 + x3_1 + x3_2 + year + (1|id), data = data2Sh) ir. Thierry Onkelinx Statisticus / Statistician Vlaamse Overheid / Government of Flanders INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE AND FOREST Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance thierry.onkelinx at inbo.be Havenlaan 88 bus 73, 1000 Brussel www.inbo.be /////////////////////////////////////////////////////////////////////////////////////////// To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey /////////////////////////////////////////////////////////////////////////////////////////// <https://www.inbo.be> Op wo 3 mrt. 2021 om 14:45 schreef Thierry Onkelinx < thierry.onkelinx at inbo.be>:
Dear Iago, The reference category of x0 (x0 == "0") contains only zeros for x3. This is why you get the rank deficiency. The solution is to create the meaningful interactions as new variables. data2Sh$x3_1 <- ifelse(data2Sh$x0 == "1", data2Sh$x3, 0) data2Sh$x3_2 <- ifelse(data2Sh$x0 == "2", data2Sh$x3, 0) lmer(outcome ~ YLD0 + YLD1 + YLD2 + YLD3 + YLD4 + SEV1 + SEV2 + SEV3 + SEV4 + SEV5 + SEV6 + x0*x1 + x2 + x3 + x3_1 + x3_2 + year + (1|id), data = data2Sh) Best regards, ir. Thierry Onkelinx Statisticus / Statistician Vlaamse Overheid / Government of Flanders INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE AND FOREST Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance thierry.onkelinx at inbo.be Havenlaan 88 bus 73, 1000 Brussel www.inbo.be /////////////////////////////////////////////////////////////////////////////////////////// To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey /////////////////////////////////////////////////////////////////////////////////////////// <https://www.inbo.be> Op wo 3 mrt. 2021 om 14:23 schreef IAGO GIN? V?ZQUEZ <i.gine at pssjd.org>:
Dear Thierry, Thank you for the ask. I send you the data attached. The model I am using is lmer(outcome ~ YLD0 + YLD1 + YLD2 + YLD3 + YLD4 + SEV1 + SEV2 + SEV3 + SEV4 + SEV5 + SEV6 + x0*x1 + x2 + x0*x3 + year + (1|id), data=data2Sh). The factor variable is x0. The dropped coefficient is x0==2 : x3 Let me know any other information I can give. Thank you! *Iago * ------------------------------ *De:* Thierry Onkelinx <thierry.onkelinx at inbo.be> *Enviat el:* dimecres, 3 de mar? de 2021 13:30 *Per a:* IAGO GIN? V?ZQUEZ <i.gine at pssjd.org> *A/c:* r-sig-mixed-models at r-project.org <r-sig-mixed-models at r-project.org
*Tema:* Re: [R-sig-ME] fixed-effect model matrix is rank deficient so dropping 1 column / coefficient Dear Iago, We only have your description of the data. It would be easier for us to help you if you provide a small dataset that illustrates the structure in your data and how you use the data in your model. Best regards, Thierry ir. Thierry Onkelinx Statisticus / Statistician Vlaamse Overheid / Government of Flanders INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE AND FOREST Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance thierry.onkelinx at inbo.be Havenlaan 88 bus 73, 1000 Brussel www.inbo.be /////////////////////////////////////////////////////////////////////////////////////////// To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey /////////////////////////////////////////////////////////////////////////////////////////// <https://www.inbo.be> Op wo 3 mrt. 2021 om 13:11 schreef IAGO GIN? V?ZQUEZ <i.gine at pssjd.org>: Dear all, I have 3 related questions, probably already answered, but which I cannot find: When computing a model with lmer I get the message fixed-effect model matrix is rank deficient so dropping 1 column / coefficient Then, my questions are, first, how can I see/compute/get the rank deficient fixed-effect model matrix, second how is that matrix computed, and third (these actually are 2 questions), if my model is yet valid (is it?) how can the dropped fixed effect explained in the results of a paper. In the example in ?fixef fm2 <- lmer(Reaction ~ Days + Days2 + (1|Subject), data=transform(sleepstudy,Days2=Days)) fixef(fm2,add.dropped=TRUE) the problem happens because 2 independent variables are equal, but in my model the numeric independent variables are not so highly correlated. In fact the problem happens with the interaction between a factor and a numeric variable, since it is one of the categories of the factor interacting with the numeric, which is dropped. Thank you and stay safe! Iago [[alternative HTML version deleted]]
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models