Modelling with uncertain (but not missing) categorical random effect values

Thanks for the suggestions Philip and Ben,

I'm coming back to this after a hiatus and this may be quite a basic question.

I managed to get the lme4 hack working with my data, but not if I
include other random effects (that are not multi membership and not in
the matrix).
i.e. in the example you provide, if I add a new random predictor, how
do I incorporate this into the model? As this line directly changes
Ztlist and Zt to be the matrix:

lmod$reTrms$Zt <- lmod$reTrms$Ztlist[[1]] <- Matrix(t(W))

Many thanks,
Mike


On Tue, 3 Aug 2021 at 23:16, Ben Bolker <bbolker at gmail.com> wrote:

    Also see
https://bbolker.github.io/mixedmodels-misc/notes/multimember.html (i.e.
you can do it in lme4, but it takes a bit of hacking)

On 8/3/21 5:19 PM, Phillip Alday wrote:

If I'm not mistaken, Thierry's suggestion is a particular case of
multi-membership models, which you can also do in brms. See e.g.:


https://rdrr.io/cran/brms/man/mm.html

https://github.com/paul-buerkner/brms/issues/130

https://discourse.mc-stan.org/t/cross-classified-multiple-membership-models-with-brms/8691


On 13/07/2021 06:09, Thierry Onkelinx via R-sig-mixed-models wrote:

Dear Michael,

Maybe something like (0 + w_1 | dad_1) + (0 + w_2 | dad_2) + (0 + w_3 |
dad_3). Where w_1 is the probability of dad_1.

Make sure that dad_1, dad_2 and dad_3 are factors with the same levels.
Then INLA allows you to add this as f(dad_1, w_1, model = "iid") + f(dad_2,
w_2, copy = "dad_"1) + f(dad_3, w_3, copy = "dad_1"). So you end up with a
single random intercept for every dad (dad_2 and dad_3 share their
estimates with dad_1).

mum_id  mum_sp  dad_sp dad_id                    con    dad_1   w_1 dad_2
w_ 2 dad_3 w_3

Af1          A              A           Am1 / Am2             1      Am1
0.6   Am2 0.4  NA 0
Af1          A              A           Am2                       1
Am2    1     NA     0     NA 0
Bf1          B             A           Am1 / Am2 / Am4   0      Am1    0.4
   Am2 0.3   Am4 0.3
Bf2          B              B          Bm1 / Bm3              1      Bm1
0.5   Bm2  0.5  NA 0

Best regards,

ir. Thierry Onkelinx
Statisticus / Statistician

Vlaamse Overheid / Government of Flanders
INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE AND
FOREST
Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance
thierry.onkelinx at inbo.be
Havenlaan 88 bus 73, 1000 Brussel
www.inbo.be

///////////////////////////////////////////////////////////////////////////////////////////
To call in the statistician after the experiment is done may be no more
than asking him to perform a post-mortem examination: he may be able to say
what the experiment died of. ~ Sir Ronald Aylmer Fisher
The plural of anecdote is not data. ~ Roger Brinner
The combination of some data and an aching desire for an answer does not
ensure that a reasonable answer can be extracted from a given body of data.
~ John Tukey
///////////////////////////////////////////////////////////////////////////////////////////

<https://www.inbo.be>


Op di 13 jul. 2021 om 12:30 schreef Michael Lawson via R-sig-mixed-models <
r-sig-mixed-models at r-project.org>:

I have a dataset where I have offspring paternity of females with
males of different species. However, many of the offspring have
ambiguous paternity - where I know the offspring must be from
particular fathers, but not from others. The data currently looks a
bit like this (but with many more rows per mum_id):

mum_id  mum_sp  dad_sp dad_id                    con

Af1          A              A           Am1 / Am2             1
Af1          A              A           Am2                       1
Bf1          B             A           Am1 / Am2 / Am4   0
Bf2          B              B          Bm1 / Bm3              1

Which I have so far run as a binomial GLMM with con (conspecific mating) as
a binary response, mum_sp and dad_sp (species) as fixed factors and
mum_id as a random factor - and have just not included dad_id as
a random factor. The ambiguously assigned fathers in dad_id is also
non-random, i.e.
certain individuals are more likely to be ambiguously assigned than
others, so just leaving these cases as NA is problematic.

For some of the ambiguous assignments, I can also extract
probabilities that a possible male is the father of the offspring,
e.g. for the first row, father Am1 is 60% likely to be the father and
Am2 40% likely.

Are there any approaches where I can include the ambiguous dad_id in
a GLMM framework? - where the uncertainty of the assignment contributes to
the
overall uncertainty in the tested relationship.

Thank you for any suggestions,
Mike

      [[alternative HTML version deleted]]

--
Dr. Benjamin Bolker
Professor, Mathematics & Statistics and Biology, McMaster University
Director, School of Computational Science and Engineering
Graduate chair, Mathematics & Statistics

    Something like:

   lmod$reTrms$Ztlist <- list(Matrix(t(W)), Z2, Z3, Z4, ...)

where the additional `Z` components are the random-effects models for
the additional terms you want (you could for example pull these from an
`lFormula()` call using a formula that included those components in the
random effects), and

   lmod$reTrms$Zt <- do.call(rbind, lmod$reTrms$Ztlist)

(I think)

On 12/20/21 12:43 PM, Mike Lawson wrote:

Thanks for the suggestions Philip and Ben,

I'm coming back to this after a hiatus and this may be quite a basic question.

I managed to get the lme4 hack working with my data, but not if I
include other random effects (that are not multi membership and not in
the matrix).
i.e. in the example you provide, if I add a new random predictor, how
do I incorporate this into the model? As this line directly changes
Ztlist and Zt to be the matrix:

lmod$reTrms$Zt <- lmod$reTrms$Ztlist[[1]] <- Matrix(t(W))

Many thanks,
Mike


On Tue, 3 Aug 2021 at 23:16, Ben Bolker <bbolker at gmail.com> wrote:

    Also see
https://bbolker.github.io/mixedmodels-misc/notes/multimember.html (i.e.
you can do it in lme4, but it takes a bit of hacking)

On 8/3/21 5:19 PM, Phillip Alday wrote:

If I'm not mistaken, Thierry's suggestion is a particular case of
multi-membership models, which you can also do in brms. See e.g.:


https://rdrr.io/cran/brms/man/mm.html

https://github.com/paul-buerkner/brms/issues/130

https://discourse.mc-stan.org/t/cross-classified-multiple-membership-models-with-brms/8691


On 13/07/2021 06:09, Thierry Onkelinx via R-sig-mixed-models wrote:

Dear Michael,

Maybe something like (0 + w_1 | dad_1) + (0 + w_2 | dad_2) + (0 + w_3 |
dad_3). Where w_1 is the probability of dad_1.

Make sure that dad_1, dad_2 and dad_3 are factors with the same levels.
Then INLA allows you to add this as f(dad_1, w_1, model = "iid") + f(dad_2,
w_2, copy = "dad_"1) + f(dad_3, w_3, copy = "dad_1"). So you end up with a
single random intercept for every dad (dad_2 and dad_3 share their
estimates with dad_1).

mum_id  mum_sp  dad_sp dad_id                    con    dad_1   w_1 dad_2
w_ 2 dad_3 w_3

Af1          A              A           Am1 / Am2             1      Am1
0.6   Am2 0.4  NA 0
Af1          A              A           Am2                       1
Am2    1     NA     0     NA 0
Bf1          B             A           Am1 / Am2 / Am4   0      Am1    0.4
   Am2 0.3   Am4 0.3
Bf2          B              B          Bm1 / Bm3              1      Bm1
0.5   Bm2  0.5  NA 0

Best regards,

ir. Thierry Onkelinx
Statisticus / Statistician

Vlaamse Overheid / Government of Flanders
INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE AND
FOREST
Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance
thierry.onkelinx at inbo.be
Havenlaan 88 bus 73, 1000 Brussel
www.inbo.be

///////////////////////////////////////////////////////////////////////////////////////////
To call in the statistician after the experiment is done may be no more
than asking him to perform a post-mortem examination: he may be able to say
what the experiment died of. ~ Sir Ronald Aylmer Fisher
The plural of anecdote is not data. ~ Roger Brinner
The combination of some data and an aching desire for an answer does not
ensure that a reasonable answer can be extracted from a given body of data.
~ John Tukey
///////////////////////////////////////////////////////////////////////////////////////////

<https://www.inbo.be>


Op di 13 jul. 2021 om 12:30 schreef Michael Lawson via R-sig-mixed-models <
r-sig-mixed-models at r-project.org>:

I have a dataset where I have offspring paternity of females with
males of different species. However, many of the offspring have
ambiguous paternity - where I know the offspring must be from
particular fathers, but not from others. The data currently looks a
bit like this (but with many more rows per mum_id):

mum_id  mum_sp  dad_sp dad_id                    con

Af1          A              A           Am1 / Am2             1
Af1          A              A           Am2                       1
Bf1          B             A           Am1 / Am2 / Am4   0
Bf2          B              B          Bm1 / Bm3              1

Which I have so far run as a binomial GLMM with con (conspecific mating) as
a binary response, mum_sp and dad_sp (species) as fixed factors and
mum_id as a random factor - and have just not included dad_id as
a random factor. The ambiguously assigned fathers in dad_id is also
non-random, i.e.
certain individuals are more likely to be ambiguously assigned than
others, so just leaving these cases as NA is problematic.

For some of the ambiguous assignments, I can also extract
probabilities that a possible male is the father of the offspring,
e.g. for the first row, father Am1 is 60% likely to be the father and
Am2 40% likely.

Are there any approaches where I can include the ambiguous dad_id in
a GLMM framework? - where the uncertainty of the assignment contributes to
the
overall uncertainty in the tested relationship.

Thank you for any suggestions,
Mike

      [[alternative HTML version deleted]]

--
Dr. Benjamin Bolker
Professor, Mathematics & Statistics and Biology, McMaster University
Director, School of Computational Science and Engineering
Graduate chair, Mathematics & Statistics

--
Dr. Benjamin Bolker
Professor, Mathematics & Statistics and Biology, McMaster University
Director, School of Computational Science and Engineering
Graduate chair, Mathematics & Statistics