A three-level GLMM with binomial link in R

Here's an example that conforms approximately to the structure of
your data: on my machine the peak RAM usage is 121 Mb, far short of your
2.3 Gb ...  so I still suspect there is something going on that we don't
know about/haven't guessed yet.

set.seed(101)
N <- 22945
ns <- 100
nf <- 100
nid <- 100
dd <- data.frame(X1=rnorm(N),
                  X2=rnorm(N),
                  X3=rnorm(N),
                  site=sample(ns, replace=TRUE, size=N),
                  family=sample(nf, replace=TRUE, size=N),
                  id=sample(nid, replace=TRUE, size=N))
form <- Y ~ 1 + X1 + X2 + X3 + (1|site/family/id)
dd$Y <- simulate(form[-2], newdata=dd, family=gaussian,
                  newparams=list(beta=rep(1,4),theta=rep(1,3),sigma=1))[[1]]
library(peakRAM)
mem <- peakRAM(
     m <- lmer(form, data=dd, REML=FALSE)
)

mem
                      Function_Call Elapsed_Time_sec Total_RAM_Used_MiB
1 m<-lmer(form,data=dd,REML=FALSE)            1.754               10.3
   Peak_RAM_Used_MiB
1             120.7


On 3/5/21 4:52 PM, Robert Long wrote:
> Perhaps because of the different ways they store objects internally while
> fitting the models.
>
> On Fri, 5 Mar 2021, 21:47 Hedyeh Ahmadi, <hedyehah at usc.edu> wrote:
>
>> Thank you for the reply Robert - If I am running out of memory in lmer(),
>> do you know why lme() runs just fine?
>>
>> Best,
>>
>> Hedyeh Ahmadi, Ph.D.
>> Statistician
>> Keck School of Medicine
>> Department of Preventive Medicine
>> University of Southern California
>>
>> Postdoctoral Scholar
>> Institute for Interdisciplinary Salivary Bioscience Research (IISBR)
>> University of California, Irvine
>>
>> LinkedIn
>> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$ >
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$ >
>>
>>
>>
>>
>> ------------------------------
>> *From:* Robert Long <longrob604 at gmail.com>
>> *Sent:* Friday, March 5, 2021 1:43 PM
>> *To:* Hedyeh Ahmadi <hedyehah at usc.edu>
>> *Cc:* Dexter Locke <dexter.locke at gmail.com>; R-mixed models mailing list <
>> r-sig-mixed-models at r-project.org>; Megan M. Herting <herting at usc.edu>;
>> Elisabeth Burnor <burnor at usc.edu>
>> *Subject:* Re: [R-sig-ME] A three-level GLMM with binomial link in R
>>
>> You've run out of memory. Try running it on a machine with much larger RAM.
>>
>> On Fri, 5 Mar 2021, 21:18 Hedyeh Ahmadi, <hedyehah at usc.edu> wrote:
>>
>> Hi - Thank you for the informative replies.
>>
>> I will try those other packages. If MASS::glmmPQL uses lme() then this
>> should work for me.
>>
>> My data structure is complex so it's hard to give reproducible example but
>> for example for two of my models I get the following errors in lmer() while
>> lme() runs smoothly.
>>
>> 1- Error: cannot allocate vector of size 2.3 Gb.
>>
>> 2- Error: couldn't evaluate grouping factor id:(family:site) within model
>> frame: try adding grouping factor to data frame explicitly if possible.
>> (note that the id variable works for simpler lmer() models so the variable
>> itself is not an issue)
>>
>> My model looks like the following nested structure:
>> lmer(Y ~ 1 + X1 + X2 + X3 + (1|site/family/id), data=dd, REML = FALSE)
>>
>> Best,
>>
>> Hedyeh Ahmadi, Ph.D.
>> Statistician
>> Keck School of Medicine
>> Department of Preventive Medicine
>> University of Southern California
>>
>> Postdoctoral Scholar
>> Institute for Interdisciplinary Salivary Bioscience Research (IISBR)
>> University of California, Irvine
>>
>> LinkedIn
>> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$>
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$>
>>>
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$>
>>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$>
>>>
>>
>>
>>
>>
>> ________________________________
>> From: Dexter Locke <dexter.locke at gmail.com>
>> Sent: Friday, March 5, 2021 1:01 PM
>> To: Hedyeh Ahmadi <hedyehah at usc.edu>
>> Cc: r-sig-mixed-models at r-project.org <r-sig-mixed-models at r-project.org>;
>> Megan M. Herting <herting at usc.edu>; Elisabeth Burnor <burnor at usc.edu>
>> Subject: Re: [R-sig-ME] A three-level GLMM with binomial link in R
>>
>> Hi Hedyeh,
>>
>> What is the problem you are having? Specifically, what is the estimation
>> issue with lmer and lme?
>>
>> Can you provide a reproducible example? Can you provide the complete
>> errors you are seeing?
>>
>> The current questions are vague, so it will be challenging for list
>> members to provide much guidance.
>>
>> -Dexter
>>
>>
>> On Fri, Mar 5, 2021 at 3:27 PM Hedyeh Ahmadi <hedyehah at usc.edu<mailto:
>> hedyehah at usc.edu>> wrote:
>> Hi All,
>> I was wondering what would be a powerful package in R to run GLMM with
>> logit link that can handle a data set with N=22945 and 3 nested random
>> intercepts. So far, I have tried glmer() from lme4 and it's giving me a lot
>> of estimation issues. Any other package I should try?
>>
>> I am asking for another package as I am having the same issue with lmer()
>> for similar LMM with continuous outcome, while lme() from nlme package runs
>> the models with no problem.
>>
>> Thank you in advance.
>>
>> Best,
>>
>> Hedyeh Ahmadi, Ph.D.
>> Statistician
>> Keck School of Medicine
>> Department of Preventive Medicine
>> University of Southern California
>>
>> Postdoctoral Scholar
>> Institute for Interdisciplinary Salivary Bioscience Research (IISBR)
>> University of California, Irvine
>>
>> LinkedIn
>> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$>
>> <
>> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQdYkAXFg$
>>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$>
>> <
>> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQdYkAXFg$
>>>>
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$>
>> <
>> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQdYkAXFg$
>>>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$>
>> <
>> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQdYkAXFg$
>>>>
>>
>>
>>
>>
>>
>>          [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-mixed-models at r-project.org<mailto:R-sig-mixed-models at r-project.org>
>> mailing list
>> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k7pcSE0ic$
>> <https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKG8rJcsmY$>
>> <
>> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQM6wKtNs$
>>>
>>
>>          [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-mixed-models at r-project.org mailing list
>> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k7pcSE0ic$
>> <https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKG8rJcsmY$>
>>
>>
>
>        [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k7pcSE0ic$
>

_______________________________________________
R-sig-mixed-models at r-project.org mailing list
https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k7pcSE0ic$