A three-level GLMM with binomial link in R
How many sites, families and Id's do you have? Please also give us the output of str(dd)
On Mon, 8 Mar 2021, 16:30 Hedyeh Ahmadi, <hedyehah at usc.edu> wrote:
Thank you for the toy example. This same example in my PC gives the following memory use output:
mem
Function_Call
Elapsed_Time_sec Total_RAM_Used_MiB Peak_RAM_Used_MiB
m<-lmer(form,data=dd,REML=FALSE) 2.64
10.3 130.1
When I run peakRAM() for my actual data, it take one hour plus then my PC
slows down, then I have to stop R to be able to use my PC.
Best,
Hedyeh Ahmadi, Ph.D.
Statistician
Keck School of Medicine
Department of Preventive Medicine
University of Southern California
Postdoctoral Scholar
Institute for Interdisciplinary Salivary Bioscience Research (IISBR)
University of California, Irvine
LinkedIn
www.linkedin.com/in/hedyeh-ahmadi<http://www.linkedin.com/in/hedyeh-ahmadi
________________________________
From: R-sig-mixed-models <r-sig-mixed-models-bounces at r-project.org> on
behalf of Ben Bolker <bbolker at gmail.com>
Sent: Friday, March 5, 2021 5:44 PM
To: r-sig-mixed-models at r-project.org <r-sig-mixed-models at r-project.org>
Subject: Re: [R-sig-ME] A three-level GLMM with binomial link in R
Here's an example that conforms approximately to the structure of
your data: on my machine the peak RAM usage is 121 Mb, far short of your
2.3 Gb ... so I still suspect there is something going on that we don't
know about/haven't guessed yet.
set.seed(101)
N <- 22945
ns <- 100
nf <- 100
nid <- 100
dd <- data.frame(X1=rnorm(N),
X2=rnorm(N),
X3=rnorm(N),
site=sample(ns, replace=TRUE, size=N),
family=sample(nf, replace=TRUE, size=N),
id=sample(nid, replace=TRUE, size=N))
form <- Y ~ 1 + X1 + X2 + X3 + (1|site/family/id)
dd$Y <- simulate(form[-2], newdata=dd, family=gaussian,
newparams=list(beta=rep(1,4),theta=rep(1,3),sigma=1))[[1]]
library(peakRAM)
mem <- peakRAM(
m <- lmer(form, data=dd, REML=FALSE)
)
mem
Function_Call Elapsed_Time_sec Total_RAM_Used_MiB
1 m<-lmer(form,data=dd,REML=FALSE) 1.754 10.3
Peak_RAM_Used_MiB
1 120.7
On 3/5/21 4:52 PM, Robert Long wrote:
Perhaps because of the different ways they store objects internally while
fitting the models.
On Fri, 5 Mar 2021, 21:47 Hedyeh Ahmadi, <hedyehah at usc.edu> wrote:
Thank you for the reply Robert - If I am running out of memory in
lmer(),
do you know why lme() runs just fine?
Best,
Hedyeh Ahmadi, Ph.D.
Statistician
Keck School of Medicine
Department of Preventive Medicine
University of Southern California
Postdoctoral Scholar
Institute for Interdisciplinary Salivary Bioscience Research (IISBR)
University of California, Irvine
LinkedIn
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
------------------------------
*From:* Robert Long <longrob604 at gmail.com>
*Sent:* Friday, March 5, 2021 1:43 PM
*To:* Hedyeh Ahmadi <hedyehah at usc.edu>
*Cc:* Dexter Locke <dexter.locke at gmail.com>; R-mixed models mailing
list <
r-sig-mixed-models at r-project.org>; Megan M. Herting <herting at usc.edu>;
Elisabeth Burnor <burnor at usc.edu>
*Subject:* Re: [R-sig-ME] A three-level GLMM with binomial link in R
You've run out of memory. Try running it on a machine with much larger
RAM.
On Fri, 5 Mar 2021, 21:18 Hedyeh Ahmadi, <hedyehah at usc.edu> wrote:
Hi - Thank you for the informative replies.
I will try those other packages. If MASS::glmmPQL uses lme() then this
should work for me.
My data structure is complex so it's hard to give reproducible example
but
for example for two of my models I get the following errors in lmer()
while
lme() runs smoothly.
1- Error: cannot allocate vector of size 2.3 Gb.
2- Error: couldn't evaluate grouping factor id:(family:site) within
model
frame: try adding grouping factor to data frame explicitly if possible.
(note that the id variable works for simpler lmer() models so the
variable
itself is not an issue)
My model looks like the following nested structure:
lmer(Y ~ 1 + X1 + X2 + X3 + (1|site/family/id), data=dd, REML = FALSE)
Best,
Hedyeh Ahmadi, Ph.D.
Statistician
Keck School of Medicine
Department of Preventive Medicine
University of Southern California
Postdoctoral Scholar
Institute for Interdisciplinary Salivary Bioscience Research (IISBR)
University of California, Irvine
LinkedIn
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$
________________________________
From: Dexter Locke <dexter.locke at gmail.com>
Sent: Friday, March 5, 2021 1:01 PM
To: Hedyeh Ahmadi <hedyehah at usc.edu>
Cc: r-sig-mixed-models at r-project.org <r-sig-mixed-models at r-project.org
;
Megan M. Herting <herting at usc.edu>; Elisabeth Burnor <burnor at usc.edu>
Subject: Re: [R-sig-ME] A three-level GLMM with binomial link in R
Hi Hedyeh,
What is the problem you are having? Specifically, what is the estimation
issue with lmer and lme?
Can you provide a reproducible example? Can you provide the complete
errors you are seeing?
The current questions are vague, so it will be challenging for list
members to provide much guidance.
-Dexter
On Fri, Mar 5, 2021 at 3:27 PM Hedyeh Ahmadi <hedyehah at usc.edu<mailto:
hedyehah at usc.edu>> wrote:
Hi All,
I was wondering what would be a powerful package in R to run GLMM with
logit link that can handle a data set with N=22945 and 3 nested random
intercepts. So far, I have tried glmer() from lme4 and it's giving me a
lot
of estimation issues. Any other package I should try?
I am asking for another package as I am having the same issue with
lmer()
for similar LMM with continuous outcome, while lme() from nlme package
runs
the models with no problem.
Thank you in advance.
Best,
Hedyeh Ahmadi, Ph.D.
Statistician
Keck School of Medicine
Department of Preventive Medicine
University of Southern California
Postdoctoral Scholar
Institute for Interdisciplinary Salivary Bioscience Research (IISBR)
University of California, Irvine
LinkedIn
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQdYkAXFg$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQdYkAXFg$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQdYkAXFg$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$
<
https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQdYkAXFg$
[[alternative HTML version deleted]]
_______________________________________________
R-sig-mixed-models at r-project.org<mailto:
R-sig-mixed-models at r-project.org>
mailing list
https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k7pcSE0ic$
<
https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKG8rJcsmY$
<
https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQM6wKtNs$
[[alternative HTML version deleted]]
_______________________________________________
R-sig-mixed-models at r-project.org mailing list
https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k7pcSE0ic$
<
https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKG8rJcsmY$
[[alternative HTML version deleted]]
_______________________________________________
R-sig-mixed-models at r-project.org mailing list
https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k7pcSE0ic$
_______________________________________________
R-sig-mixed-models at r-project.org mailing list
https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k7pcSE0ic$
[[alternative HTML version deleted]]
_______________________________________________
R-sig-mixed-models at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models