large dataset - lmer2 vector size specified is too large
Hi,
Douglas Bates wrote:
On 3/2/07, Gregor Gorjanc <gregor.gorjanc at bfro.uni-lj.si> wrote:
Douglas Bates <bates at ...> writes: ...
but even that formula means that you are estimating 448 fixed effects for which the model matrix is of size 448 * 43680 * 8 bytes (about 150 MB). In addition you are attempting to estimate
Not directly related. Is it really necessary to build model matrix i.e. X.
Necessary? For linear mixed models no - for generalized linear mixed models yes (think of how the IRLS algorithm works).
hmm, I will need to learn IRLS first :(
Building X'X directly would be more efficient i.e. only 448 * 448 * 8 bytes.
...
Generating A as a sparse matrix directly from the data or directly from a model.frame object would result in considerable savings in memory usage but doing so is far from trivial. Most useRs are not aware (nor need they be aware) of the complexity of the operations
... You are right! I am far from being aware how complex this things are and I am very grateful to you for providing us with such a great software for nothing. However, I know that in animal breeding (AB) settings A can really be huge, mainly due to hugeness of Z matrix. And I know that some packages used in AB area setup MME directly from the data. Unfortunately, I am also not up to the task to tell the details. I am providing list of packages that are used in AB and whose authors might be as generous as you to show the light. Additionally, these packages can also deal with many response variables i.e. multi-trait/multivariate models with different effects so their implementations are surely general. Though, they are mainly for linear mixed models, but some also cope with generalized mixed model. Thank you again! ASReml http://www.vsni.co.uk/products/asreml/ BGF90 http://nce.ads.uga.edu/%7Eignacy/numpub/ DMU http://www.dmu.agrsci.dk/ http://www.wcgalp8.org.br/wcgalp8/articles/paper/27_302-413.pdf PEST/VCE http://w3.tzv.fal.de/%7Eeg/ http://vce.tzv.fal.de/index.pl MATVEC http://statistics.unl.edu/faculty/steve/software/matvec/ WOMBAT http://agbu.une.edu.au/~kmeyer/wombat.html
Lep pozdrav / With regards,
Gregor Gorjanc
----------------------------------------------------------------------
University of Ljubljana PhD student
Biotechnical Faculty www: http://www.bfro.uni-lj.si/MR/ggorjan
Zootechnical Department blog: http://ggorjan.blogspot.com
Groblje 3 mail: gregor.gorjanc <at> bfro.uni-lj.si
SI-1230 Domzale fax: +386 (0)1 72 17 888
Slovenia, Europe tel: +386 (0)1 72 17 861