In an earlier thread with the subject "lme4a, glmer and all that" I stated that the results of fitting generalized linear mixed models in the lme4 and lme4a packages were different and I was beginning to doubt the results from the released version 0.999375-32 of lme4. I did discover a programming mistake in the development version, lme4a, and am working on correcting it. The results from the released version, lme4, are consistent with those reported by other software. In an unrelated development we have discovered one of the reasons that the lme4 package takes a very long time to load, relative to other packages. It has always seemed peculiar that the Matrix package with dozens of S4 classes and hundreds of methods takes a couple of seconds to load whereas the much simpler lme4 package (in terms of classes and methods) takes 5 to 10 times as long. We have isolated why this occurs and John Chambers is testing a modification in the methods package that can avoid this.
Sorry for the false alarm - results from glmer in the released lme4 are consistent with other software
3 messages · Gabor Grothendieck, Douglas Bates
In general S4 seems slow to load relative to S3. You can generally tell whether a package is S3 or S4 just by the time it takes to load. However, if there is some special situation that makes S4 take even longer than usual it would be useful to know what it is.so others can avoid the problem in situations related or unrelated to lme4a.
On Fri, Mar 5, 2010 at 9:05 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
In an earlier thread with the subject "lme4a, glmer and all that" I stated that the results of fitting generalized linear mixed models in the lme4 and lme4a packages were different and I was beginning to doubt the results from the released version 0.999375-32 of lme4. ?I did discover a programming mistake in the development version, lme4a, and am working on correcting it. ?The results from the released version, lme4, are consistent with those reported by other software. In an unrelated development we have discovered one of the reasons that the lme4 package takes a very long time to load, relative to other packages. ?It has always seemed peculiar that the Matrix package with dozens of S4 classes and hundreds of methods takes a couple of seconds to load whereas the much simpler lme4 package (in terms of classes and methods) takes 5 to 10 times as long. ?We have isolated why this occurs and John Chambers is testing a modification in the methods package that can avoid this.
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
On Fri, Mar 5, 2010 at 8:28 AM, Gabor Grothendieck
<ggrothendieck at gmail.com> wrote:
In general S4 seems slow to load relative to S3. ?You can generally tell whether a package is S3 or S4 just by the time it takes to load. However, if there is some special situation that makes S4 take even longer than usual it would be useful to know what it is.so others can avoid the problem in situations related or unrelated to lme4a.
There is a lot more involved in loading a package with S4 classes and methods than in loading a package that uses S3 methods only. S3 methods are lightweight - they only involve determining names of functions, which is going to be done anyway. S3 classes are even more lightweight since they don't exist, in a formal sense. Of course, the downside of having everything in the S3 world so lightweight is that S3 is limited (an S3 version of the Matrix package would be very limited because there is no concept of multiple dispatch) and provides no consistency checks.
From the point of view of loading a package, S3 exists only in the
namespace. S4 requires that tables of classes and methods be updated and checked for consistency, which obviously takes longer. I consider it to be analogous to the difference between calling lm and calling lm.fit directly. Of course lm.fit is much faster but it requires the user to construct the model frame and the model matrix themselves and allows them the opportunity to make any number of subtle mistakes when doing so. The thing we found out was that a large portion of the time in loading the lme4 package was being spent in the insertMethod function and being driven by the fact that lme4 imported the entire Matrix namespace. The long-term fix will be to modify or perhaps even eliminate insertMethod. It is, for the most part, vestigial. The short term fix is to import only the classes and methods from Matrix that are needed in lme4. I'm not sure of the details but I think that the problem originated in checking all the method tables imported from Matrix and not modified in lme4.
On Fri, Mar 5, 2010 at 9:05 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
In an earlier thread with the subject "lme4a, glmer and all that" I stated that the results of fitting generalized linear mixed models in the lme4 and lme4a packages were different and I was beginning to doubt the results from the released version 0.999375-32 of lme4. ?I did discover a programming mistake in the development version, lme4a, and am working on correcting it. ?The results from the released version, lme4, are consistent with those reported by other software. In an unrelated development we have discovered one of the reasons that the lme4 package takes a very long time to load, relative to other packages. ?It has always seemed peculiar that the Matrix package with dozens of S4 classes and hundreds of methods takes a couple of seconds to load whereas the much simpler lme4 package (in terms of classes and methods) takes 5 to 10 times as long. ?We have isolated why this occurs and John Chambers is testing a modification in the methods package that can avoid this.
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models