current r-forge version fails R CMD check ... ?

________________________________________
From: Martin Maechler [maechler at stat.math.ethz.ch]
Sent: Monday, August 03, 2009 6:06 AM
To: Bolker,Benjamin Michael
Cc: Reinhold Kliegl; R Mixed Models; Martin Maechler
Subject: Re: [R-sig-ME] current r-forge version fails R CMD check ... ?

? ?BB> ? Does this suggest a memory/pointer reference problem
? ?BB> (the only way I can think of getting non-deterministic
? ?BB> behavior of this type) ... ? ugh, ugh, ugh.


? ?BB> ? Tried running with valgrind, but nothing pops up.

You could try
? ?gctorture()
as well. ?But beware, that needs a lot of patience.

? ?BB> ? After running the example below to create D, I can get
? ?BB> two different results from the *same* lmer call ...

? ?>> table(replicate(40,ranef(lmer(y~(x1+x2)|ff,data=D))$ff[1,1]))

? ?BB> 8.35055995996088 8.48042553563304 17 23

aha. ?This at least seems "logical" to me, given the ?identical()
problems mentioned earlier.
Since, I've said all the time, ?the two ways really must give
identical results, and using all.equal() instead is really the
wrong test for the purpose ...


? ?BB> ? I am also worried (without much justification) that
? ?BB> the problem might lie in Matrix, which is even Deeper
? ?BB> Magic to me than lme4 ...

:-)

If I only found a way to reproduce your findings on one of my
Linux platforms,...
then I could start digging...

Martin



? ?BB> Reinhold Kliegl wrote:
? ?>> When I run Martin's example several times, using
? ?>> "set.seed(1)" before each run, I get all possible
? ?>> outcomes: (a) Error for m0 vs. m1, (b) Error for m2 vs
? ?>> m3, and (c) no error.
? ?>>
? ?>> Reinhold
? ?>>
? ?>> other attached packages: [1] lme4_0.999375-32
? ?>> Matrix_0.999375-30 lattice_0.17-25
? ?>>
? ?>>> ##------------------------------------------
# Maechler 01-08-09
set.seed(1)
##----------------------------------------------------
D <- ?data.frame(y= rnorm(20,10), ff = gl(4,5),
+ ? ? ? ? ? ? ? ? ?x1=rnorm(20,3), x2=rnorm(20,7),
+ ? ? ? ? ? ? ? ? ?x3=rnorm(20,1))
m0 <- lmer(y ~ (x1 + x2)|ff, data = D)
m1 <- lmer(y ~ x1 + x2|ff ?, data = D)
m2 <- lmer(y ~ x1 + (x2|ff), data = D)
m3 <- lmer(y ~ (x2|ff) + x1, data = D)
stopifnot(identical(ranef(m0), ranef(m1)),
+ ? ? ? ? ? identical(ranef(m2), ranef(m3)))
Fehler: identical(ranef(m2), ranef(m3)) is not TRUE
+ cat("Ok\n")
Ok
##------------------------------------------
# Maechler 01-08-09
set.seed(1)
##----------------------------------------------------
D <- ?data.frame(y= rnorm(20,10), ff = gl(4,5),
+ ? ? ? ? ? ? ? ? ?x1=rnorm(20,3), x2=rnorm(20,7),
+ ? ? ? ? ? ? ? ? ?x3=rnorm(20,1))
m0 <- lmer(y ~ (x1 + x2)|ff, data = D)
m1 <- lmer(y ~ x1 + x2|ff ?, data = D)
m2 <- lmer(y ~ x1 + (x2|ff), data = D)
m3 <- lmer(y ~ (x2|ff) + x1, data = D)
stopifnot(identical(ranef(m0), ranef(m1)),
+ ? ? ? ? ? identical(ranef(m2), ranef(m3)))
Fehler: identical(ranef(m0), ranef(m1)) is not TRUE
+ cat("Ok\n")
Ok
##------------------------------------------
# Maechler 01-08-09
set.seed(1)
##----------------------------------------------------
D <- ?data.frame(y= rnorm(20,10), ff = gl(4,5),
+ ? ? ? ? ? ? ? ? ?x1=rnorm(20,3), x2=rnorm(20,7),
+ ? ? ? ? ? ? ? ? ?x3=rnorm(20,1))
m0 <- lmer(y ~ (x1 + x2)|ff, data = D)
m1 <- lmer(y ~ x1 + x2|ff ?, data = D)
m2 <- lmer(y ~ x1 + (x2|ff), data = D)
m3 <- lmer(y ~ (x2|ff) + x1, data = D)
stopifnot(identical(ranef(m0), ranef(m1)),
+ ? ? ? ? ? identical(ranef(m2), ranef(m3)))
cat("Ok\n")
Ok
##------------------------------------------


On Sat, Aug 1, 2009 at 7:53 PM, Ben Bolker<bolker at ufl.edu> wrote:
?Thanks, Reinhold, I'm glad I'm not completely nuts. ?With Doug Bates
(quite reasonably) occupied with other things, it strikes me it might be
a little hard to dig deep enough into the guts to see what's going on
... ?I will see how far I can get, but this is the kind of problem where
**if** we understood what was going on and it looked hard to fix, it
would seem reasonable to replace the "must be identical" criterion with
"abs(difference)<1e-7" or some such in the tests ...

?Ben

Reinhold Kliegl wrote:
Just updated to Matrix_0.999375-30. The previous problem persists and
now it also reports:
Fehler: identical(ranef(om2), ranef(om3)) is not TRUE

Reinhold

stopifnot(identical(ranef(om2), ranef(om3)),
+ ? ? ? ? ?identical(deviance(om2), deviance(om3)))
Fehler: identical(ranef(om2), ranef(om3)) is not TRUE
+ if (identical(TRUE, all.equal(fixef(m2), fixef(om2))))
+ ? ?stop("offset does not change the fixed effects")
cat('Time elapsed: ', proc.time(),'\n') # for ``statistical reasons''
Time elapsed: ?13.588 0.399 14.297 0 0
sessionInfo()
R version 2.9.1 (2009-06-26)
i386-apple-darwin8.11.1

locale:
de_DE.UTF-8/en_US.UTF-8/C/C/de_DE.UTF-8/de_DE.UTF-8

attached base packages:
[1] stats ? ? graphics ?grDevices utils ? ? datasets ?methods
[7] base

other attached packages:
[1] lme4_0.999375-31 ? Matrix_0.999375-30 lattice_0.17-25


On Sat, Aug 1, 2009 at 7:37 PM, Reinhold
Kliegl<reinhold.kliegl at gmail.com> wrote:
Ben's problem shows up with my implementation, too. Info below.

Reinhold

stopifnot(identical(ranef(m0), ranef(m1)),
+ ? ? ? ? ?identical(ranef(m2), ranef(m3)),
+ ? ? ? ? ?inherits(tryCatch(lmer(y ~ x2|ff + x1, data = D), error =
function(e)e),"error"))
CHOLMOD error: xG? L?R
Fehler: identical(ranef(m0), ranef(m1)) is not TRUE
Zus?tzlich: Warnmeldung:
In Ops.factor(ff, x1) : + nicht sinnvoll f?r Faktoren
+
## Check the use of offset
om2 <- lmer(y ~ x1 + (x2|ff), data = D, offset = x3)
om3 <- lmer(y ~ x1 + (x2|ff) + offset(x3), data = D)

stopifnot(identical(ranef(om2), ranef(om3)),
+ ? ? ? ? ?identical(deviance(om2), deviance(om3)))
if (identical(TRUE, all.equal(fixef(m2), fixef(om2))))
+ ? ?stop("offset does not change the fixed effects")
cat('Time elapsed: ', proc.time(),'\n') # for ``statistical reasons''
Time elapsed: ?11.608 0.369 12.353 0 0
sessionInfo()
R version 2.9.1 (2009-06-26)
i386-apple-darwin8.11.1

locale:
de_DE.UTF-8/en_US.UTF-8/C/C/de_DE.UTF-8/de_DE.UTF-8

attached base packages:
[1] stats ? ? graphics ?grDevices utils ? ? datasets ?methods
[7] base

other attached packages:
[1] lme4_0.999375-31 ? Matrix_0.999375-29 lattice_0.17-25

loaded via a namespace (and not attached):
[1] grid_2.9.1

On Sat, Aug 1, 2009 at 6:31 PM, Ben Bolker<bolker at ufl.edu> wrote:
?I don't mind it being public.

?I got similar results with the CRAN lme4 (0.999375-31),
with Matrix ...-30. ?BATCH fails on m2 != m3 (consistently);
source() fails on m0 != m1.

?I'm probably doing something really really dumb, would appreciate
anyone else who can try this on their systems ...

?If you don't feel like downloading or running all of lmer-1.R, the
following code chunk should demonstrate the problem ...

=================
library(lme4)

set.seed(1)
## Wrong formula gave a seg.fault at times:
D <- ?data.frame(y= rnorm(20,10), ff = gl(4,5),
? ? ? ? ? ? ? ? x1=rnorm(20,3), x2=rnorm(20,7),
? ? ? ? ? ? ? ? x3=rnorm(20,1))
m0 <- lmer(y ~ (x1 + x2)|ff, data = D)
m1 <- lmer(y ~ x1 + x2|ff ?, data = D)
m2 <- lmer(y ~ x1 + (x2|ff), data = D)
m3 <- lmer(y ~ (x2|ff) + x1, data = D)
stopifnot(identical(ranef(m0), ranef(m1)),
? ? ? ? ?identical(ranef(m2), ranef(m3)),
? ? ? ? ?inherits(tryCatch(lmer(y ~ x2|ff + x1, data = D), error =
function(e)e),
? ? ? ? ? ? ? ? ? "error"))

## Check the use of offset
om2 <- lmer(y ~ x1 + (x2|ff), data = D, offset = x3)
om3 <- lmer(y ~ x1 + (x2|ff) + offset(x3), data = D)

stopifnot(identical(ranef(om2), ranef(om3)),
? ? ? ? ?identical(deviance(om2), deviance(om3)))
if (identical(TRUE, all.equal(fixef(m2), fixef(om2))))
? ?stop("offset does not change the fixed effects")

cat('Time elapsed: ', proc.time(),'\n') # for ``statistical reasons''


Martin Maechler wrote:
Hi Ben,
as you took this "private", I'd like at least Doug Bates
to be in the CC ..
Personally I would prefer to have this continue in the R-SIG-ME list
rather than privately... ?I'll be pretty offline from now till Monday
in any case

On Fri, Jul 31, 2009 at 20:17, Ben Bolker<bolker at ufl.edu> wrote:
Martin Maechler wrote:
"BB" == Ben Bolker <bolker at ufl.edu>
? ? on Thu, 30 Jul 2009 17:30:17 -0400 writes:
? ? BB> When I use the latest r-forge version of lme4
? ? BB> ( ?0.999375-32 ) it seems to fail R CMD check on a tiny
? ? BB> numerical mismatch of two objects that are supposed
? ? BB> (??) to be identical (I also
? ? BB> get a mangled CHOLMOD error message, but I suspect that
? ? BB> comes from somewhere within Matrix ...)

yes, and those should be gone with the version of Matrix
(0.999375-30) of two days ago.

? ? BB> can anyone confirm?

No. ?To the contrary.
I have had a slightly updated version of tests/lmer-1.Rout.save
ready to be committed for a while, but that's only trivial
changes.

and below, from your sessionInfo(), it looks like you are using
a current version of R and packages ...
hmm ...

Regards,
Martin


? ? BB> can anyone confirm? any ideas for a fix?


? ? BB> The offending mismatch between ranef(m2) and ranef(m3)
? ? BB> is very small ...

well; it's interesting that the offending mismatch in the error
message below is between ?m0 and m1, ?...
?hmmm indeed. ?Maybe I was already hacking things. ?I have
(1) updated Matrix, (2) installed lme4 directly from r-forge.
sessionInfo() says

?lme4_0.999375-32 ? Matrix_0.999375-30

in ../tests, I do

R --vanilla
library(lme4)
source("lmer-1.R",echo=TRUE)

or

R CMD BATCH --vanilla lmer-1.R

?oddly, the second (BATCH) always fails on m0/m1; the
first (source) fails at different comparisons (sometimes m0/m1;
sometimes m2/m3; sometimes om2/om3 in the next section ... ???
I just can't understand how that *can* happen.
It would mean that the algorithms used were slightly "random", ?or
e.g. using slightly different precision depending on memory
allocation, or ??,
???

As I said i the first e-mail: The slightly different formula should
produce absolutely identical matrices and vectors which define the
loglikelihood (or RE-LogLik.) and then the minimization really should
be 100% reproducible on a given R+Platform+Installed-Packages setup.

I assume you have tried the same with the CRAN-version of lme4 ...
which has exactly the same tests/lmer-1.R ??
....
the phenomenon looks so illogical, ?I even start to wonder if it's a
bug in your computer (hardware-low-level software combination)?
Maybe you could ask again on R-SIg-ME if others could reproduce?

BTW: Have you noticed that we (Doug Bates and I, when at the
useR/DSC meetings) have moved the former 'allcoef' branch into a
``regular R-forge package'' ?called ?'lme4a'
?yes.
But yes, that definitely does not pass 'CMD check at the moment'.

? ? >> getwd()
? ? BB> [1] "/home/ben/lib/R/pkgs/lme4/pkg/lme4/tests"

? ? >> source("lmer-1.R",echo=TRUE)

? ? BB> ...
? ? >> D <- ?data.frame(y= rnorm(20,10), ff = gl(4,5),
? ? BB> x1=rnorm(20,3), x2=rnorm(20,7),
? ? BB> x3=rnorm(20,1))
? ? >> m0 <- lmer(y ~ (x1 + x2)|ff, data = D)
? ? >> m1 <- lmer(y ~ x1 + x2|ff ?, data = D)

We had added these checks exactly *because* we wanted to be sure
that a slightly different use of formulas would lead to the
identical 'X', 'Z', .... matrices, and L(theta)
parametrizations,
so I wonder how your version of lme4 could give different
results here....

? ? >> m2 <- lmer(y ~ x1 + (x2|ff), data = D)
? ? >> m3 <- lmer(y ~ (x2|ff) + x1, data = D)
? ? >> stopifnot(identical(ranef(m0), ranef(m1)),
? ? BB> + ? ? ? ? ? identical(ranef(m2), ranef(m3)),
? ? BB> + ? ? ? ? ? inherits(tryCatch(lmer(y ~ x2|ff + x1, data = D) ....
? ? BB> [TRUNCATED]
? ? BB> CHOLMOD error: =*?1????@???T??o????
? ? BB> Error: identical(ranef(m0), ranef(m1)) is not TRUE
? ? BB> In addition: Warning message:
? ? BB> In Ops.factor(ff, x1) : + not meaningful for factors

Note that the cholmod error and warning is from the
? ?lmer(y ~ x2|ff + x1, data = D)
part {which is wrapped in ?tryCatch(...)}.

Also, if I execute

##----------------------------------------------------
D <- ?data.frame(y= rnorm(20,10), ff = gl(4,5),
? ? ? ? ? ? ? ? ?x1=rnorm(20,3), x2=rnorm(20,7),
? ? ? ? ? ? ? ? ?x3=rnorm(20,1))
m0 <- lmer(y ~ (x1 + x2)|ff, data = D)
m1 <- lmer(y ~ x1 + x2|ff ?, data = D)
m2 <- lmer(y ~ x1 + (x2|ff), data = D)
m3 <- lmer(y ~ (x2|ff) + x1, data = D)
stopifnot(identical(ranef(m0), ranef(m1)),
? ? ? ? ? identical(ranef(m2), ranef(m3)))
cat("Ok\n")
##----------------------------------------------------

many times, I never see a problem.

Are you sure you are not using your already-hacked version of
lme4 ???

Martin Maechler, ETH Zurich

?I'm not 100.0000% sure, but I don't see how I could be ...

?Ben


--
Ben Bolker
Associate professor, Biology Dep't, Univ. of Florida
bolker at ufl.edu / www.zoology.ufl.edu/bolker
GPG key: www.zoology.ufl.edu/bolker/benbolker-publickey.asc

_______________________________________________
R-sig-mixed-models at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models


--
Ben Bolker
Associate professor, Biology Dep't, Univ. of Florida
bolker at ufl.edu / www.zoology.ufl.edu/bolker
GPG key: www.zoology.ufl.edu/bolker/benbolker-publickey.asc



--
Ben Bolker
Associate professor, Biology Dep't, Univ. of Florida
bolker at ufl.edu / www.zoology.ufl.edu/bolker
GPG key: www.zoology.ufl.edu/bolker/benbolker-publickey.asc

_______________________________________________
R-sig-mixed-models at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models