lme vs. lmer
Chris,
Thanks for that - I should probably have mentioned that I'm using
family=quasipoisson in glmer since my data has Poisson distribution
as well as being overdispersed. I'm unsure how one decides which term
to drop without being informed by p-values, and so don't quite
understand how the "Likelihood ratio test using anova()" , or the AIC
or BIC model comparison will work in this case (I thought one's
supposed to remove the term with the highest p-value from the model,
and compare it with the model with the term included to see if there's
a difference, not so?).
Douglas,
Yes I have data (see attached). The response variable ('Counts'
column) is the number of seedlings per plot. There are four sites that
are not the same (since these are previously mined sites and it's
impossible to get even two sites that are exactly the same, in terms
of age, soil properties etc). Each site has 20 reps of each treatment
- Control, G, N and NG, and the data was collected over three
consecutive years. Each set of treatments were grouped together in a
'Patch' (sort of a split plot design, although I've ignored this thus
far).
So the model using glmer looks like this (I'm not sure if the 'random'
factors are correct, or if it should just be '(1|Site)', and i've
ignore the split plot design...):
ex4o_r2<-glmer(Counts~N+G+Year+N:Year+G:Year+N:G:Year+(Year|Site), data=ex4o, family=quasipoisson) ex4o_r2
Generalized linear mixed model fit by the Laplace approximation
Formula: Counts~ N + G + Year + N:Year + G:Year + N:G:Year + (Year | Site)
Data: ex4o
AIC BIC logLik deviance
5731 5823 -2846 5693
Random effects:
Groups Name Variance Std.Dev. Corr
Site (Intercept) 1.35594 1.1644
Yearthree 3.53069 1.8790 0.752
Yeartwo 0.88908 0.9429 0.169 0.777
Residual 6.76955 2.6018
Number of obs: 936, groups: Site, 4
Fixed effects:
Estimate Std. Error t value
(Intercept) 2.93093 0.58598 5.002
N 0.03767 0.09071 0.415
G 0.14927 0.08833 1.690
Yearthree -3.22170 0.98325 -3.277
Yeartwo -1.96111 0.50636 -3.873
N:Yearthree 0.15713 0.37544 0.419
N:Yeartwo 0.14736 0.24210 0.609
G:Yearthree -0.25103 0.40152 -0.625
G:Yeartwo 0.07549 0.23937 0.315
N:G:Yearone -0.31633 0.12888 -2.455
N:G:Yearthree 0.04722 0.52594 0.090
N:G:Yeartwo -0.32787 0.31260 -1.049
Correlation of Fixed Effects:
(Intr) N G Yerthr Yeartw N:Yrth N:Yrtw G:Yrth
G:Yrtw N:G:Yrn N:G:Yrth
N -0.079
G -0.081 0.523
Yearthree 0.706 0.047 0.048
Yeartwo 0.141 0.091 0.094 0.707
N:Yearthree 0.019 -0.242 -0.126 -0.209 -0.022
N:Yeartwo 0.030 -0.375 -0.196 -0.018 -0.259 0.091
G:Yearthree 0.018 -0.115 -0.220 -0.195 -0.021 0.511 0.043
G:Yeartwo 0.030 -0.193 -0.369 -0.018 -0.262 0.047 0.547 0.081
N:G:Yearone 0.056 -0.704 -0.685 -0.033 -0.064 0.170 0.264 0.151
0.253
N:G:Yearthr 0.000 0.000 0.000 0.141 0.000 -0.672 0.000 -0.726
0.000 0.000
N:G:Yeartwo 0.000 0.000 0.000 0.000 0.174 0.000 -0.666 0.000
-0.661 0.000 0.000
Hope you can work with that?
Thanks,
Raldo
On Wed, Sep 30, 2009 at 6:20 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
On Wed, Sep 30, 2009 at 4:45 AM, Raldo Kruger <raldo.kruger at gmail.com> wrote:
Hi all, I've been following this thread since it's of interest to the analyses i'm currenty doing. My question is, how does one do model simplification with lmer (or glmer in my case) if there are no p-values (since p-values are used to determine which terms to drop, and the drop1 function does not work for glmer)?
And Douglas, could you provide a working example of getting the p-values as you described, preferably with glmer (glmer does not have the REML=FALSE option)? I understand the 1st part of fitting two models, one with and one without the term of interest... So does it mean one has to do this for every term in order to get all the p-values?
Got data? (I live in "the dairy state" in the United States and the milk producers have an advertising campaign with the slogan "Got milk?") It would be easier, and probably more useful, if you could propose a data set and model for illustration. For a binary response the summary of a glmer fitted model actually provides p-values for the coefficients. ?Refitting the model without a particular term and using a likelihood ratio test may be more reasonable than using those p-values, simply because both models are being fit to the data. ?The "Wald test" statistics are based on an inferred model fit for the simpler model, which may or may not be reasonable.
On 9/29/09, Douglas Bates <bates at stat.wisc.edu> wrote:
On Tue, Sep 29, 2009 at 3:58 PM, Peter Dalgaard <p.dalgaard at biostat.ku.dk> wrote:
Ben Bolker wrote:
Douglas Bates wrote:
On Tue, Sep 29, 2009 at 1:02 PM, Ben Bolker <bolker at ufl.edu> wrote:
Christopher David Desjardins wrote:
I've started working through Pinheiro & Bates, 2000 and noticed the use of lme from the nlme package. I am curious if lmer from lme4 has superseded lme or if lme still holds its own? The reason I ask is that I have taken a few classes where we've solely used lmer and just read about lme today. If both functions are on equal footing, can the p-values from lme be trusted? Thanks! Chris
?You should read the extended discussion of p-values, degrees of freedom, etc. that is on the R wiki (I think) and referenced from the R FAQ. ?At least in my opinion, (n)lme is still fine (and indeed necessary at this stage for fitting heteroscedastic and correlated models). ?The df/p-value estimates, however, are "use at your own risk" -- you'll have to read the literature and decide for yourself. ?I still think there's room for someone to implement (at least) Satterthwaite and (possibly) Kenward-Roger corrections, at least for the sake of comparison, but I'm not volunteering.
You may need to define them first. ?Many of the formulas in the mixed models literature assume a hierarchical structure in the random effects - certainly we used such a formula for calculating the denominator degrees of freedom in the nlme package. But lme4 allows for fully or partially crossed random effects so you can't think in terms of "levels" of random effects. Referring to the "Satterthwaite and Kenward-Roger corrections" gives the impression that these are well-known formulas and implementing them would be a simple matter of writing a few lines of code. ?I don't think it is. ?I would be very pleased to incorporate such code if it could be written but, as I said, I don't even know if such things are defined in the general case, let alone easy to calculate. I am not trying to be argumentative (although of late I seem to have succeeded in being that). ?I'm just saying that I don't think this is trivial. (It I wanted to be argumentative I would say that it is difficult and, for the most part, irrelevant. :-)
?Fair enough. Actually, I'm not sure I meant implementing them in lmer -- even implementing them in nlme would be useful (and perhaps more straightforward, if not trivial). I also wouldn't impose the requirement that they have to be feasible for huge data sets -- I'm just curious if they can be implemented within lme in a relatively straightforward/ boneheaded way. ? But again, this is very far down my to-do list (and at the edge of my abilities) and completely off yours, so unless someone else bites it won't happen.
I don't think (n)lme is all that easy either; you still need to sort out the connection between its multilevel formulation and the projection matrices in the K-R paper. In both nlme and lme4, an implementation is almost certainly possible, although probably complicated and perhaps at the expense of all computational efficiency. One main problem is that even when they can be calculated, the corrections rely on a normal distribution assumption which is more than likely wrong in practice. This isn't any different from ordinary t-tests: once you get into the single-digit df regime, you really don't know what you are doing, and if there are more than 30 df, the normal approximation works well enough without the correction. Accordingly, I tend to see low df more as a warning flag than as something that gives accurate p values, and I sometimes wonder whether there is a way to raise such a flag more expediently.
I agree, wholeheartedly. My general advice to those who are required to produce a p-value for a particular fixed-effects term in a mixed-effects model is to use a likelihood ratio test. ?Fit the model including that term using maximum likelihood (i.e. REML = FALSE), fit it again without the term and compare the results using anova. The likelihood ratio statistic will be compared to a chi-squared distribution to get a p-value and this process is somewhat suspect when the degrees of freedom would be small. ?However, so many other things could be going wrong when you are fitting complex models to few observations that this may be the least of your worries. I appreciate that for Ben and others in fields like ecology the need to incorporate many different possible terms in models for comparatively small data sets may be inevitable. ?But it is also inevitable that the precision of the information one can extract from such small data sets is low. ?Reducing such analysis to a set of p-values for various terms and treating these p-values as if they were precisely determined is an oversimplification, even when journal editors insist on such an oversimplification.
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
-- Sent from my mobile device Raldo
Raldo -------------- next part -------------- Site Soil Patch Treat Plant Year Counts N G N N 1 C O one 31 0 0 N N 1 C O three 10 0 0 N N 1 C O two 6 0 0 N N 2 C O one 25 0 0 N N 2 C O three 1 0 0 N N 2 C O two 2 0 0 N N 3 C O one 22 0 0 N N 3 C O three 1 0 0 N N 3 C O two 2 0 0 N N 4 C O one 38 0 0 N N 4 C O three 2 0 0 N N 4 C O two 8 0 0 N N 5 C O one 7 0 0 N N 5 C O three 0 0 0 N N 5 C O two 5 0 0 N N 6 C O one 19 0 0 N N 6 C O three 1 0 0 N N 6 C O two 1 0 0 N N 7 C O one 10 0 0 N N 7 C O three 0 0 0 N N 7 C O two 6 0 0 N N 8 C O one 13 0 0 N N 8 C O three 1 0 0 N N 8 C O two 2 0 0 N N 9 C O one 25 0 0 N N 9 C O three 4 0 0 N N 9 C O two 1 0 0 N N 10 C O one 51 0 0 N N 10 C O three 0 0 0 N N 10 C O two 3 0 0 N N 11 C O one 21 0 0 N N 11 C O three 0 0 0 N N 11 C O two 0 0 0 N N 12 C O one 19 0 0 N N 12 C O three 2 0 0 N N 12 C O two 4 0 0 N N 13 C O one 18 0 0 N N 13 C O three 0 0 0 N N 13 C O two 2 0 0 N N 14 C O one 34 0 0 N N 14 C O three 0 0 0 N N 14 C O two 0 0 0 N N 15 C O one 28 0 0 N N 15 C O three 2 0 0 N N 15 C O two 0 0 0 N N 16 C O one 38 0 0 N N 16 C O three 8 0 0 N N 16 C O two 8 0 0 N N 17 C O one 39 0 0 N N 17 C O three 1 0 0 N N 17 C O two 7 0 0 N N 18 C O one 39 0 0 N N 18 C O three 1 0 0 N N 18 C O two 5 0 0 N N 19 C O one 8 0 0 N N 19 C O three 3 0 0 N N 19 C O two 2 0 0 N N 20 C O one 38 0 0 N N 20 C O three 1 0 0 N N 20 C O two 4 0 0 B O 1 C O one 52 0 0 B O 1 C O three 0 0 0 B O 1 C O two 4 0 0 B O 2 C O one 8 0 0 B O 2 C O three 0 0 0 B O 2 C O two 0 0 0 B O 3 C O one 6 0 0 B O 3 C O three 5 0 0 B O 3 C O two 7 0 0 B O 4 C O one 41 0 0 B O 4 C O three 0 0 0 B O 4 C O two 1 0 0 B O 5 C O one 21 0 0 B O 5 C O three 2 0 0 B O 5 C O two 9 0 0 B O 6 C O one 25 0 0 B O 6 C O three 1 0 0 B O 6 C O two 10 0 0 B O 7 C O one 5 0 0 B O 7 C O three 0 0 0 B O 7 C O two 0 0 0 B O 8 C O one 5 0 0 B O 8 C O three 9 0 0 B O 8 C O two 10 0 0 B O 9 C O one 26 0 0 B O 9 C O three 0 0 0 B O 9 C O two 1 0 0 B O 10 C O one 3 0 0 B O 10 C O three 0 0 0 B O 10 C O two 7 0 0 B O 11 C O one 5 0 0 B O 11 C O three 2 0 0 B O 11 C O two 7 0 0 B O 12 C O one 2 0 0 B O 12 C O three 3 0 0 B O 12 C O two 12 0 0 B O 13 C O one 19 0 0 B O 13 C O three 0 0 0 B O 13 C O two 5 0 0 B O 14 C O one 28 0 0 B O 14 C O three 7 0 0 B O 14 C O two 13 0 0 B O 15 C O one 3 0 0 B O 15 C O three 0 0 0 B O 15 C O two 0 0 0 B O 16 C O one 30 0 0 B O 16 C O three 1 0 0 B O 16 C O two 10 0 0 B O 17 C O one 28 0 0 B O 17 C O three 1 0 0 B O 17 C O two 5 0 0 B O 18 C O one 21 0 0 B O 18 C O three 4 0 0 B O 18 C O two 6 0 0 B O 19 C O one 21 0 0 B O 19 C O three 1 0 0 B O 19 C O two 7 0 0 B O 20 C O one 0 0 0 B O 20 C O three 0 0 0 B O 20 C O two 0 0 0 F O 2 C O one 51 0 0 F O 2 C O three 2 0 0 F O 2 C O two 4 0 0 F O 3 C O one 84 0 0 F O 3 C O three 0 0 0 F O 3 C O two 0 0 0 F O 4 C O one 35 0 0 F O 4 C O three 0 0 0 F O 4 C O two 0 0 0 F O 5 C O one 74 0 0 F O 5 C O three 3 0 0 F O 5 C O two 1 0 0 F O 6 C O one 43 0 0 F O 6 C O three 0 0 0 F O 6 C O two 1 0 0 F O 7 C O one 40 0 0 F O 7 C O three 0 0 0 F O 7 C O two 2 0 0 F O 8 C O one 2 0 0 F O 8 C O three 0 0 0 F O 8 C O two 0 0 0 F O 9 C O one 13 0 0 F O 9 C O three 0 0 0 F O 9 C O two 0 0 0 F O 10 C O one 7 0 0 F O 10 C O three 0 0 0 F O 10 C O two 0 0 0 F O 11 C O one 37 0 0 F O 11 C O three 3 0 0 F O 11 C O two 3 0 0 F O 12 C O one 8 0 0 F O 12 C O three 0 0 0 F O 12 C O two 1 0 0 F O 13 C O one 29 0 0 F O 13 C O three 0 0 0 F O 13 C O two 2 0 0 F O 14 C O one 19 0 0 F O 14 C O three 0 0 0 F O 14 C O two 0 0 0 F O 15 C O one 16 0 0 F O 15 C O three 2 0 0 F O 15 C O two 0 0 0 F O 16 C O one 23 0 0 F O 16 C O three 0 0 0 F O 16 C O two 0 0 0 F O 17 C O one 23 0 0 F O 17 C O three 0 0 0 F O 17 C O two 0 0 0 F O 18 C O one 39 0 0 F O 18 C O three 7 0 0 F O 18 C O two 10 0 0 F O 19 C O one 20 0 0 F O 19 C O three 1 0 0 F O 19 C O two 11 0 0 F O 20 C O one 3 0 0 F O 20 C O three 0 0 0 F O 20 C O two 1 0 0 P O 1 C O one 20 0 0 P O 1 C O three 0 0 0 P O 1 C O two 2 0 0 P O 2 C O one 13 0 0 P O 2 C O three 0 0 0 P O 2 C O two 0 0 0 P O 4 C O one 12 0 0 P O 4 C O three 0 0 0 P O 4 C O two 2 0 0 P O 5 C O one 20 0 0 P O 5 C O three 1 0 0 P O 5 C O two 4 0 0 P O 6 C O one 15 0 0 P O 6 C O three 0 0 0 P O 6 C O two 0 0 0 P O 7 C O one 11 0 0 P O 7 C O three 0 0 0 P O 7 C O two 0 0 0 P O 8 C O one 14 0 0 P O 8 C O three 0 0 0 P O 8 C O two 3 0 0 P O 9 C O one 14 0 0 P O 9 C O three 0 0 0 P O 9 C O two 2 0 0 P O 10 C O one 2 0 0 P O 10 C O three 0 0 0 P O 10 C O two 1 0 0 P O 11 C O one 0 0 0 P O 11 C O three 0 0 0 P O 11 C O two 0 0 0 P O 12 C O one 2 0 0 P O 12 C O three 0 0 0 P O 12 C O two 2 0 0 P O 13 C O one 0 0 0 P O 13 C O three 0 0 0 P O 13 C O two 0 0 0 P O 14 C O one 0 0 0 P O 14 C O three 0 0 0 P O 14 C O two 2 0 0 P O 15 C O one 0 0 0 P O 15 C O three 0 0 0 P O 15 C O two 0 0 0 P O 16 C O one 1 0 0 P O 16 C O three 0 0 0 P O 16 C O two 0 0 0 P O 17 C O one 24 0 0 P O 17 C O three 0 0 0 P O 17 C O two 10 0 0 P O 18 C O one 10 0 0 P O 18 C O three 0 0 0 P O 18 C O two 0 0 0 P O 19 C O one 19 0 0 P O 19 C O three 0 0 0 P O 19 C O two 0 0 0 P O 20 C O one 0 0 0 P O 20 C O three 0 0 0 P O 20 C O two 0 0 0 N N 1 G O one 13 0 1 N N 1 G O three 1 0 1 N N 1 G O two 3 0 1 N N 2 G O one 50 0 1 N N 2 G O three 1 0 1 N N 2 G O two 4 0 1 N N 3 G O one 14 0 1 N N 3 G O three 2 0 1 N N 3 G O two 5 0 1 N N 4 G O one 15 0 1 N N 4 G O three 0 0 1 N N 4 G O two 0 0 1 N N 5 G O one 36 0 1 N N 5 G O three 1 0 1 N N 5 G O two 22 0 1 N N 6 G O one 56 0 1 N N 6 G O three 1 0 1 N N 6 G O two 2 0 1 N N 7 G O one 15 0 1 N N 7 G O three 0 0 1 N N 7 G O two 10 0 1 N N 8 G O one 47 0 1 N N 8 G O three 1 0 1 N N 8 G O two 1 0 1 N N 9 G O one 26 0 1 N N 9 G O three 2 0 1 N N 9 G O two 2 0 1 N N 10 G O one 61 0 1 N N 10 G O three 2 0 1 N N 10 G O two 3 0 1 N N 11 G O one 46 0 1 N N 11 G O three 3 0 1 N N 11 G O two 9 0 1 N N 12 G O one 32 0 1 N N 12 G O three 2 0 1 N N 12 G O two 2 0 1 N N 13 G O one 10 0 1 N N 13 G O three 0 0 1 N N 13 G O two 6 0 1 N N 14 G O one 37 0 1 N N 14 G O three 1 0 1 N N 14 G O two 3 0 1 N N 15 G O one 36 0 1 N N 15 G O three 3 0 1 N N 15 G O two 6 0 1 N N 16 G O one 79 0 1 N N 16 G O three 10 0 1 N N 16 G O two 29 0 1 N N 17 G O one 53 0 1 N N 17 G O three 4 0 1 N N 17 G O two 9 0 1 N N 18 G O one 45 0 1 N N 18 G O three 1 0 1 N N 18 G O two 5 0 1 N N 19 G O one 13 0 1 N N 19 G O three 4 0 1 N N 19 G O two 13 0 1 N N 20 G O one 54 0 1 N N 20 G O three 4 0 1 N N 20 G O two 4 0 1 B O 1 G O one 31 0 1 B O 1 G O three 4 0 1 B O 1 G O two 7 0 1 B O 2 G O one 31 0 1 B O 2 G O three 0 0 1 B O 2 G O two 3 0 1 B O 3 G O one 12 0 1 B O 3 G O three 4 0 1 B O 3 G O two 1 0 1 B O 4 G O one 13 0 1 B O 4 G O three 1 0 1 B O 4 G O two 0 0 1 B O 5 G O one 39 0 1 B O 5 G O three 2 0 1 B O 5 G O two 26 0 1 B O 6 G O one 0 0 1 B O 6 G O three 0 0 1 B O 6 G O two 0 0 1 B O 7 G O one 12 0 1 B O 7 G O three 9 0 1 B O 7 G O two 3 0 1 B O 8 G O one 17 0 1 B O 8 G O three 1 0 1 B O 8 G O two 13 0 1 B O 9 G O one 19 0 1 B O 9 G O three 0 0 1 B O 9 G O two 2 0 1 B O 10 G O one 7 0 1 B O 10 G O three 1 0 1 B O 10 G O two 14 0 1 B O 11 G O one 4 0 1 B O 11 G O three 0 0 1 B O 11 G O two 3 0 1 B O 12 G O one 2 0 1 B O 12 G O three 1 0 1 B O 12 G O two 6 0 1 B O 13 G O one 6 0 1 B O 13 G O three 0 0 1 B O 13 G O two 4 0 1 B O 14 G O one 47 0 1 B O 14 G O three 0 0 1 B O 14 G O two 2 0 1 B O 15 G O one 5 0 1 B O 15 G O three 0 0 1 B O 15 G O two 0 0 1 B O 16 G O one 14 0 1 B O 16 G O three 0 0 1 B O 16 G O two 3 0 1 B O 17 G O one 22 0 1 B O 17 G O three 3 0 1 B O 17 G O two 7 0 1 B O 18 G O one 22 0 1 B O 18 G O three 1 0 1 B O 18 G O two 4 0 1 B O 19 G O one 7 0 1 B O 19 G O three 0 0 1 B O 19 G O two 0 0 1 B O 20 G O one 28 0 1 B O 20 G O three 0 0 1 B O 20 G O two 1 0 1 F O 2 G O one 65 0 1 F O 2 G O three 0 0 1 F O 2 G O two 1 0 1 F O 3 G O one 61 0 1 F O 3 G O three 1 0 1 F O 3 G O two 0 0 1 F O 4 G O one 54 0 1 F O 4 G O three 0 0 1 F O 4 G O two 0 0 1 F O 5 G O one 48 0 1 F O 5 G O three 0 0 1 F O 5 G O two 2 0 1 F O 6 G O one 37 0 1 F O 6 G O three 0 0 1 F O 6 G O two 0 0 1 F O 7 G O one 1 0 1 F O 7 G O three 0 0 1 F O 7 G O two 0 0 1 F O 8 G O one 12 0 1 F O 8 G O three 0 0 1 F O 8 G O two 0 0 1 F O 9 G O one 22 0 1 F O 9 G O three 0 0 1 F O 9 G O two 0 0 1 F O 10 G O one 15 0 1 F O 10 G O three 0 0 1 F O 10 G O two 0 0 1 F O 11 G O one 6 0 1 F O 11 G O three 0 0 1 F O 11 G O two 0 0 1 F O 12 G O one 29 0 1 F O 12 G O three 0 0 1 F O 12 G O two 0 0 1 F O 13 G O one 32 0 1 F O 13 G O three 10 0 1 F O 13 G O two 22 0 1 F O 14 G O one 8 0 1 F O 14 G O three 1 0 1 F O 14 G O two 0 0 1 F O 15 G O one 11 0 1 F O 15 G O three 0 0 1 F O 15 G O two 0 0 1 F O 16 G O one 14 0 1 F O 16 G O three 0 0 1 F O 16 G O two 0 0 1 F O 17 G O one 45 0 1 F O 17 G O three 0 0 1 F O 17 G O two 6 0 1 F O 18 G O one 32 0 1 F O 18 G O three 1 0 1 F O 18 G O two 3 0 1 F O 19 G O one 24 0 1 F O 19 G O three 0 0 1 F O 19 G O two 2 0 1 F O 20 G O one 14 0 1 F O 20 G O three 0 0 1 F O 20 G O two 5 0 1 P O 1 G O one 11 0 1 P O 1 G O three 0 0 1 P O 1 G O two 1 0 1 P O 2 G O one 21 0 1 P O 2 G O three 0 0 1 P O 2 G O two 0 0 1 P O 4 G O one 14 0 1 P O 4 G O three 0 0 1 P O 4 G O two 0 0 1 P O 5 G O one 14 0 1 P O 5 G O three 0 0 1 P O 5 G O two 0 0 1 P O 6 G O one 30 0 1 P O 6 G O three 0 0 1 P O 6 G O two 0 0 1 P O 7 G O one 15 0 1 P O 7 G O three 0 0 1 P O 7 G O two 0 0 1 P O 8 G O one 25 0 1 P O 8 G O three 0 0 1 P O 8 G O two 3 0 1 P O 9 G O one 3 0 1 P O 9 G O three 0 0 1 P O 9 G O two 0 0 1 P O 10 G O one 1 0 1 P O 10 G O three 0 0 1 P O 10 G O two 12 0 1 P O 11 G O one 5 0 1 P O 11 G O three 0 0 1 P O 11 G O two 3 0 1 P O 12 G O one 4 0 1 P O 12 G O three 0 0 1 P O 12 G O two 1 0 1 P O 13 G O one 0 0 1 P O 13 G O three 0 0 1 P O 13 G O two 0 0 1 P O 14 G O one 7 0 1 P O 14 G O three 0 0 1 P O 14 G O two 3 0 1 P O 15 G O one 0 0 1 P O 15 G O three 0 0 1 P O 15 G O two 0 0 1 P O 16 G O one 46 0 1 P O 16 G O three 0 0 1 P O 16 G O two 0 0 1 P O 17 G O one 0 0 1 P O 17 G O three 0 0 1 P O 17 G O two 1 0 1 P O 18 G O one 33 0 1 P O 18 G O three 0 0 1 P O 18 G O two 1 0 1 P O 19 G O one 30 0 1 P O 19 G O three 1 0 1 P O 19 G O two 4 0 1 P O 20 G O one 10 0 1 P O 20 G O three 0 0 1 P O 20 G O two 1 0 1 N N 1 N O one 0 1 0 N N 1 N O three 12 1 0 N N 1 N O two 9 1 0 N N 2 N O one 34 1 0 N N 2 N O three 4 1 0 N N 2 N O two 7 1 0 N N 3 N O one 48 1 0 N N 3 N O three 4 1 0 N N 3 N O two 14 1 0 N N 4 N O one 31 1 0 N N 4 N O three 0 1 0 N N 4 N O two 1 1 0 N N 5 N O one 47 1 0 N N 5 N O three 3 1 0 N N 5 N O two 15 1 0 N N 6 N O one 37 1 0 N N 6 N O three 6 1 0 N N 6 N O two 13 1 0 N N 7 N O one 77 1 0 N N 7 N O three 3 1 0 N N 7 N O two 8 1 0 N N 8 N O one 52 1 0 N N 8 N O three 5 1 0 N N 8 N O two 4 1 0 N N 9 N O one 17 1 0 N N 9 N O three 0 1 0 N N 9 N O two 0 1 0 N N 10 N O one 68 1 0 N N 10 N O three 1 1 0 N N 10 N O two 2 1 0 N N 11 N O one 47 1 0 N N 11 N O three 3 1 0 N N 11 N O two 4 1 0 N N 12 N O one 40 1 0 N N 12 N O three 4 1 0 N N 12 N O two 5 1 0 N N 13 N O one 37 1 0 N N 13 N O three 0 1 0 N N 13 N O two 0 1 0 N N 14 N O one 34 1 0 N N 14 N O three 6 1 0 N N 14 N O two 7 1 0 N N 15 N O one 31 1 0 N N 15 N O three 0 1 0 N N 15 N O two 4 1 0 N N 16 N O one 44 1 0 N N 16 N O three 5 1 0 N N 16 N O two 9 1 0 N N 17 N O one 46 1 0 N N 17 N O three 1 1 0 N N 17 N O two 1 1 0 N N 18 N O one 56 1 0 N N 18 N O three 4 1 0 N N 18 N O two 4 1 0 N N 19 N O one 39 1 0 N N 19 N O three 5 1 0 N N 19 N O two 6 1 0 N N 20 N O one 45 1 0 N N 20 N O three 4 1 0 N N 20 N O two 9 1 0 B O 1 N O one 26 1 0 B O 1 N O three 0 1 0 B O 1 N O two 14 1 0 B O 2 N O one 40 1 0 B O 2 N O three 5 1 0 B O 2 N O two 5 1 0 B O 3 N O one 12 1 0 B O 3 N O three 4 1 0 B O 3 N O two 2 1 0 B O 4 N O one 18 1 0 B O 4 N O three 2 1 0 B O 4 N O two 0 1 0 B O 5 N O one 3 1 0 B O 5 N O three 0 1 0 B O 5 N O two 1 1 0 B O 6 N O one 14 1 0 B O 6 N O three 0 1 0 B O 6 N O two 9 1 0 B O 7 N O one 15 1 0 B O 7 N O three 0 1 0 B O 7 N O two 8 1 0 B O 8 N O one 2 1 0 B O 8 N O three 4 1 0 B O 8 N O two 5 1 0 B O 9 N O one 30 1 0 B O 9 N O three 7 1 0 B O 9 N O two 17 1 0 B O 10 N O one 17 1 0 B O 10 N O three 1 1 0 B O 10 N O two 7 1 0 B O 11 N O one 13 1 0 B O 11 N O three 1 1 0 B O 11 N O two 5 1 0 B O 12 N O one 5 1 0 B O 12 N O three 1 1 0 B O 12 N O two 6 1 0 B O 13 N O one 20 1 0 B O 13 N O three 0 1 0 B O 13 N O two 2 1 0 B O 14 N O one 24 1 0 B O 14 N O three 1 1 0 B O 14 N O two 0 1 0 B O 15 N O one 7 1 0 B O 15 N O three 0 1 0 B O 15 N O two 0 1 0 B O 16 N O one 0 1 0 B O 16 N O three 0 1 0 B O 16 N O two 0 1 0 B O 17 N O one 42 1 0 B O 17 N O three 0 1 0 B O 17 N O two 4 1 0 B O 18 N O one 20 1 0 B O 18 N O three 0 1 0 B O 18 N O two 9 1 0 B O 19 N O one 16 1 0 B O 19 N O three 0 1 0 B O 19 N O two 3 1 0 B O 20 N O one 23 1 0 B O 20 N O three 2 1 0 B O 20 N O two 3 1 0 F O 2 N O one 45 1 0 F O 2 N O three 1 1 0 F O 2 N O two 3 1 0 F O 3 N O one 0 1 0 F O 3 N O three 0 1 0 F O 3 N O two 2 1 0 F O 4 N O one 16 1 0 F O 4 N O three 0 1 0 F O 4 N O two 0 1 0 F O 5 N O one 34 1 0 F O 5 N O three 1 1 0 F O 5 N O two 2 1 0 F O 6 N O one 18 1 0 F O 6 N O three 0 1 0 F O 6 N O two 5 1 0 F O 7 N O one 17 1 0 F O 7 N O three 1 1 0 F O 7 N O two 7 1 0 F O 8 N O one 35 1 0 F O 8 N O three 1 1 0 F O 8 N O two 2 1 0 F O 9 N O one 16 1 0 F O 9 N O three 0 1 0 F O 9 N O two 1 1 0 F O 10 N O one 1 1 0 F O 10 N O three 0 1 0 F O 10 N O two 0 1 0 F O 11 N O one 1 1 0 F O 11 N O three 1 1 0 F O 11 N O two 2 1 0 F O 12 N O one 34 1 0 F O 12 N O three 3 1 0 F O 12 N O two 0 1 0 F O 13 N O one 39 1 0 F O 13 N O three 2 1 0 F O 13 N O two 8 1 0 F O 14 N O one 3 1 0 F O 14 N O three 0 1 0 F O 14 N O two 1 1 0 F O 15 N O one 5 1 0 F O 15 N O three 0 1 0 F O 15 N O two 0 1 0 F O 16 N O one 23 1 0 F O 16 N O three 0 1 0 F O 16 N O two 1 1 0 F O 17 N O one 15 1 0 F O 17 N O three 0 1 0 F O 17 N O two 10 1 0 F O 18 N O one 13 1 0 F O 18 N O three 3 1 0 F O 18 N O two 12 1 0 F O 19 N O one 5 1 0 F O 19 N O three 0 1 0 F O 19 N O two 0 1 0 F O 20 N O one 0 1 0 F O 20 N O three 0 1 0 F O 20 N O two 0 1 0 P O 1 N O one 21 1 0 P O 1 N O three 0 1 0 P O 1 N O two 1 1 0 P O 2 N O one 2 1 0 P O 2 N O three 0 1 0 P O 2 N O two 0 1 0 P O 4 N O one 1 1 0 P O 4 N O three 0 1 0 P O 4 N O two 1 1 0 P O 5 N O one 19 1 0 P O 5 N O three 0 1 0 P O 5 N O two 3 1 0 P O 6 N O one 9 1 0 P O 6 N O three 0 1 0 P O 6 N O two 0 1 0 P O 7 N O one 7 1 0 P O 7 N O three 0 1 0 P O 7 N O two 0 1 0 P O 8 N O one 7 1 0 P O 8 N O three 0 1 0 P O 8 N O two 0 1 0 P O 9 N O one 21 1 0 P O 9 N O three 1 1 0 P O 9 N O two 6 1 0 P O 10 N O one 9 1 0 P O 10 N O three 0 1 0 P O 10 N O two 1 1 0 P O 11 N O one 3 1 0 P O 11 N O three 0 1 0 P O 11 N O two 0 1 0 P O 12 N O one 1 1 0 P O 12 N O three 0 1 0 P O 12 N O two 0 1 0 P O 13 N O one 0 1 0 P O 13 N O three 0 1 0 P O 13 N O two 0 1 0 P O 14 N O one 0 1 0 P O 14 N O three 1 1 0 P O 14 N O two 1 1 0 P O 15 N O one 2 1 0 P O 15 N O three 0 1 0 P O 15 N O two 0 1 0 P O 16 N O one 0 1 0 P O 16 N O three 0 1 0 P O 16 N O two 0 1 0 P O 17 N O one 19 1 0 P O 17 N O three 0 1 0 P O 17 N O two 2 1 0 P O 18 N O one 9 1 0 P O 18 N O three 0 1 0 P O 18 N O two 0 1 0 P O 19 N O one 41 1 0 P O 19 N O three 0 1 0 P O 19 N O two 3 1 0 P O 20 N O one 9 1 0 P O 20 N O three 0 1 0 P O 20 N O two 0 1 0 N N 1 NG O one 17 1 1 N N 1 NG O three 7 1 1 N N 1 NG O two 0 1 1 N N 2 NG O one 16 1 1 N N 2 NG O three 3 1 1 N N 2 NG O two 4 1 1 N N 3 NG O one 7 1 1 N N 3 NG O three 0 1 1 N N 3 NG O two 4 1 1 N N 4 NG O one 26 1 1 N N 4 NG O three 0 1 1 N N 4 NG O two 5 1 1 N N 5 NG O one 0 1 1 N N 5 NG O three 0 1 1 N N 5 NG O two 13 1 1 N N 6 NG O one 59 1 1 N N 6 NG O three 4 1 1 N N 6 NG O two 12 1 1 N N 7 NG O one 68 1 1 N N 7 NG O three 7 1 1 N N 7 NG O two 6 1 1 N N 8 NG O one 42 1 1 N N 8 NG O three 6 1 1 N N 8 NG O two 0 1 1 N N 9 NG O one 22 1 1 N N 9 NG O three 1 1 1 N N 9 NG O two 8 1 1 N N 10 NG O one 15 1 1 N N 10 NG O three 1 1 1 N N 10 NG O two 12 1 1 N N 11 NG O one 56 1 1 N N 11 NG O three 5 1 1 N N 11 NG O two 5 1 1 N N 12 NG O one 21 1 1 N N 12 NG O three 3 1 1 N N 12 NG O two 7 1 1 N N 13 NG O one 45 1 1 N N 13 NG O three 0 1 1 N N 13 NG O two 1 1 1 N N 14 NG O one 30 1 1 N N 14 NG O three 8 1 1 N N 14 NG O two 7 1 1 N N 15 NG O one 21 1 1 N N 15 NG O three 1 1 1 N N 15 NG O two 2 1 1 N N 16 NG O one 49 1 1 N N 16 NG O three 5 1 1 N N 16 NG O two 13 1 1 N N 17 NG O one 41 1 1 N N 17 NG O three 0 1 1 N N 17 NG O two 10 1 1 N N 18 NG O one 5 1 1 N N 18 NG O three 0 1 1 N N 18 NG O two 4 1 1 N N 19 NG O one 42 1 1 N N 19 NG O three 3 1 1 N N 19 NG O two 4 1 1 N N 20 NG O one 17 1 1 N N 20 NG O three 0 1 1 N N 20 NG O two 0 1 1 B O 1 NG O one 44 1 1 B O 1 NG O three 1 1 1 B O 1 NG O two 4 1 1 B O 2 NG O one 36 1 1 B O 2 NG O three 6 1 1 B O 2 NG O two 6 1 1 B O 3 NG O one 14 1 1 B O 3 NG O three 5 1 1 B O 3 NG O two 4 1 1 B O 4 NG O one 0 1 1 B O 4 NG O three 2 1 1 B O 4 NG O two 5 1 1 B O 5 NG O one 72 1 1 B O 5 NG O three 2 1 1 B O 5 NG O two 3 1 1 B O 6 NG O one 39 1 1 B O 6 NG O three 0 1 1 B O 6 NG O two 18 1 1 B O 7 NG O one 14 1 1 B O 7 NG O three 0 1 1 B O 7 NG O two 9 1 1 B O 8 NG O one 7 1 1 B O 8 NG O three 1 1 1 B O 8 NG O two 1 1 1 B O 9 NG O one 17 1 1 B O 9 NG O three 3 1 1 B O 9 NG O two 4 1 1 B O 10 NG O one 43 1 1 B O 10 NG O three 0 1 1 B O 10 NG O two 0 1 1 B O 11 NG O one 1 1 1 B O 11 NG O three 0 1 1 B O 11 NG O two 0 1 1 B O 12 NG O one 1 1 1 B O 12 NG O three 1 1 1 B O 12 NG O two 2 1 1 B O 13 NG O one 14 1 1 B O 13 NG O three 0 1 1 B O 13 NG O two 0 1 1 B O 14 NG O one 3 1 1 B O 14 NG O three 0 1 1 B O 14 NG O two 0 1 1 B O 15 NG O one 10 1 1 B O 15 NG O three 4 1 1 B O 15 NG O two 9 1 1 B O 16 NG O one 2 1 1 B O 16 NG O three 0 1 1 B O 16 NG O two 0 1 1 B O 17 NG O one 26 1 1 B O 17 NG O three 0 1 1 B O 17 NG O two 3 1 1 B O 18 NG O one 18 1 1 B O 18 NG O three 1 1 1 B O 18 NG O two 3 1 1 B O 19 NG O one 22 1 1 B O 19 NG O three 1 1 1 B O 19 NG O two 2 1 1 B O 20 NG O one 15 1 1 B O 20 NG O three 6 1 1 B O 20 NG O two 11 1 1 F O 2 NG O one 64 1 1 F O 2 NG O three 4 1 1 F O 2 NG O two 11 1 1 F O 3 NG O one 26 1 1 F O 3 NG O three 0 1 1 F O 3 NG O two 1 1 1 F O 4 NG O one 34 1 1 F O 4 NG O three 4 1 1 F O 4 NG O two 6 1 1 F O 5 NG O one 16 1 1 F O 5 NG O three 0 1 1 F O 5 NG O two 3 1 1 F O 6 NG O one 7 1 1 F O 6 NG O three 0 1 1 F O 6 NG O two 0 1 1 F O 7 NG O one 20 1 1 F O 7 NG O three 0 1 1 F O 7 NG O two 0 1 1 F O 8 NG O one 20 1 1 F O 8 NG O three 0 1 1 F O 8 NG O two 1 1 1 F O 9 NG O one 29 1 1 F O 9 NG O three 0 1 1 F O 9 NG O two 0 1 1 F O 10 NG O one 0 1 1 F O 10 NG O three 0 1 1 F O 10 NG O two 0 1 1 F O 11 NG O one 4 1 1 F O 11 NG O three 1 1 1 F O 11 NG O two 3 1 1 F O 12 NG O one 7 1 1 F O 12 NG O three 0 1 1 F O 12 NG O two 0 1 1 F O 13 NG O one 3 1 1 F O 13 NG O three 0 1 1 F O 13 NG O two 1 1 1 F O 14 NG O one 4 1 1 F O 14 NG O three 1 1 1 F O 14 NG O two 0 1 1 F O 15 NG O one 7 1 1 F O 15 NG O three 0 1 1 F O 15 NG O two 0 1 1 F O 16 NG O one 4 1 1 F O 16 NG O three 0 1 1 F O 16 NG O two 0 1 1 F O 17 NG O one 34 1 1 F O 17 NG O three 3 1 1 F O 17 NG O two 6 1 1 F O 18 NG O one 11 1 1 F O 18 NG O three 2 1 1 F O 18 NG O two 10 1 1 F O 19 NG O one 14 1 1 F O 19 NG O three 2 1 1 F O 19 NG O two 7 1 1 F O 20 NG O one 0 1 1 F O 20 NG O three 0 1 1 F O 20 NG O two 0 1 1 P O 1 NG O one 0 1 1 P O 1 NG O three 0 1 1 P O 1 NG O two 0 1 1 P O 2 NG O one 9 1 1 P O 2 NG O three 0 1 1 P O 2 NG O two 1 1 1 P O 4 NG O one 12 1 1 P O 4 NG O three 1 1 1 P O 4 NG O two 2 1 1 P O 5 NG O one 12 1 1 P O 5 NG O three 1 1 1 P O 5 NG O two 1 1 1 P O 6 NG O one 20 1 1 P O 6 NG O three 0 1 1 P O 6 NG O two 0 1 1 P O 7 NG O one 10 1 1 P O 7 NG O three 0 1 1 P O 7 NG O two 0 1 1 P O 8 NG O one 7 1 1 P O 8 NG O three 0 1 1 P O 8 NG O two 1 1 1 P O 9 NG O one 5 1 1 P O 9 NG O three 0 1 1 P O 9 NG O two 2 1 1 P O 10 NG O one 8 1 1 P O 10 NG O three 0 1 1 P O 10 NG O two 6 1 1 P O 11 NG O one 0 1 1 P O 11 NG O three 0 1 1 P O 11 NG O two 0 1 1 P O 12 NG O one 0 1 1 P O 12 NG O three 0 1 1 P O 12 NG O two 1 1 1 P O 13 NG O one 0 1 1 P O 13 NG O three 0 1 1 P O 13 NG O two 0 1 1 P O 14 NG O one 2 1 1 P O 14 NG O three 0 1 1 P O 14 NG O two 0 1 1 P O 15 NG O one 0 1 1 P O 15 NG O three 0 1 1 P O 15 NG O two 0 1 1 P O 16 NG O one 0 1 1 P O 16 NG O three 0 1 1 P O 16 NG O two 0 1 1 P O 17 NG O one 18 1 1 P O 17 NG O three 0 1 1 P O 17 NG O two 0 1 1 P O 18 NG O one 1 1 1 P O 18 NG O three 0 1 1 P O 18 NG O two 0 1 1 P O 19 NG O one 9 1 1 P O 19 NG O three 1 1 1 P O 19 NG O two 3 1 1 P O 20 NG O one 5 1 1 P O 20 NG O three 0 1 1 P O 20 NG O two 0 1 1