lme capable of running with missing data?

Kevin,

I understand that but then how is SAS accomplishing the interactions?
I have been following this conversation a little bit and this seems to be the right question to ask. I would also like to know the answer. However, this could be the wrong venue to get an answer to this question.

?
On Fri, Feb 3, 2012 at 10:58 AM, Kevin Wright <kw.stat at gmail.com> wrote:

Charles,

Here's a simple thought example.  Use a piece of graph paper (or just a
simple sketch).  Write the following letters at the coordinates specified:

A1  (1,1)
A2 (3,2)
B1 (1,2)
B2 (missing)

Draw a line from A1 to A2.  Imagine a line from B1 to the missing value of
B2.

By looking at this, you could calculate an overall mean for the A factor.
You could also estimate an overall mean for the B factor, if you assume the
lines are parallel.  This is what happens with fixed effects, as in lme (
... A + B, ...).
But, when you specify lme(... A*B, ...) which is the same as lme(... A + B
+ A:B, ...), you are essentially saying to the computer, "The A1-A2 and
B1-B2 lines are not parallel, but please give me an estimate of the slope
of the B1-B2 line."

Could _you_ draw the B1-B2 line?  No.  Neither can lme.

It's okay that B2 is missing if you don't want to fit an interaction, but
when B2 is missing, there is no way to estimate an interaction
(non-parallel slope).

Kevin

On Fri, Feb 3, 2012 at 10:18 AM, Charles Determan Jr <deter088 at umn.edu>wrote:

After the data is input, and factors are assigned,

model=lme(arginine~group*time*survival, random=~1|subj, method="REML",
data=x)

Error in MEEM(object, conLin, control$niterEM) : Singularity in backsolve
at level 0, block 1

On Fri, Feb 3, 2012 at 10:01 AM, Kevin Wright <kw.stat at gmail.com> wrote:

Providing the data is not a "reproducible example".  Complete data and
R-code are helpful.

Kevin

On Fri, Feb 3, 2012 at 9:31 AM, Charles Determan Jr <deter088 at umn.edu>wrote:

Here is the dataset, everything should be run as a factor except 'met'
which is numeric.  Thanks for the assistance,

   time group survival subj met
1      1     2        1      2   1.3954
2      2     2        1      2   1.8063
3      3     2        1      2   1.3684
4      4     2        1      2   2.0046
5      5     2        1      2   1.0334
6      6     2        1      2   0.3644
7      7     2        1      2   0.4819
8      8     2        1      2   1.4558
9      9     2        1     2   0.9718
10     1     1        2    5   0.7771
11     2     1        2    5   1.2439
12     1     2        2    8   1.0980
13     2     2        2    8   0.9511
14     1     2        1    9   1.0534
15     2     2        1    9   1.7279
16     3     2        1    9   1.4904
17     4     2        1    9   1.2737
18     5     2        1    9   0.8929
19     6     2        1    9   0.5828
20     7     2        1    9   0.3260
21     8     2        1    9   1.0373
22     9     2        1    9   0.9624
23     1     2        2   10   1.1391
24     2     2        2   10   1.3945
25     3     2        2   10   0.9414
26     4     2        2   10   1.1152
27     5     2        2   10   0.8222
28     6     2        2   10   0.4417
29     7     2        2   10   0.4126
30     1     1        1   12   1.3024
31     2     1        1   12   1.1811
32     3     1        1   12   0.9379
33     4     1        1   12   1.3000
34     5     1        1   12   1.2977
35     6     1        1   12   0.4949
36     7     1        1   12   0.5238
37     8     1        1   12   1.3862
38     1     1        1   16   1.2259
39     2     1        1   16   0.8681
40     3     1        1   16   1.2645
41     4     1        1   16   0.7316
42     5     1        1   16   0.6648
43     6     1        1   16   0.9671
44     7     1        1   16   1.0131
45     8     1        1   16   1.1762
46     9     1        1   16   0.8776
47     1     2        2   18   1.1231
48     2     2        2   18   1.2133
49     3     2        2   18   1.2005
50     4     2        2   18   0.7198
51     5     2        2   18   0.6620
52     6     2        2   18   0.5908
53     7     2        2   18   0.3945
54     1     2        2   19   0.7852
55     2     2        2   19   0.6758
56     3     2        2   19   0.5246
57     4     2        2   19   0.5263
58     1     2        2   20   1.2284
59     2     2        2   20   0.7017
60     1     2        1   23   0.9604
61     2     2        1   23   0.7977
62     3     2        1   23   1.2267
63     4     2        1   23   1.3857
64     5     2        1   23   0.9486
65     6     2        1   23   0.3571
66     7     2        1   23   0.3134
67     8     2        1   23   1.9984
68     9     2        1   23   0.4837
69     1     1        1   24   1.1793
70     2     1        1   24   1.3883
71     3     1        1   24   2.1080
72     4     1        1   24   0.8810
73     5     1        1   24   0.8825
74     6     1        1   24   0.4124
75     7     1        1   24   0.5270
76     8     1        1   24   1.9003
77     9     1        1   24   1.4344
78     1     1        1   27   1.1905
79     2     1        1   27   1.1033
80     3     1        1   27   1.4976
81     4     1        1   27   1.9018
82     5     1        1   27   0.5815
83     6     1        1   27   0.4428
84     7     1        1   27   0.4728
85     8     1        1   27   1.6309
86     9     1        1   27   0.4054
87     1     1        1   28   0.9538
88     2     1        1   28   0.7796
89     3     1        1   28   1.7906
90     5     1        1   28   0.4715
91     6     1        1   28   0.4214
92     7     1        1   28   0.4120
93     8     1        1   28   1.3111
94     9     1        1   28   0.3677
95     1     1        2    1   1.3853
96     2     1        2    1   1.5966
97     3     1        2    1   1.4542
98     4     1        2    1   1.3084
99     5     1        2    1   1.2826
100    6     1        2    1   0.6835
101    7     1        2    1   0.9709
102    1     1        1    3   1.3175
103    2     1        1    3   0.7792
104    3     1        1    3   1.8763
105    5     1        1    3   1.4633
106    6     1        1    3   0.0735
107    7     1        1    3   0.5612
108    8     1        1    3   1.3777
109    9     1        1    3   0.3810
110    1     1        2    4   1.3486
111    1     1        1    6   1.2635
112    2     1        1    6   0.7572
113    3     1        1    6   1.5011
114    5     1        1    6   0.6873
115    6     1        1    6   0.3778
116    7     1        1    6   0.4231
117    8     1        1    6   1.3817
118    9     1        1    6   0.5850
119    1     2        2    7   0.7362
120    2     2        2    7   0.5495
121    3     2        2    7   0.7621
122    4     2        2    7   0.8421
123    5     2        2    7   1.0438
124    6     2        2    7   0.9802
125    7     2        2    7   0.5627
126    1     1        1   11   1.5575
127    2     1        1   11   2.1356
128    3     1        1   11   1.3575
129    4     1        1   11   1.3056
130    5     1        1   11   0.8144
131    6     1        1   11   0.5876
132    7     1        1   11   0.4104
133    9     1        1   11   0.4942
134    1     2        1   13   1.0046
135    2     2        1   13   0.8805
136    3     2        1   13   0.7685
137    4     2        1   13   0.8786
138    5     2        1   13   1.4249
139    6     2        1   13   0.5339
140    7     2        1   13   0.5480
141    8     2        1   13   2.6369
142    9     2        1   13   1.7159
143    1     2        1   14   0.7161
144    2     2        1   14   0.3968
145    3     2        1   14   0.8142
146    4     2        1   14   0.6140
147    5     2        1   14   0.6585
148    6     2        1   14   0.7176
149    7     2        1   14   0.6613
150    8     2        1   14   1.6494
151    9     2        1   14   0.3903
152    1     1        1   15   1.4357
153    2     1        1   15   1.4772
154    3     1        1   15   1.3156
155    4     1        1   15   0.9654
156    5     1        1   15   1.2709
157    6     1        1   15   0.9330
158    7     1        1   15   0.3515
159    8     1        1   15   1.6801
160    9     1        1   15   0.3584
161    1     2        2   17   0.8077
162    2     2        2   17   0.7560
163    1     1        1   21   1.1890
164    2     1        1   21   0.9631
165    3     1        1   21   0.9753
166    4     1        1   21   0.9519
167    5     1        1   21   0.6348
168    6     1        1   21   0.8516
169    7     1        1   21   0.2366
170    8     1        1   21   1.0440
171    9     1        1   21   0.5360
172    1     2        1   22   1.0747
173    2     2        1   22   0.6451
174    3     2        1   22   0.8408
175    5     2        1   22   0.8730
176    6     2        1   22   0.3594
177    7     2        1   22   0.3019
178    9     2        1   22   1.2053
179    1     2        2   25   0.4654
180    2     2        2   25   0.3024
181    3     2        2   25   0.7525
182    4     2        2   25   0.7808
183    5     2        2   25   0.6294
184    6     2        2   25   0.3016
185    7     2        2   25   0.3223
186    1     2        1   26   0.5363
187    2     2        1   26   0.2279
188    3     2        1   26   0.4756
189    4     2        1   26   0.6644
190    5     2        1   26   0.6631
191    6     2        1   26   0.3419
192    7     2        1   26   0.4188
193    8     2        1   26   0.3199
194    9     2        1   26   0.2889
195    1     1        2   29   1.2765
196    2     1        2   29   1.0653
197    3     1        2   29   1.5607
198    1     1        1   30   0.8641
199    2     1        1   30   0.9250
200    3     1        1   30   1.0887
201    4     1        1   30   0.5537
202    5     1        1   30   0.7930
203    6     1        1   30   0.3960
204    7     1        1   30   0.3917
205    8     1        1   30   1.2687
206    9     1        1   30   0.5328
207    1     2        1   31   1.0765
208    2     2        1   31   0.8778
209    3     2        1   31   0.8228
210    4     2        1   31   1.2017
211    5     2        1   31   1.1787
212    6     2        1   31   0.4037
213    7     2        1   31   0.2625
214    8     2        1   31   2.2690
215    9     2        1   31   0.4423
216    1     1        2   32   1.2880
217    2     1        2   32   0.8537

On Fri, Feb 3, 2012 at 9:25 AM, Baldwin, Jim -FS <jbaldwin at fs.fed.us>
wrote:

I think the only way to resolve this is to provide a specific example.

Jim Baldwin
Station Statistician
USDA Forest Service
Albany, California

-----Original Message-----
From: r-sig-mixed-models-bounces at r-project.org [mailto:
r-sig-mixed-models-bounces at r-project.org] On Behalf Of Charles
Determan Jr
Sent: Friday, February 03, 2012 7:18 AM
To: Thompson,Paul; r-sig-mixed-models at r-project.org
Subject: Re: [R-sig-ME] lme capable of running with missing data?

So, is there a way in which I can alter the design matrix so the mixed
model will work or is this something that can only be done in SAS
currently?  The output from the SAS run did provide Type III fixed
effect
test values.

On Fri, Feb 3, 2012 at 9:14 AM, Thompson,Paul <
Paul.Thompson at sanfordhealth.org> wrote:

 That's interesting. SAS uses the sweep approach (it was in fact
devised by Goodnight). The method used in construction of various
types of SS does allow you to estimate when cells are missing. I
would
wonder if Type II SS can be done. Type III (despite the incorrect
statement that they are
illegitimate) and Type IV would work fine. ****

** **

It's really an issue of the manner in which the design matrix is
contructed.****

** **

*From:* Charles Determan Jr [mailto:deter088 at umn.edu](javascript:main.compose()
*Sent:* Friday, February 03, 2012 8:36 AM
*To:* Thompson,Paul; r-sig-mixed-models at r-project.org
*Subject:* Re: [R-sig-ME] lme capable of running with missing
data?****

** **

Thank you Paul, I do appreciate your response and especially your
time.
The reason I am so persistent is that I know the prior data I posted
was run in SAS (however I don't have the exact coding although I do
know it was done with PROC MIXED with an unstructured covariance
structure and REML estimation method) and it provided all the
interactions.  As such, I have scoured the web and literature as to
how this could be done with the missing data (timepoints as a result
of survival).  Perhaps this simply has not yet been done in R and I
am
stuck for the time being.  None-the-less, I want to be certain
before I
give up on running this type of analysis in R.
Thanks again,****

On Fri, Feb 3, 2012 at 8:26 AM, Thompson,Paul <
Paul.Thompson at sanfordhealth.org> wrote:****

Charles:

I did suggest the use of specific contrasts to do the analysis with
missing cells. I played around, and just have to admit that this is
not possible. I tried to use standard construction techniques to
produce main effects using contrast coding, and then multiply those
to
produce interactions. This does not work. It may be possible to use
orthonormalization and the sweep operator to produce a consistent
estimator, but I ran out of time to work on this.

What you can do is convert the design to a single factor, and do the
analysis with specific contrasts, recognizing that this will not
enable you to get to specific things like interaction effects. To
understand why, consider the situation with a 2 x 2, where one cell
is
entirely missing.
You have lost 1 df for the design, and the interaction is entirely
missing.
You can estimate and test specific contrasts, but you can't even
really test the A factor or the B factor. If Cell(2,2) is missing,
you
can test Cell (1,1) v Cell(1,2) and you can test Cell (2,1) v Cell
(1,1), but neither of these is the test of the "main effect" of A or
B. When you have larger designs with 2 or 3 factors, the comparisons
again have fewer df than should be encountered. This means that the
interactions are not defined properly.

Do you NEED the interactions for theoretical purposes, or are they
there simply for completedness? Are the cells missing due to your
design or due to happenstance?

It is the case that fractional factorial designs eliminate cells
from
the design to estimate main effects and losing the ability to
estimate
interactions. So, missing cells, when planned for appropriately, can
result in appropriate analysis. I am not sure how to run mixed
models
with fractional factorials, however.****

-----Original Message-----
From: r-sig-mixed-models-bounces at r-project.org [mailto:
r-sig-mixed-models-bounces at r-project.org] On Behalf Of Charles
Determan Jr
Sent: Friday, February 03, 2012 7:06 AM
To: r-sig-mixed-models at r-project.org
Subject: [R-sig-ME] lme capable of running with missing data?

Greetings,

Some of you may recognize my name from a few related posts but I
just
have general question that perhaps can be clarified.  I have read
several times that 'lme' and 'lmer' are techniques capable of
running
data sets with missing values.  Is this true?  I have put up similar
posts where when I try to run a two or three way interaction mixed
model I get an error of singularities or X'X not positive.  Does the
data set need to be formatted in some way where the mixed model can
be
run with all interactions?
Furthermore, if the missing values are 'not missing at random' is
there another method to follow for generating the mixed model?  I am
just confused why I see posts that lme can be run when data is
missing.
Regards,****

       [[alternative HTML version deleted]]

_______________________________________________
R-sig-mixed-models at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models

----------------------------------------------------------------------
- Confidentiality Notice: This e-mail message, including any
attachments, is for the sole use of the intended recipient(s) and
may
contain privileged and confidential information.  Any unauthorized
review, use, disclosure or distribution is prohibited.  If you are
not
the intended recipient, please contact the sender by reply e-mail
and
destroy all copies of the original message.****

** **

----------------------------------------------------------------------
- Confidentiality Notice: This e-mail message, including any
attachments, is for the sole use of the intended recipient(s) and
may
contain privileged and confidential information. Any unauthorized
review, use, disclosure or distribution is prohibited. If you are
not
the intended recipient, please contact the sender by reply e-mail
and
destroy all copies of the original message.

        [[alternative HTML version deleted]]

       [[alternative HTML version deleted]]

_______________________________________________
R-sig-mixed-models at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models

--
Kevin Wright

--
Kevin Wright

	[[alternative HTML version deleted]]

_______________________________________________
R-sig-mixed-models at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
On 02/03/12, Charles Determan Jr   wrote:
Kevin,

I understand that but then how is SAS accomplishing the interactions?
I have been following this conversation a little bit and this seems to be the right question to ask. I would also like to know the answer. However, this could be the wrong venue to get an answer to this question.
<SNIP>

It may be the case that fortune(203) is relevant here! :-)

     cheers,

         Rolf Turner
On 04/02/12 14:45, Kenneth Frost wrote:
On 02/03/12, Charles Determan Jr ? wrote:
Kevin,

I understand that but then how is SAS accomplishing the interactions?

I have been following this conversation a little bit and this seems to be
the right question to ask. I would also like to know the answer. However,
this could be the wrong venue to get an answer to this question.
<SNIP>

It may be the case that fortune(203) is relevant here! :-)
Mathematical impossibilty, no (fortune(203) refers to obtaining
negative estimates of variance components, IIRC).  The problem here is
determining a full-rank model matrix for a model with interactions and
missing cells.  Because SAS uses the sweep operator in solving least
squares problems it does not encounter problems with rank deficiency.
(I am sorely tempted to make remarks about "sweeping them under the
carpet".)  In fact, SAS expects to handle rank deficiencies because it
generates a redundant set of indicators for each factor variable then
prunes them on the fly.

The approach in R is to generate a model matrix that should be of
full-rank except in circumstances like this and to check for rank
deficiency.  There is special code in the version of the QR
decomposition used with R to detect rank deficiency and pivot the
offending columns out but keep the others in their original order.

Dirk Eddelbuettel and I explored several approaches to handling such
rank deficiency in the vignette accompanying the RcppEigen package
(http://cran.us.r-project.org/web/packages/RcppEigen/vignettes/RcppEigen-intro-nojss.pdf).
 The development version of lme4 (called lme4Eigen on the R-forge
project site) detects rank deficiency earlier in the calculation but
does not yet repair the rank deficiency.  Using the column-pivoted QR
decomposition is probably the best approach but even then it would be
necessary to find the columns that are linear dependent on columns to
their left then drop only those columns.  It is not impossible by any
means, it just requires some work and is not high on the priority list
right now.

Regarding type III tests, I have forgotten which ones they are.  Are
they the sequential sums of squares or the ones where you drop the
main effect but keep the interactions thereby rendering your null
model nonsensical is most cases?
All the silliness about Types I, II, III and IV sums of squares and
tests was formulated when fitting any model was difficult (see
fortune("JCL")).  So doing a hypothesis test by fitting the null model
and fitting the alternative model and comparing the results would take
much much longer than doing a lot of linear algebra gymnastics on the
fit of the full or alternative model.  That is no longer the case.  If
you really want to perform a hypothesis test then formulate it in
terms of models, fit them and compare them.  It's not difficult and
has the undeniable advantage of forcing you to think about the model
and whether it makes sense.  Read Bill Venables' famous unpublished
paper "Exegeses on Linear Models" (just put the name in a search
engine).  (By the way, Bill is going to be at the useR conference in
Nashville in July so maybe if a bunch of us ganged up on him he could
be convinced to submit a version of that paper for publication.)