[R-meta] AICc or variance components, which one matters more?

Hi Wolfgang,

Thank you! If we go by your ICC principle, then since almost all
studies uniquely belong to one lab (except one exception), then ~ 1 |
lab/study (as in g1) should prevail over list(~1|lab, ~1|study) [as in
g2].

But the thing is that AICc doesn't agree with this. That's exactly
where I get stuck in preferring one model over the other. Model g1
matches the data structure better, but g2 has a smaller AICc?

You proposed g3 to solve this impasse, but g3 gives the same AICc as that of g2.

Moreover, we have not yet added any moderators. If we do, then using
g2 or g3 with already 0 variance components would mean that such
zero-variance components really don't do much in the model. And that
was why I thought specifying random effects as in g1 which gives
non-zero variance components seems like a better use of the random
effects especially as we add moderators.


(g1=rma.mv(yi, vi, random = ~1|lab/study, data = dd))
(g2=rma.mv(yi, vi, random = list(~1|lab, ~1|study), data = dd))
(g3=rma.mv(yi, vi, random = list(~1|lab/study, ~ 1 | study), data = dd))

fitstats(g1,g2,g3)[5,]

             g1            g2           g3
AICc: 30.85992 29.73897 35.73897

On Mon, Nov 15, 2021 at 1:14 PM Viechtbauer, Wolfgang (SP)
<wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

The various models allow for different correlation structures for the underlying true effects. For example, the model with only ~1|lab/study implies an ICC of sigma^2_lab / (sigma^2_lab + sigma^2_study) for true effects belonging to different studies within the same lab and does not allow true effects to be correlated across labs even if these labs were involved in the same study. If one wants to account for the latter, one can add ~ 1 | study and this doesn't automatically have to replace the /study part in ~1|lab/study.

In the original data structure you showed, there were at times multiple rows for the same study within the same lab. One could then even go further and use ~1|lab/study/es_id because without this, the ~1|lab/study model implies an ICC of 1 for true effects belonging to the same study within the same lab. So one could even entertain the model:

(g5=rma.mv(yi, vi, random = ~1|lab/study/es_id, data = dd))

and then again:

(g6=rma.mv(yi, vi, random = list(~1|lab/study/es_id, ~ 1 | study), data = dd))

Of course this is all silly with the toy dataset, but even there all variance components are identifiable.

Ultimately, fully understanding these models requires writing out what they imply about the ICC for different combinations of lab, study, and es_id (e.g., same lab and same study and different es_id, same lab and different studies, different labs and same study, and so on). As any good stats book would say at this point: I leave this as an exercise to the reader.

Best,
Wolfgang

-----Original Message-----
From: Luke Martinez [mailto:martinezlukerm at gmail.com]
Sent: Monday, 15 November, 2021 19:32
To: Viechtbauer, Wolfgang (SP)
Cc: R meta
Subject: Re: [R-meta] AICc or variance components, which one matters more?

ATTACHMENT(S) REMOVED: g4.PNG

Ok, I see what you mean regarding g4. "lab" is just repetitive
(profile curves for two repetitive lab var. components are mainly
flat). Ok g4 is out.

But I still wonder about the thinking behind g3 where 'study' can be
both nested and at the same time crossed?

(g3=rma.mv(yi, vi, random = list(~1|lab/study, ~ 1 | study), data = dd))

On Mon, Nov 15, 2021 at 12:07 PM Viechtbauer, Wolfgang (SP)
<wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

Please post the profile likelihood plots for the three variance components of

model g4.

-----Original Message-----
From: Luke Martinez [mailto:martinezlukerm at gmail.com]
Sent: Monday, 15 November, 2021 18:37
To: Viechtbauer, Wolfgang (SP)
Cc: R meta
Subject: Re: [R-meta] AICc or variance components, which one matters more?

In g4, also, all variance components are identifiable (and larger in

magnitude).

But what's the thinking behind either g3 or g4 where the same grouping

variable

is both nested and crossed?

On Mon, Nov 15, 2021, 11:33 AM Viechtbauer, Wolfgang (SP)
<wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
In g3, all variance components are identifiable.

What have you found out about g4?

-----Original Message-----
From: Luke Martinez [mailto:martinezlukerm at gmail.com]
Sent: Monday, 15 November, 2021 18:28
To: Viechtbauer, Wolfgang (SP)
Cc: R meta
Subject: Re: [R-meta] AICc or variance components, which one matters more?

Hi Wolfgang,

Could you possibly elaborate a bit on "in this case, yes"?

This allows me to better justify g3 or g4 models to my co-authors.

Thanks again,
Luke

On Mon, Nov 15, 2021, 11:19 AM Viechtbauer, Wolfgang (SP)
<wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

-----Original Message-----
From: Luke Martinez [mailto:martinezlukerm at gmail.com]
Sent: Saturday, 13 November, 2021 21:10
To: Viechtbauer, Wolfgang (SP)
Cc: R meta
Subject: Re: [R-meta] AICc or variance components, which one matters more?

Interesting! To make sure I'm following you, your suggested g3 model
both considers 'study' to be nested in the 'lab', and at the same time
it considers 'study' to have its own independent crossed effect. Can
we consider the same variable (e.g., study) to be both nested and
crossed at the same time?

In this case, yes.

If so, I can then suggest the following model as well:

(g4=rma.mv(yi, vi, random = list(~1|lab/study, ~ 1 | lab), data = dd))

I suggest you profile the variance components from that model and draw
appropriate conclusions.

Doesn't this denote that one is uncertain about whether to take a
variable as nested or crossed or there are other justifications?

Thank you,
Luke

On Sat, Nov 13, 2021 at 1:11 PM Viechtbauer, Wolfgang (SP)
<wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

-----Original Message-----
From: Luke Martinez [mailto:martinezlukerm at gmail.com]
Sent: Saturday, 13 November, 2021 18:44
To: Viechtbauer, Wolfgang (SP)
Cc: Philippe Tadger; R meta
Subject: Re: [R-meta] AICc or variance components, which one matters

more?

Hi Wolfgang,

I'm fully with you, however, in my data only once 2 labs (labs 1 and
2) have collaborated on study 2. Specifically, part of study 2 has
been carried out by lab 1 (one row) and part of it by lab 2 (one row).
Except in this case, no such between-lab collaborations have ever
occurred in the data.

If such a between-lab collaboration didn't exist, I could directly go
for g1 (hierarchical model). But with this collaboration, there is
just a tiny possibility for g2 (crossed model) as well.

So, do you think AICc should be the basis of the comparison between g1
vs. g2 or the dominant data structure (ignoring the one exception)?

Using information criteria *could* be the basis. But I might be inclined

to

just ignore the issue you describe above though if this only affects one

study.

Just as a note: It's not necessarily an either-or choice. This model is

also

possible:

(g3=rma.mv(yi, vi, random = list(~1|lab/study, ~ 1 | study), data = dd))

and profile(g3) suggests that all variance components are identifiable -

although of course this is quite overfitted with so little data.

(g1=rma.mv(yi, vi, random = ~1|lab/study, data = dd))
(g2=rma.mv(yi, vi, random = list(~1|lab, ~1|study), data = dd))

Thanks,
Luke