[R-meta] nesting an inner | outer formula

Standard errors for variance components are typically not all that useful,
since the sampling distribution of a variance component estimate is
typically not normal. One can still obtain them with rma.mv() using
cvvc=TRUE (this estimates the entire var-cov matrix of the
variance/correlation components).

Variance component estimates (and corresponding CIs) are indeed forced to
be non-negative by rma.mv(). One can get around this to some extent by
using an '~ inner | outer' random effect instead of '~ 1 | outer/inner',
since in the former the correlation can be negative, which corresponds to a
negative variance for the outer term in '~ 1 | outer/inner'. See:

https://www.metafor-project.org/doku.php/analyses:konstantopoulos2011

But if you really want negative variances (for all variance components),
then you will have to use something different than rma.mv(). Allowing for
negative variances causes so many coding headaches that I decided not to
implementing this (I do allow for negative tau^2 estimates in rma.uni() but
this is a much more restricted model space, so handling all the special
circumstances around negative variances is much easier there).

And yes, I see no inherent problem when a variance component is estimated
to be essentially zero.

Best,
Wolfgang

-----Original Message-----
From: Ross Neville <ross.neville at ucd.ie>
Sent: Tuesday, February 18, 2025 14:46
To: Viechtbauer, Wolfgang (NP) <

wolfgang.viechtbauer at maastrichtuniversity.nl>

Subject: Re: nesting an inner | outer formula

Okay. Thanks for that suggestion for validating the results. My fear was

that

rma.mv was not estimating a standard error for the sigma^2.2 variance

and that

was a problem. I also feared that rma.mv was forcing the lower CI to be
positive, or exactly zero. And in the presence of the model ?wanting? to
estimate a negative variance given the data, rma.mv was resisting that

because a

variance can?t be negative. And given all this, the sigma^2.2 is not

worth

having in the analysis at all. Are you saying instead that the sigma^2.2

being 0

and having a lower CI of exactly 0 is fine and that model sensible and

something

that one could move forward with?

Dr Ross D. Neville, PhD, ProfCert University Teaching and Learning
Head of Subject - Sport Management
School of Public Health, Physiotherapy and Sport Science
University College Dublin (UCD)
Room G6 - Woodview House
Belfield, Dublin 4
mailto:ross.neville at ucd.ie
+353 (0) 1 716 3419

On Tue 18 Feb 2025 at 13:21, Viechtbauer, Wolfgang (NP)
<mailto:wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
Given the 'nlvls' values shown, you appear to have enough data to get

fairly

accurate estimates (you could check confint(model) to see how tight the

CIs

are). So I don't think the model is too complex. It could very well be

that

there isn't much heterogeneity in the Condition effect.

You could also check this by computing the mean difference within

studies for

Exp versus Control (if there are multiple informats, do this for every

type).

Strictly speaking again, these mean differences are not independent

(since

mean_trt_parent - mean_ctrl_parent and mean_trt_teacher -

mean_ctrl_teacher

involves reports on the same children by parents and teachers), but if

you

ignored V in the model results you have shown, then we can do the same

here).

You should find that those mean differences are fairly consistent (at

least not

more variable than would be expected based on their sampling variances).

Best,
Wolfgang

-----Original Message-----
From: Ross Neville <mailto:ross.neville at ucd.ie>
Sent: Tuesday, February 18, 2025 13:41
To: Viechtbauer, Wolfgang (NP)

<mailto:wolfgang.viechtbauer at maastrichtuniversity.nl>

Cc: mailto:r-sig-meta-analysis at r-project.org
Subject: Re: nesting an inner | outer formula

Thanks Wolfgang
I will take some time to digest the contents. I appreciate the detail

and

guidance.
Quickly, the suggested model provides a sigma^2.2 variance of 0, as

shown

below.
This makes me feel like the random effects random = ~ 1 | StudyID /
TreatmentGroup / Informant are too complex for the available data.
Thought on that?

On Tue, 18 Feb 2025 at 12:34, Viechtbauer, Wolfgang (NP)
<mailto:mailto:wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
Dear Ross,

Thanks for the clarification. Based on this, I would consider the

following

structure:

random = ~ 1 | StudyID / TreatmentGroup / Informant

This captures overall differences in the outcomes (across the

experimental and

control groups and across all informants) between studies. It also

allows for

heterogeneity in how much experimental and control groups differ from

each

other

across studies (I assume you will add something like mods = ~

TreatmentGroup

to

the model, since presumably you are interested in the size of the

(average)

difference between the two groups). And it allows for heterogeneity in

outcomes

that arises due to some studies using multiple informants. By nesting

Informant

within TreatmentGroup, this model automatically implies a certain

degree of

correlation in the true outcomes for different informants within
experimental/control groups. This last random effect is also the

'outcome

level'

random effect, since based on your description, every combination of

StudyID,

TreatmentGroup, and Informant should yield a unique value for each row.

Strictly speaking, this structure does not capture the correlation in

the

sampling errors that arises because multiple informants are reporting

on the

*same children*. If reports, say from parents and teachers, are

correlated,

then

this implies that the sampling errors of the means are also

correlated. Such a

correlation (or more precisely, the covariance) should go into the V

matrix.

However, to compute this covariance, you would need to know what the

correlation

(r) between the parent and teacher reports. This is probably not

reported, but

maybe can be guestimated from other or your own studies. Say the data

for the

first two studies looks like this:

Study  Group  Informant   Outcome
1      Exp    Child       .
1      Ctrl   Child       .
2      Exp    Parent      .
2      Exp    Teacher     .
2      Ctrl   Parent      .
2      Ctrl   Teacher     .

Then the corresponding V matrix would be (use a fixed width font to

view this

so

that things are lined up properly):

[s_1E^2/n_1E
        ]
[            s_1C^2/n_1C
        ]
[                        s_2EP^2/n_2E
r*s_2EP*s_2ET/n_2E                                ]
[                                     s_2ET^2/n_2E
        ]
[                                                        s_2CP^2/n_2C
r*s_2CP*s_2CT/n_2C]

[

 s_2CT^2/n_

2C      ]

where s stands for standard deviation, n for sample size, and the

subscripts

are

for study, group, and informant in that order (single letter

abbreviations).

Elements left blank are equal to 0 (zero covariance). I did not put a

subscript

on r since this is probably a single guestimated value across all

studies (and

informant pairs). Such a V matrix can be easily generated using the

vcalc()

function.

Since V is probably just going to be an approximation (especially if

you

decide

not to bother creating the V matrix, which in essence implies assuming

r=0), I

would then consider using cluster-robust inference methods

(robust(model,

cluster=StudyID, clubSandwich=TRUE)) at least as a sensitivity check.

I think the above is a good and sensible starting point. A more complex
structure might be:

dat$StudyID.TreatmentGroup <- paste0(dat$StudyID, ".",

dat$TreatmentGroup)

random = list(~ TreatmentGroup | StudyID, ~ Informant |

StudyID.TreatmentGroup),

struct="UN")

This would allow for different variances for experimental and control

groups

and

it would allow for different variances for the different types of

informants

and

allow the correlation in the random effects to differ depending on the

type of

informant pair (child-parent, child-teacher, parent-teacher, etc.).

But I

would

only attempt to fit this model if there is plenty of data. One could

use a LRT

to compare the two model structures. There is also a structure with

intermediate

complexity by using struct=c("UN","HCS"), where we assume different

variances

for the different informants, but a single correlation irrespective of

the

pair.

One might also consider TreatmentGroup and Informant to be crossed

random

effects (within studies), but I think this is overcomplicating things.

Best,
Wolfgang

-----Original Message-----
From: Ross Neville <mailto:mailto:ross.neville at ucd.ie>
Sent: Friday, February 14, 2025 14:32
To: Viechtbauer, Wolfgang (NP)

<mailto:mailto:wolfgang.viechtbauer at maastrichtuniversity.nl>

Cc: mailto:mailto:r-sig-meta-analysis at r-project.org
Subject: Re: nesting an inner | outer formula

Dear Wolfgang

Thanks for the speedy response, and for seeking clarification and

correcting

my

error.

Rather than try to correct my interpretation of the SAS code,

perhaps it

would

make more sense to tell you what structure I really want.

The data structure is such that I have studies (StudyID) reporting

post-

intervention means for children in an experimental and control group
(TreatmentGroup). The variable Informant tells us who is reporting

on behalf

of

the child. Some studies have child report only (so two rows for such

a

StudyID

corresponding to the post-intervention means for the experimental and

control

group). Studies with parent report or teacher report only are the

same (two

rows). There are also studies where there is data from child and

parent,

child

and teacher, teacher and parent, or even child parent and teacher.

So,

essentially, a StudyID could have two rows, four rows, or six rows,

depending

on

how many Informants there are. Children are in the experimental or

control

group

only, so one would expect Informants to be nested and correlated

within

TreatmentGroup within studies.

Because of the data structure (multiple rows of sample means rather

than

fewer

rows of pairwise comparisions) the degree to which control and

intervention

group means are more or less similar in a given StudyID is capture

in part

(maybe even large part) by ~ 1 | StudyID.

What is missing from this random = list (~ 1 | StudyID, ~ Informant |
TreatmentGroup) is the fact that the inner | outer is saying, for

example,

that

parents in the experimental or control group across studies share

correlated

random effects. When, in fact, one would expect parents, children,

and

teachers

in the experimental or control group to share correlated random

effects

within

a

given study.

Perhaps given the data structure, you would advise something else. Or

perhaps

I

am still being unclear in my description and understanding.

Regards
Ross

On Fri, 14 Feb 2025 at 13:00, Viechtbauer, Wolfgang (NP)
<mailto:mailto:mailto:mailto:

wolfgang.viechtbauer at maastrichtuniversity.nl>

wrote:

Dear Ross,

my proc mixed knowledge is a bit rusty, but unless I am confused,

your proc

mixed statement specifies a random intercept for StudyID and an UN

structure

for

Informant within StudyID allowing for different

variances/covariances for

the

different levels of TreatmentGroup.

random StudyID;
random Informant / subject=StudyID group=TreatmentGroup type=un;
parms  1  1 1 1  1 1 1  1 1 1  1 1 1  1 / hold=14;

I don't think this quite matches up with your description:

I would like for the different levels of Informant to be

correlated within

TreatmentGroup within StudyID, and I would like the different

levels of

TreatmentGroup to be correlated within StudyID too.

For example, there is nothing in your proc mixed statement that

allows for

"TreatmentGroup to be correlated within StudyID". Also, the second

random

statement allows for Informant to be correlated within StudyID, but

*not*

"within TreatmentGroup within StudyID".

So before I attempt to recreate the same structure, it would need to

be

clear

exactly what kind of structure you really want.

Best,
Wolfgang

-----Original Message-----
From: Ross Neville <mailto:mailto:mailto:mailto:

ross.neville at ucd.ie>

Sent: Thursday, February 13, 2025 18:28
To: Viechtbauer, Wolfgang (NP)

<mailto:mailto:mailto:mailto:

wolfgang.viechtbauer at maastrichtuniversity.nl>;

mailto:mailto:mailto:mailto:r-sig-meta-analysis at r-project.org
Subject: nesting an inner | outer formula

Hi Wolfgang

I hope this email finds you well

I was wondering if you could tell me whether the following list of

random

effects can be updated to make the the inner | outer formula

conditional.

random = list (~ 1 | StudyID, ~ Informant | TreatmentGroup)

I would like for the different levels of Informant to be

correlated within

TreatmentGroup within StudyID, and I would like the different

levels of

TreatmentGroup to be correlated within StudyID too.

In SAS Proc Mixed, I've managed to run this model and I want to

replicate

it

in

metafor http://rma.mv.

random StudyID;
random Informant/subject=StudyID, group=TreatmentGroup type=un;
parms  1  1 1 1  1 1 1  1 1 1  1 1 1  1/hold=14;

Any help you can provide to let me know if this is possible in

http://rma.mv

would be much appreciated.

Regards
Ross

--
Dr Ross D. Neville, PhD, ProfCert University Teaching and Learning
Head of Subject - Sport Management
School of Public Health, Physiotherapy and Sport Science
University College Dublin (UCD)
Room G6 - Woodview House
Belfield, Dublin 4
mailto:mailto:mailto:mailto:mailto:mailto:mailto:

ross.neville at ucd.ie

+353 (0) 1 716 3419