[R-meta] Studies with more than one control group

Oh sorry, yes they are simply labels (I thought all ID variables are
labels). In our case, none of the control groups are waitlist groups.
Basically, due to the criticisms in the literature regarding the definition
of the control groups, some newer studies have included two "active"
control groups (one doing X, the other doing Y) and then they benchmark
their treated groups against both these control groups to show that doing X
vs. Y has no bearing on the final result.

I hope my clarification helps,
Thanks again,
Jack

On Wed, Jul 21, 2021 at 10:43 AM James Pustejovsky <jepusto at gmail.com>
wrote:

I am still wondering whether control 1 versus control 2 has a specific
meaning. For example, perhaps controlID = 1 means that the study used a
wait-list control group, whereas controlID = 2 means that the study used an
attentional control group. Is this the case? Or is controlID just an
arbitrary set of labels, where you could have replaced the numerical values
as follows without losing any information?

studyID  yi  controlID
1        .1      A
1        .2      B
1        .3      A
1        .4      B
2        .5      C
2        .6      D
3        .7      E

On Wed, Jul 21, 2021 at 10:29 AM Jack Solomon <kj.jsolomon at gmail.com>
wrote:

Dear James,

Thank you for your reply. "controlID" distinguishes between effect sizes
(SMDs in this case) that have been obtained by comparing the treated groups
to control 1 vs. control 2 (see below).

I was wondering if adding such an ID variable (just like schoolID) and
the random effect associated with it would also mean that we are
generalizing beyond the levels of controlID, which then, would mean that we
anticipate that each study 'could' have any number of control groups and
not just limited to a max of 2?

Thanks again, Jack

studyID  yi  controlID
1        .1      1
1        .2      2
1        .3      1
1        .4      2
2        .5      1
2        .6      2
3        .7      1

On Wed, Jul 21, 2021 at 10:13 AM James Pustejovsky <jepusto at gmail.com>
wrote:

Hi Jack,

To make sure I follow the structure of your data, let me ask: Do
controlID = 1 or controlID = 2 correspond to specific *types* of control
groups that have the same meaning across all of your studies? Or is this
just an arbitrary ID variable?

In my earlier response, I was assuming that controlID in your data is
just an ID variable. Using random effects specified as
~ | studyID/controlID
means that you're including random *intercept* terms for each unique
control group nested within studyID. It has nothing to do with the number
of control groups.

James



On Mon, Jul 19, 2021 at 10:56 PM Jack Solomon <kj.jsolomon at gmail.com>
wrote:

Dear James,

I'm coming back to this after a while (preparing the data). A quick
follow-up. So, you mentioned that if I have several studies that have used
more than 1 control group (in my data up to 2), I can possibly add a
random-effect (controlID) to capture any heterogeneity in the effect sizes
across control groups nested within studies.

My question is that adding a controlID random-effect (a binary
indicator: 1 or 2) would also mean that we intend to generalize beyond the
possible number of control groups that a study can employ (for my data
beyond 2 control groups)?

Thank you,
Jack

On Thu, Jun 24, 2021 at 4:52 PM Jack Solomon <kj.jsolomon at gmail.com>
wrote:

Thank you very much for the clarification. That makes perfect sense.

Jack

On Thu, Jun 24, 2021 at 4:44 PM James Pustejovsky <jepusto at gmail.com>
wrote:

The random effect for controlID is capturing any heterogeneity in
the effect sizes across control groups nested within studies, *above and
beyond heterogeneity explained by covariates.* Thus, if you include a
covariate to distinguish among types of control groups, and the differences
between types of control groups are consistent across studies, then the
covariate might explain all (or nearly all) of the variation at that level,
which would obviate the purpose of including the random effect at that
level.

On Thu, Jun 24, 2021 at 9:56 AM Jack Solomon <kj.jsolomon at gmail.com>
wrote:

Thank you James. On my question 3, I was implicitly referring to my
previous question (a previous post titled: Studies with independent
samples) regarding the fact that if I decide to drop 'sampleID', then I
need to change the coding of the 'studyID' column (i.e., then, each sample
should be coded as an independent study). So, in my question 3, I really
was asking that in the case of 'controlID', removing it doesn't require
changing the coding of any other columns in my data.

Regarding adding 'controlID' as a random effect, you said: "... an
additional random effect for controlID will depend on how many studies
include multiple control groups and whether the model includes a covariate
to distinguish among types of control groups (e.g., business-as-usual
versus waitlist versus active control group)."

I understand that the number of studies with multiple control
groups is important in whether to add a random effect or not. But why
having "a covariate to distinguish among types of control groups" is
important in whether to add a random effect or not?

Thanks, Jack

On Thu, Jun 24, 2021 at 9:17 AM James Pustejovsky <
jepusto at gmail.com> wrote:

Hi Jack,

Responses inline below.

James

I have come across a couple of primary studies in my
meta-analytic pool
that have used two comparison/control groups (as the definition of
'control' has been debated in the literature I'm meta-analyzing).

(1) Given that, should I create an additional column ('control')
to
distinguish between effect sizes (SMDs in this case) that have
been
obtained by comparing the treated groups to control 1 vs. control
2 (see
below)?

Yes. Along the same lines as my response to your earlier question,
it seems prudent to include ID variables like this in order to describe the
structure of the included studies.

(2) If yes, then, does the addition of a 'control' column call
for the
addition of a random effect for 'control' of the form:  "~ |
studyID/controlID" (to be empirically tested)?

I expect you will find differences of opinion here. Pragmatically,
the feasibility of estimating a model with an additional random effect for
controlID will depend on how many studies include multiple control groups
and whether the model includes a covariate to distinguish among types of
control groups (e.g., business-as-usual versus waitlist versus active
control group).

At a conceptual level, omitting random effects for controlID leads
to essentially the same results as averaging the ES across both control
groups. If averaging like this makes conceptual sense, then omitting the
random effects might be reasonable.

(3) If I later decide to drop controlID from my dataset, I think
I can
still keep all effect sizes from both control groups intact
without any
changes to my coding scheme, right?

I don't understand what you're concern is here. Why not just keep
controlID in your dataset as a descriptor, even if it doesn't get used in
the model?