[R-meta] Meta-Analysis and Forest Plot for Multiple Treatments and Outcomes

Hi Gerta,

Thank you so much, this is incredibly helpful!

I used the common effects model for the pairwise MAs, so the tau^2 should be the same, which implies that in this case the NMA estimates should be the ones with higher precision. I think I will explain this logic in the paper then, i.e. that the NMA estimate should be more precise, but the significance of the effect is not robust in the case of a pairwise MA that only takes direct evidence into account.

Thank you for pointing out the use of forest(netsplit(net1)), this is great to see the different estimates in the same plot!

Best,
Ruth

Ruth Elisabeth Appel
Ph.D. Candidate in Political Communication and Media Psychology
Stanford University
rappel at stanford.edu<mailto:rappel at stanford.edu>

Dear Ruth,

Here are my answers, To your first question:

 If an effect was significant with the NMA estimates (TE.nma.fixed from netmeta object), but not the direct fixed effect (TE.direct.fixed from netmeta object) in a pairwise comparison, would this suggest that the interpretation should be that there is a significant effect, but it is not robust? In terms of substantive interpretation, would this mean there is insufficient direct evidence? And is it generally true that pairwise comparisons are more conservative?

While pairwise MA uses only information from the direct comparisons, NMA potentially uses information from all comparisons in the network. Therefore NMA usually provides higher precision than MA (depending on the network structure and the estimate of the random effects variance). This means that NMA-based estimates may be ?significant? when pairwise estimates are not. This is just what one would expect, and it is one of the aims of NMA to obtain more precises estimates.

Note that if each pairwise MA uses its own heterogeneity estimate (tau?), it can be the other way round, because the tau? of the NMA may be larger than that of a particular pairwise estimate (for example, if there was only one study for this comparison). If, however, all pairwise comparisons use the same tau? (for example because the common effect model is used), the NMA estimate is at least as precise as the pairwise estimate.

Second question:

 Further, if I want to use forest() to plot TE.direct.fixed instead of TE.nma.fixed of several netmeta objects combined with netbind(), what is the correct way of specifying this (i.e., in which function and with which argument)? (I realize the package was just updated, but I have been using the previous version and couldn't find the right settings yet.)

An elegant way to compare direct with indirect and network estimates in the same forest plot is to use netsplit(). If your NMA object is called net1, the simplest call is

forest(netsplit(net1))

It provides all three types of estimates in one plot, where the direct and indirect estimates are based on the same tau? which is estimated from the NMA. If you use the common effects model, tau? is set to 0 anyway. You may use the argument show (to choose the comparisons to be shown) and the arguments overall, direct and indirect (to choose the estimates to be shown). For example, with

forest(netsplit(net1), show = "with.direct", indirect = FALSE)

the output is restricted to those comparisons for which a direct estimate is available (argument show), each with its direct and network estimate (argument indirect).

Note again that in case of the random effects model the results of the pairwise meta-analyses depend on whether you use different estimates of tau? for each independent pairwise MA or take the common tau? from the NMA.

Purely separate pairwise MAs are obtained using netpairwise(), with corresponding forest function forest.netpairwise(), for example

forest(netpairwise(net1))

Best,
Gerta

Von: Ruth Elisabeth Appel <rappel at stanford.edu<mailto:rappel at stanford.edu>>
Gesendet: Mittwoch, 22. M?rz 2023 11:08
An: Dr. Gerta R?cker <ruecker at imbi.uni-freiburg.de<mailto:ruecker at imbi.uni-freiburg.de>>
Cc: r-sig-meta-analysis at r-project.org<mailto:r-sig-meta-analysis at r-project.org>
Betreff: Re: [R-meta] Meta-Analysis and Forest Plot for Multiple Treatments and Outcomes

Hi Gerta,

Thank you again for your very helpful advice! I have two follow up questions regarding the earlier thread on using netmeta. You mentioned
I would see the NMA as the primary analysis and the pairwise meta-analyses as sensitivity analyses.
and
There is another function in netmeta you may want to use to have all your 8 outcomes in one forest plot: function netbind() which is to bundle the results of several network meta-analyses into one forest plot. Here I would take the NMA estimates, not the pairwise direct comparisons.

If an effect was significant with the NMA estimates (TE.nma.fixed from netmeta object), but not the direct fixed effect (TE.direct.fixed from netmeta object) in a pairwise comparison, would this suggest that the interpretation should be that there is a significant effect, but it is not robust? In terms of substantive interpretation, would this mean there is insufficient direct evidence? And is it generally true that pairwise comparisons are more conservative?

Further, if I want to use forest() to plot TE.direct.fixed instead of TE.nma.fixed of several netmeta objects combined with netbind(), what is the correct way of specifying this (i.e., in which function and with which argument)? (I realize the package was just updated, but I have been using the previous version and couldn't find the right settings yet.)

In my current write up, I focus on the pairwise comparisons for the most part, but I want to be sure to interpret the results of the meta-analysis overall correctly. Your insights would be really appreciated!

Best,
Ruth

Ruth Elisabeth Appel
Ph.D. Candidate in Political Communication and Media Psychology
Stanford University
rappel at stanford.edu<mailto:rappel at stanford.edu>

Hi Ruth,
There is another function in netmeta you may want to use to have all your 8 outcomes in one forest plot: function netbind() which is to bundle the results of several network meta-analyses into one forest plot. Here I would take the NMA estimates, not the pairwise direct comparisons.
I cannot really answer your question related to the correlation between outcomes. This is because I am working in the medical field, also Cochrane, where it is quite unusual to put all outcomes into one model, because we almost never have any knowledge about the within-study correlations - thus the outcomes are usually analyzed separately (they also are on different scales, we rarely use SMD). A paper discussing multivariate meta-analysis is https://onlinelibrary.wiley.com/doi/10.1002/sim.4172 (with discussion).
Best,
Gerta


Am 01.02.2022 um 06:40 schrieb Ruth Appel:
Hi Gerta,

Thank you so much for your super helpful and quick reply!
Yes, that is correct, I used the netmeta package as well (I considered it a complement/extension of meta [part of the yet to be established metaverse ;)], but I should have mentioned all packages I was using). The combination of netpairwise() and forest() is very close to what I was looking for ? it would only be perfect if I could plot all 8 outcomes in the same plot rather than showing 8 separate plots, and I am not sure whether that?s possible since netpairwise seems to configure the different comparisons as subgroups and I couldn?t see another option to specify that I would like to show effects for several outcomes.

That is an important note regarding potential inconsistency issues with Hedges? g, I could use Cohen?s d in that case.

Regarding the correlation between outcomes, how strong could it potentially bias the results in your experience? I think the netpairwise() solution is great, so if the bias introduced is not too big, I might use that approach.

Best,
Ruth

Ruth Elisabeth Appel
Ph.D. Candidate in Media Psychology
Stanford University Department of Communication
rappel at stanford.edu<mailto:rappel at stanford.edu>

Hi Ruth,

First of all, if I understand it correctly, what you are aiming at is a network meta-analysis (NMA). Therefore, meta is not the appropriate R package, which would be netmeta (specialized to NMA) or metafor (more general). It seems you have in fact used netmeta, because you write about a netmeta object, is that true? I would see the NMA as the primary analysis and the pairwise meta-analyses as sensitivity analyses. These can be conducted using function netpairwise() in netmeta; for the fixed effect model, also netsplit() should provide the direct pairwise comparisons. Perhaps @Guido Schwarzer sees a convenient way to visualize the results within the same forest plot using forest.netsplit().

I would expect a problem with Hedges' g for three-arm studies because the results within a trial may become inconsistent (this holds for Hedges' g, but not for Cohen's d, as implemented in netmeta).

Note that netmeta accounts for multiple comparisons between groups with a study, however, it does not handle multivariate outcomes. Thus, if you want to account for correlation between outcomes, you need metafor. With respect to metafor, others are more expert than me.

Best,

Gerta

Am 31.01.2022 um 19:02 schrieb Ruth Appel:

Hi all,

I?m currently conducting my first meta-analysis, an internal meta-analysis to summarize the result of 3 similar studies my colleagues and I conducted.

I looked at the documentation of various meta-analysis packages and tutorials, but I am still not fully sure about the best approach.
The experiments I?m analyzing all have a similar structure (2 treatment groups, 1 control group; 8 different outcomes (measuring different constructs)). The raw data has repeated measures, but we look at outcomes at the group level, so I calculated all necessary summary statistics (mean, sd, n).
My goal is to create a forest plot that shows Hedges? g estimated using an FE model (because the studies were highly similar) for (1) all 3 studies individually and (2) across all studies. Ideally, the final result would be a single forest plot with individual study estimates and across study estimate grouped by outcome.

I managed to create such a plot with the meta package for the 2 treatment groups separately, but I realized that my SEs could be biased in this case because I?m not accounting for the correlations in the variance resulting from the comparison of two treatment groups to the same control group. Similarly, I found a workaround to show all outcomes in 1 forest plot by using subgroups for the different outcomes, but I do not take into consideration that outcomes might be correlated within studies. I also didn?t find a way to show the individual study results in addition to the overall network results in a forest plot of a netmeta object.

I then tried to calculate the correct values using metafor and following the tutorial at https://www.metafor-project.org/doku.php/analyses:gleser2009#multiple-treatment_studies, but it seems like the individual studies are not correctly identified in the output (the ids are all unique instead of matching the study variable I had created).

My questions are: (1) Did I overlook guidance somewhere on how to exactly specify a model like the one above using the metafor, meta (or another R) package, and generate a forest plot for it?
(2) If this is not easily possible, do you think the bias introduced should be sufficiently small such that acknowledging it and presenting separate meta-analyses for each treatment, and a network meta analysis with the overall effects of each treatment (separately for each outcome) in the appendix, is acceptable? (I had very similar estimates across all the approaches described above.)

Best regards, and thank you very much for your guidance,
Ruth

Ruth Elisabeth Appel
Ph.D. Candidate in Media Psychology
Stanford University Department of Communication
rappel at stanford.edu<mailto:rappel at stanford.edu>

_______________________________________________
R-sig-meta-analysis mailing list
R-sig-meta-analysis at r-project.org<mailto:R-sig-meta-analysis at r-project.org>
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

--

Dr. rer. nat. Gerta R?cker, Dipl.-Math.

Guest Scientist
Institute of Medical Biometry and Statistics,
Faculty of Medicine and Medical Center - University of Freiburg

Zinkmattenstr. 6a, D-79108 Freiburg, Germany

Mail:     ruecker at imbi.uni-freiburg.de<mailto:ruecker at imbi.uni-freiburg.de>
Homepage: https://www.uniklinik-freiburg.de/imbi-en/employees.html?imbiuser=ruecker


--



Dr. rer. nat. Gerta R?cker, Dipl.-Math.



Guest Scientist

Institute of Medical Biometry and Statistics,

Faculty of Medicine and Medical Center - University of Freiburg



Zinkmattenstr. 6a, D-79108 Freiburg, Germany



Mail:     ruecker at imbi.uni-freiburg.de<mailto:ruecker at imbi.uni-freiburg.de>

Homepage: https://www.uniklinik-freiburg.de/imbi-en/employees.html?imbiuser=ruecker

Hi all,

Just two notes:

1) If you used the common-effects model for the pairwise MAs, then tau^2 is 0 for these models by definition (so it sounds a bit odd to say that "the tau^2 should be the same"). But I assume you did not use a common-effects model for the NMA, so now we are comparing apples and oranges.

2) I don't think it is quite right to say that the significance of the effect in the pairwise MA is not "robust". It is perfectly fine, being based on the direct evidence. In fact, one could argue that the NMA is potentially less robust, since it makes assumptions about the consistency of the evidence. A better way to state this is to say that, if the assumptions underling the NMA are correct (and taking into consideration what Gerta wrote about tau^2), then the NMA can provide stronger evidence, because it can also take the indirect evidence into consideration.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of Ruth Elisabeth Appel via R-sig-meta-analysis
Sent: Thursday, 23 March, 2023 6:52
To: Dr. Gerta R?cker
Cc: Ruth Elisabeth Appel; r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] Meta-Analysis and Forest Plot for Multiple Treatments and
Outcomes

Hi Gerta,

Thank you so much, this is incredibly helpful!

I used the common effects model for the pairwise MAs, so the tau^2 should be the
same, which implies that in this case the NMA estimates should be the ones with
higher precision. I think I will explain this logic in the paper then, i.e. that
the NMA estimate should be more precise, but the significance of the effect is
not robust in the case of a pairwise MA that only takes direct evidence into
account.

Thank you for pointing out the use of forest(netsplit(net1)), this is great to
see the different estimates in the same plot!

Best,
Ruth

Ruth Elisabeth Appel
Ph.D. Candidate in Political Communication and Media Psychology
Stanford University
rappel at stanford.edu<mailto:rappel at stanford.edu>

On Mar 22, 2023, at 3:59 AM, Dr. Gerta R?cker <gerta.ruecker at uniklinik-
freiburg.de<mailto:gerta.ruecker at uniklinik-freiburg.de>> wrote:

Dear Ruth,

Here are my answers, To your first question:

 If an effect was significant with the NMA estimates (TE.nma.fixed from netmeta

object), but not the direct fixed effect (TE.direct.fixed from netmeta object) in
a pairwise comparison, would this suggest that the interpretation should be that
there is a significant effect, but it is not robust? In terms of substantive
interpretation, would this mean there is insufficient direct evidence? And is it
generally true that pairwise comparisons are more conservative?

While pairwise MA uses only information from the direct comparisons, NMA
potentially uses information from all comparisons in the network. Therefore NMA
usually provides higher precision than MA (depending on the network structure and
the estimate of the random effects variance). This means that NMA-based estimates
may be ?significant? when pairwise estimates are not. This is just what one would
expect, and it is one of the aims of NMA to obtain more precises estimates.

Note that if each pairwise MA uses its own heterogeneity estimate (tau?), it can
be the other way round, because the tau? of the NMA may be larger than that of a
particular pairwise estimate (for example, if there was only one study for this
comparison). If, however, all pairwise comparisons use the same tau? (for example
because the common effect model is used), the NMA estimate is at least as precise
as the pairwise estimate.

Second question:

 Further, if I want to use forest() to plot TE.direct.fixed instead of

TE.nma.fixed of several netmeta objects combined with netbind(), what is the
correct way of specifying this (i.e., in which function and with which argument)?
(I realize the package was just updated, but I have been using the previous
version and couldn't find the right settings yet.)

An elegant way to compare direct with indirect and network estimates in the same
forest plot is to use netsplit(). If your NMA object is called net1, the simplest
call is

forest(netsplit(net1))

It provides all three types of estimates in one plot, where the direct and
indirect estimates are based on the same tau? which is estimated from the NMA. If
you use the common effects model, tau? is set to 0 anyway. You may use the
argument show (to choose the comparisons to be shown) and the arguments overall,
direct and indirect (to choose the estimates to be shown). For example, with

forest(netsplit(net1), show = "with.direct", indirect = FALSE)

the output is restricted to those comparisons for which a direct estimate is
available (argument show), each with its direct and network estimate (argument
indirect).

Note again that in case of the random effects model the results of the pairwise
meta-analyses depend on whether you use different estimates of tau? for each
independent pairwise MA or take the common tau? from the NMA.

Purely separate pairwise MAs are obtained using netpairwise(), with corresponding
forest function forest.netpairwise(), for example

forest(netpairwise(net1))

Best,
Gerta

Hi Wolfgang,

Thank you so much for these additional notes!

On 1), yes, I used the common effects model for the pairwise MAs, so the tau^2 are all 0 and therefore the same.
I think I also used the common effects model in the NMA object, at least I specified "fixed = TRUE". I'm providing an excerpt from my code below, do you think this is an apples-to-apples comparison then?
My apologies if I misspecified or misinterpreted something, I went through some tutorials, but am still new to meta-analyses.

# NMA
net1 <- netmeta(...,
                all.treatments = TRUE,
                reference.group = "Control",
                fixed = TRUE,
                random = FALSE,
                sm = "SMD",
                method.sm = "Cohen",
                studies = c("Study 1", "Study 2", "Study 3"),
                n.arms = 3,
                ...)
# pairwise MA
netpair1 <- netpairwise(net1, sep.trts = " vs. ")

# combine NMA objects to show all outcomes in one plot
nets <- netbind(net1, net2, ..., fixed = TRUE)


As for plots, I have 1 plot with NMA results for all outcomes, and 1 plot for pairwise MA for each outcome:

# NMA for all outcomes
forest(nets,
       text.fixed = "Common Fixed Effects Model",
       ...)

# pairwise MA for one outcome
netpair1  %>%
  forest(text.fixed = "Common Fixed Effects Model", ...)

On 2), yes, I think what I meant is that I observe a significant effect with the NMA, but not the pairwise MA, so I meant to say this effect is not robust to different MA specifications. The wording you suggested is great, so maybe I could say the effect is significant when using NMA, which provides stronger evidence ? assuming the assumptions for NMA are met ? because it can take indirect evidence into account as well, but the effect is not significant in a pairwise MA that only relies on direct evidence.

Best,
Ruth

Ruth Elisabeth Appel
Ph.D. Candidate in Political Communication and Media Psychology
Stanford University
rappel at stanford.edu<mailto:rappel at stanford.edu>

Hi all,

Just two notes:

1) If you used the common-effects model for the pairwise MAs, then tau^2 is 0 for these models by definition (so it sounds a bit odd to say that "the tau^2 should be the same"). But I assume you did not use a common-effects model for the NMA, so now we are comparing apples and oranges.

2) I don't think it is quite right to say that the significance of the effect in the pairwise MA is not "robust". It is perfectly fine, being based on the direct evidence. In fact, one could argue that the NMA is potentially less robust, since it makes assumptions about the consistency of the evidence. A better way to state this is to say that, if the assumptions underling the NMA are correct (and taking into consideration what Gerta wrote about tau^2), then the NMA can provide stronger evidence, because it can also take the indirect evidence into consideration.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of Ruth Elisabeth Appel via R-sig-meta-analysis
Sent: Thursday, 23 March, 2023 6:52
To: Dr. Gerta R?cker
Cc: Ruth Elisabeth Appel; r-sig-meta-analysis at r-project.org<mailto:r-sig-meta-analysis at r-project.org>
Subject: Re: [R-meta] Meta-Analysis and Forest Plot for Multiple Treatments and
Outcomes

Hi Gerta,

Thank you so much, this is incredibly helpful!

I used the common effects model for the pairwise MAs, so the tau^2 should be the
same, which implies that in this case the NMA estimates should be the ones with
higher precision. I think I will explain this logic in the paper then, i.e. that
the NMA estimate should be more precise, but the significance of the effect is
not robust in the case of a pairwise MA that only takes direct evidence into
account.

Thank you for pointing out the use of forest(netsplit(net1)), this is great to
see the different estimates in the same plot!

Best,
Ruth

Ruth Elisabeth Appel
Ph.D. Candidate in Political Communication and Media Psychology
Stanford University
rappel at stanford.edu<mailto:rappel at stanford.edu><mailto:rappel at stanford.edu>

On Mar 22, 2023, at 3:59 AM, Dr. Gerta R?cker <gerta.ruecker at uniklinik-

Dear Ruth,

Here are my answers, To your first question:

If an effect was significant with the NMA estimates (TE.nma.fixed from netmeta
object), but not the direct fixed effect (TE.direct.fixed from netmeta object) in
a pairwise comparison, would this suggest that the interpretation should be that
there is a significant effect, but it is not robust? In terms of substantive
interpretation, would this mean there is insufficient direct evidence? And is it
generally true that pairwise comparisons are more conservative?

While pairwise MA uses only information from the direct comparisons, NMA
potentially uses information from all comparisons in the network. Therefore NMA
usually provides higher precision than MA (depending on the network structure and
the estimate of the random effects variance). This means that NMA-based estimates
may be ?significant? when pairwise estimates are not. This is just what one would
expect, and it is one of the aims of NMA to obtain more precises estimates.

Note that if each pairwise MA uses its own heterogeneity estimate (tau?), it can
be the other way round, because the tau? of the NMA may be larger than that of a
particular pairwise estimate (for example, if there was only one study for this
comparison). If, however, all pairwise comparisons use the same tau? (for example
because the common effect model is used), the NMA estimate is at least as precise
as the pairwise estimate.

Second question:

Further, if I want to use forest() to plot TE.direct.fixed instead of
TE.nma.fixed of several netmeta objects combined with netbind(), what is the
correct way of specifying this (i.e., in which function and with which argument)?
(I realize the package was just updated, but I have been using the previous
version and couldn't find the right settings yet.)

An elegant way to compare direct with indirect and network estimates in the same
forest plot is to use netsplit(). If your NMA object is called net1, the simplest
call is

forest(netsplit(net1))

It provides all three types of estimates in one plot, where the direct and
indirect estimates are based on the same tau? which is estimated from the NMA. If
you use the common effects model, tau? is set to 0 anyway. You may use the
argument show (to choose the comparisons to be shown) and the arguments overall,
direct and indirect (to choose the estimates to be shown). For example, with

forest(netsplit(net1), show = "with.direct", indirect = FALSE)

the output is restricted to those comparisons for which a direct estimate is
available (argument show), each with its direct and network estimate (argument
indirect).

Note again that in case of the random effects model the results of the pairwise
meta-analyses depend on whether you use different estimates of tau? for each
independent pairwise MA or take the common tau? from the NMA.

Purely separate pairwise MAs are obtained using netpairwise(), with corresponding
forest function forest.netpairwise(), for example

forest(netpairwise(net1))

Best,
Gerta

Dear Ruth

I think in general when you perform two analyses and they give different 
but similar results you should (a) present both, (b) present a plausible 
explanation of why. Wolfgang has given you a lead there. What ever you 
do the reader will make up their own mind which to believe but at least 
you have put all the evidence in front of them.

Michael

Hi Wolfgang,

Thank you so much for these additional notes!

On 1), yes, I used the common effects model for the pairwise MAs, so the tau^2 are all 0 and therefore the same.
I think I also used the common effects model in the NMA object, at least I specified "fixed = TRUE". I'm providing an excerpt from my code below, do you think this is an apples-to-apples comparison then?
My apologies if I misspecified or misinterpreted something, I went through some tutorials, but am still new to meta-analyses.

# NMA
net1 <- netmeta(...,
                 all.treatments = TRUE,
                 reference.group = "Control",
                 fixed = TRUE,
                 random = FALSE,
                 sm = "SMD",
                 method.sm = "Cohen",
                 studies = c("Study 1", "Study 2", "Study 3"),
                 n.arms = 3,
                 ...)
# pairwise MA
netpair1 <- netpairwise(net1, sep.trts = " vs. ")

# combine NMA objects to show all outcomes in one plot
nets <- netbind(net1, net2, ..., fixed = TRUE)


As for plots, I have 1 plot with NMA results for all outcomes, and 1 plot for pairwise MA for each outcome:

# NMA for all outcomes
forest(nets,
        text.fixed = "Common Fixed Effects Model",
        ...)

# pairwise MA for one outcome
netpair1  %>%
   forest(text.fixed = "Common Fixed Effects Model", ...)

On 2), yes, I think what I meant is that I observe a significant effect with the NMA, but not the pairwise MA, so I meant to say this effect is not robust to different MA specifications. The wording you suggested is great, so maybe I could say the effect is significant when using NMA, which provides stronger evidence ? assuming the assumptions for NMA are met ? because it can take indirect evidence into account as well, but the effect is not significant in a pairwise MA that only relies on direct evidence.

Best,
Ruth

Ruth Elisabeth Appel
Ph.D. Candidate in Political Communication and Media Psychology
Stanford University
rappel at stanford.edu<mailto:rappel at stanford.edu>

On Mar 23, 2023, at 1:07 AM, Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer at maastrichtuniversity.nl<mailto:wolfgang.viechtbauer at maastrichtuniversity.nl>> wrote:

Hi all,

Just two notes:

1) If you used the common-effects model for the pairwise MAs, then tau^2 is 0 for these models by definition (so it sounds a bit odd to say that "the tau^2 should be the same"). But I assume you did not use a common-effects model for the NMA, so now we are comparing apples and oranges.

2) I don't think it is quite right to say that the significance of the effect in the pairwise MA is not "robust". It is perfectly fine, being based on the direct evidence. In fact, one could argue that the NMA is potentially less robust, since it makes assumptions about the consistency of the evidence. A better way to state this is to say that, if the assumptions underling the NMA are correct (and taking into consideration what Gerta wrote about tau^2), then the NMA can provide stronger evidence, because it can also take the indirect evidence into consideration.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of Ruth Elisabeth Appel via R-sig-meta-analysis
Sent: Thursday, 23 March, 2023 6:52
To: Dr. Gerta R?cker
Cc: Ruth Elisabeth Appel; r-sig-meta-analysis at r-project.org<mailto:r-sig-meta-analysis at r-project.org>
Subject: Re: [R-meta] Meta-Analysis and Forest Plot for Multiple Treatments and
Outcomes

Hi Gerta,

Thank you so much, this is incredibly helpful!

I used the common effects model for the pairwise MAs, so the tau^2 should be the
same, which implies that in this case the NMA estimates should be the ones with
higher precision. I think I will explain this logic in the paper then, i.e. that
the NMA estimate should be more precise, but the significance of the effect is
not robust in the case of a pairwise MA that only takes direct evidence into
account.

Thank you for pointing out the use of forest(netsplit(net1)), this is great to
see the different estimates in the same plot!

Best,
Ruth

Ruth Elisabeth Appel
Ph.D. Candidate in Political Communication and Media Psychology
Stanford University
rappel at stanford.edu<mailto:rappel at stanford.edu><mailto:rappel at stanford.edu>

On Mar 22, 2023, at 3:59 AM, Dr. Gerta R?cker <gerta.ruecker at uniklinik-
freiburg.de<http://freiburg.de><mailto:gerta.ruecker at uniklinik-freiburg.de>> wrote:

Dear Ruth,

Here are my answers, To your first question:

If an effect was significant with the NMA estimates (TE.nma.fixed from netmeta
object), but not the direct fixed effect (TE.direct.fixed from netmeta object) in
a pairwise comparison, would this suggest that the interpretation should be that
there is a significant effect, but it is not robust? In terms of substantive
interpretation, would this mean there is insufficient direct evidence? And is it
generally true that pairwise comparisons are more conservative?

While pairwise MA uses only information from the direct comparisons, NMA
potentially uses information from all comparisons in the network. Therefore NMA
usually provides higher precision than MA (depending on the network structure and
the estimate of the random effects variance). This means that NMA-based estimates
may be ?significant? when pairwise estimates are not. This is just what one would
expect, and it is one of the aims of NMA to obtain more precises estimates.

Note that if each pairwise MA uses its own heterogeneity estimate (tau?), it can
be the other way round, because the tau? of the NMA may be larger than that of a
particular pairwise estimate (for example, if there was only one study for this
comparison). If, however, all pairwise comparisons use the same tau? (for example
because the common effect model is used), the NMA estimate is at least as precise
as the pairwise estimate.

Second question:

Further, if I want to use forest() to plot TE.direct.fixed instead of
TE.nma.fixed of several netmeta objects combined with netbind(), what is the
correct way of specifying this (i.e., in which function and with which argument)?
(I realize the package was just updated, but I have been using the previous
version and couldn't find the right settings yet.)

An elegant way to compare direct with indirect and network estimates in the same
forest plot is to use netsplit(). If your NMA object is called net1, the simplest
call is

forest(netsplit(net1))

It provides all three types of estimates in one plot, where the direct and
indirect estimates are based on the same tau? which is estimated from the NMA. If
you use the common effects model, tau? is set to 0 anyway. You may use the
argument show (to choose the comparisons to be shown) and the arguments overall,
direct and indirect (to choose the estimates to be shown). For example, with

forest(netsplit(net1), show = "with.direct", indirect = FALSE)

the output is restricted to those comparisons for which a direct estimate is
available (argument show), each with its direct and network estimate (argument
indirect).

Note again that in case of the random effects model the results of the pairwise
meta-analyses depend on whether you use different estimates of tau? for each
independent pairwise MA or take the common tau? from the NMA.

Purely separate pairwise MAs are obtained using netpairwise(), with corresponding
forest function forest.netpairwise(), for example

forest(netpairwise(net1))

Best,
Gerta


	[[alternative HTML version deleted]]

_______________________________________________
R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org
To manage your subscription to this mailing list, go to:
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

Michael
http://www.dewey.myzen.co.uk/home.html

Hi Michael,

Thanks a lot for your thoughts! Yes, I agree that presenting all the evidence and plausible explanations makes sense.
As it turned out, the results of the network meta-analysis and pairwise MA actually were not in conflict with each other ? the lower bound of a CI was very small, but with more precision/digits, it was still above zero, such that the effect remained significant in both the network and the pairwise MA.

Best,
Ruth

Ruth Elisabeth Appel
Ph.D. Candidate in Political Communication and Media Psychology
Stanford University
rappel at stanford.edu<mailto:rappel at stanford.edu>

Dear Ruth

I think in general when you perform two analyses and they give different but similar results you should (a) present both, (b) present a plausible explanation of why. Wolfgang has given you a lead there. What ever you do the reader will make up their own mind which to believe but at least you have put all the evidence in front of them.

Michael

Hi Wolfgang,
Thank you so much for these additional notes!
On 1), yes, I used the common effects model for the pairwise MAs, so the tau^2 are all 0 and therefore the same.
I think I also used the common effects model in the NMA object, at least I specified "fixed = TRUE". I'm providing an excerpt from my code below, do you think this is an apples-to-apples comparison then?
My apologies if I misspecified or misinterpreted something, I went through some tutorials, but am still new to meta-analyses.
# NMA
net1 <- netmeta(...,
                all.treatments = TRUE,
                reference.group = "Control",
                fixed = TRUE,
                random = FALSE,
                sm = "SMD",
                method.sm = "Cohen",
                studies = c("Study 1", "Study 2", "Study 3"),
                n.arms = 3,
                ...)
# pairwise MA
netpair1 <- netpairwise(net1, sep.trts = " vs. ")
# combine NMA objects to show all outcomes in one plot
nets <- netbind(net1, net2, ..., fixed = TRUE)
As for plots, I have 1 plot with NMA results for all outcomes, and 1 plot for pairwise MA for each outcome:
# NMA for all outcomes
forest(nets,
       text.fixed = "Common Fixed Effects Model",
       ...)
# pairwise MA for one outcome
netpair1  %>%
  forest(text.fixed = "Common Fixed Effects Model", ...)
On 2), yes, I think what I meant is that I observe a significant effect with the NMA, but not the pairwise MA, so I meant to say this effect is not robust to different MA specifications. The wording you suggested is great, so maybe I could say the effect is significant when using NMA, which provides stronger evidence ? assuming the assumptions for NMA are met ? because it can take indirect evidence into account as well, but the effect is not significant in a pairwise MA that only relies on direct evidence.
Best,
Ruth
Ruth Elisabeth Appel
Ph.D. Candidate in Political Communication and Media Psychology
Stanford University
rappel at stanford.edu<mailto:rappel at stanford.edu><mailto:rappel at stanford.edu>

Hi all,
Just two notes:
1) If you used the common-effects model for the pairwise MAs, then tau^2 is 0 for these models by definition (so it sounds a bit odd to say that "the tau^2 should be the same"). But I assume you did not use a common-effects model for the NMA, so now we are comparing apples and oranges.
2) I don't think it is quite right to say that the significance of the effect in the pairwise MA is not "robust". It is perfectly fine, being based on the direct evidence. In fact, one could argue that the NMA is potentially less robust, since it makes assumptions about the consistency of the evidence. A better way to state this is to say that, if the assumptions underling the NMA are correct (and taking into consideration what Gerta wrote about tau^2), then the NMA can provide stronger evidence, because it can also take the indirect evidence into consideration.
Best,
Wolfgang
-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of Ruth Elisabeth Appel via R-sig-meta-analysis
Sent: Thursday, 23 March, 2023 6:52
To: Dr. Gerta R?cker
Cc: Ruth Elisabeth Appel; r-sig-meta-analysis at r-project.org<mailto:r-sig-meta-analysis at r-project.org><mailto:r-sig-meta-analysis at r-project.org>
Subject: Re: [R-meta] Meta-Analysis and Forest Plot for Multiple Treatments and
Outcomes
Hi Gerta,
Thank you so much, this is incredibly helpful!
I used the common effects model for the pairwise MAs, so the tau^2 should be the
same, which implies that in this case the NMA estimates should be the ones with
higher precision. I think I will explain this logic in the paper then, i.e. that
the NMA estimate should be more precise, but the significance of the effect is
not robust in the case of a pairwise MA that only takes direct evidence into
account.
Thank you for pointing out the use of forest(netsplit(net1)), this is great to
see the different estimates in the same plot!
Best,
Ruth
Ruth Elisabeth Appel
Ph.D. Candidate in Political Communication and Media Psychology
Stanford University
rappel at stanford.edu<mailto:rappel at stanford.edu><mailto:rappel at stanford.edu><mailto:rappel at stanford.edu>
On Mar 22, 2023, at 3:59 AM, Dr. Gerta R?cker <gerta.ruecker at uniklinik-

Dear Ruth,
Here are my answers, To your first question:
If an effect was significant with the NMA estimates (TE.nma.fixed from netmeta
object), but not the direct fixed effect (TE.direct.fixed from netmeta object) in
a pairwise comparison, would this suggest that the interpretation should be that
there is a significant effect, but it is not robust? In terms of substantive
interpretation, would this mean there is insufficient direct evidence? And is it
generally true that pairwise comparisons are more conservative?
While pairwise MA uses only information from the direct comparisons, NMA
potentially uses information from all comparisons in the network. Therefore NMA
usually provides higher precision than MA (depending on the network structure and
the estimate of the random effects variance). This means that NMA-based estimates
may be ?significant? when pairwise estimates are not. This is just what one would
expect, and it is one of the aims of NMA to obtain more precises estimates.
Note that if each pairwise MA uses its own heterogeneity estimate (tau?), it can
be the other way round, because the tau? of the NMA may be larger than that of a
particular pairwise estimate (for example, if there was only one study for this
comparison). If, however, all pairwise comparisons use the same tau? (for example
because the common effect model is used), the NMA estimate is at least as precise
as the pairwise estimate.
Second question:
Further, if I want to use forest() to plot TE.direct.fixed instead of
TE.nma.fixed of several netmeta objects combined with netbind(), what is the
correct way of specifying this (i.e., in which function and with which argument)?
(I realize the package was just updated, but I have been using the previous
version and couldn't find the right settings yet.)
An elegant way to compare direct with indirect and network estimates in the same
forest plot is to use netsplit(). If your NMA object is called net1, the simplest
call is
forest(netsplit(net1))
It provides all three types of estimates in one plot, where the direct and
indirect estimates are based on the same tau? which is estimated from the NMA. If
you use the common effects model, tau? is set to 0 anyway. You may use the
argument show (to choose the comparisons to be shown) and the arguments overall,
direct and indirect (to choose the estimates to be shown). For example, with
forest(netsplit(net1), show = "with.direct", indirect = FALSE)
the output is restricted to those comparisons for which a direct estimate is
available (argument show), each with its direct and network estimate (argument
indirect).
Note again that in case of the random effects model the results of the pairwise
meta-analyses depend on whether you use different estimates of tau? for each
independent pairwise MA or take the common tau? from the NMA.
Purely separate pairwise MAs are obtained using netpairwise(), with corresponding
forest function forest.netpairwise(), for example
forest(netpairwise(net1))
Best,
Gerta
_______________________________________________
R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org<mailto:R-sig-meta-analysis at r-project.org>
To manage your subscription to this mailing list, go to:
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

--
Michael
http://www.dewey.myzen.co.uk/home.html