[R-meta] What asymmetry test to use? - R-SIG-meta-analysis

Thu, May 2, 2024 9:36 AM #

Hello everyone. I am performing an analysis, and when doing the asymmetry tests I see totally different results.
How do I know which one I should use?
This is a meta-analysis of SMD, with a random model given the high herogenicity.
The classic Egger test gives a p of 0.0129 and the Egger test with mixed effects model p 0.1015.
 I post the results.

Regression Test for Funnel Plot Asymmetry

model:     weighted regression with multiplicative dispersion
predictor: standard error

test for funnel plot asymmetry: t = 2.7964, df = 16, p = 0.0129

Regression Test for Funnel Plot Asymmetry

model:     mixed-effects meta-regression model
predictor: standard error

test for funnel plot asymmetry: z = 1.6378, p = 0.1015

best regards



Lorenzo Mart?n Lobo MTSAC, FACC, FESC
Especialista Jerarquizado en Cardiolog?a
Jefe de Dpto Enf. Cardiovasculares y Cardiometabolismo Hospital Militar Campo de Mayo.
Jefe de Cardiolog?a Hospital Militar Campo de Mayo
Ex Jefe de Unidad Coronaria Hospital Militar Campo de Mayo
Miembro Titular de la Sociedad Argentina de Cardiolog?a
Fellow American College of Cardiology
Fellow European Society of Cardiology
Ex Miembro del Area de Investigaci?n de la SAC
Ex Director del Consejo de Aterosclerosis y Trombosis de la SAC
Miembro Asesor del Consejo de Aterosclerosis y Trombosis de la SAC
Ex Director del Consejo de Epidemiolog?a y Prevenci?n Cardiovascular de la SAC
Miembro Asesor del Consejo de Epidemiolog?a y Prevenci?n Cardiovascular de la SAC
Experto en Lipidos de la Sociedad Argentina de Lipidos.
Miembro de la Sociedad Argentina de Lipidos.
Instructor de ACLS de la American Heart Association

De: R-sig-meta-analysis <r-sig-meta-analysis-bounces at r-project.org> en nombre de r-sig-meta-analysis-request at r-project.org <r-sig-meta-analysis-request at r-project.org>

Enviado: mi?rcoles, 1 de mayo de 2024 07:00
Para: r-sig-meta-analysis at r-project.org <r-sig-meta-analysis at r-project.org>
Asunto: R-sig-meta-analysis Digest, Vol 84, Issue 1

Send R-sig-meta-analysis mailing list submissions to
        r-sig-meta-analysis at r-project.org

To subscribe or unsubscribe via the World Wide Web, visit
        https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C02%7C%7C65b44ea4234a4e34eb6d08dc69c5a8eb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638501544879967578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=ulxX%2BrcpO0Teznp0IZzI1y%2FbHFdEUKlo8wdpCbwgpok%3D&reserved=0<https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>
or, via email, send a message with subject or body 'help' to
        r-sig-meta-analysis-request at r-project.org

You can reach the person managing the list at
        r-sig-meta-analysis-owner at r-project.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of R-sig-meta-analysis digest..."


Today's Topics:

   1. Conceptual Question: test of moderators CHE RVE vs 3LMA
      (Noah Schroeder)
   2. Re:  Conceptual Question: test of moderators CHE RVE vs 3LMA
      (James Pustejovsky)
   3. Re:  Conceptual Question: test of moderators CHE RVE vs 3LMA
      (Noah Schroeder)

----------------------------------------------------------------------

Message: 1
Date: Tue, 30 Apr 2024 14:07:55 -0700
From: Noah Schroeder <4noah.schroeder at gmail.com>
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] Conceptual Question: test of moderators CHE RVE vs
        3LMA
Message-ID:
        <CAEfFJJb9-UmWwVkJsRvkWHvsBvnSBtVvLOrzmRFdamAw-LGakQ at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi Everyone,
 I am learning about CHE RVE meta-analysis and have been experimenting
with it. Something I have noticed is that occasionally, my test of
moderators pvalue will be NA. However, if I run the same code through
a typical three level structure without CHE RVE, I get a pvalue for
the test of moderators. I have the following conceptual questions I am
hoping someone may be able to help with:
1) what might cause this NA pvalue to occur in the test of moderators
when using CHE RVE?
2) Should this NA be interpreted? If so, should it be interpreted as a
non-significant result on the test of moderators? I noticed that when
this occurs i generally have notable overlap in the CIs.
3) In cases where i do get a pval for test of moderator, the levels of
the moderator with only 1 comparison do not provide tval, pval, or CI.
Is this because there is only one comparison at that level, so it is
unable to make estimates to other comparisons within that level?

For context:
3LMA code: structured similar to this code from Harrer et al.'s book:

full.model <- rma.mv(yi = z,
                     V = var.z,
                     slab = author,
                     data = Chernobyl,
                     random = ~ 1 | author/es.id,
                     test = "t",
                     method = "REML")


Conceptual overview of CHE RVE code:
set rho
set V with impute... from clubsandwich
Similar structure for 3LMA, swapping V for the set V value.
then using robust(model,... clubSandwich = TRUE).

[sorry for not providing actual code, i don't have it on this
computer. i can provide it if it is essential to answering the
question].

I am a self-taught R user, so if I am missing something really
obvious, please don't hesitate to point it out. Many, many thanks - i
have learned a lot from reading the old messages in this mailing list
(apologies if this has been addressed before and I missed it) and the
various example documentation online, and thank you all who have
developed resources for R!

Best regards,
Noah




------------------------------

Message: 2
Date: Tue, 30 Apr 2024 16:42:36 -0500
From: James Pustejovsky <jepusto at gmail.com>
To: R Special Interest Group for Meta-Analysis
        <r-sig-meta-analysis at r-project.org>
Cc: Noah Schroeder <4noah.schroeder at gmail.com>
Subject: Re: [R-meta]  Conceptual Question: test of moderators CHE RVE
        vs 3LMA
Message-ID:
        <CAFUVuJyQdOanf7wQm2rX=_fTco20jLZRmpj7-o62SS=XaP9vtg at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

The NAs for p-values are caused by degrees of freedom (d.f.) that are very
low, zero, or negative. In the small-sample corrections used within the
robust variance estimation framework, you can think of the d.f. as a
measure of how much information is available to estimate the standard error
of the average effect size estimate or the standard error of a beta
coefficient in a meta-regression. Very low, zero, or negative degrees of
freedom means the structure of the data is such that you can't get much (if
any) information to estimate the standard errors, so RVE is not going to
work.

This mostly occurs when one of the levels of a categorical moderator only
includes a single study or a very small number of studies. robust variance
estimators are calculated using only the _between-cluster_ variation in
outcomes. Thus, if there's only one study in a category there's no
information available to look at between-study variation within that
category; if there's only two or three studies, there's only a tiny bit of
information available to look at between-study variation---this is pretty
analogous to trying to calculate a standard deviation from a sample of only
three observations.

It's possible to get very low degrees of freedom in the RVE framework in a
situation where the model-based variance estimator (i.e., what you get from
a typical 3-level model without RVE) provides a result because the latter
uses a very different approach to estimating standard errors. For one, it
is relying on different, stronger assumptions about the structure of the
model; for another, it typically also involves some large-sample
approximation so might not be as accurate an assessment of uncertainty as
the RVE framework.

In practice, the NAs for p-values based on RVEs are really an indication
that you're using an inappropriate tool for testing the specific
hypothesis---RVE-based tests will just not give you an answer, so you
shouldn't conclude anything based on their results. One thing you can do is
reconsider the hypotheses to be tested, and perhaps limit the hypotheses to
those where you have a sufficient number of studies at each level of the
moderator. For example, maybe you start out with five categories but in the
last two of those categories there's just a single study each. You could
revise the hypothesis to test for differences only between the first three
categories because it's only there that you have a sufficient sample to
even run a test.

James

On Tue, Apr 30, 2024 at 4:08?PM Noah Schroeder via R-sig-meta-analysis <
r-sig-meta-analysis at r-project.org> wrote:

> Hi Everyone,
>  I am learning about CHE RVE meta-analysis and have been experimenting
> with it. Something I have noticed is that occasionally, my test of
> moderators pvalue will be NA. However, if I run the same code through
> a typical three level structure without CHE RVE, I get a pvalue for
> the test of moderators. I have the following conceptual questions I am
> hoping someone may be able to help with:
> 1) what might cause this NA pvalue to occur in the test of moderators
> when using CHE RVE?
> 2) Should this NA be interpreted? If so, should it be interpreted as a
> non-significant result on the test of moderators? I noticed that when
> this occurs i generally have notable overlap in the CIs.
> 3) In cases where i do get a pval for test of moderator, the levels of
> the moderator with only 1 comparison do not provide tval, pval, or CI.
> Is this because there is only one comparison at that level, so it is
> unable to make estimates to other comparisons within that level?
>
> For context:
> 3LMA code: structured similar to this code from Harrer et al.'s book:
>
> full.model <- rma.mv(yi = z,
>                      V = var.z,
>                      slab = author,
>                      data = Chernobyl,
>                      random = ~ 1 | author/es.id,
>                      test = "t",
>                      method = "REML")
>
>
> Conceptual overview of CHE RVE code:
> set rho
> set V with impute... from clubsandwich
> Similar structure for 3LMA, swapping V for the set V value.
> then using robust(model,... clubSandwich = TRUE).
>
> [sorry for not providing actual code, i don't have it on this
> computer. i can provide it if it is essential to answering the
> question].
>
> I am a self-taught R user, so if I am missing something really
> obvious, please don't hesitate to point it out. Many, many thanks - i
> have learned a lot from reading the old messages in this mailing list
> (apologies if this has been addressed before and I missed it) and the
> various example documentation online, and thank you all who have
> developed resources for R!
>
> Best regards,
> Noah
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org
> To manage your subscription to this mailing list, go to:
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C02%7C%7C65b44ea4234a4e34eb6d08dc69c5a8eb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638501544879975770%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=YBoQj77C%2BaajW6357hCCrDFUbiP46k6kfb9bHhcVG%2Fw%3D&reserved=0<https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>
>

        [[alternative HTML version deleted]]




------------------------------

Message: 3
Date: Tue, 30 Apr 2024 15:10:36 -0700
From: Noah Schroeder <4noah.schroeder at gmail.com>
To: James Pustejovsky <jepusto at gmail.com>
Cc: R Special Interest Group for Meta-Analysis
        <r-sig-meta-analysis at r-project.org>
Subject: Re: [R-meta]  Conceptual Question: test of moderators CHE RVE
        vs 3LMA
Message-ID:
        <CAEfFJJYxboWNL0-asUiH7JJLeMfk=6uhYBk86TpmUkApUzc-0w at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi James,
 Thank you so much for the detailed reply, this helps a lot!
Have a great week everyone,
Noah

On Tue, Apr 30, 2024 at 2:42?PM James Pustejovsky <jepusto at gmail.com> wrote:
>
> The NAs for p-values are caused by degrees of freedom (d.f.) that are very low, zero, or negative. In the small-sample corrections used within the robust variance estimation framework, you can think of the d.f. as a measure of how much information is available to estimate the standard error of the average effect size estimate or the standard error of a beta coefficient in a meta-regression. Very low, zero, or negative degrees of freedom means the structure of the data is such that you can't get much (if any) information to estimate the standard errors, so RVE is not going to work.
>
> This mostly occurs when one of the levels of a categorical moderator only includes a single study or a very small number of studies. robust variance estimators are calculated using only the _between-cluster_ variation in outcomes. Thus, if there's only one study in a category there's no information available to look at between-study variation within that category; if there's only two or three studies, there's only a tiny bit of information available to look at between-study variation---this is pretty analogous to trying to calculate a standard deviation from a sample of only three observations.
>
> It's possible to get very low degrees of freedom in the RVE framework in a situation where the model-based variance estimator (i.e., what you get from a typical 3-level model without RVE) provides a result because the latter uses a very different approach to estimating standard errors. For one, it is relying on different, stronger assumptions about the structure of the model; for another, it typically also involves some large-sample approximation so might not be as accurate an assessment of uncertainty as the RVE framework.
>
> In practice, the NAs for p-values based on RVEs are really an indication that you're using an inappropriate tool for testing the specific hypothesis---RVE-based tests will just not give you an answer, so you shouldn't conclude anything based on their results. One thing you can do is reconsider the hypotheses to be tested, and perhaps limit the hypotheses to those where you have a sufficient number of studies at each level of the moderator. For example, maybe you start out with five categories but in the last two of those categories there's just a single study each. You could revise the hypothesis to test for differences only between the first three categories because it's only there that you have a sufficient sample to even run a test.
>
> James
>
> On Tue, Apr 30, 2024 at 4:08?PM Noah Schroeder via R-sig-meta-analysis <r-sig-meta-analysis at r-project.org> wrote:
>>
>> Hi Everyone,
>>  I am learning about CHE RVE meta-analysis and have been experimenting
>> with it. Something I have noticed is that occasionally, my test of
>> moderators pvalue will be NA. However, if I run the same code through
>> a typical three level structure without CHE RVE, I get a pvalue for
>> the test of moderators. I have the following conceptual questions I am
>> hoping someone may be able to help with:
>> 1) what might cause this NA pvalue to occur in the test of moderators
>> when using CHE RVE?
>> 2) Should this NA be interpreted? If so, should it be interpreted as a
>> non-significant result on the test of moderators? I noticed that when
>> this occurs i generally have notable overlap in the CIs.
>> 3) In cases where i do get a pval for test of moderator, the levels of
>> the moderator with only 1 comparison do not provide tval, pval, or CI.
>> Is this because there is only one comparison at that level, so it is
>> unable to make estimates to other comparisons within that level?
>>
>> For context:
>> 3LMA code: structured similar to this code from Harrer et al.'s book:
>>
>> full.model <- rma.mv(yi = z,
>>                      V = var.z,
>>                      slab = author,
>>                      data = Chernobyl,
>>                      random = ~ 1 | author/es.id,
>>                      test = "t",
>>                      method = "REML")
>>
>>
>> Conceptual overview of CHE RVE code:
>> set rho
>> set V with impute... from clubsandwich
>> Similar structure for 3LMA, swapping V for the set V value.
>> then using robust(model,... clubSandwich = TRUE).
>>
>> [sorry for not providing actual code, i don't have it on this
>> computer. i can provide it if it is essential to answering the
>> question].
>>
>> I am a self-taught R user, so if I am missing something really
>> obvious, please don't hesitate to point it out. Many, many thanks - i
>> have learned a lot from reading the old messages in this mailing list
>> (apologies if this has been addressed before and I missed it) and the
>> various example documentation online, and thank you all who have
>> developed resources for R!
>>
>> Best regards,
>> Noah
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org
>> To manage your subscription to this mailing list, go to:
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C02%7C%7C65b44ea4234a4e34eb6d08dc69c5a8eb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638501544879978481%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=yqPs1s1Ml2PWAbaJ1%2FXYlmOnGMCEeFcb%2BqIdlz0Y1ec%3D&reserved=0<https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>




------------------------------

Subject: Digest Footer

_______________________________________________
R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org
To manage your subscription to this mailing list, go to:
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C02%7C%7C65b44ea4234a4e34eb6d08dc69c5a8eb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638501544879980939%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=QEq5TgRvIrlddzbgliRZBBHBKYWXh5j73pPOIgWDYhI%3D&reserved=0<https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>


------------------------------

End of R-sig-meta-analysis Digest, Vol 84, Issue 1
**************************************************

Guido Schwarzer

Fri, May 3, 2024 12:50 AM #

Martin,

We recently published the results of a simulation study evaluating the LFK index test (https://onlinelibrary.wiley.com/doi/10.1002/jrsm.1714). We compared this method with the classic Egger and Thompson-Sharp test (aka Egger test with mixed effects model). Our simulations were for meta-analyses with the mean difference but I guess they are useful for the SMD.

The Thompson-Sharp test was conservative under homogeneity but hold the significance level under heterogeneity. The Egger test was either too liberal or conservative under heterogeneity.

Accordingly, I would lean to rely on the results of the Thompson-Sharp test as you observe high heterogeneity.

This said, typically a significance level of 10 percent is recommended for tests of funnel plot asymmetry (Sterne et al., 2011, BMJ) and the p-value of the Thompson-Sharp test is just above this value.

Best,
Guido

Martin Lobo

Fri, May 3, 2024 7:08 AM #

Dear Guido, thank you so much for your answer.

I already downloaded the article to have it among my methodological appointments.

On the other hand, when I look at the funnel plot, if there seems to be a certain asymmetry.

Another question that always arises is whether the publication bias should be carried out for each sub-analysis or for the global analysis. That is, I analyzed the subgroups of drugs vs. placebo and drugs vs. other drugs. You should evaluate the bias of all included studies or that of each subgroup separately. The results I show are from the global analysis. I share a link to see the funnel plot.

<https://1drv.ms/i/s!Ao1t9qGYkJ0rpbYRQtPrNRyOfZVb9Q?e=fSf1yD>
[https://sat02pap003files.storage.live.com/y4mcubhS2neYbUqJenfbbFJjWLPITb8eyxCNO6IIYkB3iQWIHavTNo7A9wvUXCkFxxSrGH2l0-dzxbyMRTZzpM7DzLFYAlEwDOQoCtDUCA4_WSiq0KNG29_jkdguja2QNbMpe06jxOzzTOmKqrq1oVTZv5XBYUIrjkkKG9Gg3G5De0?width=200&height=150&cropmode=center]
[https://res-h3.public.cdn.office.net/assets/mail/file-icon/png/cloud_blue_16x16.png]Funnel Inflamacion 2.jpg

thanks for your time

Best regards

Lorenzo Mart?n Lobo MTSAC, FACC, FESC
Especialista Jerarquizado en Cardiolog?a
Jefe de Dpto Enf. Cardiovasculares y Cardiometabolismo Hospital Militar Campo de Mayo.
Jefe de Cardiolog?a Hospital Militar Campo de Mayo
Ex Jefe de Unidad Coronaria Hospital Militar Campo de Mayo
Miembro Titular de la Sociedad Argentina de Cardiolog?a
Fellow American College of Cardiology
Fellow European Society of Cardiology
Ex Miembro del Area de Investigaci?n de la SAC
Ex Director del Consejo de Aterosclerosis y Trombosis de la SAC
Miembro Asesor del Consejo de Aterosclerosis y Trombosis de la SAC
Ex Director del Consejo de Epidemiolog?a y Prevenci?n Cardiovascular de la SAC
Miembro Asesor del Consejo de Epidemiolog?a y Prevenci?n Cardiovascular de la SAC
Experto en Lipidos de la Sociedad Argentina de Lipidos.
Miembro de la Sociedad Argentina de Lipidos.
Instructor de ACLS de la American Heart Association

Guido Schwarzer

Fri, May 3, 2024 8:12 AM #

Martin,

The large between-study heterogeneity might be due to including both studies comparing drugs with placebo and drugs with other drugs. Furthermore, the (slight) asymmetry in your funnel plot might be due to between-study heterogeneity which was already described in the paper introducing the Egger test as a potential source of asymmetry (Egger et al., 1997, BMJ).

You could look at a funnel plot of studies comparing drugs with placebo. However, if drugs have a different efficacy, you might get an asymmetric funnel plot due to heterogeneity.

A funnel plot of drugs vs other drugs is not well defined as you might include studies comparing A vs B or B vs A (and similarly for any other drug comparison).

All this said, I would argue that your setting (studies comparing drugs with placebo and drugs with other drugs) calls for the conduct of a network meta-analysis. If you would go this route, the paper by Chaimani and Salanti (https://onlinelibrary.wiley.com/doi/10.1002/jrsm.57) gives a nice introduction into the evalution of small study effects in network meta-analysis and introduces the comparison-adjusted funnel plot.

Best,
Guido