[R-meta] Egger's test with multilevel meta analysis

10 messages · Dylan Johnson, t@s@ueressig m@iii@g oii gmx@de, James Pustejovsky +1 more

Original

1

10

Wed, Dec 9, 2020 9:54 AM #

Hello,

I am in the process of carrying out a multilevel meta analysis using ?rma.mv?. Unfortunately, it does not seem like this type of model can be used with the dmetar ?eggers.test? function.

Does anyone have any suggestions for how I could get around this?

Many thanks!

Dylan

t@s@ueressig m@iii@g oii gmx@de

Wed, Dec 9, 2020 10:02 AM #

An HTML attachment was scrubbed...
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20201209/49e24b3e/attachment.html>

James Pustejovsky

Wed, Dec 9, 2020 10:20 AM #

We have a paper (forthcoming in Psych Methods) evaluating a similar method
for adapting Egger's test to the multilevel context, using RVE:
* Rodgers, M. A., & Pustejovsky, J. E. (In Press). Evaluating Meta-Analytic
Methods to Detect Selective Reporting in the Presence of Dependent Effect
Sizes. Psychological Methods, forthcoming.
https://doi.org/10.31222/osf.io/vqp8u

There is also a related paper by Fernandez-Castilla and colleagues:
* Fern?ndez-Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N.,
Onghena, P., & Van den Noortgate, W. (2019). Detecting selection bias in
meta-analyses with multipleoutcomes: A simulation study. The Journal of
Experimental Education, 1?20.

These tests can be implemented in rma.mv() simply by including the standard
error of the effect size (or a related measure of precision, such as the
sample size) as a moderator. Say that data includes a variable called sei
for the standard error of each effect size:

egger_multi <- rma.mv(yi = yi, V = sei^2, random = ~ 1 | studyID, effectID,
mods = ~ sei, data = dat)

Then apply cluster-robust standard errors for the RVE-based test:

coef_test(egger_multi, vcov = "CR2")

Further details available in our paper, and example code in our
supplementary materials.

James

On Wed, Dec 9, 2020 at 12:07 PM <t.saueressig at gmx.de> wrote:

_______________________________________________
R-sig-meta-analysis mailing list
R-sig-meta-analysis at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis


_______________________________________________
R-sig-meta-analysis mailing list
R-sig-meta-analysis at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

Wed, Dec 9, 2020 10:36 AM #

I have tried the following:

egger_multi <- rma.mv(HEDGE_G, HEDGE_VAR, random = ~ 1 | COHORT_ID, EFFECT_ID, mods = ~ STD_ERR, data = dataset)

coeftest(egger_multi, vcov = "CR2")

When I run the coeftest I receive the error:

Error in diag(se) : invalid 'nrow' value (too large or NA)
In addition: Warning message:
In diag(se) : NAs introduced by coercion



Dylan Johnson, MSc

MA Student, School and Clinical Child Psychology
Department of Applied Psychology and Human Development

University of Toronto
252 Bloor Street West

Toronto, ON M5S 1V6

From: James Pustejovsky<mailto:jepusto at gmail.com>
Sent: December 9, 2020 1:20 PM
To: Tobias Saueressig<mailto:t.saueressig at gmx.de>
Cc: Dylan Johnson<mailto:dylanr.johnson at mail.utoronto.ca>; R meta<mailto:r-sig-meta-analysis at r-project.org>
Subject: Re: [R-meta] Egger's test with multilevel meta analysis

EXTERNAL EMAIL:
We have a paper (forthcoming in Psych Methods) evaluating a similar method for adapting Egger's test to the multilevel context, using RVE:
* Rodgers, M. A., & Pustejovsky, J. E. (In Press). Evaluating Meta-Analytic Methods to Detect Selective Reporting in the Presence of Dependent Effect Sizes. Psychological Methods, forthcoming. https://doi.org/10.31222/osf.io/vqp8u

There is also a related paper by Fernandez-Castilla and colleagues:
* Fern?ndez-Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N., Onghena, P., & Van den Noortgate, W. (2019). Detecting selection bias in meta-analyses with multipleoutcomes: A simulation study. The Journal of Experimental Education, 1?20.

These tests can be implemented in rma.mv<http://rma.mv>() simply by including the standard error of the effect size (or a related measure of precision, such as the sample size) as a moderator. Say that data includes a variable called sei for the standard error of each effect size:

egger_multi <- rma.mv<http://rma.mv>(yi = yi, V = sei^2, random = ~ 1 | studyID, effectID, mods = ~ sei, data = dat)

Then apply cluster-robust standard errors for the RVE-based test:

coef_test(egger_multi, vcov = "CR2")

Further details available in our paper, and example code in our supplementary materials.

James

On Wed, Dec 9, 2020 at 12:07 PM <t.saueressig at gmx.de<mailto:t.saueressig at gmx.de>> wrote:

Hi Dylan,

you might want to look at this https://onlinelibrary.wiley.com/doi/abs/10.1111/biom.13342

And this

https://cran.r-project.org/web/packages/xmeta/

Regards

Tobias



Am 09.12.2020 18:54 schrieb Dylan Johnson <dylanr.johnson at mail.utoronto.ca<mailto:dylanr.johnson at mail.utoronto.ca>>:

Hello,

I am in the process of carrying out a multilevel meta analysis using ?rma.mv<http://rma.mv>?. Unfortunately, it does not seem like this type of model can be used with the dmetar ?eggers.test? function.

Does anyone have any suggestions for how I could get around this?

Many thanks!

Dylan




_______________________________________________
R-sig-meta-analysis mailing list
R-sig-meta-analysis at r-project.org<mailto:R-sig-meta-analysis at r-project.org>
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

_______________________________________________
R-sig-meta-analysis mailing list
R-sig-meta-analysis at r-project.org<mailto:R-sig-meta-analysis at r-project.org>
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

Wolfgang Viechtbauer

Wed, Dec 9, 2020 10:55 AM #

That call to rma.mv() doesn't look right. Maybe you meant:

egger_multi <- rma.mv(HEDGE_G, HEDGE_VAR, random = ~ 1 | COHORT_ID/EFFECT_ID, mods = ~ STD_ERR, data = dataset)

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org]
On Behalf Of Dylan Johnson
Sent: Wednesday, 09 December, 2020 19:37
To: James Pustejovsky; Tobias Saueressig
Cc: R meta
Subject: Re: [R-meta] Egger's test with multilevel meta analysis

I have tried the following:

egger_multi <- rma.mv(HEDGE_G, HEDGE_VAR, random = ~ 1 | COHORT_ID,
EFFECT_ID, mods = ~ STD_ERR, data = dataset)

coeftest(egger_multi, vcov = "CR2")

When I run the coeftest I receive the error:

Error in diag(se) : invalid 'nrow' value (too large or NA)
In addition: Warning message:
In diag(se) : NAs introduced by coercion

Dylan Johnson, MSc

MA Student, School and Clinical Child Psychology
Department of Applied Psychology and Human Development

University of Toronto
252 Bloor Street West

Toronto, ON M5S 1V6

From: James Pustejovsky<mailto:jepusto at gmail.com>
Sent: December 9, 2020 1:20 PM
To: Tobias Saueressig<mailto:t.saueressig at gmx.de>
Cc: Dylan Johnson<mailto:dylanr.johnson at mail.utoronto.ca>; R meta<mailto:r-
sig-meta-analysis at r-project.org>
Subject: Re: [R-meta] Egger's test with multilevel meta analysis

We have a paper (forthcoming in Psych Methods) evaluating a similar method
for adapting Egger's test to the multilevel context, using RVE:
* Rodgers, M. A., & Pustejovsky, J. E. (In Press). Evaluating Meta-Analytic
Methods to Detect Selective Reporting in the Presence of Dependent Effect
Sizes. Psychological Methods, forthcoming.
https://doi.org/10.31222/osf.io/vqp8u

There is also a related paper by Fernandez-Castilla and colleagues:
* Fern?ndez-Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N.,
Onghena, P., & Van den Noortgate, W. (2019). Detecting selection bias in
meta-analyses with multipleoutcomes: A simulation study. The Journal of
Experimental Education, 1?20.

These tests can be implemented in rma.mv<http://rma.mv>() simply by
including the standard error of the effect size (or a related measure of
precision, such as the sample size) as a moderator. Say that data includes a
variable called sei for the standard error of each effect size:

egger_multi <- rma.mv<http://rma.mv>(yi = yi, V = sei^2, random = ~ 1 |
studyID, effectID, mods = ~ sei, data = dat)

Then apply cluster-robust standard errors for the RVE-based test:

coef_test(egger_multi, vcov = "CR2")

Further details available in our paper, and example code in our
supplementary materials.

James

Wed, Dec 9, 2020 11:01 AM #

Unfortunately the error persists even after making your correction







Dylan Johnson, MSc

MA Student, School and Clinical Child Psychology
Department of Applied Psychology and Human Development

University of Toronto
252 Bloor Street West

Toronto, ON M5S 1V6



From: Viechtbauer, Wolfgang (SP)<mailto:wolfgang.viechtbauer at maastrichtuniversity.nl>
Sent: December 9, 2020 1:56 PM
To: Dylan Johnson<mailto:dylanr.johnson at mail.utoronto.ca>
Cc: R meta<mailto:r-sig-meta-analysis at r-project.org>
Subject: RE: [R-meta] Egger's test with multilevel meta analysis



EXTERNAL EMAIL:

That call to rma.mv() doesn't look right. Maybe you meant:

egger_multi <- rma.mv(HEDGE_G, HEDGE_VAR, random = ~ 1 | COHORT_ID/EFFECT_ID, mods = ~ STD_ERR, data = dataset)

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org]
On Behalf Of Dylan Johnson
Sent: Wednesday, 09 December, 2020 19:37
To: James Pustejovsky; Tobias Saueressig
Cc: R meta
Subject: Re: [R-meta] Egger's test with multilevel meta analysis

I have tried the following:

egger_multi <- rma.mv(HEDGE_G, HEDGE_VAR, random = ~ 1 | COHORT_ID,
EFFECT_ID, mods = ~ STD_ERR, data = dataset)

coeftest(egger_multi, vcov = "CR2")

When I run the coeftest I receive the error:

Error in diag(se) : invalid 'nrow' value (too large or NA)
In addition: Warning message:
In diag(se) : NAs introduced by coercion

Dylan Johnson, MSc

MA Student, School and Clinical Child Psychology
Department of Applied Psychology and Human Development

University of Toronto
252 Bloor Street West

Toronto, ON M5S 1V6

From: James Pustejovsky<mailto:jepusto at gmail.com>
Sent: December 9, 2020 1:20 PM
To: Tobias Saueressig<mailto:t.saueressig at gmx.de>
Cc: Dylan Johnson<mailto:dylanr.johnson at mail.utoronto.ca>; R meta<mailto:r-
sig-meta-analysis at r-project.org>
Subject: Re: [R-meta] Egger's test with multilevel meta analysis

We have a paper (forthcoming in Psych Methods) evaluating a similar method
for adapting Egger's test to the multilevel context, using RVE:
* Rodgers, M. A., & Pustejovsky, J. E. (In Press). Evaluating Meta-Analytic
Methods to Detect Selective Reporting in the Presence of Dependent Effect
Sizes. Psychological Methods, forthcoming.
https://doi.org/10.31222/osf.io/vqp8u

There is also a related paper by Fernandez-Castilla and colleagues:
* Fern?ndez-Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N.,
Onghena, P., & Van den Noortgate, W. (2019). Detecting selection bias in
meta-analyses with multipleoutcomes: A simulation study. The Journal of
Experimental Education, 1?20.

These tests can be implemented in rma.mv<http://rma.mv>() simply by
including the standard error of the effect size (or a related measure of
precision, such as the sample size) as a moderator. Say that data includes a
variable called sei for the standard error of each effect size:

egger_multi <- rma.mv<http://rma.mv>(yi = yi, V = sei^2, random = ~ 1 |
studyID, effectID, mods = ~ sei, data = dat)

Then apply cluster-robust standard errors for the RVE-based test:

coef_test(egger_multi, vcov = "CR2")

Further details available in our paper, and example code in our
supplementary materials.

James

Wolfgang Viechtbauer

Wed, Dec 9, 2020 11:22 AM #

Try:

coeftest(egger_multi, vcov = "CR2", cluster = dataset$COHORT_ID)

If this doesn't help, what version of clunSandwich are you using? (see sessionInfo()).

Best,
Wolfgang

Wed, Dec 9, 2020 11:25 AM #

The issue what that coeftest is a different function than the one I should have been using (I.e. coef_test).



Thanks!



Dylan Johnson, MSc

MA Student, School and Clinical Child Psychology
Department of Applied Psychology and Human Development

University of Toronto
252 Bloor Street West

Toronto, ON M5S 1V6



From: Viechtbauer, Wolfgang (SP)<mailto:wolfgang.viechtbauer at maastrichtuniversity.nl>
Sent: December 9, 2020 2:23 PM
To: Dylan Johnson<mailto:dylanr.johnson at mail.utoronto.ca>
Cc: R meta<mailto:r-sig-meta-analysis at r-project.org>
Subject: RE: [R-meta] Egger's test with multilevel meta analysis



EXTERNAL EMAIL:

Try:

coeftest(egger_multi, vcov = "CR2", cluster = dataset$COHORT_ID)

If this doesn't help, what version of clunSandwich are you using? (see sessionInfo()).

Best,
Wolfgang

Wolfgang Viechtbauer

Wed, Dec 9, 2020 11:28 AM #

Argh, of course :) Well, glad I could be of no assistance.

Best,
Wolfgang

-----Original Message-----
From: Dylan Johnson [mailto:dylanr.johnson at mail.utoronto.ca]
Sent: Wednesday, 09 December, 2020 20:26
To: Viechtbauer, Wolfgang (SP)
Cc: R meta
Subject: RE: [R-meta] Egger's test with multilevel meta analysis

The issue what that coeftest is a different function than the one I should
have been using (I.e. coef_test).

Thanks!

Dylan Johnson, MSc
MA?Student,?School and Clinical Child Psychology
Department of?Applied Psychology?and?Human Development
University of Toronto
252 Bloor Street West
Toronto, ON M5S 1V6

From: Viechtbauer, Wolfgang (SP)
Sent: December 9, 2020 2:23 PM
To: Dylan Johnson
Cc: R meta
Subject: RE: [R-meta] Egger's test with multilevel meta analysis

EXTERNAL EMAIL:

Try:

coeftest(egger_multi, vcov = "CR2", cluster = dataset$COHORT_ID)

If this doesn't help, what version of clunSandwich are you using? (see
sessionInfo()).

Best,
Wolfgang

-----Original Message-----
From: Dylan Johnson [mailto:dylanr.johnson at mail.utoronto.ca]
Sent: Wednesday, 09 December, 2020 20:01
To: Viechtbauer, Wolfgang (SP)
Cc: R meta
Subject: RE: [R-meta] Egger's test with multilevel meta analysis

Unfortunately the error persists even after making your correction

Dylan Johnson, MSc
MA Student, School and Clinical Child Psychology
Department of Applied Psychology and Human Development
University of Toronto
252 Bloor Street West
Toronto, ON M5S 1V6

From: Viechtbauer, Wolfgang (SP)
Sent: December 9, 2020 1:56 PM
To: Dylan Johnson
Cc: R meta
Subject: RE: [R-meta] Egger's test with multilevel meta analysis

That call to rma.mv() doesn't look right. Maybe you meant:

egger_multi <- rma.mv(HEDGE_G, HEDGE_VAR, random = ~ 1 |
COHORT_ID/EFFECT_ID, mods = ~ STD_ERR, data = dataset)

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-

project.org]

On Behalf Of Dylan Johnson
Sent: Wednesday, 09 December, 2020 19:37
To: James Pustejovsky; Tobias Saueressig
Cc: R meta
Subject: Re: [R-meta] Egger's test with multilevel meta analysis

I have tried the following:

egger_multi <- rma.mv(HEDGE_G, HEDGE_VAR, random = ~ 1 | COHORT_ID,
EFFECT_ID, mods = ~ STD_ERR, data = dataset)

coeftest(egger_multi, vcov = "CR2")

When I run the coeftest I receive the error:

Error in diag(se) : invalid 'nrow' value (too large or NA)
In addition: Warning message:
In diag(se) : NAs introduced by coercion

Dylan Johnson, MSc

MA Student, School and Clinical Child Psychology
Department of Applied Psychology and Human Development

University of Toronto
252 Bloor Street West

Toronto, ON M5S 1V6

James Pustejovsky

Wed, Dec 9, 2020 11:40 AM #

There was a typo in the syntax I gave too. So just for the record, here is
the correct pseudo-syntax:

egger_multi <- rma.mv(yi = yi, V = sei^2, random = ~ 1 | studyID /
effectID, mods = ~ sei, data = dat)
coef_test(egger_multi, vcov = "CR2")

On Wed, Dec 9, 2020 at 1:38 PM Viechtbauer, Wolfgang (SP) <

wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

Argh, of course :) Well, glad I could be of no assistance.

Best,
Wolfgang

-----Original Message-----
From: Dylan Johnson [mailto:dylanr.johnson at mail.utoronto.ca]
Sent: Wednesday, 09 December, 2020 20:26
To: Viechtbauer, Wolfgang (SP)
Cc: R meta
Subject: RE: [R-meta] Egger's test with multilevel meta analysis

The issue what that coeftest is a different function than the one I should
have been using (I.e. coef_test).

Thanks!

Dylan Johnson, MSc
MA Student, School and Clinical Child Psychology
Department of Applied Psychology and Human Development
University of Toronto
252 Bloor Street West
Toronto, ON M5S 1V6

From: Viechtbauer, Wolfgang (SP)
Sent: December 9, 2020 2:23 PM
To: Dylan Johnson
Cc: R meta
Subject: RE: [R-meta] Egger's test with multilevel meta analysis

EXTERNAL EMAIL:

Try:

coeftest(egger_multi, vcov = "CR2", cluster = dataset$COHORT_ID)

If this doesn't help, what version of clunSandwich are you using? (see
sessionInfo()).

Best,
Wolfgang

-----Original Message-----
From: Dylan Johnson [mailto:dylanr.johnson at mail.utoronto.ca]
Sent: Wednesday, 09 December, 2020 20:01
To: Viechtbauer, Wolfgang (SP)
Cc: R meta
Subject: RE: [R-meta] Egger's test with multilevel meta analysis

Unfortunately the error persists even after making your correction

Dylan Johnson, MSc
MA Student, School and Clinical Child Psychology
Department of Applied Psychology and Human Development
University of Toronto
252 Bloor Street West
Toronto, ON M5S 1V6

From: Viechtbauer, Wolfgang (SP)
Sent: December 9, 2020 1:56 PM
To: Dylan Johnson
Cc: R meta
Subject: RE: [R-meta] Egger's test with multilevel meta analysis

That call to rma.mv() doesn't look right. Maybe you meant:

egger_multi <- rma.mv(HEDGE_G, HEDGE_VAR, random = ~ 1 |
COHORT_ID/EFFECT_ID, mods = ~ STD_ERR, data = dataset)

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-

project.org]

On Behalf Of Dylan Johnson
Sent: Wednesday, 09 December, 2020 19:37
To: James Pustejovsky; Tobias Saueressig
Cc: R meta
Subject: Re: [R-meta] Egger's test with multilevel meta analysis

I have tried the following:

egger_multi <- rma.mv(HEDGE_G, HEDGE_VAR, random = ~ 1 | COHORT_ID,
EFFECT_ID, mods = ~ STD_ERR, data = dataset)

coeftest(egger_multi, vcov = "CR2")

When I run the coeftest I receive the error:

Error in diag(se) : invalid 'nrow' value (too large or NA)
In addition: Warning message:
In diag(se) : NAs introduced by coercion

Dylan Johnson, MSc

MA Student, School and Clinical Child Psychology
Department of Applied Psychology and Human Development

University of Toronto
252 Bloor Street West

Toronto, ON M5S 1V6

_______________________________________________
R-sig-meta-analysis mailing list
R-sig-meta-analysis at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis