Skip to content

[R-meta] Dear Wolfgang

3 messages · Wolfgang Viechtbauer, Lee, Ju

Lee, Ju
# 
Hello,

I am currently using a mixed effect meta-regression to explore the effects of different environmental variables on fish densities in coastal habitats.
For this, I am constructing a different model for individual fish species, in which my goal is to identify important predictor variables separately for each species by using aicc-based model selection (glmulti).
For each fish dataset, I usually identify 3-4 potentially important predictor variables ? which include both categorical and continuous variables ? based on a priori hypothesis and the statistical test of omnibus test (of each individual predictor variable).
I am only testing the main effects and not the interaction of multiple variables being included in the final models.

The problem I am running into is the small number of studies being available for many species, with the number of study response (effect size, not independent study) ranging from 6 to 100 for different fish species.

So my question is:

Is there a commonly used threshold for the multiple meta-regression using rma.mv to be reliable and avoid false positive or negative relationship?
Wolfgang Viechtbauer
# 
Dear Juhyung,

Briefly, I am not aware of any proper evidence based guidelines for such a threshold. You sometimes see people mention that there should be 5 or 10 studies per moderator, but these are just heuristics. One could do a proper power analysis if one is worried about false negatives. See

https://www.jepusto.com/publication/power-approximations-for-dependent-effects/

for some recent work that might be applicable to your case (since your post implies that you are dealing with a more complex data structure).

Best,
Wolfgang
Lee, Ju
# 
Dear Wolfgang,

Thank you for your response. It's helpful to know your perspectives on this, and that in general there is no hard evidence or guidelines for sample size thresholds for meta-regression models.

Also, thank you very much for directing me to possibly using power analyses for testing false relationships.

Best,
JU


Juhyung Lee
Postdoctoral Fellow
Marine Science Center, Northeastern University
430 Nahant Rd, Nahant, MA 01908, USA
Phone: +1(650)285-7614 

 

?2022. 1. 27. ?? 1:54, "Viechtbauer, Wolfgang (SP)" <wolfgang.viechtbauer at maastrichtuniversity.nl> ??:

    Dear Juhyung,

    Briefly, I am not aware of any proper evidence based guidelines for such a threshold. You sometimes see people mention that there should be 5 or 10 studies per moderator, but these are just heuristics. One could do a proper power analysis if one is worried about false negatives. See

    https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jepusto.com%2Fpublication%2Fpower-approximations-for-dependent-effects%2F&amp;data=04%7C01%7Cjuh.lee%40northeastern.edu%7C99ec6dbb24154fb76f9a08d9e1c652d8%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637789064934933614%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=fsQn8qGQ9DT7FxbNUkpjjSvJ5pwUP1vwaYgdV6K1ZCg%3D&amp;reserved=0

    for some recent work that might be applicable to your case (since your post implies that you are dealing with a more complex data structure).

    Best,
    Wolfgang

    >-----Original Message-----
    >From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On
    >Behalf Of Lee, Ju
    >Sent: Wednesday, 26 January, 2022 23:07
    >To: r-sig-meta-analysis at r-project.org
    >Subject: [R-meta] Dear Wolfgang
    >
    >Hello,
    >
    >I am currently using a mixed effect meta-regression to explore the effects of
    >different environmental variables on fish densities in coastal habitats.
    >For this, I am constructing a different model for individual fish species, in
    >which my goal is to identify important predictor variables separately for each
    >species by using aicc-based model selection (glmulti).
    >For each fish dataset, I usually identify 3-4 potentially important predictor
    >variables ? which include both categorical and continuous variables ? based on a
    >priori hypothesis and the statistical test of omnibus test (of each individual
    >predictor variable).
    >I am only testing the main effects and not the interaction of multiple variables
    >being included in the final models.
    >
    >The problem I am running into is the small number of studies being available for
    >many species, with the number of study response (effect size, not independent
    >study) ranging from 6 to 100 for different fish species.
    >
    >So my question is:
    >
    >Is there a commonly used threshold for the multiple meta-regression using rma.mv
    >to be reliable and avoid false positive or negative relationship?
    >From
    >https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fjournals.plos.org%2Fplosone%2Farticle%2Ffile%3Fid%3D10.1371%2Fjournal.pone.0229345%26ty&amp;data=04%7C01%7Cjuh.lee%40northeastern.edu%7C99ec6dbb24154fb76f9a08d9e1c652d8%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637789064934933614%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=j9OjVdDurDUJF9csQV3C89ykyv0wt10J7MFWzD7TRUY%3D&amp;reserved=0
    >pe=printable I read that, for meta-regression, n should be greater than 8 for low
    >variance data, but should be greater than 25 for high variance data.
    >I wanted to seek your advice on what would be a good threshold criteria for
    >minimum study response (effect size) number for running the meta-regression
    >models.
    >Also, could we also apply similar threshold for number of independent studies
    >(not effect size, but the actual publication) included in each dataset as well?
    >
    >Thank you.
    >Sincerely,
    >Juhyung
    >
    >Juhyung Lee
    >Postdoctoral Fellow
    >Marine Science Center, Northeastern University
    >430 Nahant Rd, Nahant, MA 01908, USA
    >Phone: +1(650)285-7614