Skip to content

[R-meta] Multivariate multi-level meta-analysis: adjusting control variables when modeling publication bias?

4 messages · Wolfgang Viechtbauer, Daniel Foster

Daniel Foster
# 
Hello Wolfgang and all,

I am carrying out multivariate multilevel meta-analysis using the
rma.mv function in the metafor package, and I have come across an
issue that has been giving me a lot of trouble. At this point I am at
a stand still and any insight would be greatly appreciated!!

When testing for publication bias using the PET approach, Doucouliagos
& Stanley (2009), suggest using the following model in a weighted
least squares formula:

ES= B1(1/SE) + SIGMA ak(Zj/SEi )+ e

Where SE is the standard error of the effect estimate,  Z is a vector
of meta-independent variables reflecting differences across studies
for the jth study in literature, ak is the meta-regression coefficient
which reflects the effect of particular study characteristics.

My confusion lies in the fact that they are suggesting that the
control variables (Z) included need to be divided by the standard
error of the effect estimate. My questions are this:

Should I be dividing my control variables by the standard error of
the effect estimate when using the rma.mv function?  I have found some
multivariate multilevel meta-analyses that follow this method
(Klomp, 2009), but then others that don't (at least explicitly;
Akgunduz, 2018)

If I do need to do this, it is not clear to me how a binary control
variable can be incorporated in my mra.mv model (i.e., 1, 0). To my
mind, it seems strange to divide a dichotomous variable by a
continuous variable. What steps do I need to execute to include these
variables in my mra.mv model?

Thank you so much in advance for your insights!

Daniel

Akgunduz, Y. E., & Plantenga, J. (2018). doi: 10.1111/joes.12192
Doucouliagos, H., & Stanley, T. D. (2009). doi: 10.1111/j.1467-8543.2009.00723.x
Klomp, J., & De Haan, J. (2010). doi: 10.1111/j.1467-6419.2009.00597.x
Wolfgang Viechtbauer
# 
Hi Daniel,

Didn't you ask the same question here?

https://stats.stackexchange.com/q/633046/1934

Please see my answer there.

Best,
Wolfgang
Daniel Foster
# 
Hi Wolfgang,

Thank you! Sorry, I am new to Stack Exchange and did not realize that you had already provided a response. I deeply appreciate your insights.

I do have a follow up question, if that's okay. I am also carrying two other multilevel analyses - one uses the t-statistic as a dependent outcome, and another uses the partial correlation coefficient (the computation of which is derived from the t-statistic) as the dependent variable.

You mention that the authors divide everything because they are working with test statistics, which makes me wonder whether the analyses I just mentioned require the same adjustment? However, you go on to say that random-effects models and models with a multilevel structure do not require this adjustment. Do you mind clarifying?

Thank you again for your time! The mailing list archives are an incredible resource!!

I will also post this response on stack exchange, so that it is available to others.

Best,
Daniel
4 days later
Wolfgang Viechtbauer
# 
Hi Daniel,

I don't think you ever got a response to this.

I don't have much to say about this anyway, except that *very rarely* does it ever make sense to run a meta-analysis on test statistics. In any case, the idea of formulating a model like in PET where we use the test statistics as the dependent variables just doesn't carry over to random-effects / multilevel models as I demonstrated in one of my previous posts. Instead, we should use the model formulation where the effect sizes estimates are the dependent variable.

Note that partial correlations are not test statistics.

Best,
Wolfgang