[R-meta] Questions regarding REML and FE models and R^2 calculation in metafor
Dear list members, I have a follow-up question. In my dataset I have about 500 experiments (i.e., observations) across 50 articles (i.e., references), but the moderators in question change across observations only within 7 of the references. Consequently, my rma.mv model that uses ~1|Reference as a random effect is over-sensitive to the data from these 7 studies compared to the others. In such a case, if I use a rma.mv (or rma.uni) model without a random effect, would it be more reliable? And if I do use such a model, how do I compute the R^2 for each moderator (as sigma^2 is inapplicable)? Thanks again, Nevo Sagi
On Mon, Jun 5, 2023 at 10:52?AM Nevo Sagi <nevosagi8 at gmail.com> wrote:
Dear Wolgang, Thank you for your feedback. It turns out that I misplaced the equation terms when calculating the pseudo-R^2. All the best, Nevo On Thu, Jun 1, 2023 at 3:30?PM Viechtbauer, Wolfgang (NP) < wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
Dear Nevo, Please see my responses below. Best, Wolfgang
-----Original Message----- From: R-sig-meta-analysis [mailto:
r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of Nevo Sagi via R-sig-meta-analysis Sent: Thursday, 04 May, 2023 11:09 To: r-sig-meta-analysis at r-project.org Cc: Nevo Sagi Subject: [R-meta] Questions regarding REML and FE models and R^2
calculation in
metafor Dear list members, I conducted a meta-analysis on the role of climate in mediating a
specific
ecological process, using the *metafor *package in R. This is actually a meta-regression, using the rma.mv function, with *temperature *and *precipitation *as moderators, along with some other moderators related to experimental design. I also use reference as a
random
effect ('random = ~1|*Reference'*), as some references include more than
one experiment.
*1. FE vs REML model:*
After reading Wolfgang Viechtbauer's blog post
<https://wviechtb.github.io/metafor/reference/misc-models.html> on the
differences between fixed-effects and random-effects models in the
*metafor *package, I decided to use the FE method, because the studies I
gathered are not a random sample of the population of hypothetical
studies.
Instead, the sample is biased by underrepresentation of some climates and overrepresentation of others. I wonder whether my interpretation of the difference between FE and REML models is correct, and would like to get some feedback on it.
I don't think this is really a good reason for using a FE model, because the underrepresentation of some climates and overrepresentation of others will affect your results either way. The bigger question is if climate is an important moderator, which you can examine via meta-regression.
*2. R^2 calculation:* Reviewers of my manuscript required that I provide R-squared values for each of the climate moderators. Using the *metafor *package, only rma.uni models (where random variables cannot be specified) provide R^2 estimation. In a previous conversation in this mailing list, Wolfgang indicated that pseudo-R^2 can be calculated based on the variance (sigma2) reported by models including and excluding the moderator in question: *(res0$sigma2 - res1$sigma2) / res0$sigma2* *where 'res0' is the model without coefficients and 'res1' the model
with.*
I have two problems with this solution: 1. FE models do not provide variance components (sigma2). Therefore, the pseudo R-squared can be calculated only for REML models. I guess this can be explained by the nature of the models, which I don't fully understand.
Yes, this approach to calculating such pseudo-R^2 values only works in RE models.
2. When using REML models and performing the above calculation, I get
weird
results. For example, one of the pseudo R^2 values was above 1. This
cannot
mean that the moderator explained more than 100% of the variance in the effect size. How comparable is this pseudo R^2 for the standard R^2 of simpler models?
This is mathematically impossible. (res0$sigma2 - res1$sigma2) / res0$sigma2 is the same as 1 - res1$sigma2 / res0$sigma2 and the second term cannot be negative, so the resulting value cannot be larger than 1.
To conclude, I will be glad to get feedback on both problems: 1. Should I use a random-effect or fixed-effect model? 2. How do I get a reliable R^2 or an alternative measure of goodness of
fit
for single-moderator models that include a random structure and a
sampling
variance? Thank you very much, Nevo Sagi -- Dr. Nevo Sagi Prof. Dror Hawlena's Risk-Management Ecology Lab Department of Ecology, Evolution & Behavior The Alexander Silberman Institute of Life Sciences The Hebrew University of Jerusalem Edmond J. Safra Campus at Givat Ram, Jerusalem 9190401, Israel.
-- Dr. Nevo Sagi Prof. Dror Hawlena's Risk-Management Ecology Lab Department of Ecology, Evolution & Behavior The Alexander Silberman Institute of Life Sciences The Hebrew University of Jerusalem Edmond J. Safra Campus at Givat Ram, Jerusalem 9190401, Israel.
Dr. Nevo Sagi Prof. Dror Hawlena's Risk-Management Ecology Lab Department of Ecology, Evolution & Behavior The Alexander Silberman Institute of Life Sciences The Hebrew University of Jerusalem Edmond J. Safra Campus at Givat Ram, Jerusalem 9190401, Israel. [[alternative HTML version deleted]]