[R-meta] Unweighted analysis, Re: [EXT] R-sig-meta-analysis Digest, Vol 82, Issue 31

Thanks for the advice, Michael.  Typically, my students who are
running unweighted analyses have no variance estimates for their
effect sizes.  Since vi is required, we need to put something into the
vi column.  Seeing the variation in 95% CIs around the mean effect
depending on what vi values I used was concerning.  Does anyone have
advice for the best practice for unweighted analyses when vi values
are not available?  Thanks in advance.  alan

-----Original Message-----
From: R-sig-meta-analysis <r-sig-meta-analysis-bounces at r-project.org>
On Behalf Of r-sig-meta-analysis-request at r-project.org
Sent: Thursday, March 21, 2024 11:19 AM
To: r-sig-meta-analysis at r-project.org
Subject: [EXT] R-sig-meta-analysis Digest, Vol 82, Issue 31

CAUTION: Email Originated Outside of Auburn.

Send R-sig-meta-analysis mailing list submissions to
         r-sig-meta-analysis at r-project.org

To subscribe or unsubscribe via the World Wide Web, visit

https://stat/
.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C02%7Cwil
son%40auburn.edu%7C2d94e8b5f80d4aa6122f08dc4d950e2b%7Cccb6deedbd294b38
8979d72780f62d3b%7C0%7C0%7C638470549845910049%7CUnknown%7CTWFpbGZsb3d8
eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0
%7C%7C%7C&sdata=vdZri81iNIDnCXfDxiA5xFkNcz2NJpXpcMjkZDEo%2Bw8%3D&reser
ved=0 or, via email, send a message with subject or body 'help' to
         r-sig-meta-analysis-request at r-project.org

You can reach the person managing the list at
         r-sig-meta-analysis-owner at r-project.org

When replying, please edit your Subject line so it is more specific than "Re: Contents of R-sig-meta-analysis digest..."


Today's Topics:

    1. unweighted analysis questions (Alan Wilson)
    2. Re: unweighted analysis questions (Michael Dewey)
    3. Re:  Prediction Intervals for estimates of random slope
       rma.mv model with predict.rma() (Zac Robinson)

----------------------------------------------------------------------

Message: 1
Date: Thu, 21 Mar 2024 14:51:06 +0000
From: Alan Wilson <wilson at auburn.edu>
To: "r-sig-meta-analysis at r-project.org"
         <r-sig-meta-analysis at r-project.org>
Subject: [R-meta] unweighted analysis questions
Message-ID:

<MW2PR1901MB4697CAA9F67A1C8E78FCBEE9BF322 at MW2PR1901MB4697.namprd19.pro
d.outlook.com>

Content-Type: text/plain; charset="utf-8"

Hey everyone - I have two questions regarding unweighted analyses.

1) Using the suggested code for unweighted analysis (rma(yi, vi, method="FE", data=dat, weighted=FALSE)) provides the same main effect but different 95% CIs when using the same or different vi values across studies.  I also found different 95% CIs when testing two different vi values (1 vs 100) that were the same across studies.  I also found that changing weighted=FALSE to weighted=TRUE provided same 95% CIs when using the same dataset.  Any idea what is causing the variation in 95% CIs?  What is best for unweighted analyses?

2) Any suggested approaches for publication bias checks for unweighted analyses?  I was thinking funnel plots of effect size and sample size.

Thanks for any advice.  alan

         [[alternative HTML version deleted]]




------------------------------

Message: 2
Date: Thu, 21 Mar 2024 15:27:00 +0000
From: Michael Dewey <lists at dewey.myzen.co.uk>
To: R Special Interest Group for Meta-Analysis
         <r-sig-meta-analysis at r-project.org>
Cc: Alan Wilson <wilson at auburn.edu>
Subject: Re: [R-meta] unweighted analysis questions
Message-ID: <b5c372ec-dbde-1825-b980-1974e04e70c5 at dewey.myzen.co.uk>
Content-Type: text/plain; charset="utf-8"; Format="flowed"

Dear Alan

On 21/03/2024 14:51, Alan Wilson via R-sig-meta-analysis wrote:

Hey everyone - I have two questions regarding unweighted analyses.

1) Using the suggested code for unweighted analysis (rma(yi, vi, method="FE", data=dat, weighted=FALSE)) provides the same main effect but different 95% CIs when using the same or different vi values across studies.  I also found different 95% CIs when testing two different vi values (1 vs 100) that were the same across studies.  I also found that changing weighted=FALSE to weighted=TRUE provided same 95% CIs when using the same dataset.  Any idea what is causing the variation in 95% CIs?  What is best for unweighted analyses?

That is what I would have expected. Although the vi are not being used as weights they do still tell you about the variability. If you combine lots of very imprecise studies you would not expect to get the same overall CI as combining lots of very precise studies. Otherwise you would be getting a free lunch.

2) Any suggested approaches for publication bias checks for unweighted analyses?  I was thinking funnel plots of effect size and sample size.

Other people will probably pitch in with evidence-based ideas here but I would go for sqrt sample size I think.


Michael

Thanks for any advice.  alan

       [[alternative HTML version deleted]]

--
Michael




------------------------------

Message: 3
Date: Thu, 21 Mar 2024 12:18:41 -0400
From: Zac Robinson <zacrobinson2015 at gmail.com>
To: R Special Interest Group for Meta-Analysis
         <r-sig-meta-analysis at r-project.org>
Subject: Re: [R-meta]  Prediction Intervals for estimates of random
         slope rma.mv model with predict.rma()
Message-ID:

<CALM2kOaeKnJ4z-Kwbu9uTyN8eU=rZuK87iD5aNo5_agi47N-fA at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Wolfgang,

Apologies for the incorrect method to reply - hopefully got it right
this time. If I'm understanding things correctly - the coding can
(potentially) end up giving me what I'm after, but I'm just specifying
the sigma / sd for the prediction interval incorrectly (i.e., the
variances can't just be summed).

Thus, I guess my follow up question is - how would you recommend going
about this calculation? The closest thing I can find in the literature
with a similar variance calculation is in the context of R2 from
Johnson et al., where they calculate the "mean random effects
variance" which seems like it could be useful here:

https://besj/
ournals.onlinelibrary.wiley.com%2Fdoi%2F10.1111%2F2041-210X.12225&data
=05%7C02%7Cwilson%40auburn.edu%7C2d94e8b5f80d4aa6122f08dc4d950e2b%7Ccc
b6deedbd294b388979d72780f62d3b%7C0%7C0%7C638470549845923786%7CUnknown%
7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
VCI6Mn0%3D%7C0%7C%7C%7C&sdata=y8aJo27gBdkFH0ZGhz%2Fpn3IjSh0T7icVj%2F6f
EAOQBiE%3D&reserved=0

Let me know what you think and thank you for your time and expertise,

Zac

On Thu, Mar 14, 2024 at 9:23?AM Viechtbauer, Wolfgang (NP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

Dear Zac,

Please respond to the mailing list, not just to me personally.

Just summing up all the variances isn't correct - the correct
computation of the prediction interval for such a random-effects
structure is unfortunately more complex.

Best,
Wolfgang

-----Original Message-----
From: Zac Robinson <zacrobinson2015 at gmail.com>
Sent: Monday, March 11, 2024 16:07
To: Viechtbauer, Wolfgang (NP) <

wolfgang.viechtbauer at maastrichtuniversity.nl>

Subject: Re: [R-meta] Prediction Intervals for estimates of random
slope

rma.mv

model with predict.rma()

Hey Wolfgang,

Thank you very much for the reply. I actually have played around
with

things a

bit more and think I've found a solution. I may not be specifying
the

sigma

argument perfectly for a random slope model - but I think in concept

this works!

Looks like emmeans / predict() have cross compatibility that allows
for prediction intervals when sigma is specified

Let me know what you think,
Zac

# SETUP

-------------------------------------------------------------------

#load packages and data
library(metafor)
library(tidyverse)
library(emmeans)

data("dat.konstantopoulos2011")
df=dat.konstantopoulos2011%>%
   as.data.frame()%>%
   mutate(factor=sample(c("A","B"),56,replace=TRUE))

# RANDOM SLOPE (x1)

-------------------------------------------------------

#fit single random slope model
m2= rma.mv(yi ~ year+factor, vi,
           random = list(~year|study, ~1|district, ~1|school),
           data = df, method = "REML",struct = "GEN",
           control = list(optimizer="bobyqa"))

#extract estimates with emmmprep + emmeans + predict
est.ci=emmprep(m2,
         at=list(year=1976:2000))%>%
   emmeans(~year,
           weights = "prop",
           sigma=sqrt(sum(
             m2$sigma2,
             m2$tau2,
             m2$gamma2
           )))

pi= est.ci%>%
   predict(interval="prediction")%>%
   select(lower.PL,upper.PL)

preds=cbind(est.ci,pi)
preds # prediction intervals are now added

ggplot(data = preds,
        aes(x=year,
            y=emmean))+theme_classic()+
   geom_point(data=df,
              aes(y=yi,
                  size=1/vi))+
   geom_ribbon(aes(ymin=lower.PL,
                   ymax=upper.PL),
               linetype="dashed",
               alpha=0,
               color="black")+
   geom_ribbon(aes(ymin=asymp.LCL,
                   ymax=asymp.UCL),
               alpha=0.25)+
   geom_line()

On Mon, Mar 11, 2024 at 2:06?AM Viechtbauer, Wolfgang (NP)
<mailto:wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
Dear Zac,

predict() currently does not provide predicition intervals for

http://rma/.
mv%2F&data=05%7C02%7Cwilson%40auburn.edu%7C2d94e8b5f80d4aa6122f08dc4d
950e2b%7Cccb6deedbd294b388979d72780f62d3b%7C0%7C0%7C63847054984592913
3%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI
6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=Pm3e0ZH72zJHoktoCm9GKnn7Ck
EZBMWMP3y2OXFXCrw%3D&reserved=0

models with a GEN random effects structure. Not sure when I will get

around to

implementing that.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis <mailto:

r-sig-meta-analysis-bounces at r-project.org>

On Behalf

Of Zac Robinson via R-sig-meta-analysis
Sent: Sunday, March 10, 2024 16:26
To: mailto:r-sig-meta-analysis at r-project.org
Cc: Zac Robinson <mailto:zacrobinson2015 at gmail.com>
Subject: [R-meta] Prediction Intervals for estimates of random
slope

http://rma/
.mv%2F&data=05%7C02%7Cwilson%40auburn.edu%7C2d94e8b5f80d4aa6122f08dc
4d950e2b%7Cccb6deedbd294b388979d72780f62d3b%7C0%7C0%7C63847054984593
3935%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ
BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=9O97NGQj6cbfYQLmYFLPg
TBhapqpa3BePe9LnzhIgBs%3D&reserved=0

model with predict.rma()

Dear All,

I am working on extracting the adjusted effects from a
dose-response multilevel meta-regression. One issue I've run into
is getting the appropriate prediction intervals on my estimates
when the model includes random slopes (specifically when struct = "GEN" is included).

From

what I gather from the help files, this comes down to appropriately
specifying the `tau2.levels` and `gamma2.levels` arguments within
predict.rma() - however, I am unsure how to get this to work.

In the code below I've tried to demonstrate my issue and explore
the

other

solutions I'm aware of through the orchaRd package). My end goal

requires

adjusting for both continuous and categorical moderators so ideally
the solution stays within the emmprep() / emmeans() framework
(which makes proportional weights and setting non-focal continuous
variables at the

mean

easy), but I realize this may not be possible.

Thank you very much in advance!

Zac

# SETUP

-------------------------------------------------------------------

#load packages and data
library(metafor)
library(tidyverse)
library(orchaRd)
library(emmeans)

data("dat.konstantopoulos2011")
df=dat.konstantopoulos2011%>%
   as.data.frame()%>%
   mutate(factor=sample(c("A","B"),nrow(df),replace=TRUE))

# RANDOM INTERCEPT

--------------------------------------------------------

#fit random intercept model
m1=http://rma.mv/(yi ~ year+factor, vi,
           random = list(~1|study, ~1|district, ~1|school),
           data = df, method = "REML")

#extract estimates with predict.rma predict(m1,
         newmods =cbind(rep(c(1976:2000),2),
                        rep(c(0,1),each=25)),
         addx = TRUE) # works fine

#extract estimates with emmprep
emmprep(m1,
         at=list(year=1976:2000))%>%
   emmeans(~year,
           weights = "prop")%>%
   pred_interval_esmeans(model=m1,
                         mm=.,
                         mod="year") # works fine

#extract estimates with orchaRd
mod_results(m1,
             mod = "year",
             group = "study") # works fine

# RANDOM SLOPE (x1)

-------------------------------------------------------

#fit single random slope model
m2=http://rma.mv/(yi ~ year+factor, vi,
           random = list(~year|study, ~1|district, ~1|school),
           data = df, method = "REML",struct = "GEN",
           control = list(optimizer="bobyqa"))

#extract estimates with predict.rma predict(m2, newmods
=cbind(rep(c(1976:2000),2),
                            rep(c(0,1),each=25)),
         addx = TRUE) # no prediction intervals

predict(m2, newmods =cbind(rep(c(1976:2000),2),
                            rep(c(0,1),each=25)),
         addx = TRUE,
         tau2.levels = levels(year), # try adding tau2.levels
         ) # still no prediction intervals

#extract estimates with emmprep
emmprep(m2,
         at=list(year=1976:2000))%>%
   emmeans(~year,
           weights = "prop")%>%
   pred_interval_esmeans(model=m2,
                         mm=.,
                         mod="year") # works but gives an error

#extract estimates with orchaRd
mod_results(m2,
             mod = "year",
             group = "study") # works but gives an error


# RANDOM SLOPE (x2)

-------------------------------------------------------

#fit double random slope model
m3=http://rma.mv/(yi ~ year+factor, vi,
           random = list(~year|study, ~year|district, ~1|school),
           data = df, method = "REML",struct = c("GEN","GEN"),
           control = list(optimizer="bobyqa"))

#extract estimates with predict.rma predict(m3, newmods
=cbind(rep(c(1976:2000),2),
                            rep(c(0,1),each=25)),
         addx = TRUE) # no prediction intervals

predict(m3, newmods =cbind(rep(c(1976:2000),2),
                             rep(c(0,1),each=25)),
         addx = TRUE,
         tau2.levels = levels(year),# try adding tau2.levels
         gamma2.levels = levels(year)# try adding gamma2.levels
         ) # still no prediction intervals

#extract estimates with emmprep
emmprep(m3,
              at=list(year=1976:2000))%>%
   emmeans(~year,
           weights = "prop")%>%
   pred_interval_esmeans(model=m3,
                         mm=.,
                         mod="year") # works but gives an error

#extract estimates with orchaRd
mod_results(m3,
             mod = "year",
             group = "study") # works but gives an error

         [[alternative HTML version deleted]]




------------------------------

Subject: Digest Footer