[R-meta] Calculating effect size for subsets of data

Thu, Dec 3, 2020 12:10 AM

Dear Tarun,

(please post in plain text; when converted to plain text, the table becomes unreadable)

Indeed, coefficients with overlapping CIs may still be significantly different from each other. You can use anova() with the 'L' argument to test pairs of coefficients against each other. See:

http://www.metafor-project.org/doku.php/tips:testing_factors_lincoms#testing_linear_combinations

and

http://www.metafor-project.org/doku.php/tips:multiple_factors_interactions

for illustrations.

Best,
Wolfgang

-----Original Message-----
From: Tarun Khanna [mailto:khanna at hertie-school.org]
Sent: Thursday, 03 December, 2020 8:02
To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis at r-project.org
Subject: Re: Calculating effect size for subsets of data

Dear Wolfgang,

I am following up on a question that we discussed a few weeks ago regarding
meta-analysis for different combinations of data. This is regarding
interpreting the results.

label
?beta
?se
?pvalues
?upper_lim
?lower_lim
Social Comparison
????????? 0.102
????????? 0.057
????????? 0.077
????????? 0.214
???????? (0.011)
Feedback
????????? 0.076
????????? 0.033
????????? 0.020
????????? 0.140
????????? 0.012
Feedback+Social
????????? 0.104
????????? 0.043
????????? 0.016
????????? 0.189
????????? 0.020
Monetary Incentives
????????? 0.261
????????? 0.042
????????? 0.000
????????? 0.344
????????? 0.178
Social+Monetary
????????? 0.034
????????? 0.081
????????? 0.674
????????? 0.193
???????? (0.125)
Feedback+Monetary
????????? 0.176
????????? 0.060
????????? 0.003
????????? 0.293
????????? 0.060
Social+Monetary+Feedback
????????? 0.338
????????? 0.139
????????? 0.015
????????? 0.611
????????? 0.065
Motivation
????????? 0.131
????????? 0.052
????????? 0.012
????????? 0.233
????????? 0.029
Feedback+Motivation
????????? 0.152
????????? 0.047
????????? 0.001
????????? 0.243
????????? 0.061
Social+Feedback+Motivation
????????? 0.212
????????? 0.087
????????? 0.015
????????? 0.383
????????? 0.041

I ran the model as you suggested. The model reveals differences in the
average effect size the different combinations but the condifence levels of
these estimates overlap. In my opinion that does not mean that the
differences are not statistically significant as we don't necessarily test
for significance of differences. Or do these results mean we can't say
anything about the differences??In a regression model I would run a F test
with Ho : b1-b2 = 0. Can we do the same here?

Best
Tarun

Tarun Khanna
Research Associate

Hertie School

Friedrichstra?e 180
10117 Berlin ? Germany
khanna at hertie-school.org ? www.hertie-school.org

________________________________________
From: Viechtbauer, Wolfgang (SP)
<wolfgang.viechtbauer at maastrichtuniversity.nl>
Sent: 02 September 2020 11:40:27
To: Tarun Khanna; r-sig-meta-analysis at r-project.org
Subject: RE: Calculating effect size for subsets of data

Dear Tarun,

If I understand you correctly, then there should be 16 different
combinations of A, B, C, and D but one of them (A=B=C=D=0) cannot occur, so
essentially there are 15 combinations that were observed. As a result, you
should have gotten a warning when fitting the model that a redundant
predictor was dropped from the model. Let's consider a simpler case with
just A and B:

set.seed(1234)
k <- 900
A <- c(rep(0,k/3), rep(1,k/3), rep(1,k/3))
B <- c(rep(1,k/3), rep(0,k/3), rep(1,k/3))
vi <- rep(.01, k)
yi <- rnorm(k, 0.5 * A + 0.1 * B + 0.3*A*B, sqrt(vi))

A <- factor(A)
B <- factor(B)

res <- rma(yi, vi, mods = ~ A*B)
res

These are the model results:

???????? estimate????? se????? zval??? pval??? ci.lb??? ci.ub
intrcpt?? -0.3019? 0.0100? -30.1904? <.0001? -0.3215? -0.2823? ***
A1???????? 0.7963? 0.0082?? 97.5319? <.0001?? 0.7803?? 0.8123? ***
B1???????? 0.4032? 0.0082?? 49.3825? <.0001?? 0.3872?? 0.4192? ***

The results are a bit tricky to interpret, so I would suggest a different
parameterization:

res <- rma(yi, vi, mods = ~ A:B + 0)
res

?????? estimate????? se????? zval??? pval?? ci.lb?? ci.ub
A1:B0??? 0.4944? 0.0058?? 85.6397? <.0001? 0.4831? 0.5058? ***
A0:B1??? 0.1013? 0.0058?? 17.5462? <.0001? 0.0900? 0.1126? ***
A1:B1??? 0.8976? 0.0058? 155.4771? <.0001? 0.8863? 0.9090? ***

Now we can clearly see that A1:B0 is the estimated effect when A is given
alone, A0:B1 is the estimated effect when B is given alone, and A1:B1 is the
estimated effect when A and B are given together.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-
project.org]
On Behalf Of Tarun Khanna
Sent: Monday, 31 August, 2020 13:11
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] Calculating effect size for subsets of data

Dear all,

I am conducting a meta-analysis of effect of certain interventions on
household energy consumption. In my data set I have a dummy variable for
each of the sub-interventions: A,B,C,D such that intersection of A=0 & B=0
&
C=0 & D=0 is zero. Each effect size may be associated with multiple
interventions though.

I have calculated an aggregate effect size across interventions and then
effect size by sub-intervention. But I also want to compare if the effect
of
the sub-interventions differs from each other. I thought about including
the
sub-regression dummies as controls in the meta regression:

rma (yi, vi, method = "REML", data = data, mods ~ A*B*C*D)

The problem in interpreting the output of this regression is that there is
no base category left for the intercept to denote. Can I perhaps run the
model by supressing the intercept? Or what would be the interpretation of
the intercept in this case?

Thanks in advance!

Best

Tarun
Tarun Khanna
PhD Researcher
Hertie School

Friedrichstra?e 180
10117 Berlin ? Germany
khanna at hertie-school.org ? www.hertie-school.org<http://www.hertie-
school.org/>

[R-meta] Calculating effect size for subsets of data

Thread (2 messages)