[R-meta] multiple models in one study

Hi James

You have perfectly described my problem of model selection: while the
dependent variable and key predictors are the same, sets of control
variables are different -- within each paper as well as between papers.

You ask two questions at the end (choosing between the results that are
closest to
the ideal and the results that are most comparable across studies; include
or exclude model without control variables).  I guess I could try this and
that and then compare results, what do you think?

Best,
Valeria

On Fri, Mar 24, 2023 at 6:58?PM James Pustejovsky via R-sig-meta-analysis <
r-sig-meta-analysis at r-project.org> wrote:

Hi Valeria,

It sounds like you're interested in synthesizing sets of regression
coefficients and the issue is that some papers report multiple regression
specifications that fit your criteria. For instance, a paper might report
three models:
Model 1: Y = b0 + b1 A + b2 B + b3 C
Model 2: Y = b0 + b1 A + b2 B + b3 C + b4 D + b5 E + b6 F
Model 3: Y = b0 + b1 A + b2 B + b3 C + b4 D + b5 E + b6 F + <a bunch of
other stuff>
And perhaps you're just interested in analyzing the coefficients (b1, b2,
b3). Does this description track with what you're wondering about?

If so, then the challenge is that the definition of regression coefficients
depends on ALL of the variables in the model, so the coefficients (b1, b2,
b3) from Model 1 aren't really estimating the same parameters as the (b1,
b2, b3) from Model 3. From your research aims and inclusion criteria, is it
possible to define an "ideal analysis" that most closely matches the
questions you're trying to investigate? If so, then perhaps you can select
results from each study that come closest to matching the ideal analysis.
This would be pretty similar to the "best-set" strategy.

Another thing to consider is how similar the regression specifications are
across studies. For example, say that you've got 10 studies meeting
inclusion criteria. For the first 8 studies, the specification that most
closely matches your ideal analysis is Model 2. But then for study 9, the
only thing that's reported is Model 1 and for study 10, Models 1, 2, and 3
are all reported and Model 3 is the one that most closely matches your
ideal analysis. For study 10, should you take the coefficients from Model 2
or Model 3? The tension is between choosing the results that are closest to
the ideal or choosing the results that are most comparable across all
included studies. And for study 1, should you take the results from Model 1
or just exclude it entirely because it doesn't report a specification that
controls for factors D, E, and F?

James


On Thu, Mar 23, 2023 at 3:17?AM Viechtbauer, Wolfgang (NP) via
R-sig-meta-analysis <r-sig-meta-analysis at r-project.org> wrote:

Dear Valeria,

I would say this depends on the aims. If there is one key predictor of
interest, then I would focus on that. If that's not the case, then I

would

extract all the ones that are of interest. Taking an average of all
coefficients (if this is what the "average-set" approach entails) doesn't
make much sense to me unless they are all measuring the same construct in
the same direction and in the same units (all unlikely).

If you extract multiple coefficients, you of course have to account for
the fact that they are not independent.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:

r-sig-meta-analysis-bounces at r-project.org] On

Behalf Of Valeria Ivaniushina via R-sig-meta-analysis
Sent: Wednesday, 22 March, 2023 17:52
To: R meta
Cc: Valeria Ivaniushina
Subject: [R-meta] multiple models in one study

Hi

I want to perform a meta-analysis of the relation between the outcome

and

key explanatory variables expressed as regression coefficients.

As a rule, authors report several models with different specifications.

I

wonder which regression coefficients should I collect?

In the book Meta-regression analysis in economics and business (Stanley

&

Doucouliagos, 2012)
several approaches are described:
- The best-set = ONE estimate from each study, using the KEY regression
from each paper
- The average-set = an average of all coefficients reported in the study
- The all-set = all relevant estimates reported in the study

Which approach is preferable? Are there additional considerations that I
have to take into account?

Regards,
Valeria

         [[alternative HTML version deleted]]

	[[alternative HTML version deleted]]