Many thanks for the insights Wolfgang.
Apologies for my imprecise questions. By "agreed upon" & "what
conclusions/interpretations", I was thinking if there is a minimum sample
size whose pooled estimate can be considered somewhat reliable to produce
robust inferences e.g. inferences drawn from just 2 studies can be
drastically changed by the publication of a third study for instance - but
it seems like there isn't. But I guess readers have to then check this for
themselves to access how much weight they can place on the conclusions of
specific meta-analyses.
Again, I appreciate it!
Sincerely,
nelly
On Thu, Aug 27, 2020 at 3:43 PM Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
Dear nelly,
See my responses below.
-----Original Message-----
From: R-sig-meta-analysis [mailto:
r-sig-meta-analysis-bounces at r-project.org]
On Behalf Of ne gic
Sent: Wednesday, 26 August, 2020 10:16
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] Sample size and continuity correction
Dear List,
I have general meta-analysis questions that are not
platform/software related.
*=======================*
*1. Issue of few included studies *
* =======================*
It seems common to see published meta-analyses with few studies e.g. :
(A). An analysis of only 2 studies.
(B). In another, subgroup analyses ending up with only one study in one of
the subgroups.
Nevertheless, they still end up providing a pooled estimate in their
respective forest plots.
So my question is, is there an agreed upon (or rule of thumb, or in your
view) minimum number of studies below which meta-analysis becomes
unacceptable?
Agreed upon? Not that I am aware of. Some may want at least 5 studies (per
group or overall), some 10, others may be fine with if one group only
contains 1 or 2 studies.
What interpretations/conclusions can one really draw from such analyses?
That's a vague question, so I can't really answer this in general. Of
course, estimates will be imprecise when k is small (overall or within
groups).
*===================*
*2. Continuity correction *
* ===================*
In studies of rare events, zero events tend to occur and it seems common
add a small value so that the zero is taken care of somehow.
If for instance, the inclusion of this small value via continuity
correction leads to differing results e.g. from non-significant results
when not using correction, to significant results when using it, what does
make of that? Can we trust such results?
If this happens, then the p-value is probably fluctuating around 0.05 (or
whatever cutoff is used for declaring results as significant). The
difference between p=.06 and p=.04 is (very very unlikely) to be
significant (Gelman & Stern, 2006). Or, to use the words of Rosnow and
Rosenthal (1989): "[...] surely, God loves the .06 nearly as much as the
.05".
Gelman, A., & Stern, H. (2006). The difference between "significant" and
"not significant" is not itself statistically significant. American
Statistician, 60(4), 328-331.
Rosnow, R.L. & Rosenthal, R. (1989). Statistical procedures and the
justification of knowledge in psychological science. American Psychologist,
44, 1276-1284.
If one instead opts to calculate a risk difference instead, and test that
for significance, would this be a better solution (more reliable result?)
to the continuity correction problem above?
If one is worried about the use of 'continuity corrections', then I think
the more appropriate reaction is to use 'exact likelihood' methods (such as
using (mixed-effects) logistic regression models or beta-binomial models)
instead of switching to risk differences (nothing wrong with the latter,
but risk differences are really a fudamentally different effect size
measure compared to risk/odds ratios).
Looking forward to hearing your views as diverse as they may be in cases
where there is no consensus.
Sincerely,
nelly
[[alternative HTML version deleted]]