-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org]
On Behalf Of ne gic
Sent: Wednesday, 26 August, 2020 10:16
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] Sample size and continuity correction
Dear List,
I have general meta-analysis questions that are not
platform/software related.
*=======================*
*1. Issue of few included studies *
* =======================*
It seems common to see published meta-analyses with few studies e.g. :
(A). An analysis of only 2 studies.
(B). In another, subgroup analyses ending up with only one study in one of
the subgroups.
Nevertheless, they still end up providing a pooled estimate in their
respective forest plots.
So my question is, is there an agreed upon (or rule of thumb, or in your
view) minimum number of studies below which meta-analysis becomes
unacceptable?
Agreed upon? Not that I am aware of. Some may want at least 5 studies (per group or overall), some 10, others may be fine with if one group only contains 1 or 2 studies.
What interpretations/conclusions can one really draw from such analyses?
That's a vague question, so I can't really answer this in general. Of course, estimates will be imprecise when k is small (overall or within groups).
*===================*
*2. Continuity correction *
* ===================*
In studies of rare events, zero events tend to occur and it seems common to
add a small value so that the zero is taken care of somehow.
If for instance, the inclusion of this small value via continuity
correction leads to differing results e.g. from non-significant results
when not using correction, to significant results when using it, what does
make of that? Can we trust such results?
If this happens, then the p-value is probably fluctuating around 0.05 (or whatever cutoff is used for declaring results as significant). The difference between p=.06 and p=.04 is (very very unlikely) to be significant (Gelman & Stern, 2006). Or, to use the words of Rosnow and Rosenthal (1989): "[...] surely, God loves the .06 nearly as much as the .05".
Gelman, A., & Stern, H. (2006). The difference between "significant" and "not significant" is not itself statistically significant. American Statistician, 60(4), 328-331.
Rosnow, R.L. & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284.
If one instead opts to calculate a risk difference instead, and test that
for significance, would this be a better solution (more reliable result?)
to the continuity correction problem above?
If one is worried about the use of 'continuity corrections', then I think the more appropriate reaction is to use 'exact likelihood' methods (such as using (mixed-effects) logistic regression models or beta-binomial models) instead of switching to risk differences (nothing wrong with the latter, but risk differences are really a fudamentally different effect size measure compared to risk/odds ratios).
Looking forward to hearing your views as diverse as they may be in cases
where there is no consensus.
Sincerely,
nelly