[R-meta] Sample size and continuity correction
Many thanks for the insights Wolfgang. Apologies for my imprecise questions. By "agreed upon" & "what conclusions/interpretations", I was thinking if there is a minimum sample size whose pooled estimate can be considered somewhat reliable to produce robust inferences e.g. inferences drawn from just 2 studies can be drastically changed by the publication of a third study for instance - but it seems like there isn't. But I guess readers have to then check this for themselves to access how much weight they can place on the conclusions of specific meta-analyses. Again, I appreciate it! Sincerely, nelly On Thu, Aug 27, 2020 at 3:43 PM Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
Dear nelly, See my responses below.
-----Original Message----- From: R-sig-meta-analysis [mailto:
r-sig-meta-analysis-bounces at r-project.org]
On Behalf Of ne gic Sent: Wednesday, 26 August, 2020 10:16 To: r-sig-meta-analysis at r-project.org Subject: [R-meta] Sample size and continuity correction Dear List, I have general meta-analysis questions that are not platform/software related. *=======================* *1. Issue of few included studies * * =======================* It seems common to see published meta-analyses with few studies e.g. : (A). An analysis of only 2 studies. (B). In another, subgroup analyses ending up with only one study in one of the subgroups. Nevertheless, they still end up providing a pooled estimate in their respective forest plots. So my question is, is there an agreed upon (or rule of thumb, or in your view) minimum number of studies below which meta-analysis becomes unacceptable?
Agreed upon? Not that I am aware of. Some may want at least 5 studies (per group or overall), some 10, others may be fine with if one group only contains 1 or 2 studies.
What interpretations/conclusions can one really draw from such analyses?
That's a vague question, so I can't really answer this in general. Of course, estimates will be imprecise when k is small (overall or within groups).
*===================* *2. Continuity correction * * ===================* In studies of rare events, zero events tend to occur and it seems common
to
add a small value so that the zero is taken care of somehow. If for instance, the inclusion of this small value via continuity correction leads to differing results e.g. from non-significant results when not using correction, to significant results when using it, what does make of that? Can we trust such results?
If this happens, then the p-value is probably fluctuating around 0.05 (or whatever cutoff is used for declaring results as significant). The difference between p=.06 and p=.04 is (very very unlikely) to be significant (Gelman & Stern, 2006). Or, to use the words of Rosnow and Rosenthal (1989): "[...] surely, God loves the .06 nearly as much as the .05". Gelman, A., & Stern, H. (2006). The difference between "significant" and "not significant" is not itself statistically significant. American Statistician, 60(4), 328-331. Rosnow, R.L. & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284.
If one instead opts to calculate a risk difference instead, and test that for significance, would this be a better solution (more reliable result?) to the continuity correction problem above?
If one is worried about the use of 'continuity corrections', then I think the more appropriate reaction is to use 'exact likelihood' methods (such as using (mixed-effects) logistic regression models or beta-binomial models) instead of switching to risk differences (nothing wrong with the latter, but risk differences are really a fudamentally different effect size measure compared to risk/odds ratios).
Looking forward to hearing your views as diverse as they may be in cases where there is no consensus. Sincerely, nelly